Apparatus and method for detecting associations between different types of datasets

Abstract

An apparatus for detecting associations between different types of datasets is disclosed. The apparatus includes at least a processor and a memory communicatively coupled to the at least one processor. The memory instructs the processor to receive a plurality of datasets from a user. The memory instructs the processor to identify a first set of associations between the plurality of datasets. The memory instructs the processor to generate a second set of associations in response to the first set of associations using a second association classifier. Generating the second set of associations includes training the second association classifier using second association training data, the second association training data including a plurality of data entries including a first set of associations as input that correlates to a second set of associations as output. The memory instructs the processor to display the second set of associations using a display device.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of priority of U.S. Provisional Application No. 63 / 357,978, filed on July 1, 2022, entitled "SYSTEMS AND METHODS FOR DETECTING ASSOCIATIONS AMONG DATASETS OF DIFFERENT TYPES", and claims the benefit of priority of U.S. Non - Provisional / Provisional Application No. 18 / 217,378, filed on June 30, 2023, entitled "APPARATUS AND A METHOD FOR DETECTING ASSOCIATIONS AMONG DATASETS OF DIFFERENT TYPES". Each of U.S. Non - Provisional / Provisional Application No. 18 / 217,378 and U.S. Provisional Patent Application No. 63 / 357,978 is hereby incorporated by reference in its entirety.

[0002] The present invention generally relates to the field of detecting associations among different types of datasets. In particular, the present invention is directed to an apparatus and a method for detecting associations among different types of datasets.

Background Art

[0003] In various fields such as healthcare, pathology, logistics, and document management, the need for accurate and reliable information extraction and association across multiple media is significant. However, current techniques used to create relationships between data across two or more media are flawed and can introduce errors due to factors such as image quality, font variations, or data inconsistencies.

Summary of the Invention

[0004] In one aspect, an exemplary apparatus for detecting associations between different types of datasets includes at least a processor and a memory communicatively coupled to the at least one processor, the memory configured to receive a plurality of datasets including a first dataset and a second dataset, identify a first set of associations between a first subset of the first dataset and a first subset of the second dataset, generate a second set of associations between a second subset of the first dataset and a second subset of the second dataset in response to the first set of associations using a second association classifier, wherein generating the second set of associations includes training the second association classifier using second association training data, the second association training data including a plurality of data entries including the first set of associations, and generating the second set of associations in response to the first set of associations using the trained second association classifier, and configure the at least one processor to display the second set of associations using a display device.

[0005] In another embodiment, an exemplary method for detecting associations between different types of datasets includes: receiving a plurality of datasets, including a first dataset and a second dataset, using at least a processor; identifying a first set of associations between a first subset of the first dataset and a first subset of the second dataset, using at least a processor; generating a second set of associations between a second subset of the first dataset and a second subset of the second dataset, in accordance with the first set of associations, using a second association classifier, wherein generating the second set of associations includes training the second association classifier using second association training data, the second association training data including a plurality of data entries including the first set of associations; and generating the second set of associations in accordance with the first set of associations, using the trained second association classifier; and displaying the second set of associations using a display device.

[0006] In another embodiment, another exemplary apparatus for detecting associations between different types of datasets comprises at least a processor and a memory communicably connected to the at least processor, the memory including instructions to configure at least the processor to receive a plurality of datasets, including a first dataset and a second dataset, wherein the first and second datasets include data elements of different types; identify an initial set of associations between the first dataset and the second dataset, wherein each association includes one or more first data elements from the first dataset and one or more second data elements from the second dataset; train a neural network model to detect additional associations between the first dataset and the second dataset, wherein the initial set of associations is used as training data for training the neural network model; and use the trained neural network model to detect one or more additional associations between the first dataset and the second dataset.

[0007] In another embodiment, another exemplary method for detecting associations between different types of datasets includes receiving a plurality of datasets, including a first dataset and a second dataset, using at least a processor, wherein the first and second datasets include data elements of different types; identifying an initial set of associations between the first dataset and the second dataset, wherein each association includes one or more first data elements from the first dataset and one or more second data elements from the second dataset; training a neural network model to detect additional associations between the first dataset and the second dataset, using at least a processor, wherein the initial set of associations is used as training data for training the neural network model; and detecting one or more additional associations between the first dataset and the second dataset, using at least a processor and the trained neural network model.

[0008] Details of one or more variations of the subject matter described herein are given in the accompanying drawings and the following description. Other features and advantages of the subject matter described herein will become apparent from the description and drawings, as well as the claims.

[0009] For the purpose of illustrating the present invention, the drawings illustrate aspects of one or more embodiments of the present invention. However, it should be understood that the present invention is not limited to the exact arrangement and means shown in the drawings. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram of an exemplary embodiment of a device for detecting associations between different types of datasets. [Figure 2] Figure 2 is a block diagram of an exemplary machine learning process. [Figure 3] Figure 3 is a block diagram of an exemplary embodiment of the association database. [Figure 4] Figure 4 is a diagram illustrating an exemplary embodiment of a neural network. [Figure 5] Figure 5 shows an exemplary embodiment of a neural network node. [Figure 6] Figure 6 is a diagram illustrating an exemplary embodiment of fuzzy set comparison. [Figure 7] Figure 7 is a flowchart illustrating an exemplary method for detecting associations between different types of datasets. [Figure 8] Figure 8 is a block diagram illustrating an exemplary apparatus for detecting associations between different types of datasets. [Figure 9] Figure 9 is a flowchart illustrating an exemplary method for detecting associations between different types of datasets. [Figure 10] Figure 10 is a block diagram of a computing system that may be used to implement any one or more of the methodologies disclosed herein and any one or more of them. [Modes for carrying out the invention]

[0011] Drawings are not necessarily to scale and may be represented by imaginary lines, graphic symbols, and partial drawings. In certain cases, details not necessary for understanding the embodiment, or details that would make it difficult to perceive other details, may be omitted. Similar reference numerals in different drawings indicate similar elements.

[0012] Broadly speaking, aspects of this disclosure relate to apparatus and methods for detecting associations between different types of datasets. The apparatus includes at least a processor and memory communicably connected to at least the processor. The memory instructs the processor to receive a plurality of datasets from a user. The memory instructs the processor to identify a first set of associations between the plurality of datasets. The memory instructs the processor to use a second association classifier to generate a second set of associations in accordance with the first set of associations. Generating the second set of associations includes training the second association classifier using second association training data, which includes a plurality of data entries, with the first set of associations as inputs correlated to the second set of associations as outputs. The memory instructs the processor to use a display device to display the second set of associations. Exemplary embodiments illustrating aspects of this disclosure are described below in the context of several specific examples.

[0013] Referring here to Figure 1, an exemplary embodiment of the apparatus 100 for detecting associations between different types of datasets is shown. The apparatus 100 includes a processor 104. The processor 104 may include any computing device as described in this disclosure, including, but not limited to, a microcontroller, microprocessor, digital signal processor (DSP), and / or system-on-a-chip (SoC) as described in this disclosure. The computing device may include a mobile device such as a mobile phone or smartphone, be included in a mobile device, and / or communicate with a mobile device. The processor 104 may include a single computing device operating independently, or two or more computing devices operating in cooperation, in parallel, sequentially, etc., and two or more computing devices may be included in a single computing device or together in two or more computing devices. The processor 104 may interface with or communicate with one or more additional devices via a network interface device, as will be described in more detail later. The network interface device may be used to connect the processor 104 to one or more of various networks and one or more devices. Examples of network interface devices include, but are not limited to, network interface cards (e.g., mobile network interface cards, LAN cards), modems, and any combination thereof. Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, corporate networks), local area networks (e.g., networks associated with offices, buildings, campuses, or other relatively small geographical spaces), telephone networks, data networks associated with telephone / voice providers (e.g., mobile communications provider data and / or voice networks), direct connections between two computing devices, and any combination thereof. Networks may use wired and / or wireless communication modes. In general, any network topology may be used.Information (e.g., data, software, etc.) can be communicated to and from a computer and / or computing device. The processor 104 may include, for example, a computing device or cluster of computing devices at a first location and a second computing device or cluster of computing devices at a second location, but is not limited to these. The processor 104 may include one or more computing devices dedicated to data storage, security, traffic distribution for load balancing, etc. The processor 104 can distribute one or more computing tasks, as described below, across multiple computing devices of computing devices that can operate in parallel, serial, redundantly, or in any other way used for task or memory distribution between computing devices. The processor 104 may be implemented using a “shared nothing” architecture in which data is cached at the worker, which in one embodiment may enable scalability of the device 100 and / or computing devices.

[0014] Continuing with reference to Figure 1, the processor 104 may be designed and / or configured to execute any method, process, or sequence of process steps in any embodiment described herein in any order and at any degree of iteration. For example, the processor 104 may be configured to repeatedly execute a single process or sequence until a desired or instructed result is achieved, and the iteration of the process or sequence of processes is performed using the output of the previous iteration as input to the subsequent iteration, and the inputs and / or outputs of the iterations can be aggregated to produce an aggregated result, one or more variables such as global variables can be decreased or decimated, and / or larger processing tasks can be divided into a set of smaller processing tasks that are iteratively addressed. The processor 104 may execute any process or sequence of processes as described herein in parallel, such as executing the process simultaneously and / or substantially simultaneously two or more times using two or more parallel threads, processor cores, etc., and the task division between parallel threads and / or processes may be performed according to any protocol suitable for task division between iterations. Those skilled in the art, upon reviewing the entirety of this disclosure, will recognize a variety of ways in which processes, sequences of processes, processing tasks, and / or data can be subdivided, shared, or otherwise processed using iterative, recursive, and / or parallel processing.

[0015] Continuing to refer to Figure 1, the apparatus 100 includes memory. The memory is communicatively connected to the processor 104. The memory may include instructions that configure the processor 104 to perform the tasks disclosed herein. Where used herein, “communicatively connected” means connected by a connection, attachment or link between two or more relational terms that enables the reception and / or transmission of information between them. For example, but not limited to, such a connection may be between two or more components, circuits, devices, systems, apparatus, etc., wired or wireless, direct or indirect, enabling the reception and / or transmission of data and / or signals between them. The data and / or signals between them may include, but are not limited to, electrical, electromagnetic, magnetic, video, audio, radio, and microwave data and / or signals, combinations thereof, among others. The communication connection may be achieved, for example, directly or through one or more intervening devices or components via wired or wireless electronic, digital, or analog communications. Furthermore, a communication connection may include electrically coupling or connecting at least one output of one device, component, or circuit to at least one input of another device, component, or circuit, for example, via a bus or other equipment for intercommunication between elements of a computing device, for example, but not limited to. A communication connection may also include indirect connections via, for example, wireless connections, radio communications, low-power wide-area networks, optical communications, magnetic coupling, capacitive coupling, or optical coupling, for example, but not limited to. In some cases, the term “communicatively coupled” may be used in this disclosure instead of “communicatively connected.”

[0016] Continuing to refer to Figure 1, the processor 104 is configured to receive multiple datasets 108 from the user. As used in this disclosure, “dataset” is a collection of data. A dataset 108 is a structured collection of data organized and presented in a specific format for analysis and interpretation. It consists of individual data points or observations, each representing specific information. A dataset 108 can be generated through a variety of means, including manual data entry, data collection from sensors or devices, scraping data from websites, or extracting data from existing databases. A dataset may contain multiple individual data points, often referred to as records, instances, or observations. Each data point represents a distinct unit of information, such as a customer, transaction, measurement, or any other relevant entity.

[0017] Continuing to refer to Figure 1, multiple datasets 108 may contain multiple metadata. As used in this disclosure, “metadata” refers to descriptive information or attributes that provide context, structure, and meaning to data. Metadata is, in essence, data about data. Metadata helps to understand and manage various aspects of data, such as its origin, content, format, quality, and use. This plays a crucial role in effectively organizing, retrieving, and interpreting data. Metadata may include descriptive metadata, structural metadata, administrative metadata, technical metadata, providence metadata, and usage metadata. Metadata can be organized and managed through metadata schemas, standards, or frameworks. These provide guidelines and specifications for capturing, storing, and exchanging metadata in a consistent and structured manner. Common metadata standards include Dublin Core, Metadata Object Description Schema (MODS), and the Federal Geographic Data Commission (FGDC) metadata standards. In some cases, metadata may be associated with text data or image data. Metadata may also be associated with pathology slides. Metadata may provide additional descriptive information or attributes linked to the image data or text data associated with the pathology slides. Metadata associated with multiple datasets 108 may include patient information. Patient information may include data such as the patient's name, unique patient identifier (ID), age, sex, and any other relevant demographic information. Patient information helps identify slides and associate them with the correct individual's medical record. Metadata may also include case-specific details, which may include information about a specific case or clinical scenario related to the slide. Case-specific details may include information about the case number, attending physician, clinical history, related symptoms, or any other relevant details that help understand the context of the slide. In some cases, metadata may include information about the specific specimen type of the slide. This may include the type of tissue or sample represented by the slide. Metadata may include notes, comments, or observations made by pathologists or other healthcare professionals.These annotations can highlight specific features, anomalies, or noteworthy aspects of the slide that are important for interpretation or follow-up analysis. The date and time the slide was prepared, analyzed, or labeled can be associated as metadata. This information helps track and maintain a chronological record of slide-related activities. This could be breast tissue, lung biopsy, skin lesion, or any other anatomical or pathological specimen. In some embodiments, the metadata may include information about staining or preparation techniques, pathological diagnosis, etc.

[0018] Continuing to refer to Figure 1, the multiple datasets 108 include the first dataset 112. As used in this disclosure, “first dataset” is dataset 108 containing multiple data. In one embodiment, the first dataset 112 may be dataset 108 containing multiple text data. As used in this disclosure, “text data” is a collection of data consisting of text-based information. Examples of text data may include documents, captions, sentences, paragraphs, free text fields, transcripts, prognosis labels, etc. In some embodiments, the text data in the first dataset may relate to pathology slides. As used in this disclosure, “pathology slide” is a glass slide containing a portion of biological material biopsied from a patient. The pathology slide may contain biopsied tissue from a patient, which is sliced ​​into very thin layers and placed on a glass slide. The first dataset 112 may include written descriptions of various aspects of the pathology slide. The first dataset 112 may include documents surrounding the pathology slide. These documents may include information regarding the examination, analysis, storage, and disposal of pathology slides by healthcare professionals. The first dataset 112 may include information that describes and provides context for the pathology slides, enabling researchers, clinicians, or data analysts to effectively understand and analyze the slides. In non-limiting examples, the first dataset 112 may include slide identifiers, slide descriptions, clinical information, pathology reports, annotations, notes, laboratory findings, etc. The first dataset 112 may include a brief description or summary of the pathology slides, providing an overview of their content, specimen type, and relevant characteristics. This description may include details such as tissue type, staining techniques used, and any specific features or abnormalities present in the slides. In other embodiments, the first dataset 112 may include a pathology report generated by a healthcare professional or data scientist after the slides have been analyzed. This report may provide detailed findings, observations, interpretations, and diagnoses based on the examination of the slides. This may include descriptions of tissue structure, cell morphology, tumor grading, staging, and other relevant pathological features.

[0019] Continuing to refer to Figure 1, the multiple datasets 108 include a second dataset 116. As used in this disclosure, “second dataset” is dataset 108 containing multiple data. In one embodiment, the second dataset 116 may be dataset 108 containing multiple image data. As used in this disclosure, “image data” is a collection of data consisting of data associated with multiple images. The second dataset 116 may be a collection of multiple images organized and presented in a particular format for analysis, training machine learning models, or any other image-related task. It may consist of diverse images captured from various sources such as digital cameras, satellites, medical imaging devices, microscopes, or other image acquisition methods. The image data may be stored in a digital format such as JPEG, PNG, or TIFF. The image data may include both color images and / or grayscale images. The second dataset 116 may include multiple medical images. Medical images may include X-rays, CT scans, MRI, ultrasound, PET scans, electrocardiogram scans, etc. In some cases, the image data may include either 2D and / or 3D medical images. In one embodiment, the second dataset 116 may include multiple images associated with a pathology slide. The image data associated with the pathology slide may include a visual representation of the slide captured by imaging technology. The image data may represent cellular structure, tissue morphology, and any pathological features observed on the slide. This may include a series of images capturing the pathology slide. In one embodiment, this may include multiple images of the pathology slide at various levels of magnification and resolution. The image data associated with the pathology slide may be captured at various magnification levels ranging from low to high magnification. Lower magnification images may provide a wider view of the tissue and help identify the overall structure, while higher magnification images may provide a more detailed examination of individual cells and cellular structures. The image data may include images obtained by digital scanning technology or digital microscopy technology. In some cases, each image may represent a specific pathology slide and include visual information regarding cellular structure, tissue morphology, and any abnormalities or features present on the slide.The second dataset 116 may also include metadata that provides additional information about each pathology slide image. This metadata may include details such as the slide specimen type (e.g., tissue, cell, biopsy), the staining technique used, the magnification level, the imaging modality, and any relevant contextual information about the slide. In some embodiments, the image data may, in some cases, include annotations or overlays that highlight specific areas or features of interest. These annotations may be added manually by a pathologist or generated by an automated algorithm to assist in the identification and analysis of specific pathological findings.

[0020] Continuing to refer to Figure 1, multiple datasets 108 may contain multiple media data. As used in this disclosure, “media data” refers to data elements associated with multiple media. Media may include audio recordings, video recordings, images, digital media, graphs, and interrelated data structures. In some embodiments, media data may include video data. Video data can capture sequences of frames that often show temporal changes or dynamic processes. Video data may be used for applications such as surveillance, behavior recognition, and motion analysis. Each video in a dataset may contain multiple frames, and associated metadata may provide details such as duration, frame rate, or timestamp. In some cases, media data may include audio data. Audio data may include audio files, which are another type of media data found in a dataset. Audio files may include recorded sounds, speech, or sounds associated with pathological slides. Audio data is frequently used in speech recognition, speech classification, and acoustic analysis. Metadata for audio data may include attributes such as duration, sample rate, or audio format. In some cases, media data may include document data. Document data may include text documents such as PDFs, Word files, or plain text files. These documents may include research papers, reports, or any other form of written information. Document data is often used in natural language processing tasks, information retrieval, or text classification. Metadata for document data may include information such as document title, author, publication date, or word count. In some cases, media data may be associated with pathology slides. Media data associated with pathology slides may refer to additional information and records related to the slides, such as images, videos, or metadata. These media elements complement the slides themselves and help provide further insight into the underlying pathology.

[0021] Continuing with Figure 1, media data may include graphs or other interrelated data structures. In some cases, the contents of graphs and data structures may be associated with pathology slides. Media data may include tissue graphs, where individual cells or regions within the tissue are represented as nodes, and the connections or edges between them indicate spatial relationships or interactions. This graph structure allows for the exploration of cell networks, patterns of cell organization, and identification of abnormal cell clusters. Furthermore, media data may include diagnostic pathway graphs that map the progression of the diagnosis based on observation and test results. This represents the decision-making process followed by the pathologist and indicates potential branching pathways and alternative diagnoses. Data structures such as databases or file systems are used to store and manage various data associated with pathology slides. These structures may include patient information, slide metadata, diagnostic annotations, digital images, and other media data. These facilitate the efficient retrieval, organization, and retrieval of relevant information for clinical purposes, research, and education.

[0022] Continuing to refer to Figure 1, the processor 104 is configured to identify a first set 120 of associations between the first dataset 112 and the second dataset 116. As used in this disclosure, “first set of associations” refers to a relationship or connection established between two or more data types. This may include a relationship or connection established between image data and text data. This may include linking or integrating image data of the second dataset 116 with descriptive or explanatory text data of the first dataset 112 to provide additional context, enhance understanding, and convey relevant information. In one embodiment, the first set 120 of associations may include identifying an object or set of objects in the image data of the second dataset 116 and identifying that object using the text data of the first dataset 116. In a non-limiting example, the processor 104 may identify a group of unusual objects in a plurality of image data associated with the second dataset 116. The processor 104 can identify an object based on a combination of metadata and image data from a second dataset 116, where the metadata identifies the biopsied tissue from the slide and the image data includes an image of the anomaly. The processor 104 can then pair the anomaly with text data from a first dataset 112, where the text data includes a written description of the anomaly. The pairing of the text data from the first dataset 112 with the image data from the second dataset 116 can be described as a first set 120 of associations. In one embodiment, each association in the first set 120 of associations may include a feature in the image data of the second dataset 116 that correlates with a string from the first dataset 112, where the string may linguistically describe this feature. In a non-limiting example, the first set of associations may include relationships or connections established between several different data types. This may include relationships between two or more of the following: audio data, image data, text data, video data, graphs, data structures, etc.

[0023] Continuing to refer to Figure 1, processor 104 may be configured to generate a first set 120 of associations using a bootstrap process. Where used in this disclosure, “bootstrap process” is a resampling technique used to estimate the sampling distribution of a statistic or to assess the uncertainty associated with a sample. The bootstrap process may involve generating multiple resamples of the original dataset by random sampling with substitution. Each resample is the same size as the original dataset, except that some observations may appear multiple times and others may be excluded. This process enables the creation of a pseudo-population from which statistical estimates can be derived. Once resamples are obtained, a desired statistic is calculated for each resample. This statistic may be the mean, median, standard deviation, correlation coefficient, or any other measure of association. By repeating this resampling process many times (often thousands of times), a distribution of the statistic known as the bootstrap distribution is obtained. In this case, the bootstrap process of processor 104 can be initiated by generating multiple resamples of the dataset. For example, each resample may consist of paired samples of text data and image data, which may be any other data referred to herein. These resamples are created by randomly selecting instances from the first and second datasets with substitution, ensuring that both text and image data are retained together in each resample. For each resample, the text and image data pairs are analyzed together to explore relationships or associations between them. Various techniques can be applied based on specific tasks or objectives. The strength of the relationship or association between text and image data can be evaluated by measuring performance metrics or statistical measures. For classification tasks, accuracy, precision, recall, or F1 score can be calculated. Alternatively, correlation coefficients, mutual information, or other statistical measures can be used to quantify the relationship between the two data types.A first association may be created between text data and image data if the strength of the relationship or association between the text data and image data exceeds a predetermined threshold. The bootstrap process may be repeated multiple times, each time generating a different resample. This iteration allows for estimation of variability and uncertainty in the relationship or association metric. By analyzing the results across the resampled dataset, confidence intervals can be constructed, hypothesis tests can be performed, or stability assessments can be conducted to evaluate the significance and robustness of the relationship.

[0024] Continuing to refer to Figure 1, the first set of associations 120 may include one or more direct correlations between the first dataset 112 and the second dataset 116. As used in this disclosure, “direct correlation” is a direct correspondence or alignment between two or more data types. This may include a direct correspondence or alignment between the visual content of image data and the accompanying text data. A direct correlation between the first dataset 112 and the second dataset 116 may include a clear and explicit correspondence between the image data and the accompanying text data. In a non-limiting example, a direct connection may include pairing an image with a caption that accurately describes an object, scene, or visual feature depicted in the image. The text data is directly related to and closely aligned with the visual information in the image. In the context of pathology slides, this connection is intended to provide a comprehensive description of the visual pathological features observed in the second dataset 116 via the associated text data in the first dataset. A non-limiting example of a direct correlation may include annotations on a pathology slide. Image data from the second dataset 116 may be annotated with text data from the first dataset 112 that explicitly describes observed features, structures, or abnormalities in the images. The annotations may function as a direct link between the image data and the associated text data, providing specific descriptions aligned with the visual content. Direct correlation may include pairing prognostic labels with all or part of the images. Prognostic labels may describe abnormalities, inflammation, discoloration, size, tissue structure, etc. The prognostic labels function as a direct representation of visual findings in the image data.

[0025] Continuing to refer to Figure 1, the processor 104 can generate a first set 120 of associations by extracting visual features from a second dataset 116. The processor 104 may use machine vision to extract visual features from image data. As used in this disclosure, “visual features” are one or more objects of interest located within the image data. Visual features within image data may include the presence or absence of cellular structures or tissue morphologies. The second dataset 116 may include microscopic images of a tissue sample, and the visual features may include various elements that provide information about cellular composition and structure. Non-limiting examples of visual features may include cell nuclei, tissue structures, cell arrangement, cell differentiation, inflammatory cells, cellular abnormalities, staining patterns, etc. In some cases, the morphology and characteristics of the cell nucleus may serve as visual features. This may include size, shape, chromatin pattern, presence of nucleoli, and any abnormalities such as nuclear enlargement or irregularity. In other cases, the arrangement and organization of tissue components may be identified as visual features. This may include the presence of glands, tubules, tubes, blood vessels, or other anatomical structures within a tissue sample. Spatial distribution and relationships between cells can also be identified as visual features. Spatial distribution and relationships may indicate a specific pathological condition. Features such as cell density, overlap, irregular patterns, or loss of normal tissue structure may be observed and analyzed. The degree of cell differentiation or maturation may be a visual feature assessed by a pathologist. This may include assessing the similarity of cells to their normal counterparts and identifying any abnormal or undifferentiated cells. The presence and distribution of inflammatory cells such as lymphocytes, neutrophils, or macrophages in a tissue sample may be a visual feature indicating an immune response or inflammation. Visual features may include the presence of abnormal cells such as cancer cells or cells with abnormal shape, size, or staining patterns. These abnormalities may indicate a neoplasm or other pathological process. Different staining techniques are used in pathology to highlight specific components or structures. Staining patterns such as eosinophilia, basophilia, or immunohistochemical staining may serve as visual features to identify specific cellular characteristics or pathological markers.

[0026] Continuing to refer to Figure 1, the processor 104 may use a machine vision system to identify visual features in the second dataset 116. The machine vision system may identify characteristic points or regions in the image that can be used as visual features. The machine vision system may use images from the second dataset 116 to make decisions about scenes, spaces, and / or objects in the image data. For example, the machine vision system may be used for world modeling or alignment of objects in space. Alignment may include, but is not limited to, image processing such as object recognition, feature detection, and edge / corner detection. Non-exclusive examples of feature detection may include scale-invariant feature transformation (SIFT), Canny edge detection, and Shi Tomasi corner detection. Alignment may include one or more transformations for orienting the camera frame (or image or video stream) to a three-dimensional coordinate system, and exemplary transformations include, but is not limited to, homography transformations and affine transformations. In one embodiment, the alignment of the first frame to the coordinate system can be verified and / or corrected using object recognition and / or computer vision, as described above. However, initial alignment to two dimensions, for example, represented as alignment to x and y coordinates, may be performed using a two-dimensional projection of a three-dimensional point onto a first frame. A third dimension of alignment, representing depth and / or the z-axis, can be detected by comparing two frames. For example, if the first frame includes a pair of frames captured using a pair of cameras (e.g., stereo cameras, also referred to in this disclosure as stereo cameras), image recognition and / or edge detection software can be used to detect a pair of stereoscopic images of an object, compare the two stereoscopic images to derive z-axis values ​​of points on the object, and, for example, allow the derivation of further z-axis points inside and / or around the object using interpolation. This can be repeated with multiple objects in the field of view, including environmental features of interest identified by an object classifier and / or indicated by operators.In one embodiment, the x and y axes can be selected to span a common plane and / or the xy-plane of the first frame for two cameras used for stereoscopic image capture, and as a result, the x and y translation components and φ can be pre-inputted into the translation matrix and rotation matrix for the affine transformation of the object's coordinates, as also described above. The initial x and y coordinates and / or estimations in the transformation matrix may be performed alternatively or additionally between the first frame and the second frame, as described above. As described above, for multiple points on the object and / or edges of the object and / or points on multiple edges, the x and y coordinates of the first stereoscopic frame may be input with an initial estimate of the z coordinate based on assumptions about the object, such as the assumption that the ground is substantially parallel to the xy-plane, as selected above. Next, the Z-coordinate and / or x, y, and z-coordinates aligned using the image capture and / or object recognition process described above can be compared with the predicted coordinates using the initial inference in the transformation matrix, and an error function can be calculated by comparing the two sets of points, and new x, y, and / or z-coordinates can be iteratively estimated and compared until the error function falls below a threshold level. In some cases, the machine vision system may use a classifier such as any classifier described throughout this disclosure.

[0027] Continuing to refer to Figure 1, the processor 104 can match visual features extracted from the second dataset 116 with similar visual features. Feature matching involves comparing visual features extracted from different image data to identify corresponding or similar features. This aims to establish correspondences between features in different image data, which may be useful for tasks such as image registration, object recognition, or image retrieval. Once the extracted visual features are identified, the processor 104 may generate a description of the extracted visual features. The description may capture the local appearance or characteristics around each visual feature. The description may include what type of tissue is located within the slide, a description of any visual feature, etc. The description may include information about the shape, texture, color, inflammation, staining pattern, or gradient of the region surrounding the visual feature. The processor 104 may pair the descriptions of the extracted visual features with descriptions of other visual features that have been previously identified. In some cases, the processor 104 can compare measures of similarity or correspondence between visual features in different pathology slides. This information can be used to quantify the similarity or dissimilarity between slides based on shared or distinct visual features. This allows for the comparison of slides based on common morphological patterns, cellular characteristics, or other visual attributes. In some embodiments, visual feature pairing can be used to find similar or corresponding regions of interest (ROIs) within different pathology slides. For example, in tumor detection or tracking, matching features across different slides can identify the same tumor region over time or across different patient samples.

[0028] Continuing to refer to Figure 1, the processor 104 may be configured to generate a number of named entities in accordance with a first dataset 112 using a named entity recognition process. As used in this disclosure, “named entity” is a specific type of word or phrase that represents a real-world object with a unique identity. Named entities may be people, places, ideas, concepts, or things that represent a specific person, organization, place, date, time, product, event, quantity, disease, tissue sample, and other uniquely identifiable entities. These entities play a crucial role in understanding context and extracting meaningful information from text. Named entities can provide contextual information and serve as reference points for understanding meaning and relationships within text. Recognizing and extracting named entities from text data is a fundamental task in natural language processing (NLP), information extraction, text mining, and various other applications where understanding the semantics of text and identifying key elements is important. Named entities generated from text data associated with the first dataset 112 may include specific terms or entities that provide information about a slide, its characteristics, or observed pathological findings. Non-exclusive examples of named entities may include diseases, conditions, tissue or organ names, cell types, cell structures, staining methods, staining techniques, gene names, protein names, diagnostic terms, and medical abbreviations.

[0029] Continuing to refer to Figure 1, the processor 104 may be configured to generate multiple named entities using a Named Entity Recognition (NER) system. Where used in this disclosure, “Named Entity Recognition (NER) system” is software that identifies multiple named entities from text. The NER system may be configured to identify multiple named entities from a first dataset 112. Inputs to the NER system may include multiple datasets 108, the first dataset 112, metadata, text data, etc. Outputs to the Named Entity Recognition system may include multiple named entities. Named entities may typically include structured representations of identified named entities in the form of annotations or tags attached to the original text.

[0030] Continuing to refer to Figure 1, the NER system can generate multiple named entities using a natural language processing model. Where used in this disclosure, “natural language processing (NLP) model” is a computational model designed to process and understand human language. It leverages techniques from machine learning, linguistics, and computer science to enable computers to understand, interpret, and generate natural language text. An NLP model may preprocess text data, where the input text may include a first dataset 112, or any other data referred to herein. Preprocessing the input text may include tasks such as tokenization (dividing the text into individual word or subword units), normalization of the text (e.g., lowercase, punctuation removal), and encoding the text into a numerical representation suitable for the model. An NLP model may include a transformer architecture, where the transformer is a deep learning model that uses an attention mechanism to capture relationships between word or subword units in a text sequence. These consist of multiple layers of self-attention and feedforward neural networks. An NLP model can weight the importance of different word or subword units in a text sequence, taking context into account. This allows the model to capture dependencies and relationships between words, taking into account both local and global contexts. This process can be used to identify multiple named entities. The language processing model may include a program that automatically generates associations between one or more important terms extracted from a first dataset 112, and detects associations between such important terms, including, but not limited to, mathematical associations. Associations between language elements (language elements including important terms extracted for the purposes of this specification), relationships between such categories and other such terms, may include, but not limited to, mathematical associations, including statistical correlations between any language element and any other language element and / or multiple language elements.Statistical correlations and / or mathematical associations may include, for example, probabilistic formulas or relationships that indicate the possibility that a given extracted key term represents a given category of semantic meaning. As a further example, statistical correlations and / or mathematical associations may include probabilistic formulas or relationships that indicate positive and / or negative associations between at least the extracted key terms and / or a given semantic relationship, and the positive or negative index may include an index that indicates whether a given document exhibits or does not exhibit a categorical semantic relationship. Whether a phrase, sentence, word, or other text element in the first dataset 112 constitutes a positive or negative index may be determined in one embodiment by, for example, mathematical associations between detected key terms, or by comparison with phrases and / or words that exhibit positive and / or negative indexes stored in the memory of the processor 104.

[0031] Continuing to refer to Figure 1, the processor 104 can generate a first set 120 of associations by pairing one or more visual features with multiple named entities. In some cases, the first set 120 of associations may include a joint representation of named entities and multiple visual features. A joint representation of a prognostic slide may be created by combining visual features from a second dataset 116 with named entities from the first dataset 112. In some cases, the joint representation can be used as training data for a machine learning model, as described later herein. In some cases, the pairing of one or more visual features with multiple named entities may include annotations overlaid on the image that function as a direct correlation between the visual data and the text data. These annotations can highlight specific areas or structures of interest within the pathology slide, and the corresponding text provides a detailed description or diagnosis associated with those areas. The annotations may indicate the presence of a tumor, grading or staging information, or specific abnormalities observed. In other cases, the pairing of one or more visual features with multiple named entities may include a pathology report that directly describes the visual features observed in the image. This report may provide detailed findings, interpretations, and diagnoses based on examination of pathology slides. It directly aligns with visual information describing histological structure, cellular morphology, tumor characteristics, and any other relevant pathological observations.

[0032] Continuing to refer to Figure 1, the processor 104 can generate a first set 120 of associations using a first association classifier 124. As used in this disclosure, “first association classifier” is a classifier configured to generate a first set 120 of associations. The first association classifier 124 may be consistent with the classifier described later in Figure 2. Inputs to the first association classifier 124 may include multiple datasets 108, a first dataset 112, a second dataset 116, and examples of the first set 120 of associations. Outputs to the first association classifier 124 may include the first set 120 of associations adjusted to the first datasets 112 and the second datasets 116. The first association training data may include multiple data entries, each containing multiple inputs correlated to multiple outputs for training the processor by a machine learning process. In one embodiment, the first association training data may include multiple first datasets 112 and multiple second datasets 116 correlated to examples of the first set 120 of associations. In another embodiment, the first association training data may include multiple named entities that correlate to multiple visual features. The first association training data may be received from database 300. The first association training data may include information about multiple datasets 108, a first dataset 112, a second dataset 116, and examples of a first set of associations 120. In one embodiment, the first association training data may be iteratively updated in accordance with past input and output results of the first association classifier or any other classifier referred to throughout this disclosure. The classifier may be a linear classifier such as a logistic regression classifier and / or a naive Bayes classifier, a nearest neighbor classifier such as a k nearest neighbor classifier, a support vector machine, a least squares support vector machine, a Fisher linear discriminant, a quadratic classifier, a decision tree, a boosted tree, a random forest classifier, and the like.

[0033] Continuing to refer to Figure 1, the processor 104 may assign a pathology identifier 128 to each visual feature or named entity. Where used in this disclosure, “pathology identifier” is a unique identification code or label assigned to both image data and text data associated with a pathology slide. This identifier serves as a common reference or link between the image data and the corresponding text information, enabling their association and retrieval. The pathology identifier 128 may represent the content of both the image data and the text data. In non-limiting examples, the pathology identifier 128 may identify both images and text containing information about the structure of tissue within a pathology slide. The pathology identifier 128 may be a numeric code, an alphanumeric code, a barcode, a QR code (registered trademark), etc. Using the same identifier for both modalities establishes direct or indirect associations, ensuring that the image data and text data represent the same pathology slide. In some cases, the pathology identifier 128 may be used to represent a first set of associations 120 or a second set of associations. The pathology identifier 128 may be assigned to a first set of associations 120 or a second set of associations 132 using the processor 104. This identifier serves as a common reference point for establishing connections between the visual content captured in the image data and the corresponding text information, such as clinical notes and images associated with the pathology report or pathology slide. This identifier facilitates the seamless integration and retrieval of information, enabling efficient organization, analysis, and retrieval of pathology data. Researchers, pathologists, and healthcare professionals can use the pathology identifier 128 to search and access image data and text data together, enabling a comprehensive understanding of the contents of pathology slides and facilitating accurate diagnosis, research, and collaboration.

[0034] Continuing to refer to Figure 1, the processor 104 is configured to identify a second set of associations 132 between the first dataset 112 and the second dataset 116, in accordance with a first set of associations 120. As used in this disclosure, “second set of associations” refers to a relationship or connection established between two or more types of data. This may include a relationship or connection established between image data and text data. The second set of associations 128 may be generated in a similar manner to the first set of associations. The second set of associations 132 may include linking or integrating image data from the second dataset 116 with descriptive or explanatory text data from the first dataset 112 to provide additional context, enhance understanding, and convey relevant information. The second set of associations 132 may be similar to the first set of associations 120. However, the second set of associations 132 may include multiple indirect correlations between the first dataset 112 and the second dataset 116. Where used in this disclosure, “indirect correlation” refers to a relationship in which the first dataset 112 provides text data or supplementary information to the image data of the second dataset 112, but does not explicitly describe the visual features observed in the images. Indirect correlations may provide contextual information related to the image content. This may include patient demographics, medical history, primary symptoms, diagnosis, treatment details, or other relevant clinical information associated with the pathology slide. While not directly describing the visual appearance, this information helps provide context and insight into the underlying pathology. In one embodiment, indirect correlations may include diagnostic findings or interpretations made by a pathologist or healthcare professional based on examination of the pathology slide. These findings may not explicitly describe the visual appearance, but provide insight into the presence of abnormalities, disease status, tumor grading or staging, or other diagnostic observations related to the image data. Indirect correlations between the first dataset 112 and the second dataset 116 may enable a broader understanding of diagnostic interpretations associated with the clinical context, patient medical history, and visual information captured in the images.While text data may not explicitly describe visual features, it provides supplementary information that enhances the interpretation and analysis of pathology slides.

[0035] Continuing to refer to Figure 1, the processor 104 is configured to identify a second set of associations 132 between a first dataset 112 and a second dataset 116 as a function of a first set of associations 120. The processor 104 may generate a second set of associations 132a as a function of a plurality of pathology identifiers 128. In one embodiment, the processor 104 pairs data from the first dataset 112 and the second dataset 116 based on the degree of similarity by the pathology identifiers 128. The processor 104 can generate similarity scores between query image features and features of other text descriptions in the datasets, and vice versa. This enables the processor 104 to identify the most similar and relevant image-text pairs. In some embodiments, the processor can perform image-text similarity calculations or text-image similarity calculations. These calculations can be used to evaluate how similar the content of image data is to the content of text data. As used in this disclosure, “similarity score” is a score that reflects the degree of similarity between the content of image data and the content of text data. Similarity scores can be generated by preprocessing both image data and text data. Preprocessing may include resizing, normalizing, and extracting relevant visual features using computer vision techniques such as convolutional neural networks (CNNs). Alternatively, preprocessing of text data may be done by tokenizing and removing stop words and converting them into numerical representations using techniques such as word embeddings or language models. Processor 104 can then associate each visual feature and named entity with a unique pathology identifier 128. This identifier serves as a link between image and text data associated with the same pathology slide. Processor 104 then generates similarity scores between the visual features of the second dataset 116 and the text features of the first dataset 112. Various similarity metrics can be used, such as cosine similarity, Euclidean distance, or other distance measures that capture similarity between feature vectors.Next, processor 104 can sort pairs of visual features and named entities in descending order based on their similarity scores. This identifies the most similar image-text pairs. Processor 104 can further filter the associations by considering only pairs that share the same pathology identifier. This ensures that the generated associations are based on the content similarity of the image and text data associated with the same pathology slide.

[0036] Continuing to refer to Figure 1, the processor 104 can generate a second set 132 of associations using a second association classifier 136. Where used in this disclosure, “second association classifier” is a classifier configured to generate a second set 132 of associations. The second association classifier 136 may be consistent with the classifier described later in Figure 2. Inputs to the second association classifier 136 may include multiple datasets 108, a first dataset 112, a second dataset 116, a first set 120 of associations, visual features, named entities, pathology identifiers, examples of the second set 132 of associations, etc. Outputs to the second association classifier 136 may include a second set 136 of associations adjusted to match the first dataset 112 and the second dataset 116. Outputs to the second association classifier 136 may further include similarity scores. The second association training data may include multiple data entries, each containing multiple inputs correlated to multiple outputs for training the processor by a machine learning process. In one embodiment, the second association training data may include a first set of associations 120 that correlates with a second set of associations 136. In another embodiment, the second association training data may include a plurality of visual features that correlate with a plurality of named entities. The second association training data may be received from a database 300. The second association training data may include information about a plurality of datasets 108, a first dataset 112, a second dataset 116, a first set of associations 120, visual features, named entities, pathology identifiers, a second set of associations 132, and so on. In one embodiment, the second association training data may be iteratively updated in accordance with past input and output results of a first association classifier or any other classifier referred to throughout this disclosure. The classifier may be a linear classifier such as a logistic regression classifier and / or a Naive Bayes classifier, a nearest neighbor classifier such as a k-nearest neighbor classifier, a support vector machine, a least squares support vector machine, a Fisher linear discriminant, a quadratic classifier, a decision tree, a boosted tree, a random forest classifier, and so on.

[0037] Continuing to refer to Figure 1, processor 104 can generate a second set 132 of associations using comparative fuzzy inference. As used in this disclosure, “comparative fuzzy inference” is a method of interpreting values ​​in an input vector (i.e., visual features and named entities) and assigning values ​​to an output vector based on a set of rules. The set of fuzzy rules may include a set of linguistic variables that describe how the system should make decisions regarding the classification of inputs or the control of outputs. Fuzzy inference rules operate on a fuzzy set and provide a framework for mapping input variables to output variables via linguistic rules. Fuzzy inference rules may operate using linguistic variables that represent inaccurate or ambiguous concepts rather than exact numerical values. Linguistic variables are defined by membership functions that describe membership or degree of truth about different linguistic terms or categories. In a non-limiting example, linguistic variables associated with the second set 132 of associations may have linguistic terms such as “highly relevant,” “moderately relevant,” and / or “not relevant,” each having its corresponding membership function. Fuzzy inference rules typically follow a conditional "IF-THEN" structure, consisting of an antecedent (IF part) and a consequent (THEN part). The antecedent specifies the condition or criterion under which the rule applies, and the consequent determines the output or conclusion of the rule. In one embodiment, a second set of associations 132 can be determined by comparing the degree of agreement between the first fuzzy set and the second fuzzy set, and / or between single values ​​within them, with each other or with either set, which is sufficient for the purposes of the matching process.

[0038] Referring further to Figure 1, the second set of associations 132 can be determined in accordance with the intersection of two fuzzy sets, where each fuzzy set can represent a visual feature and a named entity, respectively. Comparing the visual features and named entities may involve using a fuzzy set inference system as described later in this specification, or any scoring method as described throughout this disclosure. For example, but not limited to, the processor 104 may use a fuzzy logic model to determine the first set of associations 120 or the second set of associations 136 in accordance with a fuzzy set comparison technique as described in this disclosure. In some embodiments, each piece of information associated with a visual feature may be compared with a named entity, and the second set of associations 132 may be represented using a linguistic variable on a range of potential numerical values, where the values ​​of the linguistic variable may be represented as a fuzzy set on that range, where a “good” or “ideal” fuzzy set may correspond to a range of values ​​that can be characterized as ideal, while other fuzzy sets may correspond to a range that can be characterized as ordinary, bad, or other unideal ranges and / or values. In embodiments, these variables may be used to compare visual features with named entities to determine a second set 132 of associations specific to the visual features. A fuzzy inference system can combine such language variable values ​​according to one or more fuzzy inference rules, including any type of fuzzy inference system and / or rules described herein, to determine the degree of membership in one or more output language variables having values ​​representing ideal overall performance, average or moderate overall performance, and / or low or insufficient overall performance, and such mappings can be “defuzzy” as described below in more detail to provide an overall output and / or evaluation.

[0039] Referring further to Figure 1, the processor may be configured to generate machine learning models, such as a second association classifier 136, using a naive Bayes classification algorithm. The naive Bayes classification algorithm generates classifiers by assigning class labels to problem instances, which are represented as vectors of element values. The class labels are drawn from a finite set. The naive Bayes classification algorithm may involve generating a family of algorithms that, given class variables, assume that the values ​​of certain elements are independent of the values ​​of any other elements. The naive Bayes classification algorithm is derived from Bayes' theorem, expressed as P(A / B) = P(B / A)P(A)÷P(B), where P(A / B) is the probability of hypothesis A given data B, also known as the posterior probability; P(B / A) is the probability of data B assuming hypothesis A is true; P(A) is the probability that hypothesis A is true regardless of the data, also known as the prior probability of A; and P(B) is the probability of the data unrelated to the hypothesis. A naive Bayes algorithm can be generated by first converting the training data into a frequency distribution table. The processor 104 can then compute a likelihood table by calculating the probabilities of different data entries and classification labels. The processor 104 can then use the naive Bayes equation to compute the posterior probability of each class. The class with the highest posterior probability is the prediction result. A naive Bayes classification algorithm may include a Gaussian model following a normal distribution. A naive Bayes classification algorithm may include a multinomial model used for discrete counts. A naive Bayes classification algorithm may include a Bernoulli model, which can be used when the vectors are binary.

[0040] Referring further to Figure 1, the processor 104 may be configured to generate a machine learning model, such as a second association classifier 136, using the K-Nearest Neighbors (KNN) algorithm. When used in this disclosure, the "K-Nearest Neighbors algorithm" includes a classification method that leverages feature similarity to analyze how similar out-of-sample features are to the training data and classifies the input data into one or more clusters and / or categories of features as represented in the training data, which may be performed by representing both the training data and the input data in vector form and using one or more measures of vector similarity to identify classifications in the training data and determine the classification of the input data. The K-Nearest Neighbors algorithm may include specifying a K value, or a number that instructs the classifier to select k most similar entries for a given sample; determining the most common classifier for entries in a database; and classifying known samples, which may be performed recursively and / or iteratively to generate classifiers that can be used to classify the input data as further samples. For example, an initial set of samples may be run to cover initial heuristics and / or “first inferences” in the output and / or relationships, which may be seeded with expert inputs received according to any process as described herein, but not limited to such processes. As a non-limiting example, the initial heuristics may include ranking the associations between input and training data elements. The heuristics may include selecting some of the highest-ranking associations and / or training data elements.

[0041] Continuing to refer to Figure 1, the k-nearest neighbor algorithm generates a first vector output containing the data entry cluster, a second vector output containing the input data, and the distance between the first and second vector outputs can be calculated using any appropriate norm, such as cosine similarity or Euclidean distance measure. Each vector output can be represented as an n-tuple of values, where n is at least two values. Each value in the n-tuple of values ​​can represent a measured or other quantitative value associated with a given category or attribute of the data, examples of which are provided in further detail below. The vectors can be represented in n-dimensional space using a per-category axis of the values ​​represented in the n-tuple of values, such that the vectors have a geometric direction that characterizes the relative amounts of the attributes in the n-tuple compared to each other, although this is not limited to the vectors. Two vectors can be considered equivalent if their directions and / or the relative amounts of the values ​​in each vector compared to each other are the same. Thus, as a non-restrictive example, a vector represented as [5,10,15] can be treated as equivalent to a vector represented as [1,2,3] for the purposes of this disclosure. Vectors may be more similar if their directions are more similar, and more different if their directions are more diverse, but vector similarity may alternatively or additionally be determined by the mean of similarity between similar attributes, or any other measure of similarity suitable for any n tuple of values, or by an aggregation of numerical similarity measures for the purpose of a loss function, as described in more detail below. Any vectors as described herein can be scaled so that each vector represents each attribute along an equivalent scale of values. Each vector may be “normalized,” or Pythagorean norm

number

[0042] Continuing to refer to Figure 1, the processor 104 may be configured to generate a third dataset 140 using a second set of associations 132. As used in this disclosure, the “third dataset” is a dataset 108 containing multiple text data and / or multiple image data. The third dataset 140 is generated based on the second set of associations 132. In one embodiment, the text data and / or image data may be used together with the second set of associations 132 to identify additional text data or image data that has an indirect or direct correlation to the initial text data and / or image data. The additional text data and / or image data may be identified from the first dataset 112, the second dataset 116, or the database 300. The processor 104 may identify the third dataset using a cross-modal search process. As used in this disclosure, a “cross-modal search process” refers to the process of retrieving data from one modality (such as images) based on a query from another modality (such as text), or vice versa. The cross-modal search process involves finding related instances within one modality that are semantically related to queries in different modalities. The cross-modal search process may be performed depending on the pathology identifier 128. The processor 104 may assign pathology identifiers 128 to initial text data and / or image data. Pathology identifiers 128 associated with initial text data and / or image data may be classified into other related pathology identifiers 128 to identify image data and / or text data in a third dataset 140. In an unrestricted example, the processor 104 may receive image data depicting inflamed tissue along with a second set 132 of associations. The processor uses the cross-modal search process to generate a third dataset 140 containing multiple text data indirectly correlated with the image data.

[0043] Continuing to refer to Figure 1, the processor 104 can generate a third dataset 140 using a dataset classifier. As used in this disclosure, “dataset classifier” is a classifier configured to generate a third dataset 140. The dataset classifier may be consistent with the classifier described later in Figure 2. Inputs to the dataset classifier may include multiple datasets 108, a first dataset 112, a second dataset 116, a first set of associations 120, visual features, named entities, pathology identifiers, a second set of associations 132, and an example of the third dataset 140. Outputs to the dataset classifier may include the third dataset 140, which includes text data and / or image data. The dataset training data may include multiple data entries, each containing multiple inputs that correlate to multiple outputs for training the processor by a machine learning process. In one embodiment, the dataset training data may include a second set of associations 136 that correlate to an example of the third dataset 140. In another embodiment, the dataset training data may include multiple visual features that correlate to multiple named entities. Dataset training data may be received from database 300. The dataset training data may include information about multiple datasets 108, a first dataset 112, a second dataset 116, a first set of associations 120, visual features, named entities, pathology identifiers, a second set of associations 132, a third dataset 140, and so on. In one embodiment, the dataset training data may be iteratively updated in accordance with the input and output results of past first association classifiers or any other classifiers referred to throughout this disclosure. Classifiers may include, but are not limited to, linear classifiers such as logistic regression classifiers and / or naive Bayes classifiers, nearest neighbor classifiers such as k nearest neighbor classifiers, support vector machines, least squares support vector machines, Fisher linear discriminants, quadratic classifiers, decision trees, boosted trees, random forest classifiers, and so on.

[0044] Referring further to Figure 1, the processor 104 may be configured to display a second set 132 of associations using a display device 144. As used in this disclosure, “display device” is a device used to display content. The display device 144 may include a user interface. As used herein, a “user interface” is a means by which a user and a computer system interact, for example, through the use of an input device and software. A user interface may include a graphical user interface (GUI), a command-line interface (CLI), a menu-driven user interface, a touch user interface, a voice user interface (VUI), a form-based user interface, or any combination thereof. A user interface may include a smartphone, smart tablet, desktop, or laptop operated by a user. In one embodiment, a user interface may include a graphical user interface. As used herein, a “graphical user interface (GUI)” is a graphical form of a user interface that enables a user to interact with an electronic device. In some embodiments, a GUI may include icons, menus, other visual indicators, or representations (graphics), audio indicators such as primary notation, and display information and associated user controls. A menu may contain a list of options from which the user can select one. A menu bar, such as a pull-down menu, may appear horizontally across the screen. When any option in this menu is clicked, a pull-down menu may appear. A menu may include a context menu that appears only when the user performs a specific action. An example of this is pressing the right mouse button. When this is done, a menu may appear under the cursor. Files, programs, web pages, etc., can be represented using small pictures within a graphical user interface. For example, a link to a decentralized platform, such as those described in this disclosure, may be incorporated using an icon.Icons can be a quick way to open documents, run programs, and so on, because clicking them provides immediate access. Information contained in a user interface can be directly affected using graphical control elements such as widgets. “Widget,” as used herein, is a user control element that allows a user to control and change the appearance of an element within a user interface. In this context, a widget can refer to a common GUI element such as a checkbox, button, or scroll bar to an instance of that element, or a customized set of such elements used for a particular function or application (such as a dialog box for a user to customize the appearance of a computer screen). User interface controls can include software components that a user interacts with directly through manipulation to read or edit information displayed through the user interface. Widgets can be used to display lists of related items, navigate a system using links and tabs, and manipulate data using checkboxes, radio buttons, and so on.

[0045] Continuing to refer to Figure 1, the device 100 for detecting associations between different types of datasets may include at least a processor 104 and memory communicably connected to at least this processor. The memory may include instructions that configure the processor 104 to receive a plurality of datasets 108. The plurality of datasets 108 may include a first dataset 112 and a second dataset 116. Instructions may instruct the processor to identify a first set 120 of associations between a first subset of the first dataset 112 and a first subset of the second dataset 116. In some cases, generating the first set of associations includes identifying a plurality of named entities in the first dataset 112, where each named entity is associated with at least a data element in the second dataset 116. For example, the first set of associations may be generated through explicit naming of configuration data in one or more of the first dataset 112 and the second dataset 116. Alternatively or additionally, the first set of associations may be received from expert input or user input.

[0046] Continuing to refer to Figure 1, the instruction may further instruct the processor 104 to generate a second set 132 of associations between a second subset of the first dataset 112 and a second subset of the second dataset 116. In some cases, one or more of the second subsets of the first dataset 112 and the second subsets of the second dataset 116 are larger (e.g., have more data or data elements) than one or more of the first subsets of the first dataset 112 and the first subsets of the second dataset 116. Similarly, in some cases, the second subset may include some or all of the first dataset 112 and / or the first subset of the second dataset. In some cases, the generation of the second set 132 of associations may be performed in accordance with the first set 120 of associations, for example, by using a second association classifier 136. In some cases, generating the second set 132 of associations may include training the second association classifier 136 using second association training data 300. The second association classifier 136 may include any classifier described in this disclosure, for example with reference to Figure 2. The second association training data may include any training data described in this disclosure, for example with reference to Figure 2. In some versions, the second association training data may include multiple data entries, including a first set of associations 120. In some cases, generating a second set of associations 132 may be performed using the trained second association classifier 136, depending on the first set of associations. Finally, an instruction may instruct the processor 104 to display the second set of associations using a display device. In some cases, displaying the second set of associations may include displaying only a portion of the second set of associations 132 and / or displaying data that requires the second set of associations 132 but does not explicitly represent the second set of associations 132.

[0047] Referring further to Figure 1, in some embodiments, the second association training data 300 may include a first subset of the first dataset 112 as input that correlates with a first subset of the second dataset 116 as output, i.e., the second association training data 300 may include a first set 120 of associations.

[0048] Referring further to Figure 1, in some embodiments, generating a second set of associations 132 may further include training a generative machine learning process using a first dataset 112, synthesizing a first synthetic data in accordance with the first dataset using generative machine learning, and generating a second set of associations 132 in accordance with the first synthetic data and the first set of associations. The generative machine learning process may include any generative machine learning process described in this disclosure, for example with reference to Figure 2. The synthetic data may include any generative data described in this disclosure, for example with reference to Figure 2. In some cases, the second association training data may include the first synthetic data as input correlated to one or more subsets of the first or second dataset as output.

[0049] Referring further to Figure 1, in some embodiments, one or more of the first and second datasets may contain text or text data. In some cases, generating a second set of associations 132 may further include using a natural language processing model to associate text data in one or more of the first and second datasets 112 and 116, and generating a second set of associations 132 in accordance with the associated text data. The natural language processing model may include any language processing model or process described herein, for example, with reference to Figure 2.

[0050] Referring further to Figure 1, in some embodiments, generating a second set of associations 132 may further include calculating distances between data elements in one or more of the first dataset 112 and the second dataset 116, and generating a second set of associations 132 according to the distances between the data elements. The distances may include any distances described in this disclosure, such as vector distances, including the disclosure in Figure 2.

[0051] Referring further to Figure 1, in some embodiments, one or more of the first dataset 112 and the second dataset 116 include metadata, and generating a second set of associations 132 may further include associating metadata within one or more of the first dataset and the second dataset, and generating a second set of associations 132 in accordance with the associated metadata. The metadata may include any metadata or contextual data as described in this disclosure, for example, referring to Figure 2.

[0052] Referring further to Figure 1, in some embodiments, one or more of the first dataset 112 and the second dataset 116 may contain image data, and generating a second set 132 of associations may involve using a machine vision system to identify one or more visual features in one or more of the first dataset and the second dataset.

[0053] Referring here to Figure 2, an exemplary embodiment of a machine learning module 200 capable of performing one or more machine learning processes as described in this disclosure is shown. The machine learning module may use the machine learning processes to perform decision, classification, and / or analysis steps, methods, processes, etc., as described in this disclosure. A “machine learning process,” as used in this disclosure, is a process that automatically uses training data 204 to generate an algorithm instantiated with hardware or software logic, data structures, and / or functions, which are executed by a computing device / module to produce an output 208 given data provided as input 212, in contrast to a non-machine learning software program, where the commands to be executed are predetermined by the user and written in a programming language.

[0054] Referring further to Figure 2, “training data,” as used herein, is data containing correlations that a machine learning process can use to model relationships between two or more categories of data elements. For example, but not limited to, training data 204 may contain multiple data entries, also known as “training examples,” each entry representing a set of data elements recorded, received, and / or generated together, and the data elements may be correlated by the presence of commons in a given data entry, by proximity in a given data entry, etc. Multiple data entries within training data 204 may reveal one or more tendencies of correlation between categories of data elements, for example, but not limited to, a higher value of a first data element belonging to a first category of data elements may tend to correlate with a higher value of a second data element belonging to a second category of data elements, and may show a potential proportional relationship or other mathematical relationship linking values ​​belonging to two categories. Multiple categories of data elements can be related in the training data 204 according to various correlations, which may indicate causal and / or predictive links between categories of data elements, and which can be modeled as relationships such as mathematical relationships by a machine learning process, as will be described in more detail below. The training data 204 can be formatted and / or organized by categories of data elements, for example, by associating data elements with one or more descriptors corresponding to categories of data elements. As a non-limiting example, the training data 204 may include data entered in a standardized format by a person or process such that entries of a given data element in a given field in a form can be mapped to one or more descriptors of a category.Elements within the training data 204 may be linked to category descriptors by tags, tokens, or other data elements, for example, the training data 204 may be provided in a fixed-length format, a format that links the data location to categories such as Comma-Separated Value (CSV) format, and / or a self-describing format such as Extensible Markup Language (XML), JavaScript® Object Notation (JSON), enabling a process or device to discover the categories of the data.

[0055] Alternatively or additionally, continuing to refer to Figure 2, the training data 204 may include one or more unclassified elements; that is, the training data 204 may not be formatted or may not include descriptors for some elements of the data. Machine learning algorithms and / or other processes can sort the training data 204 according to one or more classifications, for example, using natural language processing algorithms, tokenization, or detection of correlation values ​​in the raw data, and the categories may be generated using correlation and / or other processing algorithms. As a non-limiting example, in a text corpus, phrases constituting a number "n" of compound words, such as nouns modified by other nouns, may be identified according to the statistically significant prevalence of n-grams containing such words in a particular order, and such n-grams may be classified as linguistic elements such as "words," which will be tracked as well as single words, and new categories may be generated as a result of statistical analysis. Similarly, in data entries containing some text data, a person's name may be identified by referencing a list, dictionary, or other list of terms, which may enable ad-hoc classification by machine learning algorithms and / or automated association of data within the data entry with descriptors or to a given format. The ability to automatically classify data entries may make the same training data 204 applicable to two or more separate machine learning algorithms, as will be described in more detail below. The training data 204 used by the machine learning module 200 may correlate any input data as described in this disclosure to any output data as described in this disclosure. Examples of non-limiting and illustrative examples include the first set of associations 120, direct correlation, the second set of associations 132, or any data referred to throughout this disclosure.

[0056] Referring further to Figure 2, one or more supervised and / or unsupervised machine learning processes and / or models can be used to filter, sort, and / or select training data, and such models may include, but are not limited to, a training data classifier 216. The training data classifier 216 may include a machine learning model, such as a “classifier,” which, when used in this disclosure, represents and / or uses a data structure, such as a mathematical model, neural network, or program generated by a machine learning algorithm known as a “classification algorithm,” as defined below, which sorts the input into categories or bins of data and outputs categories or bins of data and / or labels associated therewith. The classifier may be configured to output at least one data set that labels or otherwise identifies, such as datasets that have been clustered together and found to be close under a distance metric, as described below. The distance metric may include, but are not limited to, any norm, such as the Pythagorean norm. The machine learning module 200 may generate a classifier using a classification algorithm defined as a process by which a computing device and / or any module and / or component operating therein derives a classifier from the training data 204. Classification may be performed using, but not limited to, linear classifiers such as logistic regression classifiers and / or naive Bayesian classifiers, nearest neighbor classifiers such as k-nearest neighbor classifiers, support vector machines, least squares support vector machines, Fisher linear discriminant, quadratic classifiers, decision trees, boosted trees, random forest classifiers, learning vector quantization, and / or neural network-based classifiers. As an unrestricted example, the training data classifier 216 may classify elements of the training data into a first set 120 of associations.

[0057] Referring further to Figure 2, training examples for use as training data may be selected from a population of potential examples according to a cohort related to the analytical problem or classification task to be solved. Alternatively or additionally, training data may be selected to span a set of possible situations or inputs for the machine learning model and / or process encountered during deployment. For example, for each category of input data to a machine learning process or model that may exist within a range of values ​​in a set of phenomena such as images, user data, process data, and physical data, the computing device, processor, and / or machine learning model may select training examples that represent each possible value and / or a representative sample of values ​​in such a range. The selection of representative samples may include, for example, selecting training examples in proportion to a statistically determined and / or predicted distribution of values ​​according to relative frequency, such that values ​​that are encountered more frequently in the population of data thus analyzed are represented by more training examples than values ​​that are encountered less frequently. Alternatively or additionally, the set of training examples may be compared against and / or presented to the user a set of representative values ​​in a database so that the process can automatically or via user input detect one or more values ​​that are not included in the set of training examples. Computing devices, processors, and / or modules can automatically generate missing training examples, which may be done by receiving and / or retrieving missing input and / or output values ​​and correlating the missing input and / or output values ​​with the retrieved values ​​and corresponding output and / or input values ​​collated within the data record, provided by the user and / or other devices, etc.

[0058] Referring further to Figure 2, a computer, processor, and / or module may be configured to sanitize training data. "Sanitizing training data," as used in this disclosure, is the process of removing training examples that would hinder the convergence of a machine learning model and / or processing to useful results. For example, but not limited to, training examples may include input and / or output values ​​that are outliers from values ​​typically encountered so that a machine learning algorithm using the training examples fits less likely quantities as input and / or output, for example, values ​​exceeding a threshold number of standard deviations from the mean, mean, or expected value may be excluded. Alternatively or additionally, one or more training examples may be identified as having low-quality data, where "low-quality" is defined as having a signal-to-noise ratio below a threshold.

[0059] As a non-limiting example, and referring further to Figure 2, images used to train an image classifier or other machine learning models and / or processes that take images as input or produce images as output may be rejected if their image quality falls below a threshold. For example, but not limited to, computing devices, processors, and / or modules may perform blur detection and reject one or more blur detections, as a non-limiting example, by performing an approximation such as a Fourier transform or fast Fourier transform (FFT) of the image and analyzing the distribution of low and high frequencies in the resulting frequency domain depiction of the image, where the number of high-frequency values ​​below a threshold level may indicate blur. As a further non-limiting example, blur detection may be performed by convolving the image or the channels of the image etc. with a Laplacian kernel, which may generate a numerical score that reflects some rapid changes in intensity shown in the image, such that a high score indicates clarity and a low score indicates blur. Blur detection can be performed using gradient-based operators that measure the operator based on the gradient or first derivative of an image, based on the hypothesis that abrupt changes indicate sharp edges in an image and therefore indicate a lower degree of blur. Blur detection can be performed using wavelet-based operators that take advantage of the ability of discrete wavelet transform coefficients to describe the frequency and spatial content of an image. Blur detection can be performed using statistics-based operators that take advantage of several image statistics as texture descriptors to calculate the focus level. Blur detection can be performed by using discrete cosine transform (DCT) coefficients to calculate the focus level of an image from its frequency components.

[0060] Continuing to refer to Figure 2, the computing device, processor, and / or module may be configured to precondition one or more training examples. For example, if a machine learning model and / or process has one or more inputs and / or outputs that transmit or receive, requiring a certain number of bits, samples, or other units of data, then the elements of one or more training examples that will be used as inputs and / or outputs, or compared to them, can be modified to have such a number of data units. For example, the computing device, processor, and / or module may convert a smaller number of units, such as in a low-pixel-count image, into a desired number of units, for example, by upsampling and interpolation. As a non-limiting example, a low-pixel-count image may have 100 pixels, but the desired number of pixels may be 128. The processor can interpolate the low-pixel-count image to convert 100 pixels to 128 pixels. Those skilled in the art should note that by reading this disclosure, they will be able to understand various methods for interpolating a smaller number of data units, such as samples, pixels, or bits, into a desired number of such units. In some cases, the set of interpolation rules may be trained by a neural network or other machine learning model trained to predict interpolated pixel values ​​using training data, along with very detailed inputs and / or outputs, as well as corresponding sets of inputs and / or outputs downsampled to fewer units. As a non-limiting example, sample inputs and / or outputs, such as a sample picture with sample augmented data units (e.g., pixels added between the original pixels), may be input to a neural network or machine learning model and output a pseudo-replica sample picture with dummy values ​​assigned to the pixels between the original pixels based on the set of interpolation rules.As a non-limiting example, in the context of an image classifier, a machine learning model may have a set of interpolation rules trained on a set of very detailed images and images downsampled to fewer pixels, and a neural network or other machine learning model trained using those examples to predict interpolated pixel values ​​in a face picture context. As a result, an input with sample-expanded data units (added between the original data units, with dummy values) may be processed through the trained neural network and / or model, which may be input with values ​​to replace the dummy values. Alternatively or additionally, a processor, computing device, and / or module may utilize a sample expander method, a low-pass filter, or both. As used in this disclosure, a “low-pass filter” is a filter that allows signals below a selected cutoff frequency to pass through and attenuates signals above a cutoff frequency. The exact frequency response of the filter depends on the filter design. A computing device, processor, and / or module may use averaging, such as luma or chroma averaging, in the image to fill in the data units between the original data units.

[0061] In some embodiments, continuing with reference to Figure 2, a computing device, processor, and / or module may downsample elements of a training example to a desired number of fewer data elements. As a non-limiting example, a high-pixel-count image may have 256 pixels, but the desired number of pixels may be 128. The processor can downsample the high-pixel-count image to convert 256 pixels to 128 pixels. In some embodiments, the processor may be configured to perform downsampling on the data. Downsampling, also known as decimation, may involve a process known as “compression,” which removes every Nth entry in a sequence of samples, all but the every Nth entry, etc., and can be performed, for example, by an N-sample compressor implemented using hardware or software. Anti-aliasing filters and / or anti-imaging filters and / or low-pass filters may be used to remove the side effects of compression.

[0062] Referring further to Figure 2, the machine learning module 200 may be configured to perform a delayed learning process 220 and / or protocol, which may alternatively be called a “delayed loading” or “invoke on demand” process and / or protocol, and may be a process in which machine learning is performed by receiving an input which will be converted into an output, and then combining this input and training set to derive an algorithm which will be used to generate the output on demand. For example, an initial set of simulations may be performed to cover an initial heuristic and / or “first guess” on the output and / or relationships. As a non-limiting example, the initial heuristic may include ranking the associations between the input and elements of the training data 204. The heuristic may include selecting some of the highest-ranking associations and / or elements of the training data 204. Delayed learning can implement any suitable delayed learning algorithm, including, but not limited to, the K-nearest neighbor algorithm, the delayed Naive Bayes algorithm, etc., and those skilled in the art will recognize, upon considering the entirety of this disclosure, a variety of delayed learning algorithms that can be applied to produce the output described herein, including, but not limited to, delayed learning applications of machine learning algorithms as described below in more detail.

[0063] Alternatively or additionally, referring again to Figure 2, a machine learning model 224 can be generated using a machine learning process such as those described in this disclosure. “Machine learning model” is, as used in this disclosure, a data structure that represents and / or instantiates a mathematical and / or algorithmic representation of a relationship between inputs and outputs, such that it is generated and stored in memory using any machine learning process, including, but not limited to, any of the processes described above, and the inputs are submitted to a previously created machine learning model 224 that generates outputs based on the derived relationships. For example, a linear regression model generated using a linear regression algorithm may compute a linear combination of input data using coefficients derived during the machine learning process to compute the output data. As a further non-limiting example, a machine learning model 224 may be generated by creating an artificial neural network, such as a convolutional neural network, which includes an input layer of nodes, one or more hidden layers, and an output layer of nodes. The connections between nodes can be created through the process of "training" the network, where elements from a set of training data 204 are applied to the input nodes. Then, using an appropriate training algorithm (such as the Levenberg-Marquardt method, conjugate gradient, simulated annealing, or other algorithms), the connections and weights between nodes in adjacent layers of the neural network are adjusted to produce the desired values ​​in the output nodes. This process is sometimes called deep learning.

[0064] Referring further to Figure 2, the machine learning algorithm may include at least a supervised machine learning process 228. At least a supervised machine learning process 228 as defined herein includes an algorithm that takes a training set of several inputs relating to several outputs and attempts to generate one or more data structures that represent and / or instantiate one or more mathematical relations relating the inputs to the outputs, each of which is optimal for some criteria specified for the algorithm using some scoring function. For example, a supervised learning algorithm may include a first set 120 of such associations as inputs, a second set 120 of associations as outputs, and a scoring function that represents a desired form of relationship to be detected between the inputs and outputs, the scoring function may, for example, attempt to maximize the probability that a given input and / or combination input of elements is related to a given output and minimize the probability that a given input is not related to a given output. The scoring function can be expressed as a risk function representing the “expected loss” of an algorithm relating inputs to outputs, where the loss is calculated as an error function representing the degree to which the predictions generated by the relationship are inaccurate when compared to a given input-output pair provided to the training data 204. Those skilled in the art will recognize, upon considering the entirety of this disclosure, various possible variations of at least the supervised machine learning process 228 that can be used to determine the relationship between inputs and outputs. The supervised machine learning process may include a classification algorithm such as the one defined above.

[0065] Referring further to Figure 2, training a supervised machine learning process may include, but is not limited to, iteratively updating coefficients, biases, and weights based on an error function, expected loss, and / or risk function. For example, the output generated by a supervised machine learning model using input examples in training examples may be compared with output examples from training examples, and an error function may be generated based on the comparison, which may include any error function suitable for use in any machine learning algorithm described herein, such as the square of the difference between one or more sets of comparison values. Such an error function may be used to update one or more weights, biases, coefficients, or other parameters of a machine learning model via any suitable process, including, but is not limited to, a gradient descent process, a least-squares process, and / or other processes described herein. This may be done iteratively and / or recursively to gradually adjust such weights, biases, coefficients, or other parameters. The updates may be performed in a neural network using one or more backpropagation algorithms. The iterative and / or recursive updates of weights, biases, coefficients, or other parameters described above can be performed until the currently available training data is exhausted and / or until a convergence test is passed. A “convergence test” is a test of a condition selected to indicate that the model and / or its weights, biases, coefficients, or other parameters have reached a certain degree of accuracy. In a convergence test, for example, the difference between two or more consecutive error or error function values ​​can be compared, where a difference below a threshold amount may be interpreted as indicating convergence. Alternatively or additionally, one or more error and / or error function values ​​evaluated in training iterations can be compared to a threshold.

[0066] Referring further to Figure 2, computing devices, processors, and / or modules may be configured to execute methods, process steps, sequences of process steps, and / or algorithms described with reference to this figure in any order and at any degree of iteration. For example, computing devices, processors, and / or modules may be configured to repeatedly execute a single process, sequence, and / or algorithm until a desired or instructed result is achieved, and the iteration of a process or sequence of process steps is performed iteratively and / or recursively using the output of the previous iteration as input to the subsequent iteration, aggregating the inputs and / or outputs of the iterations to produce an aggregated result, decreasing or phasing out one or more variables such as global variables, and / or dividing a larger processing task into a set of smaller processing tasks that are iteratively dealt with. Computing devices, processors, and / or modules may execute any process, sequence of process, or algorithm in parallel, such as executing a process simultaneously and / or substantially simultaneously two or more times using two or more parallel threads, processor cores, etc., and task division between parallel threads and / or processes may be performed according to any protocol suitable for task division between iterations. Those skilled in the art, upon reviewing the entirety of this disclosure, will recognize a variety of ways in which processes, sequences of processes, processing tasks, and / or data can be subdivided, shared, or otherwise processed using iterative, recursive, and / or parallel processing.

[0067] Referring further to Figure 2, the machine learning process may include at least an unsupervised machine learning process 232. An unsupervised machine learning process, as used herein, is a process that derives inferences within a dataset regardless of labels, and as a result, it can freely discover any structures, relationships, and / or correlations provided in the data. An unsupervised process may not require a response variable, and can be used to find interesting patterns and / or inferences between variables, determine the degree of correlation between two or more variables, and so on.

[0068] Referring further to Figure 2, the machine learning module 200 may be designed and configured to create a machine learning model 224 using techniques for developing linear regression models. A linear regression model may include a standard least-squares regression, which aims to minimize the square of the difference between the predicted and actual results according to a suitable norm (e.g., the vector space distance norm) for measuring such a difference, and the coefficients of the resulting linear equation can be modified to improve the minimization. A linear regression model may include a ridge regression method, where the function to be minimized includes a least-squares function and a term that multiplies the square of each coefficient by a scalar quantity to penalize large coefficients. A linear regression model may include a least absolute contraction selection operator (LASSO) model, where ridge regression is combined with multiplying the least-squares term by a coefficient obtained by dividing 1 by twice the number of samples. A linear regression model may include a multitask lasso model, where the norm applied to the least-squares term of the lasso model is the Frobenius norm, which corresponds to the square root of the sum of the squares of all terms. Linear regression models may include elastic network models, multitask elastic network models, minimum angle regression models, LARS Lasso models, orthogonal matching tracking models, Bayesian regression models, logistic regression models, stochastic gradient descent models, perceptron models, passive attack algorithms, robust regression models, Huber regression models, or any other suitable models that a person skilled in the art may conceive of by considering the entirety of this disclosure. In one embodiment, the linear regression model can be generalized to a polynomial regression model, thereby finding a polynomial (e.g., a quadratic, cubic, or higher-order equation) that provides the best predictive output / actual output fit. As will be apparent to a person skilled in the art by considering the entirety of this disclosure, similar methods as described above can be applied to minimize the error function.

[0069] Continuing to refer to Figure 2, machine learning algorithms may include, but are not limited to, linear discriminant analysis. Machine learning algorithms may include quadratic discriminant analysis. Machine learning algorithms may include kernel ridge regression. Machine learning algorithms may include, but are not limited to, support vector machines, including support vector classification-based regression processes. Machine learning algorithms may include stochastic gradient descent algorithms, including classification and regression algorithms based on stochastic gradient descent. Machine learning algorithms may include nearest neighbor algorithms. Machine learning algorithms may include various forms of latent space regularization, such as variational regularization. Machine learning algorithms may include Gaussian processes, such as Gaussian process regression. Machine learning algorithms may include cross-decomposition algorithms, including partial least squares and / or canonical correlation analysis. Machine learning algorithms may include naive Bayes methods. Machine learning algorithms may include decision tree-based algorithms, such as decision tree classification or regression algorithms. Machine learning algorithms may include ensemble methods, such as bagging meta-estimators, random tree forests, AdaBoost, gradient tree boosting, and / or voting classifier methods. Machine learning algorithms may include neural network algorithms, including convolutional neural network processes.

[0070] Referring further to Figure 2, machine learning models and / or processes can be deployed or instantiated by being incorporated into programs, devices, systems, and / or modules. For example, but not limited to, machine learning models, neural networks, and / or some or all of their parameters can be stored and / or deployed in any memory or circuit. Parameters such as coefficients, weights, and / or biases can be stored as circuit-based constants such as arrays of wires set to logical "1" and "0" voltage levels in a logic circuit and / or binary inputs and / or outputs to represent numbers in any suitable encoding system, including two's complement, or they can be stored in any volatile and / or non-volatile memory. Similarly, mathematical operations and inputs and / or outputs of data to and from models, neural network layers, etc., can be instantiated in the form of machine code such as instructions, binary arithmetic code instructions, assembly language, or any higher-order programming language in hardware circuits and / or firmware. Machine learning processes and / or models can be instantiated using any technology for hardware and / or software instantiation of memory, instructions, data structures, and / or algorithms, which includes, but are not limited to, the manufacture and / or configuration of non-reconfigurable hardware elements, circuits, and / or modules such as ASICs, but are not limited to, the manufacture and / or configuration of reconfigurable hardware elements, circuits, and / or modules such as FPGAs, but are not limited to, the manufacture and / or configuration of non-reconfigurable and / or configuration non-rewritable memory elements, circuits, and / or modules such as non-rewritable ROMs, but are not limited to, the manufacture and / or configuration of reconfigurable and / or rewritable memory elements, circuits, and / or modules such as rewritable ROMs or other memory technologies described herein, and / or any combination of the manufacture and / or configuration of any computing devices and / or components described herein.Such deployments and / or instantiated machine learning models and / or algorithms can receive input from any other processes, modules, and / or components described in this disclosure and generate outputs to any other processes, modules, and / or components described in this disclosure.

[0071] Continuing to refer to Figure 2, any process of training, retraining, deploying, and / or instantiating any machine learning model and / or algorithm can be performed and / or repeated after initial deployment and / or instantiation to modify, refine, and / or improve the machine learning model and / or algorithm. Such retraining, deployment, and / or instantiation may be performed as a periodic or regular process, such as retraining, deployment, and / or instantiation in a periodic elapsed period, after some measure of quantity, such as the number of bytes of data processed or other measures, the number of uses or executions of the processes described herein, and / or according to a software, firmware, or other update schedule. Alternatively or additionally, retraining, deployment, and / or instantiation may be event-based and / or triggered by user input exhibiting suboptimal or other problematic performance, and / or by automated field testing and / or audit processes, which may compare the output of the machine learning model and / or algorithm, and / or its error and / or error function, to any threshold, convergence test, etc., and / or the output of the processes described herein to similar threshold, convergence test, etc. Event-based retraining, deployment, and / or instantiation may, alternatively or additionally, be triggered by the receipt and / or generation of one or more new training examples, where some new training examples can be compared to a pre-configured threshold, and if the threshold is exceeded, retraining, deployment, and / or instantiation can be triggered.

[0072] Referring further to Figure 2, retraining and / or additional training can be performed using any currently or previously deployed version of the machine learning model and / or algorithm as a starting point, and using any process for training described above. Training data for retraining can be collected, pre-conditioned, sorted, classified, sanitized, or otherwise processed according to any process described herein. Training data includes, but is not limited to, training examples including inputs and correlated outputs used, received, and / or generated from any version of any system, module, machine learning model or algorithm, apparatus, and / or method described herein, such examples can be modified and / or labeled according to user feedback or other processes to show desired results, and / or have actual or measured results from processes modeled and / or predicted by the system, module, machine learning model or algorithm, apparatus, and / or method as “desired” results to be compared with the outputs for the training process as described above.

[0073] Redeployment can be performed by any reconfiguration and / or rewriting of reconfigurable and / or rewritable circuit and / or memory elements, or by generating new hardware and / or software components, circuits, instructions, etc., which may be added to and / or replace existing hardware and / or software components, circuits, instructions, etc.

[0074] Continuing to refer to Figure 2, the machine learning process may include a generative machine learning process. As used in this disclosure, “generative machine learning process” is a process that takes prompts (i.e., inputs) to automatically generate outputs that are consistent with the training data, in contrast to non-machine learning software programs whose outputs are predetermined by the user and written in a programming language. Typically, a generative machine learning process determines patterns and structures from the training data and uses these patterns and structures to synthesize new data with similar properties, depending on the input.

[0075] Continuing to refer to Figure 2, generative machine learning processes can synthesize data of different types or domains, including, but not limited to, text, code, images, molecules, audio (e.g., music), video, and robotic motion (e.g., electromechanical system motion). Exemplary generative machine learning systems trained on words or word tokens operating in the text domain include, but not limited to, GPT-3, LaMDA, LLaMA, BLOOM, and GPT-4. Exemplary machine learning processes trained on programming language text (i.e., code) include, but not limited to, OpenAI Codex. Exemplary machine learning processes trained on sets of images (e.g., with text captions) include Imagen, DALL-E, Midjourney, Adobe Firefly, and Stable Diffusion, and image-generative machine learning processes may, in some cases, be trained for text-image generation and / or neural style transfer. Exemplary generative machine learning processes trained on molecular data include, but not limited to, AlphaFold, which may be used for protein structure prediction and drug discovery. Generative machine learning processes trained on audio training data include MusicLM, which can be trained on audio waveforms of music correlated with text annotations, and music generative machine learning processes can, in some cases, generate new music samples based on text descriptions. Exemplary generative machine learning processes trained on video include, but are not limited to, RunwayML and Make-A-Video by Meta Platforms. Finally, exemplary generative machine learning processes trained using robot motion data include, but are not limited to, UniPi from Google Research.

[0076] Continuing to refer to Figure 2, in some cases, a generative machine learning process may include a generative adversarial network. As used in this disclosure, “Generative Adversarial Network (GAN)” is a machine learning process that includes at least two adversarial networks configured to synthesize data according to a set of rules (e.g., rules of a game). In some cases, a generative adversarial network may include a generative network and a discriminative network, where the generative network generates candidate data and the discriminative network evaluates the candidate data. An exemplary GAN can be described according to the following game: each in a probability space (Ω, μ) ref The GAN game is defined as having two adversarial networks, namely a generator network and a discriminator network. The generator network's strategy set is P(Ω), i.e., the set of all probability measures μ on Ω. G The discriminator network strategy set is the set of Markov kernels μ. D Ω→P[0,1], where P[0,1] is a set of probability measures on [0,1]. A GAN game can be a zero-sum game with the following objective function:

[0077]

number

[0078] Referring further to FIG. 2, in some embodiments, the generative machine learning process may include, but is not limited to, adversarial generative networks (GANs) such as CycleGAN. CycleGAN may use a pair of mutually dependent neural network generators that depend on each other's outputs to be used as inputs. The CycleGAN process allows forward and reverse transformations between domains to occur simultaneously. The CycleGAN process may include a set of computations. CycleGAN may be different from paired training data, where paired training data is training examples where a correspondence already exists between x i and y i and consists of. In unpaired training data, the source set

Number

Number

Number

Number

Number

number

[0079] This process is repeatable in the reverse direction and is motivated by reducing cycle consistency loss as follows:

number

number

[0080] Referring further to Figure 2, in a separate, non-limiting embodiment, a generative machine learning model may include a diffusion process and / or a diffusion model. An exemplary diffusion model may include an energy-induced stochastic differential equation (EGSDE) process. EGSDE relies on a score-based diffusion model to transform data from one domain to another. In contrast to the CycleGAN process described above, EGSDE uses a score-based diffusion model (SBDM) to perturb an initial dataset to Gaussian noise and then reverse the process to transform the noise back into a data distribution. This diffusion model employs a pre-trained energy function based on data from an initial source domain and data from a final target domain to derive a pre-trained stochastic differential equation (SDE) inference process. As used herein, “energy function” is defined as an equational approximation of a transfer function for transforming data from a first dataset from a source domain to usable data in a target domain (e.g., the domain of a second dataset). The energy function consists of two terms. The first induced term is a realistic expert, prioritizing that the energy function focus on discarding source domain-specific features. The second inductive term is the faithful expert, which prioritizes the preservation of domain-independent features of the energy function. By combining these two features, the result is a target domain dataset that is independent of the source domain data protocol but preserves the substantial features exhibited by the data. Combined with the pre-trained energy function, the EGSDE method uses three experts (energy function, realistic expert, and faithful expert) so that all contribute to the generation of best-fit output data. A mathematical explanation of this process is described in detail for time series data. q(y0)

number

number

number

number

number

number

number

[0081] Next, this can be discretized using the Euler-Maruyama solver. By formally adopting a step size of h, the iteration rule from s to t=sh is as follows:

number

[0082] See Figure 2 for more information, and also see EGSDE, a set of unpaired images from the source domain.

number

number

number

number

number

number

[0083] Continuing to refer to Figure 2 and further to the use of EGSDE, the energy function is derived by balancing the need to retain domain-independent features of the initial time-series data 108 while appropriately modifying domain-specific features. Based on this balance, the energy function is the sum of two log-potential functions:

number

number

number

number

number

number

number

number

number

number

number

[0084] It doesn't just function as a simple low-pass filter, but is a more sophisticated E i You can use this to unravel the learning methods for different domains.

[0085] Furthermore, referring to Figure 2 and the use of EGSDE, solving the energy-induced inverse time SDE can be achieved using a pre-trained score-based model s(y,t) and energy function E(y,x,t), and an example 116 can be generated from the conditional distribution p(y0|x0). A numerical solver can be used to approximate the trajectory from the SDE to achieve a fair comparison. In a non-restrictive embodiment, an Euler-Maruyama solver employing a step size h is used herein, where the iteration rule from s to t=sh is as follows:

number

[0086] The Monte Carlo method is used to estimate the expected value of a single sample for efficiency. In a non-restrictive embodiment, dispersed conserved energy-induced SDE can be used to modify the noise prediction network and incorporate it into the sampling procedure of the denoised diffuse stochastic model.

[0087] Furthermore, referring to Figure 2 and the use of EGSDE, the expert product is used in the discretized sampling process. The conditional distribution at time t is defined as follows:

number

number

number

number

number

number

number

[0088] Referring further to Figure 2, one or more of the processes or algorithms described above can be performed by at least one dedicated hardware unit 232. For the purposes of this figure, “dedicated hardware unit” is a hardware component, circuit, etc., separate from the main control circuit and / or processor that performs the steps of the method described herein, which is specifically designated or selected to perform one or more particular tasks and / or processes described with reference to this figure, such as preconditioning and / or sanitizing training data, and / or training machine learning algorithms and / or models. The dedicated hardware unit 232 may include, but is not limited to, a hardware unit that can perform iterative or centralized computations, such as matrix-based computations for updating or adjusting parameters, weights, coefficients, and / or biases of machine learning models and / or neural networks, using pipelined, parallel processing, etc., efficiently, and such a hardware unit may be optimized for such processes by including dedicated circuitry for matrix and / or signal processing operations, which includes, for example, multiple arithmetic and / or logic circuit units, such as multipliers and / or adders, that can operate simultaneously and / or in parallel. Such dedicated hardware units 232 may include, but are not limited to, a graphical processing unit (GPU), a dedicated signal processing module, an FPGA, or other reconfigurable hardware configured to instantiate parallel processing units for one or more specific tasks. A computing device, processor, apparatus, or module may be configured to instruct one or more dedicated hardware units 232 to perform one or more operations described herein, such as evaluating model and / or algorithm outputs, one-time or iterative updates to parameters, coefficients, weights, and / or biases, and / or any other operations such as vector and / or matrix operations as described herein.

[0089] Referring here to Figure 3, an exemplary association database 300 is shown by a block diagram. In one embodiment, any past or present version of any data disclosed herein may be stored in an association database 300, which includes, but is not limited to, multiple datasets 108, a first dataset 112, a second dataset 116, a first set of associations 120, visual features, named entities, pathology identifiers, a second set of associations 132, a third dataset 140, and so on. The processor 104 can be communicatively connected to the association database 300. For example, in some cases, the database 300 may be local to the processor 104. Alternatively or additionally, in some cases, the database 300 may be remote to the processor 104 and communicative to the processor 104 via one or more networks. The networks may include, but are not limited to, cloud networks, mesh networks, and so on. For example, “cloud-based” system, as used herein, may refer to a system containing software and / or data that is stored, managed, and / or processed on a network of remote servers hosted in the “cloud,” for example via the Internet, rather than on a local server or personal computer. “Mesh network,” as used in this disclosure, is a local network topology in which the infrastructure processor 104 connects directly, dynamically, and non-hierarchically to as many other computing devices as possible. “Network topology,” as used in this disclosure, is the arrangement of elements in a communication network. The association database 300 may be implemented as, but not limited to, a relational database, a key-value lookup database such as a NoSQL database, or any other format or structure for use as a database that a person skilled in the art would find appropriate in consideration of the entirety of this disclosure. The association database 300 may also be implemented using a distributed data storage protocol and / or data structure, such as a distributed hash table, as an alternative or additional method.As described above, the association database 300 may contain multiple data entries and / or records. Data entries in the database may be flagged with or linked to one or more additional elements of information, and those additional elements may be reflected in linked tables, such as tables associated by data entry cells and / or one or more indexes in the relational database. A person skilled in the art will recognize, upon reviewing the entirety of this disclosure, a variety of ways in which data entries in a database can store, retrieve, organize, and / or reflect data and / or records as used herein, as well as categories and / or sets of data consistent with this disclosure.

[0090] Referring here to Figure 4, an exemplary embodiment of neural network 400 is shown. Also known as an artificial neural network, neural network 400 is a network of “nodes,” or a data structure having one or more inputs, one or more outputs, and a function that determines the output based on the input. Such nodes can be organized into a network, such as a convolutional neural network, which includes, but is not limited to, an input layer of node 404, one or more hidden layers 408, and an output layer of node 412. Connections between nodes can be created through a process of “training” the network, where elements from a training dataset are applied to the input nodes, and then, using a suitable training algorithm (such as the Levenberg-Marquardt method, conjugate gradient, simulated annealing, or other algorithms), the connections and weights between nodes in adjacent layers of the neural network are adjusted to produce desired values ​​in the output nodes. This process is sometimes called deep learning. Connections may be made only from input nodes to output nodes in a “feedforward” network, or the output of one layer may be fed back to the input of the same or different layers in a “recurrent network.” As a further non-limiting example, a neural network may include a convolutional neural network having an input layer, one or more hidden layers, and an output layer. When used in this disclosure, a “convolutional neural network” is a neural network in which at least one hidden layer is a convolutional layer that convolves the input to that layer with a subset of inputs known as a “kernel,” along with one or more additional layers such as a pooling layer or a fully connected layer.

[0091] Referring now to Figure 5, an exemplary embodiment of a neural network node is shown. A node can receive multiple inputs x from inputs to the neural network containing the node and / or from other nodes, although this is not limited to the node itself. i It may include. The node is each input x i Weight multiplied by w iA weighted sum of inputs can be performed using . Additionally or alternatively, a bias b may be added to the weighted sum of inputs such that an offset is added to each unit in the neural network layer, independent of the input to the layer. The weighted sum can then be input to a function φ, which can generate one or more outputs y. Input x i The weight applied w i The weight w can indicate whether the input is "excitatory" (e.g., by a corresponding weight having a large number), which strongly influences one or more outputs y, and / or "inhibitory" (e.g., by a corresponding weight having a small number), which strongly influences another input y. i The value of can be determined by training a neural network using training data, which can be done using any appropriate process as described above.

[0092] Referring here to Figure 6, an exemplary embodiment of the fuzzy set comparison 600 is shown. In a non-limiting embodiment, the fuzzy set comparison 600 may be consistent with the fuzzy set comparison of Figure 1. In another non-limiting example, the fuzzy set comparison 600 may be consistent with name / version matching as described herein. For example, but not limiting, the parameters, weights, and / or coefficients of the membership function may be adjusted using any machine learning method for name / version matching as described herein. In another non-limiting embodiment, the fuzzy set may represent the visual features and named entities from Figure 1.

[0093] Alternatively or additionally, referring further to Figure 6, a fuzzy set comparison 600 may be generated in response to determining a data compatibility threshold. The compatibility threshold may be determined by a computing device. In some embodiments, the computing device may determine the compatibility threshold and / or version certifier using a logic comparison program, such as a fuzzy logic model, but is not limited to this. Each such compatibility threshold may be represented as a value of a post variable representing the compatibility threshold, or in other words, as the above fuzzy set corresponding to a degree of compatibility and / or acceptability, calculated using a statistical method, a machine learning method, or any other method that a person skilled in the art could conceive of by considering the whole of this disclosure. In some embodiments, determining the compatibility threshold and / or version certifier may involve using a linear regression model. The linear regression model may include a machine learning model. The linear regression model may map statistics, such as the frequency of version numbers in the same range, to the compatibility threshold and / or version certifier, but is not limited to this. In some embodiments, determining the compatibility threshold for any post may involve using a classification model. The classification model may be configured to input collected data and cluster data into a centroid based on, but is not limited to, the frequency of occurrence of a range of version numbers, language indicators of compatibility and / or acceptability, etc. The centroid may include scores assigned to each of the compatibility thresholds, so that a score can be assigned to each of them. In some embodiments, the classification model may include a K-means clustering model. In some embodiments, the classification model may include a particle swarm optimization model. In some embodiments, determining compatibility thresholds may include using a fuzzy inference engine. The fuzzy inference engine may be configured to map one or more compatibility thresholds using fuzzy logic. In some embodiments, multiple computing devices may be arranged in a compatibility configuration by a logic comparison program. "Compatibility configuration," as used in this disclosure, is any grouping of objects and / or data based on skill levels and / or output scores.The membership function coefficients and / or constants described above can be adjusted according to classification and / or clustering algorithms. For example, a clustering algorithm can determine a Gaussian or other distribution of questions relating to centroids corresponding to a given compatibility threshold and / or version authenticator, and using iteration or other methods, find a membership function of any of the membership function types described above that minimizes the mean error from a statistically determined distribution, such as a triangle or Gaussian membership function relating to a centroid representing the center of the distribution that best matches the distribution. The error function to be minimized and / or the method of minimization can be performed according to an error function and / or error function minimization process and / or method described herein, but not limited to these.

[0094] Referring further to Figure 6, the inference engine may be implemented according to the input of multiple scanned user labels 120 and multiple named entities. For example, the acceptance variable may represent a first measurable value relating to the classification of multiple visual features for named entities. Continuing the example, the output variable may represent a second set 132 of associations. In one embodiment, multiple visual features and / or named entities may be represented by their own fuzzy sets. In other embodiments, evaluation factors may be represented according to two intersecting fuzzy sets, as shown in Figure 6. The inference engine can combine its arbitrary rules such as semantic versioning, semantic language, and version range. The degree to which a given input function membership conforms to a given rule can be determined by the triangular norm or "T-norm" of the output function having a rule or input function that satisfies the requirements of commutativity (T(a,b)=T(b,a)), monotonicity (T(a,b)≦T(c,d) for a≦c and b≦d), (associativity: T(a,T(b,c))=T(T(a,b),c)), and that the number 1 functions as an identity element, such as min(a,b), the product of a and b, the intense product of a and b, and the Hamacher product of a and b. The combination of rules (the "and" or "or" combination of rule membership decisions) may be performed using any T-conorm represented by the inverted T symbol or "⊥", such as max(a,b), the stochastic sum of a and b (a+ba*b), the bounded sum, and / or the fierce T-conorm, and any T-conorm satisfying the properties of commutativity: ⊥(a,b)=⊥(b,a), monotonicity: ⊥(a,b)≦⊥(c,d) for a≦c and b≦d, associativity: ⊥(a,⊥(b,c))=⊥(⊥(a,b),c), and identity element 0. Alternatively or additionally, the T-conorm may be approximated by a sum, such as in a "product-sum" inference engine where the T-norm is a product and the T-conorm is a sum. The final output score or other fuzzy inference output can be determined from the output membership function described above using any appropriate defuzzy process, including but not limited to the mean of the maximum defuzzy, the centroid of the area / centroid defuzzy, the mean of the central defuzzy, the bisector of the area defuzzy, etc.Alternatively or additionally, the output rules may be replaced with functions based on the Takagi-Sugeno-King (TSK) fuzzy model.

[0095] The first fuzzy set 604 can be expressed according to a first membership function 608 that represents the probability that an input falling into a first range of values ​​612 is a member of the first fuzzy set 604, where the first membership function 608 has values ​​on a range of probabilities, such as the interval [0,1], and the area below the first membership function 608 can represent a set of values ​​in the first fuzzy set 604. In this illustrative description, for clarity, the first range of values ​​612 is shown as a range on a single numerical line or axis, but the first range of values ​​612 may be defined in two or more dimensions, for example, representing the Cartesian product between multiple ranges, curves, axes, spaces, dimensions, etc. The first membership function 608 may include any suitable function that maps the first range 612 to a probability interval, including a triangulation function defined by two linear elements, such as a line segment or plane, intersecting at or below the top of the probability interval. As a non-restrictive example, a triangular membership function can be defined as follows:

number

number

number

number

number

[0096] Those skilled in the art will, upon reviewing the entirety of this disclosure, recognize a variety of alternative or additional membership functions that may be used in accordance with this disclosure.

[0097] The first fuzzy set 604 can represent any value or combination of values ​​as described above, including any multiple visual features and named entities. A second fuzzy set 616, which can represent any value that can be represented by the first fuzzy set 604, may be defined by a second membership function 620 on a second range 624, the second range 624 may be identical to and / or overlap with the first range 612, and / or be combined with the first range via a Cartesian product or the like to produce a mapping that allows for overlapping evaluations of the first fuzzy set 604 and the second fuzzy set 616. If the first fuzzy set 604 and the second fuzzy set 616 have an overlapping region 636, the first membership function 608 and the second membership function 620 may intersect at a point 632 that represents the probability of agreement between the first fuzzy set 604 and the second fuzzy set 616, as defined on a probability interval. Alternatively or additionally, a single value of the first and / or second fuzzy set may be located at point 636 on the first range 612 and / or the second range 624, where the probability of membership may be taken by evaluating the first membership function 608 and / or the second membership function 620 at that range point. The probability at 628 and / or 632 can be compared with a threshold 640 to determine whether a positive match is indicated. In a non-restrictive example, the threshold 640 may represent the degree of agreement between the first fuzzy set 604 and the second fuzzy set 616, and / or a single value having each other or either set within them, which is sufficient for the purposes of the matching process. For example, the second set of associations 132 may show a sufficient degree of overlap with the fuzzy sets representing multiple visual features and named entities for the combinations that occur as described above. Each threshold may be established by one or more user inputs. Alternatively or additionally, each threshold can be adjusted by machine learning and / or statistical processes, for example, not as a limitation, but as will be described in more detail below.

[0098] In one embodiment, the degree of matching between fuzzy sets can be used to rank one resource against another. For example, if both multiple visual features and named entities have fuzzy sets, a second set of associations 132 may be generated by having a degree of overlap exceeding a prediction threshold, and the processor 104 may further rank the two resources by ranking the resource with the higher degree of matching higher than the resource with the lower degree of matching. If multiple fuzzy matches are performed, the degree of matching for each fuzzy set can be calculated and aggregated, for example by addition, averaging, etc., to determine an overall degree of matching, which can then be used to rank the resources, and the selection between two or more matching resources may be performed by selecting the highest-ranked resource, and / or multiple notifications may be presented to the user in the order of ranking.

[0099] Referring here to Figure 7, a flowchart of an exemplary method 800 for detecting associations between different types of datasets is shown. In step 705, method 700 includes receiving multiple datasets from a user, using at least a processor. The multiple datasets include a first dataset containing multiple text data and a second dataset containing multiple image data. This can be carried out as described with reference to Figures 1 to 6. In one embodiment, the multiple datasets include multiple data associated with one or more prognostic slides.

[0100] Referring further to Figure 7, in step 710, method 700 includes identifying a first set of associations between a first dataset and a second dataset, using at least a processor. This can be carried out as described with reference to Figures 1 to 6. In one embodiment, the first set of associations includes a plurality of direct correlations between a plurality of datasets. In another embodiment, the first set of associations includes a plurality of pathological identifiers. Generating the first set of associations may include identifying one or more visual features in the second dataset, using at least a processor. Generating the first set of associations may include generating a plurality of named entities associated with the first dataset, using at least a processor. Generating the first set of associations may further include generating a plurality of named entities associated with the first dataset, using at least a processor.

[0101] Referring further to Figure 7, in step 715, method 700 includes generating a second set of associations in accordance with a first set of associations using a second association classifier, with at least a processor. Generating a second set of associations includes training a second association classifier using second association training data, the second association training data includes a first set of associations as inputs correlated to a second set of associations as outputs, and includes multiple data entries that generate a second set of associations in accordance with the first set of associations of the trained second association classifier. In some versions, the second association training data may include a first subset of the first dataset as inputs correlated to a first subset of the second dataset as outputs. This can be implemented as described with reference to Figures 1 to 6. In one embodiment, the second set of associations includes multiple abstract correlations between multiple datasets. In another embodiment, the method further includes generating a third dataset in accordance with a second set of associations, with at least a processor. In some cases, the method includes generating a second set of associations using a fuzzy inference set, with at least a processor.

[0102] Referring further to Figure 7, in step 720, method 700 includes displaying a second set of associations using a display device. This can be carried out as described with reference to Figures 1 to 6.

[0103] Referring further to Figure 7, in some embodiments, generating a second set of associations further includes, in step 715, training a generative machine learning process using the first dataset, synthesizing a first synthetic data in accordance with the first dataset using generative machine learning, and generating a second set of associations in accordance with the first synthetic data and the first set of associations. The generative machine learning process may include any generative machine learning process described in this disclosure, including referring to Figures 1 to 6 above. The first synthetic data may include any data generated by a machine learning process, such as the generative machine learning process described in this disclosure, including referring to Figures 1 to 6 above. In some cases, the second association training data may include the first synthetic data as input correlated to a first subset of the second dataset as output.

[0104] Referring further to Figure 7, in some embodiments, the first dataset includes text. The text may include any text or text data described in this disclosure, for example, referring to Figures 1 to 6. In some cases, generating a second set of associations may further include, in step 715, using a natural language processing model to associate text data in the first dataset and generating a second set of associations in accordance with the associated text data and the first set of associations in the first dataset.

[0105] Referring further to Figure 7, in some embodiments, generating a second set of associations may further include, in step 715, calculating the distances between data elements in the first dataset and generating a second set of associations in accordance with the distances between data elements in the first dataset and the first set of associations. The distances may include any distance, such as vector distances, as described in this disclosure, for example, referring to Figures 1 to 6.

[0106] Referring further to Figure 7, in some embodiments, the first dataset may include metadata. The metadata may include any metadata or contextual data described in this disclosure, for example, referring to Figures 1 to 6. In some cases, generating a second set of associations may further include, in step 715, associating metadata in the first dataset and generating a second set of associations in accordance with the associated metadata and the first set of associations in the first dataset.

[0107] Referring further to Figure 7, in some embodiments, the second dataset may include image data. The image data may include any representative data, such as graphics, as described in this disclosure, for example, referring to Figures 1 to 6. In some cases, generating a second set of associations may include, in step 715, using a machine vision system to identify one or more visual features in the second dataset. The machine vision system may include any machine vision system described in this disclosure, for example, referring to Figures 1 to 6. The visual features may include any visual features described in this disclosure, for example, referring to Figures 1 to 6.

[0108] Referring further to Figure 7, in some embodiments, generating a first set of associations may include, in step 710, identifying a plurality of named entities in the first dataset, where each of the plurality of named entities is associated with at least one data element in the second dataset.

[0109] Referring now to Figure 8, which is a simplified diagram of a device 800 for detecting associations between different types of datasets. For example, the device 800 may be configured to detect associations (e.g., text-image pairings) between a first dataset (e.g., text data 804) and a second dataset (e.g., image data 808). That is, for a given element in the first dataset (e.g., a word, phrase, sentence, or set of sentences in text data 804), the device 800 may detect an associated element in the second dataset (e.g., an image in image data 808). These may be implemented as disclosed with respect to Figures 1 to 7.

[0110] Continuing to refer to Figure 8, in some embodiments, large general-purpose datasets consisting of image-text pairs have become widely available in recent years. For example, such datasets may include pairings of images with their captions. However, in more specialized fields such as medical imaging, the use of these general-purpose datasets is limited. For example, general-purpose datasets often consist of naturally occurring images. In practice, neural network models, such as the Contrastive Language Image Pre-trained (CLIP) model, trained using naturally occurring images provide poor performance. To achieve a performance level suitable for practical use, it is desirable to train the neural network model using image-text pairs from the same specialized field. For example, it is desirable to use image-text pairs from the medical field to train a neural network model used for medical applications. Doing so allows for improved performance of the trained model and improved computational efficiency (e.g., reduced GPU time for training).

[0111] Continuing to refer to Figure 8, in some embodiments, an exemplary source of image-text pairs in the medical field is patient notes. Generally, patient notes may contain unstructured text and image data, along with associations between text and images, which may be direct or indirect. An example of a direct association is an identifier present in both the text and the image (e.g., the identifier may be embedded in the image itself or included in accompanying metadata). An example of an indirect association is a textual description (e.g., a word, phrase, or sentence) of a pathological condition corresponding to an image that visually represents that condition. Often, direct associations between text and images can be easily parsed using known techniques (e.g., an algorithm can be programmed to detect text and images with matching identifiers). However, such techniques may not be suitable for detecting indirect associations.

[0112] Continuing to refer to Figure 8, in some embodiments, and according to some embodiments, indirect associations between image and text data are detected using a neural network model to learn representations (e.g., vector representations) of the text and image data. The learned representations are used to map the image and text data into a joint latent space (e.g., a vector space of the learned representations). Pairings are then identified within the latent space. One challenge of this method is that the mapping of text and image data into the latent space is a learned relationship, not a function, regardless of whether we consider image or text as the domain of the mapping. Therefore, the techniques described later may address these challenges.

[0113] Continuing to refer to Figure 8, in some embodiments, for simplicity and clarity, the following examples focus on pairing image and text data, but those skilled in the art will understand that this technique can be easily adapted to detect associations between a wide variety of other types of data. For example, the technique can be applied to multimedia data (e.g., audio and video), time-series data (e.g., electrocardiogram data), structured data (e.g., data tables, graphs, databases, models, etc.), medical scans (e.g., 2D or 3D medical imaging data), etc. Furthermore, the technique can be used to detect associations between three or more datasets (e.g., groups of text, images, and time-series data).

[0114] Continuing to refer to Figure 8, in some embodiments, the text data 804 may correspond to a set of documents. For example, the text data 804 may include patient notes or other types of text-based medical records. The image data 808 may include a set of images related to the text data 804. For example, the text data 804 and the image data 808 may belong to the same set of medical records (for example, a medical record may include both written and image-based components).

[0115] Continuing to refer to Figure 8, in some embodiments, the result of pairing text data 804 with image data 808 is a set of text-image pairs 812. These text-image pairs 812 can be used for various downstream tasks, such as supervised machine learning tasks, self-supervised machine learning tasks, or semi-supervised machine learning tasks. In some embodiments, a text-image pair 812 includes a set of one or more elements of text data 804 and a corresponding set of one or more elements of image data 808, for example, the device 800 can pair an "image tuple" with a corresponding "text tuple". For example, a given image may be associated with multiple text elements, such as a caption placed with the image and a description of the image appearing elsewhere in the body of the document. Similarly, text elements can describe or compare multiple images, such as a caption applied to a group of images. Tuples may have particular advantages in certain downstream tasks, such as sequence prediction tasks.

[0116] Continuing to refer to Figure 8, in some embodiments, the device 800 may be configured to improve the detection of text-image pairs 812 based on text data 804 and image data 808. For example, the device 800 can detect text-image pairs 812 more accurately and efficiently (e.g., in less GPU time) than existing techniques. By detecting a more complete set of text-image pairs 812 than existing techniques, the technique used by the device 800 can generate a sufficiently large set of text-image pairs 812 using less text data 804 and image data 808, thereby reducing the amount of storage and bandwidth used to store and transmit the data. Furthermore, by generating a larger and more accurate set of text-image pairs 812, the device 800 can similarly improve the accuracy and efficiency of downstream tasks, for example, for machine learning tasks, such as faster training with fewer GPU cycles.

[0117] Continuing to refer to Figure 8, in some embodiments, the apparatus 800 can perform a bootstrapping process in which an initial set of text-image pairs 816 is generated based on text data 804 and image data 808. Various strategies can be used to generate the initial text-image pairs 816. For example, a pre-trained neural network model 820 can be used to generate the initial text-image pairs 816 using transfer learning. The pre-trained model 820 may be a model trained in a similar domain, or a model that otherwise provides sufficient accuracy to generate the initial set. Additionally or alternatively, the initial image-text pairs 816 may be identified using direct associations, such as explicit links in the text data 804 or identifiers present in both the text data 804 and image data 808 (e.g., the identifiers may be embedded in the images themselves or included in accompanying metadata). Parsing the dataset to identify direct associations can be done using existing symbolic methods. In some embodiments, the identifiers may include timestamps, where the text-image pairs have matching timestamps.

[0118] Continuing to refer to Figure 8, in some embodiments, the initial text-image pairs 816 correspond to fewer than the total number of text-image pairs in the text data 804 and image data 808. Therefore, using the “bootstrap” set of initial text-image pairs 816, the device 800 performs further processes to detect additional text-image pairs. For example, additional text-image pairs that are desirable to detect may include indirect pairs (e.g., text passages describing aspects of an image without directly linking to the image) or pairs that would not be detected using the pre-trained model 820 (e.g., because the pre-trained model 820 is a general-purpose model or otherwise not optimized for the text data 804 and / or image data 808).

[0119] Continuing to refer to Figure 8, in some embodiments, the device 800 can train a pairing model 824 using the initial image-text pair 816 to detect additional text-image pairs. For example, the pairing model 824 can jointly learn representations of image data 808 and text data 804 in the same representation space (e.g., latent space). Examples of such techniques are described in more detail in Radford, et al., "Learning Transferable Visual Models From Natural Language Supervision" (https: / / arxiv.org / pdf / 2103.00020.pdf).

[0120] Continuing to refer to Figure 8, in some embodiments, the apparatus 800 can train a generative model 828 using image data 808 to synthesize additional image data. That is, the generative model 828 can generate a synthesized image having similar attributes to the images in image data 808. For example, the generative model 828 may correspond to a generative adversarial network (GAN) or a diffusion model. To efficiently train the generative model 828, the initial stage of training may include training another model (e.g., an autoencoder) to create a low-dimensional representation of the image data 808, which is then provided to the generative model 828. In some embodiments, the generative model 828 can generate image data unconditionally or conditionally. For example, image synthesis may be conditioned on text, such as text from text data 804. The initial text-image pair 816 can be used as a training set to train the generative model 828 to generate images conditionally. In some embodiments, the generative model 828 can be used to edit images in image data 808, for example, by performing inpainting to fill in masked portions of the image. Generative models, training techniques, and examples of conditional and unconditional image generation are described in more detail in Rombach, et al., "High-Resolution Image Synthesis with Latent Diffusion Models" (https: / / arxiv.org / pdf / 2112.10752.pdf).

[0121] Continuing to refer to Figure 8, in some embodiments, the apparatus 800 can use text data 804 used to pre-train a language model 832. For example, the language model 832 may be configured to generate representations (e.g., vector representations) of elements of the text data 808. Exemplary examples of language models include autoregressive models such as the generative pre-trained transformer 3 (GPT-3) model, described in more detail in Brown, et al., "Language Models are Few-Shot Learners" (https: / / arxiv.org / pdf / 2005.14165.pdt), which is entirely incorporated herein by reference as Appendix Candis, and masked language models (MLMs) such as the bidirectional encoder representation (BERT) model from transformers, described in more detail in Bao, et al., "BEiT:BERT Pre-Training of Image Transformers" (https: / / arxiv.org / pdf / 2106.08254.pdf).

[0122] Continuing to refer to Figure 8, in some embodiments, the device 800 can proceed to detect additional pairs of text 836 and image 840 using one or more of the language model 832, pairing model 824, and generation model 828. For example, the device 800 may use one or more of the following techniques to detect additional pairs.

[0123] Continuing to refer to Figure 8, in some embodiments, the apparatus 800 can conditionally generate images based on elements of text data 804 using the generative model 828 by using text representations to find matching images using the generative model 828. In some embodiments, the conditions may be based on representations of text elements generated by the language model 832. The generated images may be used to identify zero or more other images in image data 808 that are similar to the generated images. Similar images can be identified using various techniques, such as determining vector similarity (e.g., cosine similarity) in the embedding space. In some embodiments, images in image data 808 that are similar to the generated images above a predetermined threshold are determined to be paired with corresponding text elements in text data 804.

[0124] Continuing to refer to Figure 8, in some embodiments, the apparatus 800 can use the pairing model 824 to identify one or more candidate text elements in text data 804 for a particular image in image data 808 by using the image representation to find matching text using a combined, trained pairing model 824. To improve the efficiency of this method, the text data 804 may be culled by various techniques so that a reduced number of text elements are considered. For example, a language model 832 may be used to cluster elements of text data 804 so that the culled set of text data contains cluster centroids. Further improvements can be obtained by varying the granularity of text elements (e.g., words, phrases, sentences, etc.) to identify an appropriate level of granularity. In some embodiments, one or more of the text candidates can be paired with a corresponding image by pairing, for example, the closest match, one or more candidates exceeding a predetermined threshold, etc. In some embodiments, a generative model 828 can be used to conditionally generate images based on candidate text elements. In a consistent manner with such embodiments, the decision of whether to pair a candidate text element with an image may be based on the similarity between the composite image and the actual image.

[0125] Continuing to refer to Figure 8, in some embodiments, the apparatus 800 can use one or more context cues to make pairing decisions by using implicit image magnification levels and temporal order in the text to find candidate images. Context cues can be used alone or in combination with other techniques, such as those shown in blocks 107 and 108. For example, the magnification level of a microscopic image can be used as a context cue for pairing images with text that explicitly or implicitly reference a consistent magnification level, for instance, text describing cellular-level features or text counting cellular-level features suggesting a higher level of magnification, while a broader description of a pathological condition suggests a lower level of magnification.

[0126] Continuing to refer to Figure 8, in some embodiments, the time information in the text data 804 can be used as a queue for finding candidate images in the image data 808. For example, when the text in a patient note refers to a time or period (e.g., the text includes a time phrase such as a specific date, "the next day," or "the next week"), these queues can be used to identify candidate images based on whether the image is associated with time information that matches the time information in the text (e.g., an image with a timestamp for a date one week after the patient note refers to a scan that will be performed "the next week"). In some examples, the time information in the text data 804 may indicate images taken consecutively in time (e.g., the text may indicate "the patient was sent back for a scan"), which can be a queue for identifying candidate images with a matching sequence. For example, an initial diagnostic core biopsy may be paired with text reporting the initial histological grade of the tumor, and a subsequent excised specimen may be paired with text reporting a different grade than the previously reported grade. In this example, the two events may be separated by days or months.

[0127] Continuing to refer to Figure 8, in some embodiments, these context cues may be automatically picked up by the neural network model during training, but configuring the device 800 to explicitly use predetermined cues may offer further advantages. In some embodiments, the device 800 can perform processes that facilitate the use of context cues, such as determining the magnification level of an image (for example, by detecting and counting features in the image).

[0128] Continuing with reference to Figure 8, in some embodiments, the subject matter described herein can be implemented in digital electronic circuits, computer software, firmware, or hardware, or combinations thereof, including structural means and structural equivalents disclosed herein. The subject matter described herein can be implemented as one or more computer program products, such as one or more computer programs tangibly embodied in an information carrier (e.g., in a machine-readable storage device) or embodied in propagated signals, for execution by or control of the operation of a data processing device (e.g., a programmable processor, a computer, or a group of computers). A computer program (also known as a program, software, software application, or code) can be written in any form of programming language, including compiled or interpreted languages, and can be deployed as a standalone program or in any form, including modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file. A program can be stored in part of a file that holds other programs or data, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files that store one or more modules, subprograms, or parts of code). Computer programs can be deployed to run on a single computer, on multiple computers at a single site, or distributed across multiple sites and interconnected by a communication network.

[0129] Continuing to refer to Figure 8, in some embodiments, the processes and logic flows described herein, including the method steps of the subject described herein, can be executed by one or more programmable processors that execute one or more computer programs that perform the functions of the subject described herein by manipulating input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuits such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the devices of the subject described herein can be implemented as such dedicated logic circuits.

[0130] Continuing to refer to Figure 8, in some embodiments, a processor suitable for executing a computer program includes, for example, both general-purpose microprocessors and dedicated microprocessors, and any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random-access memory or both. Essential elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, e.g., magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to them to receive data from them, transfer data to them, or both. Information carriers suitable for embodying computer program instructions and data include, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and optical disks (e.g., CDs and DVDs), and all forms of non-volatile memory. Processors and memory may be complemented by or incorporated into dedicated logic circuits.

[0131] Continuing to refer to Figure 8, in some embodiments, the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device (e.g., mouse or trackball) by which the user can provide input to the computer, in order to provide interaction with the user. Interaction with the user can also be provided using other types of devices. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0132] Continuing to refer to Figure 8, in some embodiments, the subject matter described herein can be implemented in a computing system including backend components (e.g., data servers), middleware components (e.g., application servers), or frontend components (e.g., client computers having a graphical user interface or web browser through which a user can interact with the implementation of the subject matter described herein), or any combination of such backend, middleware, and frontend components. The components of the system can be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0133] Referring here to Figure 9, a flowchart of an exemplary method 900 for detecting associations between different types of datasets is shown. Method 900 includes a step 905 of receiving a first dataset and a second dataset using at least a processor, wherein the first and second datasets include data elements of different types. In some embodiments, the first dataset may include a set of text data, the second dataset may include a set of image data, and the association may include a pair of text images. In some embodiments, the set of image data may include a visual representation of a pathology slide captured by an imaging technique. In some embodiments, the second dataset may include metadata, which is associated with the pathology slide, and which may include annotations. In some embodiments, the datasets may include media data, which may include one or more audio data, video data, and document data associated with the pathology slide. In some embodiments, the media data may further include a tissue graph. In some embodiments, the media data may further include a diagnostic pathway graph. In some embodiments, one or more first data elements may include visual features, and one or more second data elements may include named entities. In some embodiments, Method 900 may further include assigning pathological identifiers to an initial set of associations, which may include visual features and named entities, using at least a processor. 1. In some embodiments, Method 900 may further include identifying visual features using a machine vision system, using at least a processor. These may be implemented as disclosed with respect to Figures 1 to 8.

[0134] Continuing to refer to Figure 9, Method 900 includes a step 905 of using at least a processor to identify an initial set of associations between a first dataset and a second dataset, each association including one or more first data elements from the first dataset and one or more second data elements from the second dataset. In some embodiments, Method 900 may further include identifying an initial set of associations using a bootstrap process, which includes using at least a processor to generate a plurality of resamples of the first dataset and the second dataset, each resample potentially including paired samples of the first dataset and the second dataset, and determining an initial set of associations between the first dataset and the second dataset in accordance with the plurality of resamples. In some embodiments, Method 900 may further include using at least a processor to determine the strength of the initial set of associations between the first dataset and the second dataset in accordance with the plurality of resamples, and using at least a processor to determine the initial set of associations between the first dataset and the second dataset in accordance with the strength and a predetermined threshold. In some embodiments, method 900 may further include, at least using a processor, pairing prognostic labels to a second dataset, and at least using a processor, identifying a first dataset associated with the prognostic labels. In some embodiments, method 900 may further include, at least using a processor, generating an initial set of associations using a first association classifier trained on training data comprising a plurality of first datasets and a plurality of second datasets correlated to an initial set of associations.In some embodiments, Method 900 may further include comparing a first visual feature of a first second dataset with a second visual feature of a second second dataset, using at least a processor, such that the second second dataset may include an initial set of associations with the second first dataset; matching the first visual feature to the second visual feature in response to this comparison, using at least a processor; and determining an initial set of associations for the first second dataset in response to this match, using at least a processor. In some embodiments, Method 900 may further include identifying named entities in the first dataset using a named entity recognition process, using at least a processor. In some embodiments, Method 900 may further include preprocessing the first and second datasets using at least a processor; and generating similarity scores between the preprocessed first dataset and the preprocessed second dataset, using at least a processor. These may be implemented as disclosed with respect to Figures 1 to 8.

[0135] Continuing with Figure 9, method 900 includes step 905 of using at least a processor to train a neural network model to detect additional associations between a first dataset and a second dataset, the initial set of these associations being used as training data for training the neural network model. These can be implemented as disclosed with respect to Figures 1 to 8.

[0136] Continuing with reference to Figure 9, Method 900 includes step 905 of detecting one or more additional associations between a first dataset and a second dataset using at least a processor and a trained neural network model. In some embodiments, Method 800 may further include training a generative model to conditionally generate images based on text, wherein an initial set of associations is used as training data for training this generative model. In some embodiments, Method 800 may further include using the generative model to generate one or more images corresponding to elements from a set of text data, and comparing the one or more generated images with one or more images in a set of image data. In some embodiments, Method 900 may further include using at least a processor to generate additional associations in accordance with pathological identifiers in the initial set of associations. These may be implemented as disclosed with respect to Figures 1 to 8.

[0137] It should be noted that any one or more of the embodiments and models described herein can be conveniently implemented using one or more machines programmed according to the teachings herein (e.g., one or more computing devices used as a user computing device for electronic documents, one or more server devices such as a document server, etc.), as will be obvious to those skilled in the computer art. As will be obvious to those skilled in the software art, appropriate software coding can be readily produced by a skilled programmer based on the teachings of this disclosure. The above embodiments and implementations using software and / or software modules may also include appropriate hardware to assist in the implementation of machine-executable instructions of the software and / or software modules.

[0138] Such software may be a computer program product that uses a machine-readable storage medium. A machine-readable storage medium may be any medium capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and causing a machine to execute any one of the methods and / or embodiments described herein. Examples of machine-readable storage mediums include, but are not limited to, magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state memory devices, EPROMs, EEPROMs, and any combination thereof. As used herein, a machine-readable medium is intended to include a single medium, as well as a collection of physically separate media, such as a compact disk in combination with computer memory or a collection of one or more hard disk drives. As used herein, a machine-readable storage medium does not include transient forms of signal transmission.

[0139] Such software may also include information (e.g., data) carried as data signals on a data carrier, such as a carrier wave. For example, machine-executable information may be included as data-carrying signals embodied on a data carrier, where the signals encode a sequence or portion thereof of instructions for execution by a machine (e.g., a computing device), and any relevant information (e.g., data structures and data) that causes the machine to execute any one of the methods and / or embodiments described herein.

[0140] Examples of computing devices include, but are not limited to, e-book readers, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smartphones, etc.), web appliances, network routers, network switches, network bridges, any machine capable of executing a sequence of instructions specifying the actions it should take, and any combination thereof. For example, a computing device may include and / or be included in a kiosk.

[0141] Figure 10 shows a graphical representation of one embodiment of a computing device in an exemplary form of computer device 1000, in which a set of instructions for causing a control system to execute any one or more aspects and / or methodologies of the present disclosure can be executed internally. It is also conceivable that multiple computing devices could be used to implement a specially configured set of instructions for causing one or more of the devices to execute any one or more aspects and / or methodologies of the present disclosure. Computer device 1000 includes a processor 1004 and memory 1008 that communicate with each other and with other components via a bus 1012. The bus 1012 may include any of several types of bus structures, including but not limited to a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, using any of various bus architectures.

[0142] Processor 1004 may include, but is not limited to, any suitable processor, such as a processor incorporating logic circuits for performing arithmetic and logical operations, such as an arithmetic logic unit (ALU), which may be coordinated by a state machine and directed by operational inputs from memory and / or sensors, and processor 1004 may, as an unspecified example, be organized according to the von Neumann architecture and / or the Harvard architecture. Processor 1004 may include, but is not limited to, microcontrollers, microprocessors, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), complex-programmable logic devices (CPLDs), graphical processing units (GPUs), general-purpose GPUs, tensor processing units (TPUs), analog or mixed-signal processors, trusted platform modules (TPMs), floating-point units (FPUs), and / or systems on a chip (SoCs), which may be incorporated into and / or integrated therein.

[0143] Memory 1008 may include a variety of components (e.g., machine-readable media) including, but not limited to, random-access memory components, read-only components, and any combination thereof. For example, a basic input / output system 1016 (BIOS) containing basic routines useful for transferring information between elements within the computer device 1000 during startup, etc., may be stored in memory 1008. Memory 1008 may also include instructions (e.g., software) 1020 (e.g., stored in one or more machine-readable media) that embody any one or more aspects and / or methodologies of this disclosure. In another example, memory 1008 may further include, but not limited to, an operating system, one or more application programs, other program modules, program data, and any number of program modules including any combination thereof.

[0144] The computer device 1000 may also include a storage device 1024. Examples of storage devices (e.g., storage device 1024) include, but are not limited to, hard disk drives, magnetic disk drives, optical disk drives combined with optical media, solid-state memory devices, and any combination thereof. The storage device 1024 may be connected to the bus 1012 by a suitable interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE®), and any combination thereof. In one example, the storage device 1024 (or one or more of its components) may be detachably interfaced with the computer device 1000 (e.g., via an external port connector (not shown)). In particular, the storage device 1024 and associated machine-readable media 1028 can provide non-volatile and / or volatile storage for machine-readable instructions, data structures, program modules, and / or other data for the computer device 1000. In one example, the software 1020 may reside entirely or partially within a machine-readable medium 1028. In another example, the software 1020 may reside entirely or partially within a processor 1004.

[0145] Computer device 1000 may also include an input device 1032. In one example, a user of computer device 1000 can input commands and / or other information to computer device 1000 via the input device 1032. Examples of input devices 1032 include, but are not limited to, alphanumeric input devices (e.g., keyboards), pointing devices, joysticks, gamepads, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, video cameras), touchscreens, and any combination thereof. Input device 1032 can interface to bus 1012 via any of a variety of interfaces (not shown) including, but not limited to, serial interfaces, parallel interfaces, game ports, USB interfaces, FIREWIRE® interfaces, direct interfaces to bus 1012, and any combination thereof. Input device 1032 may include a touchscreen interface, which may be part of or separate from the display 1036, as will be further described below. As described above, the input device 1032 can be used as a user selection device for selecting one or more graphical representations within the graphical interface.

[0146] The user can also input commands and / or other information to the computer device 1000 via a storage device 1024 (e.g., a removable disk drive, flash drive, etc.) and / or a network interface device 1040. Network interface devices such as network interface device 1040 can be used to connect the computer device 1000 to one or more of various networks, such as network 1044, and one or more remote devices 1048 connected to it. Examples of network interface devices include, but are not limited to, network interface cards (e.g., mobile network interface cards, LAN cards), modems, and any combination thereof. Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, corporate networks), local area networks (e.g., networks associated with offices, buildings, campuses, or other relatively small geographical spaces), telephone networks, data networks associated with telephone / voice providers (e.g., mobile communications provider data and / or voice networks), direct connections between two computing devices, and any combination thereof. Networks such as network 1044 may use wired and / or wireless communication modes. In general, any network topology may be used. Information (e.g., data, software 1020, etc.) can be communicated to and from the computer device 1000 via the network interface device 1040.

[0147] The computer device 1000 may further include a video display adapter 1052 for communicating displayable images to a display device such as a display device 1036. Examples of display devices include, but are not limited to, liquid crystal displays (LCDs), cathode ray tubes (CRTs), plasma displays, light-emitting diode (LED) displays, and any combination thereof. The display adapter 1052 and the display device 1036 may be used in combination with a processor 1004 to provide a graphical representation of an aspect of the disclosure. In addition to the display device, the computer device 1000 may include one or more other peripheral output devices, including, but not limited to, audio speakers, printers, and any combination thereof. Such peripheral output devices may be connected to the bus 1012 via a peripheral interface 1056. Examples of peripheral interfaces include, but are not limited to, serial ports, USB connections, FireWire® connections, parallel connections, and any combination thereof.

[0148] The above has been a detailed description of exemplary embodiments of the present invention. Various modifications and additions can be made without departing from the spirit and scope of the invention. Each feature of the various embodiments described above can be combined as necessary with features of other described embodiments to provide a number of feature combinations in the relevant new embodiments. Furthermore, although several distinct embodiments have been described above, those described herein are merely illustrative of the application of the principles of the present invention. Furthermore, certain methods herein may be illustrated and / or described as being performed in a particular order, but the order is highly variable in the ordinary art to achieve the methods, systems, and software according to this disclosure. Therefore, this description is intended to be construed as illustrative only and does not limit the scope of the invention.

[0149] Exemplary embodiments are disclosed above and shown in the accompanying drawings. Those skilled in the art will understand that various modifications, omissions, and additions can be made to those specifically disclosed herein without departing from the spirit and scope of the invention.

Claims

1. A device for detecting associations between different types of datasets, wherein the device, At least a processor, The system comprises at least a memory that is communicably connected to the processor, and the memory is Receiving multiple datasets, including the first dataset and the second dataset, Identifying a first set of associations between a first subset of the first dataset and a first subset of the second dataset, Using a second association classifier, generate a second set of associations between a second subset of the first dataset and a second subset of the second dataset, in accordance with the first set of associations, wherein generating the second set of associations is Training the second association classifier using second association training data, wherein the second association training data includes a plurality of data entries, each including a first set of associations, and Using the trained second association classifier, generate a second set of associations in accordance with the first set of associations. Including generating, Displaying the second set of associations using a display device The instructions include at least the instructions that constitute the processor to perform the following: The second association training data includes the first subset of the first dataset as input, which correlates with the first subset of the second dataset as output. Device.

2. To generate a second set of associations, Training a generative machine learning process using the first dataset, Using the generative machine learning described above, first synthetic data is synthesized according to the first dataset, To generate a second set of associations in accordance with the first composite data and the first set of associations. The apparatus according to claim 1, further comprising:

3. The apparatus according to claim 2, wherein the second association training data includes the first composite data as input, which correlates with the first subset of the second dataset as output.

4. The first dataset includes text, and the generation of the second set of associations is: Using a natural language processing model, the text data in the first dataset is associated, To generate a second set of associations in accordance with the associated text data and the first set of associations in the first dataset. The apparatus according to claim 1, further comprising:

5. To generate a second set of associations, Calculating the distance between data elements in the first dataset, To generate a second set of associations in accordance with the distance between data elements in the first dataset and the first set of associations. The apparatus according to claim 1, further comprising:

6. The first dataset includes metadata and generates a second set of associations. Associating metadata within the first dataset, To generate a second set of associations in accordance with the associated metadata and a first set of associations in the first dataset. The apparatus according to claim 1, further comprising:

7. The apparatus according to claim 1, wherein the first set of associations includes a plurality of pathological identifiers.

8. The apparatus according to claim 1, wherein the second dataset includes image data, and generating the second set of associations includes using a machine vision system to identify one or more visual features in the second dataset.

9. The apparatus according to claim 1, wherein identifying a first set of associations includes identifying a plurality of named entities in a first dataset, and each named entity of the plurality of named entities is associated with at least one data element in a second dataset.

10. A method for detecting associations between different types of datasets, wherein the method is Using at least a processor, receive multiple datasets including a first dataset and a second dataset, Using at least the processor, a first set of associations between a first subset of the first dataset and a first subset of the second dataset is identified, Using at least the processor and using a second association classifier, generate a second set of associations between a second subset of the first dataset and a second subset of the second dataset, in accordance with the first set of associations, wherein generating the second set of associations is Training the second association classifier using second association training data, wherein the second association training data includes a plurality of data entries, each including a first set of associations, and Using the trained second association classifier, generate a second set of associations in accordance with the first set of associations. Including generating, Displaying the second set of associations using a display device Includes, The second association training data includes the first subset of the first dataset as input, which correlates with the first subset of the second dataset as output. method.

11. To generate a second set of associations, Training a generative machine learning process using the first dataset, Using the generative machine learning described above, first synthetic data is synthesized according to the first dataset, The first composite data and the first set of associations, in accordance with the second set of associations Generating a set and The method according to claim 10, further comprising:

12. The method according to claim 11, wherein the second association training data includes the first composite data as input, which correlates with the first subset of the second dataset as output.

13. The first dataset includes text, and the generation of the second set of associations is: Using a natural language processing model, the text data in the first dataset is associated, To generate a second set of associations in accordance with the associated text data and the first set of associations in the first dataset. The method according to claim 10, further comprising:

14. To generate a second set of associations, Calculating the distance between data elements in the first dataset, To generate a second set of associations in accordance with the distance between data elements in the first dataset and the first set of associations. The method according to claim 10, further comprising:

15. The first dataset includes metadata and generates a second set of associations. Associating metadata within the first dataset, To generate a second set of associations in accordance with the associated metadata and a first set of associations in the first dataset. The method according to claim 10, further comprising:

16. The method according to claim 10, wherein the first set of associations includes a plurality of pathological identifiers.

17. The method according to claim 10, wherein the second dataset includes image data, and generating the second set of associations involves using a machine vision system to identify one or more visual features in the second dataset.

18. The method according to claim 10, wherein identifying a first set of associations includes identifying a plurality of named entities in a first dataset, and each named entity of the plurality of named entities is associated with at least one data element in a second dataset.

19. A device for detecting associations between different types of datasets, wherein the device, At least a processor, The system comprises at least a memory that is communicably connected to the processor, and the memory is Receiving a plurality of datasets, including a first dataset and a second dataset, wherein the first and second datasets include data elements of different types. Identifying an initial set of associations between the first dataset and the second dataset, wherein each association includes one or more first data elements from the first dataset and one or more second data elements from the second dataset. Training a neural network model to detect additional associations between the first dataset and the second dataset, wherein an initial set of associations is used as training data for training the neural network model. Using the trained neural network model, detect one or more additional associations between the first dataset and the second dataset. The instructions include at least the instructions that constitute the processor to perform the following: The first dataset includes a set of text data, The second dataset includes a set of image data, The aforementioned association includes a text-image pair, The memory includes instructions for further configuring the at least processor to train a generative model to conditionally generate images based on text, and the initial set of associations is used as training data for training the generative model. Device.

20. The apparatus according to claim 19, wherein the set of image data includes a visual representation of a pathology slide captured by imaging technology.

21. The detection of the one or more additional associations is Using the aforementioned generation model, generate one or more images corresponding to elements from the set of text data, The one or more generated images are compared with one or more images in the set of image data. The apparatus according to claim 19, further comprising:

22. The apparatus according to claim 19, wherein the second dataset includes metadata, the metadata is associated with pathology slides, and the metadata includes annotations.

23. The apparatus according to claim 19, wherein the plurality of datasets include media data, and the media data includes one or more audio data, video data and document data associated with pathology slides.

24. The apparatus according to claim 23, wherein the media data further includes an organizational graph.

25. The apparatus according to claim 23, wherein the media data further includes a diagnostic path graph.

26. The memory includes instructions to further configure at least the processor to identify the initial set of associations using a bootstrap process, and the bootstrap process To generate a plurality of resamples of the first dataset and the second dataset, wherein each resample includes paired samples from the first dataset and the second dataset. The initial set of associations between the first dataset and the second dataset is determined in accordance with the plurality of resampling. The apparatus according to claim 19, including the apparatus described in claim 19.

27. The aforementioned memory, The strength of the initial set of associations between the first dataset and the second dataset is determined according to the plurality of resamplings, The initial set of associations between the first dataset and the second dataset is determined according to the strength and a predetermined threshold. The apparatus according to claim 26, further comprising instructions for configuring at least the processor to perform the following:

28. The aforementioned memory, Pairing prognostic labels with the second dataset, Identifying the first dataset associated with the prognostic label The apparatus according to claim 19, further comprising instructions for configuring at least the processor to perform the following:

29. The apparatus according to claim 19, wherein the memory further includes instructions to configure the at least processor to generate the initial set of associations using a first association classifier trained on training data comprising a plurality of first datasets and a plurality of second datasets correlated to the initial set of associations.

30. The one or more first data elements include visual features, The one or more second data elements include named entities, The apparatus according to claim 19.

31. The apparatus according to claim 30, wherein the memory includes instructions for further configuring the at least processor to assign pathological identifiers to an initial set of associations, the initial set of associations including the visual features and the named entities.

32. A method for detecting associations between different types of datasets, the method being: Receiving multiple datasets, including a first dataset and a second dataset, using at least a processor, wherein the first and second datasets include data elements of different types. Identifying an initial set of associations between the first dataset and the second dataset using at least the aforementioned processor, wherein each association includes one or more first data elements from the first dataset and one or more second data elements from the second dataset. Training a neural network model to detect additional associations between the first dataset and the second dataset using at least the aforementioned processor, wherein an initial set of associations is used as training data for training the neural network model. Using at least the processor and the trained neural network model, one or more additional associations between the first dataset and the second dataset are detected. Includes, The first dataset includes a set of text data, The second dataset includes a set of image data, The aforementioned association includes a text-image pair, Training a generative model to conditionally generate images based on text using at least the aforementioned processor, wherein the initial set of associations is used as training data for training the generative model. Further including, method.

33. The method according to claim 32, wherein the set of image data includes a visual representation of a pathology slide captured by imaging technology.

34. Using at least the processor and the generation model, generate one or more images corresponding to elements from the set of text data, Using at least the processor, compare the one or more generated images with one or more images in the set of image data. The method according to claim 32, further comprising:

35. The method according to claim 32, wherein the second dataset includes metadata, the metadata is associated with pathology slides, and the metadata includes annotations.

36. The method according to claim 32, wherein the plurality of datasets include media data, and the media data includes one or more audio data, video data and document data associated with pathology slides.

37. The method according to claim 36, wherein the media data further includes an organizational graph.

38. The method according to claim 36, wherein the media data further includes a diagnostic path graph.

39. Using at least the processor, to identify the initial set of associations using a bootstrap process, wherein the bootstrap process is To generate a plurality of resamples of the first dataset and the second dataset, wherein each resample includes paired samples from the first dataset and the second dataset. The initial set of associations between the first dataset and the second dataset is determined in accordance with the plurality of resampling. to identify The method according to claim 32, further comprising:

40. Using at least the processor, the strength of the initial set of associations between the first dataset and the second dataset is determined in accordance with the plurality of resamplings, Using at least the processor, an initial set of associations between the first dataset and the second dataset is determined according to the strength and a predetermined threshold. The method according to claim 39, further comprising:

41. Using at least the aforementioned processor, the prognostic labels are paired with the second dataset, Using at least the processor, identify the first dataset associated with the prognostic label. The method according to claim 32, further comprising:

42. Using at least the aforementioned processor, an initial set of associations is generated using a first association classifier trained on training data that includes a plurality of first datasets and a plurality of second datasets correlated with an initial set of associations. The method according to claim 32, further comprising:

43. The one or more first data elements include visual features, The one or more second data elements include named entities, The method according to claim 32.

44. Assigning pathological identifiers to an initial set of associations using at least the processor, wherein the initial set of associations includes the visual features and the named entities. The method according to claim 43, further comprising:

45. Using at least the processor, generate the additional associations according to the pathology identifiers of the initial set of associations. The method according to claim 44, further comprising:

46. Using at least the aforementioned processor, a machine vision system is used to identify the visual features. The method according to claim 43, further comprising:

47. Using at least the aforementioned processor, the comparison involves comparing a first visual feature of a first second dataset with a second visual feature of a second second dataset, wherein the second second dataset includes an initial set of associations with the second first dataset. Using at least the aforementioned processor, the first visual feature is matched to the second visual feature in accordance with the comparison, Using at least the processor, determine an initial set of associations for the first and second datasets in accordance with the match. The method according to claim 46, further comprising:

48. Using at least the processor, a named entity recognition process is used to identify the named entities in the first dataset. The method according to claim 43, further comprising:

49. Using at least the aforementioned processor, the first dataset and the second dataset are preprocessed. Using at least the aforementioned processor, a similarity score is generated between the preprocessed first dataset and the preprocessed second dataset. The method according to claim 32, further comprising:

Images