Classifying discipline-specific content using brain response models of general content

JP2024505611A5Inactive Publication Date: 2024-12-09BRAINVIVO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023534244
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-01-14
Filing Date
2021-12-27
Publication Date
2024-12-09
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing automatic inspection technologies are unable to replace the capabilities of human experts in identifying defects and determining their consequences, particularly in fields like electronic circuit defect inspection and medical examinations, necessitating manual human review.

Method used

A method and device using a general brain response model derived from human brain measurements to classify content items by extracting features and applying a mapping between feature sets and brain activities, enabling automated classification based on discipline-specific brain responses.

Benefits of technology

Enables automated classification of content items with high accuracy by mimicking human expert review, reducing the need for manual inspection and enhancing efficiency in disciplines such as semiconductor defect inspection and medical diagnostics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The method for categorizing content comprises receiving content items (200) from categories of a particular domain of expertise and extracting respective features from each said content item. Labels of said content items of a particular domain of expertise performed by a human observer are received, the labels indicating the respective categories assigned to said content items by the human observer. A general content brain response model (212) is uploaded, the model being estimated using human brain measurements presented with a database of general content (204), the database of general content being defined using the features and including a mapping between features and brain activity. The model is applied to the extracted features and the labels are used to calculate brain responses for the particular domain of expertise. Given a new content item associated with a domain of expertise, a category that best matches the new content item is estimated based on the model and the domain-specific brain responses.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to artificial intelligence (AI). More specifically, the present invention is directed to techniques for automatic classification of domain-specific content using brain response models. [Background technology]

[0002] The human brain and its underlying biological neural networks have efficient ways of processing small amounts of data to make rapid cognitive classifications. Therefore, machine learning (ML) algorithms that can mimic the responses of the human brain may have particular advantages, such as relying on very limited training available to classify content.

[0003] For example, in the field of electronic circuit defect inspection, some inspection tasks still must be performed manually by human experts because there are still capabilities of human experts that cannot be replaced by existing automated inspection techniques, such as identifying defects, such as rare defects, and determining the consequences of the defects (e.g., scrapping a die due to a particular defect).

[0004] Another example can be seen in the field of medical diagnostics, where most of the diagnostic process is performed manually by medical professionals. Although AI models for diagnosis may become more available, it is expected that human experts will continue to be part of the diagnostic process, as data collection, structuring and labeling will limit AI from becoming an autonomous tool under the control of the medical system.

[0005] Therefore, there is a need for automated visual inspection techniques that can replace such human expert review. Summary of the Invention

[0006] An embodiment of the present invention described herein provides a method for categorizing content comprising receiving a set of content items belonging to a plurality of predefined categories of a particular domain of expertise and extracting respective features from each content item of the particular domain of expertise. Labels of the content items of the particular domain of expertise performed by one or more human observers are received, the labels indicating, for each content item, a respective category assigned to the content item by the one or more human observers from among the plurality of predefined categories. A database of general content is uploaded with a generic content brain response model estimated using human brain measurements, the database of general content being defined using a set of features and including a mapping between the set of features and the set of extracted brain activities. The generic content brain response model is applied to the extracted features and the labels are used to calculate a set of category-wise brain responses for the particular domain of expertise. Given a new content item associated with a domain of expertise, a category that best matches the new content item from among the plurality of predefined categories is estimated based on the generic content brain response model and the domain-specific brain responses.

[0007] In some embodiments, inferring a category comprises: (a) extracting a plurality of features from the new content item; (b) applying a general content brain response model to the features extracted from the new content item to compute a new content brain response; and (c) using the set of domain-specific brain responses and the new content brain response to infer a category that best matches the new content item. In some embodiments, estimating the category comprises estimating a respective set of distances between the new content brain responses and the domain-specific brain responses in a brain activity coordinate system, and using the label to classify the new content item into one of the predefined categories according to the set of distances. In one embodiment, the step of estimating the category comprises the steps of calculating a respective set of probabilities that the new content item has the same label as one of the given categories, and classifying the new content item into one of the predefined categories according to the calculated set of probabilities.

[0008] In another embodiment, extracting features comprises omitting from the extracted features one or more predetermined features that are deemed statistically insignificant. In some embodiments, the extracted features include at least one of color shading, characteristic spatial frequency, contrast level, and prevalence. In some embodiments, the content classification method further comprises deriving the general content brain response model using a statistical model that is one of linear regression and non-linear regression. In one embodiment, the human brain measurements include a brain connectivity matrix. In another embodiment, human brain measurements are modeled based on cognitive layers combined with connectivity association matrices. In some embodiments, the content items in a particular field include images of semiconductor dies, the categories are predefined quality bins, and a human review label is assigned to each image as representing a die that belongs to one of the predefined quality bins.

[0009] In some embodiments, the method includes determining a use of a semiconductor die for which the image of the die is classified as representing a die that belongs to one of the predefined bins. In one embodiment, brain measurements are performed by one or more of anatomical magnetic resonance imaging (MRI), diffusion tensor imaging (DTI), functional MRI (fMRI), electroencephalogram (EEG), magnetoencephalogram (MEG), infrared imaging, ultraviolet imaging, computed tomography (CT), brain mapping ultrasound, in vivo cellular data, in vivo molecular data, genomic data, and optical imaging. In some embodiments, the label comprises a label for at least one of a sequence of frames, an image, an audio, a tactile signal, a smell, a taste, and an abstract content type. In some embodiments, the abstract content type is an emotion. In one embodiment, the features are represented as a first space, the set of brain activities is represented as a second space, and a general content brain response model is defined as a linear transformation between the first space and the second space.

[0010] Another embodiment of the present invention provides an apparatus for classifying content, the apparatus comprising an interface and a processor. The interface is configured to perform the steps of: (i) receiving a set of content items belonging to a plurality of predefined categories of a particular domain of expertise and extracting respective features from each of the content items of the particular domain of expertise; (ii) receiving labels of the content items of the particular domain of expertise performed by one or more human observers, the labels indicating, for each of the content items, a respective category from among the plurality of predefined categories assigned to the content item by the one or more human observers; and the processor is configured to perform the steps of: (i) uploading a database of generic content and a generic content brain response model estimated using presented human brain measurements, the database of generic content being defined using a set of features and including a mapping between the set of features and a set of extracted brain activities; (ii) applying the generic content brain response model to the extracted features and using the labels to calculate a set of category-specific brain responses for the particular domain of expertise; and (iii) given a new content item associated with a domain of expertise, inferring a category from among the plurality of predefined categories that best matches the new content item based on the generic content brain response model and the domain-specific brain responses. [Brief description of the drawings]

[0011] The present invention will be more fully understood from the detailed description of the embodiments, taken in conjunction with the following figures: [Figure 1] FIG. 1 is a block diagram that illustrates generally an apparatus configured to classify new content items that belong to a particular domain of expertise using a general brain response model and a domain-specific database of labeled brain responses, in accordance with an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram accompanied by a flow chart that illustrates generally the elements of the apparatus of FIG. 1 and the processing scheme applied to generate a general brain response model and use it to derive the database of labeled brain responses of FIG. 1 , according to an embodiment of the present invention. [Diagram 3] 2 is a flow chart that illustrates generally a method of classifying new content items that belong to a particular domain using the apparatus of FIG. 1, according to an embodiment of the present invention; [Figure 4] FIG. 2 is a pictorial schematic diagram of a set of estimated distances between brain responses of new content items and the variance of domain-specific labeled brain responses used to classify new content items, in accordance with one embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (overview) The embodiments of the invention described below provide an apparatus and method for classifying content of a given domain of expertise using a general brain response model. An example domain of expertise includes inspection of semiconductor processing defects, but in general the disclosed techniques can be used to classify content related to other suitable domains of expertise. The content items can include, for example, images (e.g., of processed semiconductor dies). However, in general the disclosed techniques can be used to classify other types of content, such as content items that are audio files.

[0013] In general, the disclosed techniques generate and use a generic brain response model (hereinafter also referred to as a "naked brain response model") to compute the required discipline-specific brain responses. A generic brain response model is generic, or naked, in the sense that it is not discipline-specific. Generally speaking, a naked brain response model is composed of responses of thousands of brain regions, each region uniquely activated by different attributes of the content. Responses can be estimated from functional MRI scans of the brains of human observers presented with a discipline-free content database, e.g., a sufficiently large image set.

[0014] The general brain response model covers a very large number of attributes, hereafter referred to as "features." This coverage allows for the simulation of content-specific brain responses without the need to scan human brains exposed to content in a particular domain and add content-specific attributes to the model. For example, a feature such as shades of gray is covered by the general model to the extent that the brain response to a particular content does not require the addition of new shades of gray to the shades of gray feature list.

[0015] For a given classification task, a set of labeled, domain-specific brain responses is computed, along with the brain responses computed when given a new content item that belongs to that domain but whose label is unknown.

[0016] Using statistical analysis, the category to which the new content belongs (e.g., its label) is inferred by the processor, for example by comparing the mapped responses to the variability of domain-specific brain responses. In this way, new content items in the domain of expertise can be classified. Naked brain response models, described in detail below, typically include one or more mappings (e.g., operators) between a space of features (that define the content items) and a brain activity space that contains fMRI-extracted brain amplitudes.

[0017] As indicated above, the disclosed technology is based on the observation that a general brain response model can be constructed by presenting a human with sufficiently comprehensive general content under certain conditions and scanning the brain in the process. The general brain response model can then be used to compute respective discipline-specific brain responses using a database of discipline-specific labeled content items. The discipline-specific brain responses can then be used to classify new content in a particular discipline.

[0018] For example, a database of general content, such as a set of images capturing sophisticated urban environments, may be presented to a large number of people during a brain scan. fMRI scans, and the brain responses extracted by fMRI, can be used to simulate brain responses to domain-specific images (e.g., electronic circuits).

[0019] Thus, the disclosed technology leverages actual human brains to generate "expert machines" that more closely (e.g., more closely than fully artificial models) employ and mimic the capabilities of human experts in many different disciplines to automatically classify new content. Nonetheless, this expert performance is achieved by using mostly non-experts for the generation of the bare brain response models and a small number of domain-specific humans ("experts") for labeling.

[0020] Using content from complex environments ensures that a sufficiently large set F of features for any same type of content (e.g., images) is covered (e.g., sampled) that is collectively sufficient for the process to generate different simulated brain responses that each serve as the basis for successfully classifying content from different disciplines. Indeed, the inventors have empirically found that it is possible to generate such a "universal" set F of features and successfully apply a general brain response model to discipline-specific brain response classification problems.

[0021] Such a universal set of features F can include, among others, geometry, color, hue, contrast, characteristic spatial frequencies, etc., obtained using spectral analysis of the image and feature prominence. The feature extraction step typically involves performing initial image pre-processing steps such as smoothing, biasing, dilation, etc.

[0022] An example of multiple categories for a particular domain of expertise is the category of images of processed semiconductor dies. The domain of expertise in such an application is the inspection of semiconductor process defects. In this example, one or more experienced human observers label the content, and each category contains only images labeled as representing dies of similar (e.g., same bin) quality and is labeled in the category accordingly. In other words, different images labeled as showing different die qualities (i.e., dies belonging to different category bins) will be labeled with different labels L for a set G of M labels. s ∈ G, s=1,2,…M (e.g., each group of images with the same label corresponds to a single predefined labeled quality bin).

[0023] Thus, in some embodiments, the preparation stage of the disclosed techniques is divided into two phases: (i) The first "universal" phase involves actual brain scans of humans presented with content from the general content database. Using statistical models, a "naked" brain response model is generated to establish an optimal transformation between a "universal" feature set F that defines the general content and a set of brain response amplitudes (i.e., F). For example, the set of features could be one that defines a feature space that represents the general content by spanning each feature vector of the content items. (ii) A domain-specific phase, in which one or more human experts label content items in a particular domain of expertise as belonging to one of a group G of predefined categories for that domain of expertise. A naked brain response model is applied to compute brain responses to the labeled content items according to the categories of the labeled content items, thereby obtaining a database of estimated brain responses of human experts in that domain of expertise.

[0024] In one embodiment, during an initial preparation phase, the processor extracts from each (digitized) content of the general content database a set of features (e.g., feature vectors) f that belong to the aforementioned predefined "universal" content set (f ∈ F). For each content item, each of the set of features f ∈ F specifies a different weight, which can be binary (checked / not checked) features of the set F (i.e., in some applications some members of f are nulled out and not used).

[0025] The naked brain response model uses the feature f i It can be constructed from a set of relations (e.g. equations), such as a linear relationship between ∈ F and the estimated brain amplitudes {A}. The latter is also called a "brain activity map": Formula (1) A j (R j )=Σ (i=1) n α ij f i ,1≦j≦k Here, A j (R j ) corresponds to brain region R, which is further described below. j The processor then statistically estimates the coefficients α by, for example, performing linear regression. ij is derived.

[0026] In another expression, a vector of image features f, f=(f1,f2,…,f n ) is extracted for each image shown to the subject during the fMRI scan, from which a respective brain activation feature vector, A=(A1,A2,…,A k ) is extracted. Unknown mapping:

number

number

[0027] The statistically solved relation in equation (1) thereby defines a set of “best fit” coefficients of the relation, which in the following is the array [α ij F,0 ], which can also be described as the aforementioned mapping (e.g., operator) that transforms any vector in the feature space into a respective brain activity vector in the brain activity space.

[0028] It should be noted that the structure of equation (1) depends on the underlying brain model and statistical model used, as explained below. The equations shown are provided merely as examples for ease of explanation. For example, a set of nonlinear relationships could be substituted for equation (1).

[0029] Therefore, more generally, a statistical algorithm (in particular, a linear or nonlinear regression, or a general linear model (GLM), hereafter referred to as Stat), statistically solves a large set of relations (such as the relations in Eq. (1)) between the extracted features and the respective estimated brain amplitudes for each image to obtain a general content brain response model (or operator): Formula (2) [α ij F,0 ]=Stat({[α ij F,0 ]} f∈F )

[0030] Coefficient [α ij F,0 This model, or statistically estimated operator, described by the array , is still “naked” or implicit, in the sense that the content (e.g., images) is not application-specific, and thus the set of extracted features does not represent the actual classification target.

[0031] The dimensions of the above bare brain response models are given by way of example only, and the actual dimensions may vary depending on the complexity (anatomical, functional, etc.) of the underlying brain model used. At the same time, regardless of the exact mathematical description, a sufficiently comprehensive (e.g., of a large database) general content ensures that any bare brain response model used is "complete" (e.g., the coefficients [α ij F,0 ] is not affected by database growth).

[0032] As mentioned above, to build the aforementioned expert machine (e.g., a database of domain-specific brain responses), a processor may generate a set of labels L sThe user uploads a subject-specific set of (digitized) content items with Ns labeled categories ∈G. Each labeled category is assumed to contain Ns content items. The processor extracts from each of the subject-specific content items a subset F of features from the universal set of predefined features mentioned above. ds , F ds Extract the subset F ⊆ F ds can be the same as the predefined set F, or a smaller, optimized subset. For example, in the case of detecting or receiving a representation of a black and white image, the subset of extracted features may be the same set, but with zero weights for shades of gray (e.g., shades of red). As another example, f ds and / or the elements of f are respectively the set F ds and / or F can be binary (checked / unchecked) features. Alternatively, the subset size can be reduced to consist of only shades of gray, i.e., saving memory space and reducing computation by not processing invalid shades of other colors.

[0033] In any case, for each discipline-specific content, feature f ds ∈F ds Each vector f ds The non-zero elements (i.e., weights) of differ between content items.

[0034] Feature Set F ds The extraction of involves the same preprocessing steps as above, followed by the extraction of a smaller feature subset F of the aforementioned feature set F. ds In addition to the possibility of reducing the model to , it also optimizes the model for a specific application (e.g. image classification based on grayscale image content).

[0035] Next, the processor calculates the naked brain response model [α ij F,0 ] (Feature f i∈F), and reapply the s Compute labeled, domain-specific brain responses for each ∈G: Formula (3) {[α ij F,0 ]f ds} fds∈Fds,Ls∈G In one embodiment, the set of discipline-specific brain responses given by equation (3) can be described as a scattering of vectors in brain response space.

[0036] Given a new content item with an as yet unknown label x∈G that needs to be classified (e.g., given a new image of a processed semiconductor die), the new content item is assumed to belong to one of multiple categories in the domain of expertise. Thus, the processor computes a feature set (e.g., a feature vector) f nc ∈ F ds Extract.

[0037] The processor then applies the brain response model to the new content item to calculate the brain response to the new content: Formula (4) {[α ij F,0 ]f nc Here, as mentioned above, f nc ∈ F ds is the extracted feature set of the new content. If the new content is a feature set of the same content, such as an extended set of images of semiconductor defects, f nc ∈ Fν ⊆ F ds When generating (e.g., by extension) a set of different "views" of the same content item, the brain response to the new content can be statistically refined as follows: Formula (5) Stat({[α ij F,0 ]f nc} fnc,∈Fν)

[0038] N discipline-specific content items per category s So, the average of these brain responses is: Formula (6)

number

[0039] In other words, a hypothesized labeled dataset from a particular domain is {f ,L s} s=1 M Here, we assume that s ∈ G and that the set of categories G has M categories. In some applications, some members of f are nulled and not used, so such a reduced feature vector can be written as f above. ds Note again that the sigma-based approach is shown as

[0040] The brain activations belonging to the same category (i.e., class) are averaged, and the average is expressed as: Formula (6´)

number

number

number

[0041] In the above description, the processor then applies a statistical test, such as a similarity test, or in some embodiments, a vector [α ijF,0 ]f nc Calculate the dot product between and the mean vector for each label, and the unknown label x is converted to the label L s Generate the probability that ∈G: Formula (7)

number

[0042] More generally, similarity may be related to novel content-based brain responses [α ij F,0 ]f nc and labeled scattering {[α ij F,0 ]f ds} fds∈Fds,Ls∈ Similarity can be estimated by calculating the distance between x and y. Other distances such as the L1 or Hellinger distance can be used to estimate similarity.

[0043] When the dot product is used, the labels are used by the processor to classify the content according to a probability distribution and a classification criterion applied to that distribution. For example, the processor may classify the content according to a probability distribution and a classification criterion applied to that distribution. The processor may select the most likely label based on the results of the mapping of the new content to the labeled content. More generally, the processor classifies the content according to the level of similarity between the mapped new content and the mapped labeled content, as described further below.

[0044] In some embodiments, the processor further outputs a decision based on the classification, for example, the processor can make a decision to reject a semiconductor die based on the classification using pre-specified acceptance criteria and direct the die for an in-house root cause analysis.

[0045] To classify content of very different disciplines, including non-visual disciplines such as emotion, tone, etc., the content pre-processing step may include converting content between content types for further analysis using the disclosed techniques. For example, pre-processing may include encoding an audio file into an image.

[0046] Alternatively, content belonging to a non-visual category may have its own type of database of defined associated features, such as, for audio, loudness levels, and characteristic sound frequencies. However, the inventors have discovered that the processor can instead be programmed to perform a different pre-processing step, for example, encoding audio information into visual information and using the extracted features of the visual content in a verifiable manner. Similarly, other non-visual content types, such as smells and abstract content types (e.g., emotions), can also be successfully encoded and classified into multi-dimensional visual scales using the disclosed techniques.

[0047] (Description of the device) 1 is a schematic block diagram of an apparatus 10 configured to classify new content items 102 belonging to a particular domain using a general brain response model 212 and a domain-specific database of labeled brain responses 200, according to an embodiment of the present invention. The domain-specific new content items 102 may be, for example, images of processed semiconductor dies.

[0048] In order for the device 10 to perform its intended use, the memory 112 of the device 10 holds a database 200. The database 200 includes pre-computed, category-specific, labeled brain responses, as illustrated in FIG. {[α ij F,0 ]f ds} fds∈Fds,Ls∈ In other words, the processor 110 maintains in the memory 112 an "expert machine" for that area of ​​expertise.

[0049] Digitization module 104 digitizes new content 102 into a digital data set, hereinafter referred to as a digital image, for input to device 10. The digital image may be communicated (e.g., over a network or over a local communications link) to processor 110 of device 10. The digital image may be stored in memory 112.

[0050] The feature extraction module 122 of the processor 110 executes a feature recognition algorithm to extract from the new content 102 each new content feature set (e.g., feature vector) f nc Typically, the feature extraction module 122 extracts a feature vector f nc To aid in the extraction of the content, the method may include one or more of the pre-processing steps described above, as appropriate for use with content of a particular domain.

[0051] The processor 110 calculates a feature vector f nc is input to the multiplication module 124, which multiplies the new content, [α ij F,0 ]f nc At this stage, new content is labeled with the previously unknown labels x∈G. Upon receiving the input, the statistical testing module 125 derives a labeled mapped brain response model. {[α ij F,0 ]f ds} fds∈Fds,Ls∈ Mapped brain response data points (e.g. vectors) to [α ij F,0 ]f nc , where these can all be vectors in brain activity space, or more generally, data points in a brain activity coordinate system as shown in FIG. 4. The statistical testing module 125 then performs a similarity test by computing the vector dot product, for example using equation (7), to determine whether the human experts have associated the new content 102 with category L s The probability that we would have classified and labeled it as belonging to p = p s , x = L s More generally, as described below, the two sets are subjected to a similarity check using a set of estimated distances as shown in Figure 4.

[0052] To cover all labels, the statistical test module 125 (time logic not shown) tests category L s We call all stored sets 200 of domain-specific sets of brain responses for each ∈G and generate a probability distribution that a new content item 102 belongs to one of the labeled sets of domain-specific content.

[0053] The content classifier 126 prepares results 150 including labeled and ranked probability distributions for selection. To this end, the content classifier 126 can apply predefined criteria, such as selecting the label with the greatest probability, the smallest distance, etc. The content classifier 126 outputs a decision depending on the results, for example to a user display (not shown).

[0054] For example, for an image of a processed semiconductor die, the results may be limited by module 126 to three options: a 97% probability that the die will be labeled as belonging to the "good" bin, a 2% probability that the die will be labeled as belonging to the "acceptable" bin, and a 1% probability that the die should be labeled as belonging to the "fail" bin. This decision may be based on criteria stored in memory 112, for example, having a 0.1% criterion as the maximum probability allowed for the "fail" bin, rejecting the die based on the above classification, and transferring the die to human inspection for root cause analysis.

[0055] The processor 110 can be engaged locally or remotely using a communication and user interface 114, such as queries from a remote program / user, or selections / queries by a user using, for example, a touch screen.

[0056] In various embodiments, the different electronic elements of the apparatus shown in FIG. 1 may be implemented using suitable hardware, such as using one or more discrete components, one or more application specific integrated circuits (ASICs) and / or one or more field programmable gate arrays (FPGAs). Some of the functionality of the disclosed processors, units, and modules, e.g., some or all of the functionality of processor 110 and its modules 122-126, may be implemented in one or more general-purpose processors, which are programmed with software to perform the processes described herein. The software may be downloaded to the processor in electronic form, e.g., over a network or from a host, or alternatively or additionally, may be provided and / or stored on non-transitory tangible media, such as magnetic, optical, magnetic media, or electronic memory. In particular, processor 110 executes dedicated algorithms that enable processor 110 to execute the dedicated algorithms disclosed herein, including FIG. 3, as further described below.

[0057] (Generation of content-specific labeled brain response sets) FIG. 2 is a block diagram accompanied by a flow chart that illustrates, in accordance with an embodiment of the present invention, generally the elements (122, 124) of the device 10 of FIG. 1 and the processing scheme applied to generate a general brain response model 212 and use it to derive the labeled brain response database 200 of FIG. 1 .

[0058] The process shown is divided into two phases: (i) In phase 201, we take actual brain scans and use a statistical model 222 to model the general content brain response model [α ij F,0 ] (212), which connects features from the content database 204 with each set 211 of fMRI-extracted brain amplitudes. (ii) In phase 202, a general content brain response model 212 is applied to Brain response for each category Ls∈G {[α ij F,0 ]f ds} fds∈Fds,Ls∈ or a vector of features f ds 226. In this way, the power of the human brain in prediction or classification is harnessed to model multi-domain expert machines.

[0059] According to the embodiment presented, the generation of the database 200 includes algorithms used by the device 10 to execute a process that begins with receiving a database 204 of digitized content. Two examples of digitized content are a set of color images capturing a complex urban environment and / or other complex environments, including various human activities and gestures.

[0060] Using the feature extraction module 122 of FIG. 1, the processor 110 extracts from each content (i.e., digitized image) a predefined set of features 206 F=(f1, f2, ..., f k ) features (filled circles). Such extraction usually involves preliminary steps of preprocessing and pattern recognition.

[0061] As shown above, for each content, we define a vector (f1, f2, …, f k ) features are checked and not checked, and the content of the feature vector fcontent For example, color features are not checked in grayscale images, and some geometric features may or may not be checked, such as circular arcs that are missing in a target image that contains only straight lines and right angles.

[0062] The processor 110 further receives a respective database of functional MRI (fMRI) data (208) that includes fMRI images of the brain of the person to whom the contents of the database 204 are presented. The data 208 is presented as fMRI data, but may be of other types, such as electroencephalography (EEG), magnetoencephalography (MEG), infrared images, ultraviolet images, computed tomography (CT), ultrasound images, in vivo cellular data, in vivo molecular data, genomic data, or optical imaging.

[0063] Using a pre-specified brain model, such as a map of brain regions and / or neural pathways in the brain that may be activated, the pattern recognition module 210 generates a brain model from the brain measurements (e.g., fills in such data into the model) for each set 211 of brain response amplitudes 213 for each image presented to the human, where the brain response amplitudes 213 are, for example, the brain activity amplitudes A j (R j ).

[0064] Mathematically, modeling the activation of specific brain regions and / or neural pathways (the latter also called the "brain connectome") by medical images and other measurement tools can be achieved by replacing a set of matrices of fMRI with a general human brain response model. We generate a brain activity map by extracting response values ​​that correlate with features of the presented content, and then set amplitudes A of the aforementioned general content brain response model (operator j (R j ) provided.

[0065] In the embodiment shown, a set of fMRI extracted brain amplitudes 211 (e.g., a set of brain activity maps 213) are generated for each brain region (R1, R2, ..., Rk ) are shown to include grey-level values ​​of the response for each brain region R j , while (another array (not shown) may indicate the level of activation by taking into account the interactions (e.g., couplings) between brain regions (i.e., weighted according to the connectome and represented, for example, as off-diagonal elements of a set of matrices of a general human brain response model, where the diagonal elements represent values ​​for brain regions).

[0066] Thus, the statistical model 222 computes the vector of extracted features f content , respectively, into sequences of brain response amplitudes 213, and then we use equations (1)-(2) to calculate the general content brain response model [α ij F,0 ] (212) to store the operator in memory for use in generating the best-fit bare brain response model 212 (i.e., the best-fit operator [α ij F,0 ] ) are derived statistically (e.g., using multidimensional regression).

[0067] In phase 202, one or more experts review and label (228) a set of domain-specific content 224. For example, if the content is images of fabricated semiconductor dies, such an expert could be a defect inspection technician. The experts review and label (228) a set of domain-specific labels G=(L1, L2, …, L m ), any label can be selected from the set 230 G=(L1, L2, …, L3). A predefined group of labels may include, for example, labels L1=“highest bin” (e.g., completely clean die), L2=“high bin”, L3=“medium bin”, L4=“low bin”, L5=“lowest bin”, and L6=“reject”. Here, in general, the results of the review and labeling process 228 are stored in a set 230 G=(L1, L2, …, L m ) labels must be covered.

[0068] In parallel, the processor 110 extracts from the set 224 the domain-specific content, a respective subset 226 of domain-specific features, using the feature extraction module 122 .

[0069] In this step, the estimated naked model 212 is modeled on a subset of the applied domain-specific features 226 and each label L s Using the computation for each ∈G, we s Labeled discipline-specific brain responses for each {[α ij F,0 ]f ds} fds∈Fds,Ls∈ get.

[0070] Returning to Figure 1, suppose we have a new content item in a domain that has not yet been classified, say with an as yet unknown label x∈G, and a feature set f nc , the general content brain response model 212 is applied to calculate the brain response to the new content using equation (4). Next, the statistical testing module 125 determines whether the label x is related to the label L, as described above. s Output the probability for each category in ∈G.

[0071] Thus, the result 150 in Figure 1 is a probability distribution p = (p1, p2, …, p m ), whose labels are the labels (L1, L2, …, L m ) or x=L s The probability that p= p s Such probabilities can be calculated, for example, using equation (7).

[0072] For the semiconductor die example, the result 150 in FIG. 1 may be a probability distribution p=(p1, p2, ..., p6) of the label of the new image being one of the labels (L1, L2, ..., L6) listed above.

[0073] The probability distribution is then translated into a decision-making process (e.g., a commercial decision to determine whether the imaged die is suitable for high-end applications, low-end applications, or needs to be discarded) based on acceptance criteria, e.g., thresholding of probability amplitudes or occurrence levels per label.

[0074] (How to categorize new content) Figure 3 is a flow chart that illustrates in schematic form a method for classifying new content items belonging to a particular domain of expertise using the device 10 of Figure 1 in accordance with an embodiment of the present invention. The algorithm according to this embodiment performs a process that uses the brain response model 212 of general content and the labeled domain-specific brain responses 200 for each category, stored in the memory 112 of the device 10. An example of new content is the aforementioned image of a processed semiconductor die taken by an inspection machine to detect semiconductor processing defects.

[0075] The process begins with a discipline-specific content upload step 302, where the processor 110 uploads different categories L of a particular discipline, such as semiconductor processing defect inspection. s Upload multiple content items, labeled as categories, L ∈G. Each category is labeled with different labels L , ranging from benign defects to critical defects. s In particular, one category of content may be a reference image of a "golden die" (i.e., an image of a defect-free die). It is assumed that the content is already in digital form.

[0076] Next, in a domain-specific feature extraction step 304, the feature extraction module 122 of the processor 110 performs preprocessing to extract the feature vector f ds ∈ F ds Each set of is extracted and stored in memory 112.

[0077] Next, in a naked brain response model upload step 306, the processor 110 uploads the naked brain response model 212, [α ij F,0 ] to upload.

[0078] Next, in the computation step 308 of the domain-specific brain responses, we use a labeled (230) feature vector f ds ∈ F ds Using the set of {[α ij F,0 ]f ds} fds∈Fds,Ls∈ is calculated, i.e., in the aforementioned expert machine database 200 of the field of expertise.

[0079] In a new content feature extraction step 310, given a new content item with an as yet unknown label x ∈ G (e.g., given a new image of a processed semiconductor die), the processor 110 extracts a feature vector f nc ∈ F ds Extract.

[0080] In the new content mapping step 312, the brain response to the new content [α ij F,0 ]f nc Calculate.

[0081] In a labeling step 314, the processor 110 determines, using a statistical test including, for example, equation (7), that s The probability that p = p s Specifically, using equation (7), the processor 110, using the module 125, generates a vector [α ij F,0 ]f nc [α ij F,0 ]f ds Calculate the directional components.

[0082] Finally, in a classification step 316, module 126 of processor 110 classifies the new content item (e.g., into the most probable label or into the label that best fits) based on the probability distribution P and other pre-specified classification criteria (e.g., the set of distances described above and shown in FIG. 4).

[0083] The flow chart of Figure 3 is provided by way of example only for clarity. Although the inspection process described by Figure 3 is illustrated for a wafer die, other electronic circuits, such as circuits on a PCB, may be inspected in a similar manner. Although the example of the disclosed method shown in Figure 3 is of a particular discipline, the method can be built upon and applied, mutatis mutandis, to numerous disciplines ranging from engineering to psychology, law enforcement to entertainment.

[0084] FIG. 4 is a pictorial schematic diagram of a set of estimated distances 404 between brain responses 410 of new content items and the variance of domain-specific labeled brain responses used to classify new content items, according to one embodiment of the present invention.

[0085] As can be seen, the new content data points 410 have different distances (e.g., L1 distances) from the labeled sets of brain responses 402, with distance d2 from the L2 labeled set 402 being the smallest of these and distance d4 from the L4 labeled set 402 being the largest.

[0086] In one embodiment, the processor 110 labels the new content with the label L2. In another embodiment, the processor 110 labels the new content with the distances d1, d2, ..., d s , …, d m , and generate a probability distribution based on the inverse of , and assign probabilities to the labels L1, L2, ..., L according to the probability distribution derived from the distance. s , … , L m Label the data point 410 with .

[0087] The diagram in Figure 4 is provided as an example only for clarity. Additional steps can be performed but are omitted for simplicity. For example, in one embodiment, each of the scatters 402 is first averaged into a single data point and the distance between the average values ​​is calculated. The example shown in Figure 4 can be built upon and applied in many disciplines.

[0088] It will therefore be understood that the above-described embodiments have been cited by way of example, and that the present invention is not limited to what has been particularly shown and described above, but rather the scope of the present invention includes both combinations and subcombinations of the various features described above, as well as variations and modifications thereof that will occur to those skilled in the art upon reading the above description and that are not disclosed in the prior art.

Claims

1. A method for classifying content comprising: receiving a set of content items belonging to a plurality of predefined categories of a particular field of expertise and extracting respective features from each of the content items of the particular field of expertise; receiving labels of the content items of the particular field of expertise, performed by one or more human observers, the labels indicating, for each content item, a respective category assigned to the content item by the one or more human observers from among the plurality of predefined categories; uploading a generic content brain response model estimated using human brain measurements presented to a database of generic content, the database of generic content being defined using the set of features and including a mapping between the set of features and the set of extracted brain activities; applying the general content brain response model to the extracted features and using the labels to compute the set of categorical brain responses for the particular domain of expertise; and When given a new content item associated with the domain of expertise, estimating a category from among the plurality of predefined categories that best matches the new content item based on the content-general brain response model and the domain-specific brain responses; A content classification method comprising:

2. The step of estimating the category comprises: extracting a plurality of said features from said new content item; applying the generic content brain response model to the features extracted from the new content item to calculate a new content brain response; and using the set of domain-specific brain responses and the new content brain responses to estimate the category that best matches the new content item; 2. The method of claim 1, further comprising:

3. 3. The method of claim 2, wherein the step of estimating categories comprises estimating a respective set of distances between the new content brain responses and the domain-specific brain responses in a brain activity coordinate system, and using the label to classify the new content item into one of the predefined categories according to the set of distances.

4. 2. The method of claim 1, wherein the step of estimating a category comprises the steps of: calculating a respective set of probabilities that the new content item has the same label as one of the given categories; and classifying the new content item into one of the given categories according to the calculated set of probabilities.

5. 5. A content classification method according to claim 1, wherein the step of extracting features includes a step of omitting one or more predetermined features from the extracted features that are deemed to be statistically insignificant.

6. The method of claim 1, wherein the extracted features comprise at least one of color shading, characteristic spatial frequency, contrast level, and prevalence.

7. 5. The method of claim 1, further comprising deriving the general content brain response model using a statistical model that is one of linear regression and non-linear regression.

8. 5. The method of claim 1, wherein the human brain measurements include a brain connectivity matrix.

9. 10. The method of claim 8, wherein the brain connectivity matrix has connectivity matrix weights based on microstructure estimation of brain tissue to form a brain connectome.

10. 5. The method of claim 1, wherein the human brain measurements are modeled based on cognitive layers combined with connectivity association matrices.

11. 5. The method of claim 1, wherein the content items in the particular domain include images of semiconductor dies, the categories are predefined quality bins, and the human review labels assign each image as representing a die belonging to one of the predefined quality bins.

12. 12. The method of claim 11, further comprising determining a use of a semiconductor die, the image of which is classified as representing a die belonging to one of the predetermined bins.

13. 5. The method of claim 1, wherein the brain measurements are performed by one or more of anatomical magnetic resonance imaging (MRI), diffusion tensor imaging (DTI), functional MRI (fMRI), electroencephalography (EEG), magnetoencephalography (MEG), infrared imaging, ultraviolet imaging, computed tomography (CT), brain mapping ultrasound, in vivo cellular data, in vivo molecular data, genomic data, and optical imaging.

14. 5. The method of claim 1, wherein the labels comprise at least one of a sequence of frames, an image, an audio signal, a tactile signal, a smell, a taste, and an abstract content type.

15. 15. The method of claim 14, wherein the abstract content type is an emotion.

16. 5. The content classification method according to claim 1, wherein the features are represented as a first space, the set of brain activities is represented as a second space, and the general content brain response model is defined as a linear transformation between the first space and the second space.

17. 1. A content classification device, comprising: an interface; and a processor; The interface: receiving a set of content items belonging to a plurality of predefined categories of a particular field of expertise and extracting respective features from each of the content items of the particular field of expertise; receiving labels of the content items of the particular field of expertise, performed by one or more human observers, the labels indicating, for each content item, a respective category assigned to the content item by the one or more human observers from among the plurality of predefined categories; configured to run The processor: uploading a generic content brain response model estimated using human brain measurements presented to a database of generic content, the database of generic content being defined using the set of features and including a mapping between the set of features and the set of extracted brain activities; applying the general content brain response model to the extracted features and using the labels to compute the set of categorical brain responses for the particular domain of expertise; and When given a new content item associated with the domain of expertise, estimating a category from among the plurality of predefined categories that best matches the new content item based on the content-general brain response model and the domain-specific brain responses; configured to perform A content classification device characterized by:

18. The processor: extracting a plurality of said features from said new content item; applying the generic content brain response model to the features extracted from the new content item to calculate a new content brain response; and using the set of domain-specific brain responses and the new content brain responses to estimate the category that best matches the new content item; The category is estimated by 18. The content classification device according to claim 17.

19. 20. The content classification device of claim 18, wherein the processor is configured to estimate the category by estimating a respective set of distances between the new content brain responses and the domain-specific brain responses in a brain activity coordinate system, and using the label to classify the new content item into one of the predefined categories according to the set of distances.

20. 18. The content classification device of claim 17, wherein the processor is configured to estimate the category by calculating a respective set of probabilities that the new content item has the same label as one of the given categories, and classifying the new content item into one of the given categories according to the calculated set of probabilities.

21. 21. The content classification device of claim 17, wherein the processor is configured to extract the features by omitting from the extracted features one or more predetermined features that are deemed statistically insignificant.

22. 21. The content classification device according to claim 17, wherein the extracted features comprise at least one of color shading, characteristic spatial frequency, contrast level, and prevalence.

23. 21. The content classification device of claim 17, wherein the processor is further configured to derive the general content brain response model using a statistical model that is one of linear regression and non-linear regression.

24. 21. The content classification device of claim 17, wherein the human brain measurements include a brain connectivity matrix.

25. 25. The content classification device of claim 24, wherein the brain connectivity matrix has connectivity matrix weights based on microstructural estimation of brain tissue to form a brain connectome.

26. 21. The content classification device according to any of claims 17 to 20, wherein the human brain measurements are modelled based on cognitive layers combined with connectivity association matrices.

27. 21. The content classification apparatus of claim 17, wherein the content items of the particular field of expertise include images of semiconductor dies, the categories are predefined quality bins, and the human review labels assign each image as representing a die belonging to one of the predefined quality bins.

28. 28. The apparatus of claim 27, wherein the processor is further configured to determine a use of a semiconductor die for which the image of the die is classified as representing a die belonging to one of the predetermined bins.

29. 21. The content classification device of claim 17, wherein the brain measurements are performed by one or more of anatomical magnetic resonance imaging (MRI), diffusion tensor imaging (DTI), functional MRI (fMRI), electroencephalography (EEG), magnetoencephalography (MEG), infrared imaging, ultraviolet imaging, computed tomography (CT), brain mapping ultrasound, in vivo cellular data, in vivo molecular data, genomic data, and optical imaging.

30. 21. The content classification device according to claim 17, wherein the labels comprise at least one of a sequence of frames, an image, an audio, a tactile signal, a smell, a taste, and an abstract content type.

31. 31. The apparatus of claim 30, wherein the abstract content type is an emotion.

32. 21. The content classification device of claim 17, wherein the features are represented as a first space, the set of brain activities is represented as a second space, and the general content brain response model is defined as a linear transformation between the first space and the second space.