Improvement of sample segmentation

Abstract

To perform image segmentation with high accuracy and efficiency. A method, system, and apparatus, including a computer program encoded on a computer storage medium, for improving image segmentation using hyperspectral imaging. In some implementations, the system acquires image data for a hyperspectral image, the image data including image data for each of a plurality of wavelength bands. The system accesses stored segmentation profile data for a particular object type indicating a predetermined subset of wavelength bands designated for segmenting different region types for an image of an object of the particular object type. The system segments the image data into a plurality of regions using the predetermined subset of wavelength bands designated in the stored segmentation profile data for segmenting the different region types. The system provides output data indicative of the plurality of regions and the region types for each of the plurality of regions.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit and priority of U.S. Non - Provisional Patent Application No. 17 / 383,278, filed on July 22, 2021, the entire disclosure of which is incorporated herein by reference in its entirety. <>

[0002] This specification generally relates to the digital processing of images, and more particularly to improved image segmentation based on hyperspectral images.

Background Art

[0003] Image segmentation is a digital image processing technique that divides an image into meaningful parts such that pixels belonging to a particular part share similar characteristics. This enables the analysis of digital images by defining the shape and boundaries of objects within the image. Image segmentation is widely used in a number of areas such as autonomous vehicles, medical image diagnosis, and satellite imaging.

[0004] Hyperspectral imaging technology can provide image data regarding a subject for multiple bands of light having different wavelengths (e.g., "wavelength bands", "spectral bands", or simply "bands"). This provides significantly more information than grayscale images (e.g., indicating intensity over a single, typically large band) and standard color images (e.g., RGB images containing image information for the visible red, green, and blue bands). The additional data provided in hyperspectral images provides more information about the subject, but the much larger amount of resultant data in hyperspectral images (often for 5, 10, 20, or more different wavelength bands) is often not processed efficiently or effectively applied to image processing tasks such as segmentation.

Summary of the Invention

[0005] According to one innovative aspect of the subject matter described herein, a computer system can perform image segmentation with greater accuracy and efficiency than previous methods using hyperspectral images. Hyperspectral images contain far more information than conventional color images. This information is provided in many forms and often includes more bandwidth than a typical RGB image, information on a narrower spectral band than a conventional RGB Bayer filter band, and information on bands outside the visible range (e.g., infrared, ultraviolet, etc.).

[0006] However, not all wavelength bands in a hyperspectral image are relevant to the boundaries of each type of segmentation. As a result, depending on the type of object being imaged and its characteristics (e.g., material, composition, structure, texture, etc.), image data for different hyperspectral wavelength bands may indicate regional boundaries. Similarly, for some object types and regional types, information in some wavelength bands may add noise or actually obscure the desired boundaries, resulting in reduced segmentation accuracy and increased computational cost of segmentation analysis.

[0007] The techniques described below illustrate how computer systems can generate and use profiles that specify different combinations of wavelength bands to provide accurate and efficient segmentation of different object and region types. Using these profiles, the system can selectively use image data within a hyperspectral image, thereby using different combinations of image bands to locate different types of regions or boundaries within the image. For example, for a particular object type, a profile may indicate that for objects of that object type, a first type of region should be segmented using image data in bands 1, 2, and 3, and a second type of region should be segmented using image data in bands 3, 4, and 5. When processing a hyperspectral image of a particular object type, segmentation parameters specified in the profile are used, including subsets of bands for each region type, e.g., image data for bands 1, 2, and 3 to identify a first type of region, and image data for bands 3, 4, and 5 to identify a second type of region.

[0008] As an example, using fruit image segmentation, a computer vision system can automatically evaluate the characteristics and quality of fruit. Beyond simply segmenting fruit from the background, the system can be used to segment different parts of fruit from one another. In the case of a strawberry, the exterior includes leaves (e.g., calyx, sepals, pedicel), seeds (e.g., achenes), and pulp (e.g., husk). The pulp may have areas in different states, e.g., ripe, unripe, damaged, moldy, rotten, etc. To facilitate rapid and efficient machine vision analysis of individual strawberries for quality control or other purposes, the system can generate strawberry object type profiles that specify the type of area of ​​interest (e.g., leaves, seeds, and pulp) and subsets of hyperspectral image bandwidths to be used to segment or identify areas of each area type. These subsets of bandwidths can be determined through data-driven analysis of training examples, which include hyperspectral images and ground truth segmentation showing the area types for the examples. The profiles may specify other parameters for each area type, such as functions to apply to image data of different bandwidths and thresholds to use. Using defined profiles, the system can accurately and efficiently process hyperspectral images of strawberries and segment each region type. For each region type, the system can define boundaries for instances of that region type using a subset of bandwidth and other parameters specified in the profile. As a result, each region type can be accurately segmented using the subset of bandwidth that best represents the region boundary, and processing becomes more efficient by limiting the number of bandwidths used for segmentation of each region type.

[0009] As another example, image segmentation of waste materials can be used to better identify and characterize recyclable materials. For instance, a system can be used to accurately segment regions of image data representing different types of plastics (e.g., polyethylene (PE), polyethylene terephthalate (PET), polyvinyl chloride (PVC), polypropylene (PP), etc.) to automatically detect the material of an object and identify where different types of objects are located. Furthermore, segmentation techniques can be used to identify and characterize additives and instances of contamination within materials. For example, in addition to identifying regions containing one or more main materials (e.g., PE vs. PET), or instead, segmentation techniques can also identify objects or parts of objects that contain different additives (e.g., phthalates, bromides, chlorates, UV-resistant coatings) or contaminants (e.g., oil, food residues, etc.). To better characterize different types of regions, the system can generate and store profiles for different types of objects and materials that specify the type of region (e.g., different types of materials, different additives present, different contaminants), as well as for subsets of hyperspectral image bandwidths used to segment or identify regions of each region type. These subsets of bandwidths can be determined through data-driven analysis of training examples, which may include hyperspectral images and ground truth segmentation indicating region types for the examples. The profiles may specify other parameters for each region type, such as functions to apply to image data of different bandwidths and thresholds to use. Once the profiles are defined, the system can process hyperspectral images and segment each region type accurately and efficiently. For each region type, the system can define boundaries for regions composed of different materials, regions where different contaminants are detected, regions where different types of contaminants exist, and so on.As a result, each domain type can be precisely segmented using a subset of bandwidth that best represents the domain boundary, and the process becomes more efficient by limiting the number of bandwidths used for segmentation of each domain type.

[0010] As will be further explained below, the system can also define a composite band that modifies the bands before performing segmentation. The composite band can be based on one or more image bands in the hyperspectral image, but one or more functions or transformations may be applied to it. For example, the composite band may be a combination or aggregation of two or more bands to which a function is applied (e.g., addition, subtraction, multiplication, division, etc.). One example is to calculate a normalized index based on two bands, such as dividing the difference between the two bands by the sum of the two bands, as the composite band. For a hyperspectral image where the image for each band has dimensions of 500 pixels × 500 pixels, the result of generating normalized indices for band 1 and band 2 may be a 500 pixels × 500 pixels 2D image, where each pixel in the result is given by equation (P 帯域1 -P 帯域2 ) / (P 帯域1 +P 帯域2 ) Two pixels P at the same position in the source image according to 帯域1 and P 帯域2 It is calculated by combining these. Of course, this is just one way of combining image data of different bandwidths, and many different functions can be used.

[0011] The composite band, along with other parameters, can be used by the system to amplify or enhance the type of information indicating region boundaries while filtering or reducing the influence of image information that does not indicate region boundaries. This provides an enhanced image to which a segmentation algorithm can be applied. By predefining the band and function for each target region, the segmentation process can be much faster and less computationally expensive than other techniques, such as processing each hyperspectral image using a neural network. Generally, the composite band can combine information about region boundaries distributed across image data of various different bands, allowing the system to extract hyperspectral image components that best signal region boundaries from various bands and combine them into one or more composite images that enable high-accuracy, high-confidence segmentation. Another advantage of this method is that it enables an empirical, data-driven approach to customizing segmentation for different object and region types, while requiring far less training data and training computation than is typically needed to train neural networks and similar models.

[0012] To generate a profile, the system can perform a selection process to identify subsets of wavelength bands in a hyperspectral image that enable more accurate segmentation of different region types. This process may include multiple phases or iterations applied to training examples. A first phase may involve evaluating the image data for each band and selecting a subset of individual bands that most clearly demonstrate the difference between target regions (e.g., the bands that show the highest or most consistent difference between a particular region to be segmented and one or more other region types represented in the training data). A predetermined number of bands, or a subset of bands that meet certain criteria, may be selected for further evaluation in a second phase. The second phase may include generating composite bands based on the application of different functions to the individually selected bands from the first phase. For example, if bands 1 and 2 were selected in phase 1, the system may generate several different candidate bands based on different ways of combining those two bands (e.g., band 1 - band 2, band 1 + band 2, normalized index of bands 1 and 2, etc.). The system then evaluates how clearly and consistently the normalized bands distinguish the target region from other regions and can select a subset of these composite bands (e.g., a predetermined number with the highest scores, or those with region type identification scores above a minimum threshold). The selection process can optionally continue into further phases to evaluate different combinations of composite bands and select from them, with each additional phase selecting a new combination that provides higher accuracy and / or consistency in identifying the target region type.

[0013] In applications of optical sorting and classification of plastics, the system's discriminative power can be significantly improved by generating combined or composite bands of image data and determining which bands should be used to detect different materials. For example, analysis may be performed to determine which bands best identify different base plastic types and to distinguish them from other common materials. Similarly, analysis can be used to select bands that best identify a base type of plastic without additives (e.g., pure PE) from plastics of the same type containing one or more additives (e.g., phthalates, bromides, chlorates, etc.) and to distinguish uncontaminated areas from areas with different types of surface contaminants. Band selection may depend on the type of base plastic that sets the baseline amount of reflectance and variation in a particular spectral region. Thus, different combinations of bands may be selected to identify areas of different additives or contaminants present. In some implementations, band selection may be indicated by the set of materials to be identified and the specific types of additives and contaminants of interest.

[0014] In one general embodiment, a method performed by one or more computers includes: acquiring image data of a hyperspectral image, wherein the image data includes image data for each of a plurality of wavelength bands; accessing stored segmentation profile data for a particular object type, which indicates a predetermined subset of wavelength bands designated for segmenting images of a particular object type; segmenting the image data into a plurality of regions using the predetermined subset of wavelength bands designated in the stored segmentation profile data, for segmenting images of a particular object type; and providing output data indicating the plurality of regions and the respective region types of each of the plurality of regions.

[0015] In some implementations, a given subset of different wavelengths includes different combinations of wavelength bands, each of which includes two or more of the wavelength bands.

[0016] In some implementations, a given subset of wavelengths with different bandwidths includes pairs with different wavelengths.

[0017] In some implementations, the accessed data specifies a combination of two wavelength bands, for at least one of the region types, which represents the difference between the image data for the two wavelength bands divided by the sum of the image data for the two wavelength bands.

[0018] In some implementations, the method involves accessing data indicating one or more operations to be performed on image data for a predetermined subset of wavelength bands corresponding to each of the different domain types, and generating a modified set of image data by performing one or more operations corresponding to each of the different domain types on a predetermined subset of wavelength bands specified for the domain type, and segmenting the image data into multiple domains involves segmenting the domains of the corresponding domain types using the modified set of image data for each domain type.

[0019] In some implementations, providing output data involves providing a set of image data for each region type, where each set of image data isolates a region of the corresponding region type.

[0020] In some implementations, at least one of the image data sets includes image data derived from one or more wavelength bands different from a given subset of the wavelength bands used for region type segmentation.

[0021] In some implementations, the domain type corresponds to different materials.

[0022] In some implementations, the region types correspond to different conditions or objects.

[0023] In some implementations, providing the output data includes providing the output data to a classification system configured to determine a classification of an object represented in a hyperspectral image.

[0024] In some implementations, the method includes using one or more of the segmented regions to determine a classification or a state of an object represented in a hyperspectral image.

[0025] In some implementations, the image data represents one or more waste items, the accessed segmentation profile data specifies a subset of wavelength bands for segmenting regions where a particular type of recyclable material is present, and at least one of the plurality of regions indicates a region where a recyclable material is present.

[0026] In some implementations, the method includes accessing segmentation profile data for a plurality of different plastics, the accessed segmentation profile data indicates different subsets of wavelength bands to use for segmenting the different plastics, segmenting the image data into a plurality of regions includes segmenting the image data into regions representing different plastics, and the regions for the different plastics are segmented using respective subsets of wavelength bands for the different plastics.

[0027] In some implementations, the method includes detecting at least one of a type of plastic, an additive for the plastic, or a contaminant on the plastic, based on segmented image data. For example, segmentation of an area corresponding to an area having a contaminant can indicate the presence of the contaminant. In some implementations, the segmented image data for an area, or a feature value derived from the segmented data for the area, is processed by a machine learning model to detect material properties, such as classifying the material, identifying a particular additive or contaminant, estimating the amount or concentration of a chemical substance present, and the like.

[0028] In some implementations, the method includes controlling a machine based on segmentation performed using segmentation profile data to sort or convey one or more objects described by hyperspectral image data. For example, based on image data of one or more objects segmented as corresponding to an area type of a particular material, an instruction can be provided to the machine to move the one or more objects to an area or container designated for objects of the particular material. As another example, segmented image data can be used to generate an input provided to a trained machine learning model, and the output resulting from the machine learning model can be used to classify or label the one or more objects, and the classification or label can be provided to the machine to cause the machine to manipulate the one or more objects based on the classification or label.

[0029] Other implementations of this embodiment and other embodiments include corresponding systems, devices, and computer programs configured to perform the actions of the method and encoded on computer storage devices. One or more computer systems can be configured in this way by software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions during operation. One or more computer programs can be configured in this way by having instructions that cause the device to perform actions when executed by a data processing device.

[0030] Details of one or more embodiments of the subject matter described herein are given in the accompanying drawings and in the following description. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0031] [Figure 1] This block diagram shows an example of a system implemented to perform image segmentation using hyperspectral images. [Figure 2] This figure illustrates an example of performing enhanced image segmentation using profiles to select different subsets of wavelength bands in a hyperspectral image in order to segment different types of regions of an object. [Figure 3] This figure shows an example of automatically generating and selecting different bandwidths of image data for performing image segmentation. [Figure 4] This flowchart illustrates the process of automatically generating and selecting the wavelength bands used when segmenting an image of an object. [Modes for carrying out the invention]

[0032] Figure 1 is a block diagram of an exemplary system 100 implemented to perform band selection and image segmentation of hyperspectral images. System 100 includes a camera system 110 for capturing hyperspectral images of an object such that each hyperspectral image contains image data for each of several bands, each band representing a measurement of reflected light in a specific wavelength band. Figure 1 further shows an exemplary data flow shown in steps (A) to (E). Steps (A) to (E) may occur in the order shown, or may occur in a different order.

[0033] System 130 can be used to select bandwidths to be used for performing image segmentation for many different applications. For example, the system can be used to select bandwidths to identify and evaluate different types of fruits, vegetables, meats, and other foods. As another example, the system can be used to select bandwidths to identify and evaluate waste materials, such as detecting recyclable material types, as well as detecting the presence and amount or concentration of additives or contaminants.

[0034] In the example in Figure 1, the camera system 110 captures a hyperspectral image 115 of an object 101, which is a strawberry in the figure. Each hyperspectral image 115 contains image data for N bands. Generally, a hyperspectral image can be thought of as having three dimensions x, y, and z, where x and y represent the spatial dimensions of a 2D image for a single band, and z represents an index or step through the number of wavelength bands. Thus, a hyperspectral image contains multiple 2D images, each represented by the x and y spatial dimensions, and each image represents the captured light intensity (e.g., reflectance) of the same scene for different spectral bands of light.

[0035] Most hyperspectral images contain image data for each of several or tens of wavelength bands, depending on the imaging technique. In many applications, it is desirable to reduce the number of bands in a hyperspectral image to a manageable level, primarily because processing images with a large number of bands is computationally expensive, resulting in delays and high power consumption. Many different dimensionality reduction techniques have been proposed in the past, such as principal component analysis (PCA) and pooling. However, these techniques are still computationally expensive, require specialized training, and often do not provide the desired accuracy in applications such as image segmentation. Furthermore, many techniques attempt to use almost all or all bands for segmentation decisions, even though different wavelength bands often have dramatically different information values ​​for segmenting different types of boundaries (e.g., boundaries of different types of regions with different properties such as material, composition, structure, and texture). This has traditionally resulted in the inefficiency of processing image data for more wavelength bands than are required for segmentation analysis. In addition, data from bands with low relevance to the segmentation boundary is noise and obscures key signals in data with slightly relevant data, thus limiting accuracy.

[0036] In particular, the importance of different wavelength bands to segmentation decisions varies significantly depending on the type of region. Of the 20 different wavelength bands, one type of region (e.g., having a specific material or composition) may strongly interact with only a portion of the total band being imaged, while a second type of region (e.g., having a different material or composition) may strongly interact with different subsets of the total band being imaged. Many conventional systems lack the ability to determine, store, and use region-dependent variations in which a subset of bands produces the best segmentation results, which often results in inefficient handling of band data that is only slightly relevant or irrelevant to the segmentation of at least some of the target regions. As described below, the technique described herein allows segmentation parameters for each object type and region type to be determined and stored based on an analysis of training examples, and then used to better identify and distinguish each type of region for a given type of object. This can be done for many different object types, and allows the system to select profiles for different objects or scenes and segment the various region types that may exist for different objects or scenes using appropriate sets of bands and parameters.

[0037] In the example in Figure 1, the camera system 110 includes or is associated with a computer or other device that can communicate via a network 120 with a server system 130 that processes hyperspectral image data and returns segmented images or other data derived from segmented images. In other implementations, the functions of the computer system 130 (e.g., generating profiles, processing hyperspectral image data, performing segmentation, etc.) may be performed locally at the location of the camera system 110. For example, system 100 can be implemented as a standalone unit housing the camera system 110 and the computer system 130.

[0038] Network 120 may include a local area network (LAN), a wide area network (WAN), the internet, or a combination thereof. Network 120 may also include any type of wired and / or wireless network, satellite network, cable network, Wi-Fi network, mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. Network 120 may utilize communication protocols, including packet-based and / or datagram-based protocols such as Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or other types of protocols. Network 120 may further include several devices that facilitate network communication and / or form the hardware infrastructure for the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, or a combination thereof.

[0039] In some implementations, the computer system 130 provides a bandwidth selection and image segmentation module that analyzes images and provides a selected bandwidth configuration and segmented images as output. In some implementations, the computer system 130 may be implemented by a single remote server or by a group of multiple different servers distributed locally or globally. In such implementations, the functions performed by the computer system 130 can be performed by multiple distributed computer systems, and the machine learning model is provided as a software service via the network 120.

[0040] In short, Figure 1 shows an example of how the computer system 130 generates segmentation profiles of object types and / or region types through the analysis of various training examples. The computer system 130 then receives additional hyperspectral images and uses the object type profiles of the objects in the images to efficiently generate accurate segmentation results. Figures 1 and 2 show strawberries as the type of object being detected and evaluated, but the same technique described can be used to process other types of objects.

[0041] During stage (A), as part of the setup process, the computer system 130 generates profiles of the types of objects whose images will be segmented. For example, to enable the system to segment an image of a strawberry, a profile 153 of the strawberry object type can be created. Profile 153 can specify a subset of bandwidth to use when segmenting the strawberry, or more potentially, different bandwidths to use for segmenting different types of regions of the strawberry.

[0042] To generate object type profiles, the computer system 130 processes various training examples 151, which include hyperspectral images of instances of the object type to be profiled. In some implementations, a bandwidth evaluation module 150 performs a bandwidth selection process in which each of the bandwidths of the processed hyperspectral images is analyzed to generate a selected bandwidth configuration that enables high accuracy while performing hyperspectral image segmentation.

[0043] The band evaluation module 150 can perform an iterative process of band selection 152 for object type and / or region type. During the first iteration, individual bands of a hyperspectral image undergo selection process 152a. Process 152a selects a subset of bands from multiple bands of a hyperspectral training image. For example, during the first iteration, module 150 evaluates bands 1 to N of various hyperspectral image training examples 151 and assigns each band a score indicating how well it distinguishes between a particular type of target region (e.g., strawberry flesh) and other regions (e.g., leaves, seeds, background, etc.). In this example, the first iteration of process 152 selects bands 1 and 3 from bands 1 to N.

[0044] In some implementations, after selecting a subset of individual bandwidths, a composite bandwidth or modified bandwidth is generated. A composite bandwidth may be generated by processing image data on one or more of the bandwidths selected in the first iteration. For example, each bandwidth in the subset of bandwidths can undergo one or more operations (e.g., image processing operations, mathematical operations, etc.) which may include operations that combine data from two or more different bandwidths (e.g., the bandwidths selected in the first iteration). Each of various predetermined functions can be applied to the image data for different combinations of the selected bandwidths (e.g., for each pair of bandwidths or each permutation within the selected subset of bandwidths). This can create a new set of composite bandwidths, each representing a different modification or combination of bandwidths to the bandwidths selected in the first iteration. For example, during selection by selection process 152a, module 150 performs operations on bandwidths 1 and 3 to create three new composite bandwidths, including (1) bandwidth 1 + bandwidth 3, (2) bandwidth 1 + bandwidth 3, and (3) bandwidth 1 - bandwidth 3.

[0045] Next, the composite bands thus created are evaluated, for example, scored, to determine the level at which they distinguish between the target region type (e.g., strawberry pulp) and other region types. Then, for a second iteration, the computer system 130 selects from the composite bands in the selection process 152b. In this example, the composite band created as band 1-band 3 is selected by process 152b. The iterative process of generating new modified or composite bands and then selecting the most effective one from among them can continue until the desired level of accuracy is reached.

[0046] In this example, the information for segmenting the strawberry flesh is distilled or aggregated into a single 2D image. However, this is not mandatory, and in some implementations, profile 153 may indicate that multiple separate bands (e.g., original bands or combined / modified bands) should be generated and used for segmentation. For example, the system may specify that segmentation should use image data of three bands: band 1 + band 3, band 1 / band 3, and band 1-band 3.

[0047] The bandwidth evaluation module 150 performs a selection process for each of the multiple region types of the object type from which the profile 153 is being generated. This generates a selected subset of bandwidths to be used for each region type. If the selected bandwidths are composite bandwidths, the component input bandwidths and the functions applied to generate the composite bandwidths are stored in the profile. As a result, the profile 153 for an object type may include the selected bandwidth configurations to be used for each region type of the object, enabling high accuracy for image segmentation for each region type. For example, in the case of a profile 153 for segmenting a strawberry, the region types may be leaves, seeds, and pulp. As another example, in a profile for segmenting elements of a dining room, the multiple region types may include chairs as the first region, a table as the second region, and walls as the third region.

[0048] In some implementations, the region type of profile 153 represents regions of different materials, and as a result, segmentation can easily distinguish between regions of different materials in the image. The bandwidth evaluation module 140 generates bandwidth configurations for each material type to enable high accuracy while performing image segmentation. For example, to evaluate furniture, multiple material types include wood, plastic, and leather. More generally, the selection can determine image bandwidth parameters for any of various properties, including material, composition, texture, density, and structure.

[0049] In some implementations, the bandwidth evaluation module 150 performs a bandwidth selection process 152 for each of several condition types of an object. For example, the flesh of a strawberry may be thought to have different types of regions, such as ripe, unripe, damaged, and powdery mildew. The bandwidth evaluation module 150 may generate bandwidth configurations for each condition type and store them in a profile 153, enabling high-precision distinction between regions of different conditions while performing image segmentation.

[0050] The process for generating the profile described in stage (A) can be performed for many different object types to create a library of segmentation profiles 153, which can be stored and retrieved by system 130 to accurately segment each of the different object types. For each object type, multiple different region types may be specified, each region type having a corresponding wavelength band, operator, algorithm, and other parameters specified for use in segmenting the image region of that region type.

[0051] During stage (B), the camera system 110 captures a hyperspectral image of the object. For example, the camera system 110 takes a hyperspectral image 115 of the strawberry 101, which includes image data for each of N different wavelength bands. In some implementations, the hyperspectral image 115 may be sent as one of many images in a series of images of different objects, such as objects on a conveyor belt for manufacturing, packaging, or quality assurance.

[0052] During stage (C), the hyperspectral image 115 is transmitted from the camera system 110 to the computer system 130, for example, using the network 120. The hyperspectral image 115 may be transmitted in connection with a request to process the image, such as to generate a segmented image, to inspect the characteristics or quality of an object represented in the image, or for other purposes.

[0053] During stage (D), upon receiving the hyperspectral image 115, the computer system 130 performs processing to identify and generate image data required for segmenting different types of regions. This may include a pre-segmentation step to identify the object types represented in the hyperspectral image 115, a step to retrieve object type profiles 153 (e.g., from a database and from profiles of multiple different object types), a step to pre-process image data of different bands (e.g., a step to generate a composite or combined image, a step to apply thresholds, functions, or filters, etc.), a step to reduce the number of images (e.g., a step to project or combine image data from multiple bands into fewer images or a single image), and / or a step to prepare the hyperspectral image 115 for segmentation processing using parameters in the profile 153.

[0054] The computer system 130 identifies the object type of the object 101 represented in the hyperspectral image 115, and then selects and retrieves a segmentation profile 153 corresponding to that image type. The data provided in relation to the hyperspectral image 115 can indicate the type of object 101 represented in the image. For example, a request to process the hyperspectral image 115 may include an instruction that the object being evaluated is of the "strawberry" object type. In another example, system 100 may be configured to repeatedly process hyperspectral images showing the same type of object, and as a result, computer system 130 is already configured to interpret or process incoming hyperspectral images 115 as images of strawberries. This may be the case in a manufacturing facility or packaging workflow where items of the same type are processed sequentially. In yet another example, computer system 130 may use an object recognition model to detect the type of object represented in the hyperspectral image 115, and then automatically select a profile corresponding to the identified object type.

[0055] Once an appropriate profile 153 is selected for the object type of the object depicted in the hyperspectral image 115, the computer system 130 processes the hyperspectral image 115 by applying the information in the selected profile 153. For example, the profile 153 may specify different composite or combined images to be generated from image data of different bands within the hyperspectral image 115. The computer system 130 may generate these images and may apply any other algorithms or operations specified by the profile. As a result, module 140 prepares one or more images to which segmentation processing has been applied. In some cases, this may result in a single 2D image, or different 2D images for each of several different region types to be segmented, or multiple 2D images for each of several different region types. In practice, module 140 can use the profile 153 to act as a preprocessing step, filtering out image data of bands unrelated to a given region type and processing the image data into a suitable format for segmentation.

[0056] During stage (E), the segmentation module 160 performs segmentation based on the processed image data from the image processing module 140. Segmentation can determine the boundaries of different objects and different types of regions of those objects. One way to see the segmentation process is that the module 160 can classify different regions of the image it receives (representing the corresponding hyperspectral image 115) into classes or categories, for example, assigning pixels as one of various types, background, or not part of an object, such as leaves, seeds, or pulp. For example, image data of a selected bandwidth configuration generated by the image processing module 140 can be subjected to any of the following segmentation algorithms: threshold segmentation, clustering segmentation, compression-based segmentation, histogram-based segmentation, edge detection, region extension techniques, partial differential equation-based methods (e.g., curve propagation, parametric methods, level-set methods, fast marching methods, etc.), graph partitioning segmentation, watershed segmentation, model-based segmentation, multiscale segmentation, and multispectral segmentation. The segmentation algorithm may use parameters specified by the profile 153 for each region type (e.g., threshold, weight, criterion, different model or model training state, etc.), and as a result, different region types can be identified using different segmentation parameters or different segmentation algorithms. The results of the segmentation can be represented as an image. One example is an image providing a 2D pixel grid, where pixels are given a value of "1" when they correspond to a particular region type (e.g., leaf) and a value of "0" otherwise.

[0057] In this example, profile 153 specifies three region types: strawberry leaves, seeds, and pulp. Profile 153 specifies these three regions to be segmented, as well as the bandwidth and parameters to be used to identify where these three regions exist. The segmentation module 160 generates output 160 containing three images, one for each of the three different region types. Thus, each image corresponds to a different region type and specifies the region occupied by an instance of a particular region type from the 2D field of view of the hyperspectral image 115. In other words, the segmented images may include an image mask, or otherwise specify the boundaries of the region identified as containing a certain region type. In some cases, regions of different region types may all be specified in a single image using different values ​​that classify different pixels as corresponding to different regions (e.g., 0 for things that are not part of the background or object, 1 for leaves, 2 for seeds, 3 for strawberry pulp, etc.).

[0058] During stage (F), system 130 stores the segmentation results 160 and uses them to generate and provide output. Even if the segmentation boundaries are generated using image data for a subset of bands in the hyperspectral image 115, the determined boundaries can be used to process the image data for each of the bands in the hyperspectral image 115. For example, the hyperspectral image may have image data for 20 different bands, and the segmentation process may use image data only for bands 1 and 2. The resulting determined region boundaries can then be applied to segment or select defined regions in the image data for all 20 different bands. Since the images for different bands of the hyperspectral image 115 share the same view and perspective of object 101, segmentation based on image data for one band can be directly applied (e.g., overlay, project, or otherwise mapped) to the image data for the other bands. In this way, segmentation can be consistently applied across all images in the hyperspectral image 115.

[0059] The segmentation results 160 can be stored in a database or other data storage, associated with the sample identifier of object 101 and the captured hyperspectral image 115. For quality control and manufacturing applications, system 130 can use the association as part of detecting and recording defects, tracking the quality and characteristics of specific objects as they move through a facility, and assisting in sampled analysis of lots or batches of objects. Some common functions that system 130 can perform using segmented hyperspectral image data include characterizing object 101 or specific parts thereof, such as assigning scores for composition, quality, size, shape, texture, or other characteristics. Based on these scores, or potentially as a direct output of image analysis without intermediate scores, computer system 130 can classify objects based on the segmented hyperspectral image data. For example, system 130 can classify objects into categories such as different quality grades and direct the objects to different areas using a conveyor system based on the assigned categories. Similarly, system 130 can detect defective objects and remove them from the manufacturing or packaging pipeline.

[0060] The segmentation results 160, the results of applying segmentation to the hyperspectral image 115, and / or other information generated using them can be provided. In some implementations, one or more images showing the segmented boundaries are sent to the camera system 110 or another computing device for display or further processing. For example, boundaries of different region types determined through segmentation can be overlaid on the region boundaries and specified in annotation data indicating the region type for the hyperspectral image 115 or composite image or standard color (e.g., RGB) image of the object 101.

[0061] In some implementations, the computer system 130 performs further processing on the segmented image, such as generating input feature values ​​from the pixel values ​​of specific segmented regions and providing the input feature values ​​to a machine learning model. For example, the machine learning model may be trained to classify objects such as strawberries based on characteristics such as size, shape, color, consistency of appearance, and absence of defects. The computer system 130 may use the region boundaries determined through segmentation to separate image data in various spectral bands from the hyperspectral image 115 that correspond to individual strawberries and / or specific types of regions of strawberries. Thus, the computer system 130 can provide an input image as input to the trained machine learning model that excludes background elements and other objects and instead provides only regions showing parts of a strawberry, or only regions showing specific parts of a strawberry (e.g., omitting the flesh, seeds, and leaves of the strawberry).

[0062] In some cases, the input provided to a machine learning model can be derived from segmented images without providing the image data itself to the model. Examples include the ratio of the number of pixels classified as one region type to the number of pixels classified as another region type (e.g., the ratio of the number of seed region pixels to the number of pulp region pixels), the average intensity of pixels segmented as a certain region type (e.g., strawberry pulp) (potentially for each of the various spectral bands), and the distribution of the intensity of pixels segmented as a certain region type.

[0063] The machine learning model 170 can be trained to perform a variety of different functions, such as classifying the state of an object or estimating its properties. In the case of strawberries, the machine learning model can use segmented input image data to determine classifications (e.g., good condition, unripe, damaged, powdery mildew, etc.). The machine learning model can also be trained to provide scores or classifications for hyperspectral image-based predictions of specific properties such as chemical composition, intensity, defect type or density, texture, and color. For example, one or more models can be trained to predict the amount or concentration of chemicals present in an object. One example is a model trained to non-destructively predict the sugar concentration in the juice from strawberries, for example, in Brix degrees or mass fraction, based on input features that characterize the hyperspectral imaging data of strawberries. This model can be trained based on examples of hyperspectral images of strawberries and the corresponding sugar content measured, so that these examples show the relationship between reflectance levels in different spectral bands and the sugar content present. Using the same training and modeling techniques, models can be generated to predict the amount or concentration of other chemicals, as well as chemicals in other fruits, foods, and non-food objects.

[0064] The machine learning model 170 may be a neural network, classifier, decision tree, random forest model, support vector machine, or other type of model. The results of the machine learning model that process segmented hyperspectral image data or input features derived from hyperspectral image data may be stored in a database for later use and provided to any of various devices, for example, a client device for display to a user, a conveyor system for orienting object 101 to one of several locations, a tracking system, etc. For example, the results of a machine learning model that classifies object 101 may be used to generate a sorting device 180 (for example, causing the sorting device 180 to physically move or group objects according to the characteristics indicated by the machine learning model results), a packaging device that specifies how and where to package object 101, a robotic arm or other automated manipulator for moving or adjusting object 101, or instructions to be sent to operate object 101.

[0065] The technique shown in Figure 1 can also be applied to evaluate other types of materials, such as plastics and other recyclable materials. For example, these techniques can be used to improve the efficiency and accuracy of characterizing the chemical or material identity of waste materials, making it possible to sort items by material type, presence and amount of additives, presence and amount of contaminants, and other properties determined through computer vision. This analysis can be used to improve both mechanical and chemical recycling processes.

[0066] Mechanical recycling is a viable strategy for recycling plastics, involving crushing, melting, and re-extruding plastic waste. Recycling facilities are often designed to handle streams of highly purified and sorted materials to maintain high levels of material performance in recycled products. However, impurities in the raw materials, including complex formulations with additives, as well as the physical decomposition of the material, reduce the effectiveness of recycling, even immediately after several cycles of mechanical recycling. For example, among plastic materials, polylactic acid (PLA) is a common waste plastic that is often not detected in polyethylene terephthalate (PET) sorting and mechanical recycling operations. Another example is chlorinated compounds such as polyvinyl chloride (PVC), which are unacceptable in both mechanical and chemical recycling operations because they generate corrosive compounds during the recycling process, which limit the value of the hydrocarbon output.

[0067] Mechanical recycling is limited in its applicability to mixed, complex, and contaminated waste flows, partly due to its use of mechanical separation and reforming processes that are insensitive to chemical contaminants and may not be able to alter the chemical structure of waste materials. System 130 can improve the effectiveness of mechanical recycling through improved identification and classification of plastics and other materials, resulting in more accurate sorting of materials and, therefore, higher purity and more valuable recycled raw materials. Furthermore, System 130 can use imaging data to detect the presence and type of additives and contaminants, enabling materials in which these compounds are present to be treated or removed differently.

[0068] Chemical recycling can overcome the limitations of mechanical recycling by breaking down the chemical bonds of waste materials into smaller molecules. For example, in the case of polymer materials, chemical recycling can provide a means of recovering oligomers, monomers, or even basic molecules from plastic waste raw materials. In the case of polymers, the chemical recycling process may include operations to depolymerize and dissociate the chemical composition of complex plastic products so that their by-products can be upcycled into raw materials for new materials. Elements of chemical recycling can enable the material to be repeatedly dissociated into primary raw materials. In this way, chemical recycling can be integrated into an "end-to-end" platform to facilitate the reuse of molecular components of recyclable materials, rather than being limited to a limited number of physical processes by chemical structure and material integrity, as in the case of mechanical recycling. For example, products of chemical recycling may include basic monomers (ethylene, acrylic acid, butyric acid, vinyl, etc.), raw material gases (carbon monoxide, methane, ethane, etc.), or elemental materials (sulfur, carbon, etc.). Instead of being limited to a single group of recycled products, products that can be synthesized from intermediate chemicals that can be produced from the waste by chemical reactions can be identified based on the molecular structure of the input waste material. In this way, the end-to-end platform can manage waste flows by generating chemical reaction schemes that convert waste materials into one or more target products. For example, the end-to-end platform could direct waste raw materials to a chemical recycling facility for the chemical conversion of waste materials into target products.

[0069] The capabilities of system 130 can also improve the effectiveness of chemical recycling. For example, system 130 can capture hyperspectral images of the waste flow on a conveyor belt and detect which materials are present using the technique shown in Figure 1. System 130 can also estimate the amount of different materials present, for example, based on the size and shape of different types of regions identified, as well as the proportion of different materials present. System 130 can also detect which additives and contaminants are present. From this information regarding the composition of the waste flow, system 130 can modify or update chemical processing parameters to change the target product quantity, endpoint, or chemical structure. Some of these parameters may include changes in processing conditions (e.g., residence time, reaction temperature, reaction pressure, or mixing rate and pattern) and the type and concentration of chemical agents used (e.g., including input molecules, output molecules, catalysts, reagents, and solvents). System 130 can store tables, formulas, models, or other data specifying processing parameters for different input types (e.g., different mixtures or conditions of input materials), and can use the stored data to determine instructions to the processing machine to implement the required processing conditions. In this way, System 130 can use the analysis of hyperspectral imaging data of the waste flow to adjust the waste processing parameters so that the chemical recycling processing parameters match the material properties of the waste flow. Monitoring can be performed continuously so that if the mixture of materials in the waste flow changes, System 130 can appropriately change the processing parameters for the incoming mixture of materials.

[0070] As an example of applying the technology shown in Figure 1 to recycling, the camera system 110 can be configured to capture hyperspectral images of waste materials as objects 101 to be imaged. In some implementations, the camera system 110 is configured to capture hyperspectral images of waste materials on a conveyor. The waste materials may include many objects of different types and compositions imaged within a single image. The results of processing the images are used to sort the waste materials and generate instructions for equipment to process the waste materials mechanically or chemically, if optional.

[0071] During stage (A), as part of the setup process, the computer system 130 generates profiles for one or more types of materials of interest. Different profiles may be generated for different plastic types (e.g., PE, PET, PVC, etc.). Information to facilitate the segmentation and detection of additives and contaminants may be included in the base material profile or other profiles. For example, in this application, profile 153 shown in Figure 1 may represent a profile for PET and may show a subset of spectral bands used when segmenting regional PET. Profile 153 may also show different subsets of bands used to segment different types or variations of PET, or to segment regions having different types of additives or contaminants, respectively.

[0072] Using the techniques described above, the computer system 130 generates a profile of a material by processing various training examples 151, which include hyperspectral images of instances of the material to be profiled. The training examples may include examples showing the target material (e.g., PET) identified in the presence of various other different materials, including other waste materials such as other types of plastics. The training examples 151 may include at least several examples of images of the material having regions where additives or contaminants are present, so that the system 130 can learn which bands and properties distinguish the clean or pure regions of the material from the regions where various additives or contaminants are present.

[0073] The Bandwidth Evaluation Module 150 performs a bandwidth selection process in which each band of the processed hyperspectral image is analyzed to generate a selected bandwidth configuration that enables high accuracy in segmenting the desired material from other types of materials, in particular other types of plastics and other waste materials that may be imaged together with the material of interest. As described above, the Bandwidth Evaluation Module 150 can perform an iterative process of bandwidth selection 152 for each of several different material types and / or region types. For region segmentation for different base plastic types, the system 130 can use principal component analysis or support vector machine (SVM) to determine the changes or differences between different pixel groups resulting from identifying materials using different bandwidths. In some cases, the pixel intensities of each type of plastic can be clustered together, and the differences between clusters (e.g., between the mean values ​​of clusters) can be determined. In general, bandwidth selection analysis can attempt to maximize the differences or margins between pixel groups for different materials. For example, to distinguish between PE, PET, PVC, and PP, the band evaluation module 150 can evaluate the difference in pixel intensity in different spectral bands and identify a band that provides the largest and most consistent amount of difference (e.g., margin) between the reflectance intensities of different plastics. For example, the band evaluation module 150 may determine that a first band has a similar average reflectance for each of the four plastic types mentioned above, but a second band shows a larger amount of reflectance difference for at least some of the plastic types. The analysis may be performed pairwise to identify which band is most effective in distinguishing which pair of materials. In various iterations, the band selection module 150 can select the band with the greatest discriminative power (e.g., the highest margin between pixel intensity groupings), and different combinations of these can be made to generate composite bands, which are similarly evaluated until the maximum number of iterations is reached or until the margin meets a minimum threshold of discriminative power.Similarly, this process can be used to determine the bands and synthetic bands that best distinguish plastics with additives or contaminants from pure base plastics.

[0074] As a result of band selection, system 130 may determine, for example, that a first subset of bands provides the best identification of a desired plastic type (e.g., PET) from other plastics or waste materials, and that a second, different subset of bands is most effective for segmenting a certain additive or contaminant (e.g., oil as food residue). This may include generating and evaluating a composite band that combines data from multiple different spectral bands of the original hyperspectral image. This results in a repository of profiles for different materials, each showing the best parameters and band subsets identified by the system to distinguish the target material from other materials it is likely to be near.

[0075] Continuing the application of the technology in Figure 1 to recycling applications, during stage (B), the camera system 110 captures hyperspectral images of the waste flow. For example, the camera system 110 takes hyperspectral images 115 of the waste flow on a conveyor in the process of sorting or other processing. During stage (C), the hyperspectral images 115 are transmitted from the camera system 110 to the computer system 130, for example, using the network 120. The hyperspectral images 115 may be transmitted in connection with a request to process the image, such as to generate segmented images, to identify materials represented in the image 115, to determine the amount of one or more specific materials represented by the image, to evaluate the level or type of contamination of the sample, or for other purposes.

[0076] During step (D), the computer system 130 retrieves profiles of different types of materials to be identified in the hyperspectral image data 115. Based on the information in the profiles, the computer system 130 selects bands and generates composite bands of image data to identify each of the different types of regions to be detected. For example, the profiles may specify different sets of bands for different plastic types and for different additives and contaminants. For example, one set of bands may be used to segment clean PET regions, another set of bands may be used to segment oil-contaminated PET regions, a third set of bands may be used to segment regions of PET with specific additives, and so on.

[0077] During stage (E), the segmentation module 160 performs segmentation based on the processed image data from the image processing module 140. In the case of plastic recycling, the segmented regions may be regions of different plastic types (e.g., PET, PE, PVC, etc.), as well as regions where additives or contamination are present (e.g., PE with oil contamination, PET with UV-resistant additives, etc.). The system 130 can perform further processing to interpret the segmented results, such as counting the number of different items or regions of each type, or determining the area covered by each region type (e.g., as an indicator of the amount and proportion of different materials).

[0078] During stage (F), system 130 stores the segmentation results 160 and other data characterizing the imaged region. System 130 can store metadata that marks specific objects or regions within the imaged region (e.g., waste material on a conveyor) along with the material type determined through the segmentation analysis. It can also store the boundaries of different regions, as well as the area or proportion of different region types. System 130 can then use this information to generate instructions for processing the waste material. For example, system 130 can provide information to a mechanical sorting machine to direct different plastic pieces to different bins, conveyors, or other devices according to the type of plastic detected. As another example, system 130 can identify plastic items that have one or more additives and separate them from plastics that do not have additives. Similarly, system 130 can identify items with at least a minimum amount of contamination (e.g., at least a minimum area where contaminants are present) and remove these items to avoid contamination of the rest of the recycled material. More generally, the system 130 can characterize various properties of waste material, such as assigning a score to any of the various properties of the detected material, using segmented hyperspectral image data. Based on these scores, or as a direct output of image analysis without potentially intermediate scores, the computer system 130 can classify parts of the waste material or classify a set of waste material as a whole.

[0079] As described above, the segmentation results 160, the results of applying segmentation to the hyperspectral image 115, and / or other information generated using them may be provided to other devices for display or further processing. The segmented image may also be used by computer system 130 or another system to generate input for one or more machine learning models. For example, computer system 130 may generate input feature values ​​from the pixel values ​​of a segmented region and provide the input feature values ​​to a machine learning model. For example, a machine learning model may be trained to classify whether an item or set of items is recyclable or not based on the type of plastic and the amount and type of additives and / or contaminants present. Computer system 130 may use the region boundaries determined through segmentation to separate data from the hyperspectral image 115 for regions of different material types, different additives, or different contaminants, respectively. In some cases, information from the hyperspectral image data for a segmented region may indicate further material properties that may be relevant to the classification decision, such as material density, thickness, quality, etc. Therefore, the computer system 130 can provide one or more input images as input to the trained machine learning model, excluding background elements (e.g., a conveyor belt) and other irrelevant objects, and instead providing only the regions relevant to the model's classification decision. The bandwidth of the information provided may differ from that used for segmentation. The spectral bandwidth that best distinguishes one material from another may be entirely different from the spectral bandwidth that exhibits the properties of that material, or the spectral bandwidth that distinguishes different states or changes in that material.

[0080] Segmented hyperspectral image data can be processed by trained machine learning models or other computational methods such as procedural or rule-based models to search for patterns in the signal that relate to material signatures, additive or contaminant signatures, or other information indicating chemical type, composition, morphology, structure, or purity. For materials incorporating multiple different additives, contaminants, or impurities into the main material, such as different forms of recycled PET objects containing various plasticizers often received by recycling facilities, data for multiple region types can be provided. Data for multiple bands of interest can also be provided, including image data for subsets of spectral bands that exclude less useful spectral bands that have similar properties across many forms of recycled raw materials. As an example, data for specific bands of interest can be provided to a classifier implementing an SVM trained to classify materials, with different types of segmented regions marked or otherwise indicated.

[0081] Machine learning models can be used to predict the amount or concentration of different chemical substances, such as additives and contaminants. When analyzing plastics, one or more models can be trained with an appropriate set of training data and sufficient training iterations to predict the mass fraction of each of different additives or contaminants with an accuracy that matches or exceeds the level provided by typical destructive testing. The model can then be provided with input based on hyperspectral data within one or more segmented regions, and as a result, the model produces an output that shows the level of prediction (e.g., inference or estimation) of the content of one or more chemical substances that the model was trained to predict. For example, different models may each provide regression outputs (e.g., numerical values) showing the mass fraction of different additives or contaminants. As another example, a model can generate an estimate of the amount of a chemical substance present (e.g., in grams, moles, or other appropriate units), which can be shown by the amount in the region where the chemical substance is present, spectral characteristics indicating the concentration, and the variation in concentration across the region. As yet another example, a model can classify objects or regions based on the amount or concentration of a chemical substance present (e.g., assigning a first classification to a first concentration range and a second classification to a second concentration range). The type of measurement output in the prediction may be the same type used as the ground truth label during model training.

[0082] Different classifiers can be trained to predict different chemical properties based on information from different spectral bands and different segmented region types. For example, a first classifier can be configured to predict the amount or concentration present of a first contaminant. The input provided to the first classifier can be derived from a region segmented as containing the first contaminant, where only data from a predetermined set of spectral bands (e.g., bands 1, 2, 3, and 4) in that segmented region is used to generate the input to the first classifier. A second classifier can be configured to predict the amount or concentration present of a second contaminant. The input provided to the second classifier can be derived from a region segmented as containing the second contaminant, where only data from a second predetermined set of spectral bands (e.g., bands 1, 3, 5, and 7) in that segmented region is used to generate the input to the second classifier. Thus, different contaminants, additives, and other chemicals can be predicted using models that use specific spectral bands most relevant to the chemical being evaluated and inputs that focus on the most relevant segmented spatial region.

[0083] In some cases, the input provided to a machine learning model can be derived from segmented images without providing the image data itself to the model. Examples include the ratio of the number of pixels classified as one region type to the number of pixels classified as another region type (e.g., the amount of pixels indicating plastic to the amount of pixels indicating non-plastic, the ratio of pixels representing PET to the amount of pixels representing PE, the amount of pixels representing clean PET to the amount of pixels representing contaminated PET), the average intensity of pixels segmented as a certain region type (potentially for each of various spectral bands), and the distribution of the intensity of pixels segmented as a certain region type. Any output of the machine learning model can be used by system 130 to control machines for sorting and processing waste materials. For example, this can be done by labeling items with metadata indicating their classification, or by generating instructions and sending them to sorting devices to operate specific items in a specified manner.

[0084] In some embodiments, the waste material being imaged and analyzed may include, but is not limited to, polymers, plastics, plastic-containing composites, non-plastics, lignocellulosic materials, metals, glass, and / or rare earth materials. Polymer materials and plastic materials may include materials formed by one or more polymerization processes, and may include highly crosslinked polymers and linear polymers. In some cases, the waste material may include additives or contaminants. For example, plastic materials may include, for example, plasticizers, flame retardants, impact modifiers, rheology modifiers, or other additives contained in the waste material 111 to impart desired properties or promote formation properties. In some cases, the waste material may incorporate constituent chemicals or elements that may not be compatible with a wide range of chemical recycling processes, and therefore the characterization data 113 may include information specific to such chemicals. For example, the decomposition of halogen or sulfur-containing polymers may produce corrosive byproducts, which may hinder or impair the chemical recycling of waste materials containing such elements. An example of a waste material containing halogen components is polyvinyl chloride (PVC). For example, the decomposition of PVC can produce chlorine-containing compounds that can act as corrosive byproducts.

[0085] Figure 2 is an exemplary diagram showing region type segmentation based on the band configuration specified by profile 153. Figure 2 provides additional detail following the example in Figure 1, where a strawberry is the object 101 being imaged, and profile 153 has already been defined to specify a subset of bands used to segment different region types within the strawberry. Here, the profile specifies three different composite or combined images to be created for use in segmentation, each labeled as components A, B, and C, each derived from image data of two or more bands of the hyperspectral image 115. Of course, the profile is not required to specify an image that combines multiple bands; instead, it may, in some cases, simply specify the original bands selected from the hyperspectral image 115 from which the image data should be passed to the segmentation module 160. As indicated by profile 153, the image processing module 140 generates various composite images 220a to 220c, which are then used by the segmentation module to generate masks 230a to 230c that show the segmentation results (e.g., regions or boundaries of regions determined to correspond to different region types). The masks can then be applied to parts or all of the image for different bands of the hyperspectral image.

[0086] System 100 utilizes the fact that regions with different compositions, structures, or other properties can have very different responses to light of different wavelengths. In other words, two different types of regions (e.g., seeds versus leaves) can each strongly reflect light in different bands. For example, the first band may be largely absorbed by the first region type (e.g., seeds) but reflected much more strongly by the second region type (e.g., leaves), making it a good band to use when segmenting the second region type (e.g., leaves). In this example, the different reflectivity characteristics captured in the image data showing the intensity values ​​captured for the first band of light tend to reduce or remove the first type of region at least partially, leaving a signal that more strongly corresponds to the second region type. Different bands can show the opposite if the first region type has a higher reflectivity than the second region type. However, in many cases, the level of differential reflectivity between two region types is not as clear as illustrated. In particular, different types of regions may not be effectively distinguishable based on only a single spectral band.

[0087] The bandwidth evaluation module 140 receives the hyperspectral image 115 as input and uses the profile 153 to determine the bandwidth configuration to use for each of three different region types. For example, the bandwidth evaluation module 140 generates three composite images 220a to 220c. The pulp and seeds are most prominently shown in composite image A, generated by (band 1 + band 3) / (band 1 - band 3), the strawberry seeds are most prominently shown in composite image B (band 1 + band 3), and the strawberry leaves are most prominently shown with the bandwidth configuration (band 1 - band 3) / (band 1 / band 5). In each of these, a reference to a bandwidth refers to image data for that bandwidth, and consequently, for example, "band 1 + band 3" represents the sum of image data for both bandwidth 1 and bandwidth 3 (for example, summing the pixel intensity value for each pixel in the bandwidth 1 image with the corresponding pixel intensity value for the image for bandwidth 3). Profile 153 can specify the transformation and aggregation of image data for different bandwidths that best highlight different region types, and more specifically, it can highlight the differences between different region types to make boundaries clearer for segmentation.

[0088] The segmentation module 160 also receives information from the profile 153, such as instructions on which bands or combinations thereof should be used to segment different region types, and what other parameters should be used for different region types (e.g., thresholds, which algorithms or models to use). The segmentation module 160 can perform the segmentation operation and determine masks 230a to 230c for each target region type. Each mask for a region type identifies the region corresponding to that region type (e.g., specifies pixels that are classified as depicting that region type). As seen in mask 230a, the segmentation process can remove seeds, leaves, and background, leaving regions where the strawberry flesh is clearly identified and shown. Next, masks 230a to 230c can each be applied to any or all of the images in the hyperspectral image 115 to generate images 240a to 240c that show segmented images, e.g., variations in intensity values ​​for each band, but restrict the data to image data corresponding to the desired region type.

[0089] As described above, the segmentation results can be used to evaluate object 101, such as determining whether its shape, size, proportion, or other characteristics meet predetermined criteria. The segmented regions can also be provided for analysis, including by machine learning models, to determine other characteristics. For example, by isolating regions corresponding to a particular region type, the system can limit the analytical processes that should be performed using regions of that region type. For instance, an analysis of the chemical composition of a strawberry (e.g., Brix degree or sugar content in other units) can exclude pixels corresponding to seeds, leaves, or background that would distort the results if considered, based on a set of pixels identified as corresponding to the strawberry pulp. For each image data of one or more spectral bands of the hyperspectral image 115, information about pixels corresponding to the strawberry pulp region type is provided to a machine learning model that makes estimates about the strawberry's sugar content or other characteristics. Similarly, segmented data can be used to evaluate other characteristics such as ripeness, overall quality, and expected shelf life.

[0090] The spectral bands used for segmentation may differ from those used for subsequent analysis. For example, segmentation to distinguish between pulp and seeds may use image data in bands 1 and 2. The segmentation results may then be applied to image data in band 3, which shows chemical properties such as sugar content, and band 4, which shows water content. In general, this method allows for segmenting each region type using image data in the band(s) that most accurately distinguish the region type. Analysis of any property of the subject (e.g., presence or concentration of different chemicals, surface features, structural properties, texture, etc.) can benefit from its segmentation, regardless of the set of bands that best provides image data of the property to be evaluated.

[0091] Figure 3 is an exemplary process diagram illustrating an example of iteratively selecting different bandwidth configurations to specify a segmentation profile. As previously mentioned, a hyperspectral image consists of multiple 2D images, each representing the reflectance measured for a different wavelength band. For simplicity, the example in Figure 3 uses a hyperspectral image 301 with image data for only three bands, band 1, band 2, and band 3, although many implementations use more bands. Furthermore, the example in Figure 3 shows an analysis for selecting the bandwidth configuration to use for a single-region type of a single-object type. The same process can be performed for each of multiple region types and for each of various different object types.

[0092] In some implementations, during the first iteration of the selection process, the system can evaluate image data for each individual band within the source hyperspectral image 301 to determine how well a desired region type can be segmented from that band. For example, each image 302-304 of the hyperspectral image 301 has segmentation applied to generate segmentation results 305-307. The scoring module compares the segmentation results based on the images of each band with the segmentation ground truth for the region and generates scores 312-314 representing the performance of the segmentation for a particular band. For example, band 1 image 302 of the hyperspectral image 301 is provided as input to the segmentation module 160. The segmentation module 160 performs segmentation based on the processing of band 1 image 302 and generates segmentation result 305. The scoring module 308 compares segmentation result 305 with ground truth segmentation 310 and generates a score 312 indicating 95% accuracy. The same process can be performed to evaluate each of the other bands in the hyperspectral image 301.

[0093] Next, the system compares scores 312–314 (e.g., segmentation accuracy scores) for each bandwidth and selects the one showing the highest accuracy. For example, a predetermined number of bandwidths can be selected (e.g., n bandwidths with the highest scores), or a threshold can be applied (e.g., selecting bandwidths with accuracy above 80%). The selected bandwidths are then used in a second iteration of the selection process to evaluate potential combinations of bandwidths.

[0094] For the second selection iteration, the system generates new combinations of bands to test. This may involve combining different pairs of bands selected in the first iteration using different functions (e.g., sum, difference, product, quotient, maximum, minimum values ​​of images of different band pairs). This results in a composite or combined image that combines the selected bands with a specific function as a candidate in segmentation. For example, during iteration 1, bands 1 and 3 of hyperspectral image 301 were selected based on these bands having the highest precision scores 312 and 332. In the second iteration, the image data of these two bands are combined in different ways to generate a new composite or aggregated image. The selected bands 1 and 3 are combined to form three new images: (1) band 1 + band 3, (2) band 1 - band 3, and (3) band 1 / band 3. The images obtained from these three new band combinations constitute the image collection 351.

[0095] The second iteration performs the same steps as in iteration 1 on the images in image collection 351, for example, performing segmentation on each image, comparing the segmentation result with ground truth segmentation 310, generating a score for segmentation accuracy, comparing the scores, and finally selecting a subset of images 352-354 from image collection 351 that provides the best accuracy. For example, during iteration 2, each image in image collection 351 undergoes the same segmentation and selection process as described for iteration 1. For example, image 352 (formed by adding image data of band 1 and image data of band 3) is provided as input to segmentation module 160. Segmentation module 160 performs segmentation and generates a segmentation result 355 for this band combination (band 1 + band 3). Scoring module 308 compares the segmentation result 355 with ground truth 310 and generates a score 322 indicating an accuracy of 96%. For each of the other images generated using different operators that combine data from Band 1 and Band 3, the segmentation result and score are determined. The system then selects the bandwidth configuration(s) that provides the best segmentation accuracy.

[0096] The process can be continued for additional iterations as needed, for example, until the maximum number of iterations is reached, until the minimum level of accuracy is reached, until the candidate band combination reaches the maximum number of operations or bands, or until another condition is met, provided that the highest accuracy achieved in each iteration increases by at least a threshold amount. The highest-accuracy bands across various iterations are selected and added to the profiles of the region type and object type being evaluated. This may include specifying a subset of the original bands of the hyperspectral image 301 and / or a specific composite band, for example, a subset of bands combined with a specific operator or function.

[0097] Figure 4 is a flowchart illustrating an example of process 400 for band selection and hyperspectral image segmentation. Process 400 is an iterative process, and for each region of interest, the process is repeated multiple times until the termination criteria are met. During each iteration, process 400 performs hyperspectral image segmentation, selects a set of bands based on the segmentation, combines bands within the set to generate new bands, and generates a new hyperspectral image with the new bands. In short, process 400 involves accessing hyperspectral image data containing multiple wavelength bands. Based on the hyperspectral image data, it generates image data for each of several different combinations of wavelength bands. It performs segmentation on each of the generated image datasets to obtain segmentation results for each of the several different combinations of wavelength bands. It determines a precision metric for each segmentation result for each of the several different combinations of wavelength bands. It selects one of the wavelength band combinations based on the precision metric. It provides an output showing the selected combination of wavelength bands.

[0098] More specifically, hyperspectral image data is acquired that includes multiple images having different wavelength bands (410). As previously mentioned, a hyperspectral image has three dimensions x, y, and z, where x and y represent spatial dimensions and z represents the number of spectral / wavelength bands. In one interpretation, a hyperspectral image includes multiple two-dimensional images, each represented by the spatial dimensions x and y, and each of the two-dimensional images has a different spectral / wavelength band represented by z. Most hyperspectral images have hundreds or possibly thousands of bands, depending on the imaging technique. For example, camera system 110 captures a hyperspectral image 115 containing N images, each of which represents data from a different wavelength band.

[0099] For each of the multiple region types, process 400 performs hyperspectral image segmentation and generates segmentation results (420). As previously mentioned, a hyperspectral image includes multiple images having different wavelength bands. Each image of the hyperspectral image having a particular wavelength band undergoes segmentation. For example, during iteration 1, image 301 includes three images 302, 303, and 304 for different wavelength bands, which are provided as input to the segmentation module 160. Segmenting each image at a specific wavelength generates segmentation results. For example, segmentation of image 302 generates segmentation result 305. Similarly, segmentation of images 303 and 304 generates segmentation results 306 and 307, respectively.

[0100] The segmentation results are compared to ground truth segmentation to generate performance scores (430). For example, scoring module 308 compares the segmentation result 305 of image 302 to ground truth segmentation to generate segmentation accuracy for image 302 having bandwidth 1. Similarly, scoring module 308 compares segmentation results 306 and 307 to segmentation ground truth 310 to generate accuracy 322 and 332.

[0101] The segmentation accuracy of the hyperspectral image in any given band is compared to a desired criterion (440). For example, if the user desires a 99% segmentation accuracy, but the highest segmentation accuracy in a band for a particular iteration is not 99% or higher, process 400 performs a selection process from the bands with the highest accuracy. However, if the highest segmentation accuracy in any of the bands meets the desired criterion, process 400 provides that band as output.

[0102] If the segmentation accuracy of the wavelength band does not meet the desired criteria, process 400 selects several different bands from among those having a specific performance score and uses these different bands to generate a new wavelength band (450). For example, the segmentation of three images 302, 303, and 304 with different bands in iteration 1 produces accuracy scores of 95%, 70%, and 98%, respectively, thereby failing to meet the desired criterion of 99%. Based on their high accuracy, process 450 selects bands 1 and 3 to generate new bands 352, 353, and 354. These three new bands form a new hyperspectral image 351 in iteration 2.

[0103] Various implementations of the systems and technologies described herein may be realized in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general-purpose and coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0104] These computer programs (also known as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly language / machine code. As used herein, the terms “machine-readable medium” and “computer-readable medium” mean any computer program product, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, and include machine-readable mediums that receive machine instructions as machine-readable signals. The term “machine-readable signal” means any signal used to provide machine instructions and / or data to a programmable processor.

[0105] To provide user interaction, the systems and technologies described herein may be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) on which the user can provide input to the computer. Other types of devices may be used similarly to provide user interaction; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0106] The systems and technologies described herein may be implemented in computing systems that include backend components (e.g., as data servers), middleware components (e.g., application servers), or frontend components (e.g., client computers having a graphical user interface or web browser that allows users to interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., communication networks). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), peer-to-peer networks (with ad-hoc or static members), grid computing infrastructure, and the Internet.

[0107] A computing system can include clients and servers. Clients and servers are generally geographically separated from each other and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other.

[0108] Several implementations have been described. Nevertheless, it should be understood that various modifications are possible. For example, the various forms of the flow shown above can be used with steps rearranged, added, or removed. Also, while several applications and methods for providing incentives for media sharing have been described, it should be recognized that numerous other applications are conceived. Therefore, other implementations are within the scope of the following claims.

Claims

1. A method that is performed by one or more computers, Acquiring image data of a hyperspectral image using one or more computers, wherein the image data includes image data for each of a plurality of wavelength bands. Accessing stored segmentation profile data for a particular object type, which indicates a predetermined subset of the wavelength bands designated for segmenting different region types for images of an object of a particular object type, by one or more computers; The process involves one or more computers segmenting the image data into multiple regions using a predetermined subset of the wavelength bands specified in the stored segmentation profile data, wherein the regions of the region type correspond to the type of material, additive, or contaminant, and the image data is segmented into multiple regions. A method comprising providing output data indicating the plurality of regions and the region type of each of the plurality of regions using one or more computers, the method comprising providing a set of image data for each region type, wherein each set of image data isolates the region of the corresponding region type.

2. The method according to claim 1, wherein a predetermined subset of different wavelength bands includes different combinations of the wavelength bands, and each of the different combinations includes two or more of the wavelength bands.

3. The method according to claim 1, wherein the different predetermined subsets of wavelength bands include different pairs of wavelength bands.

4. The method according to claim 1, wherein the accessed data specifies a combination of two wavelength bands, which for at least one of the region types represents the difference between the image data for the two wavelength bands divided by the sum of the image data for the two wavelength bands.

5. For each of the different domain types, access data indicating one or more actions to be performed on image data for a predetermined subset of the wavelength band corresponding to the domain type, For each of the different region types, the process includes generating a modified set of image data by performing one or more operations corresponding to the region type on a predetermined subset of the wavelength band specified for the region type, Segmenting the image data into multiple regions means segmenting the regions of the corresponding region type using the modified set of image data for each region type. The method according to claim 1, including the method described in claim 1.

6. The method according to claim 1, wherein at least one of the sets of image data includes image data derived from one or more wavelength bands different from the predetermined subset of wavelength bands used for segmentation of the region type.

7. The method according to claim 1, which corresponds to materials with different region types.

8. The method according to claim 1, wherein the region type corresponds to different conditions or objects.

9. The method according to claim 1, wherein providing the output data includes providing the output data to a classification system configured to determine the classification of an object represented in the hyperspectral image.

10. The method according to claim 1, comprising determining the classification or state of an object represented in the hyperspectral image using one or more of the segmented regions.

11. It is a system, One or more computers, The system comprises one or more computer-readable media that, when executed by one or more computers, stores instructions that can be operated to cause the system to perform an operation, and the operation is Acquiring image data of a hyperspectral image, wherein the image data includes image data for each of a plurality of wavelength bands. Accessing stored segmentation profile data for a particular object type, which indicates a predetermined subset of the wavelength bands specified for segmenting images of objects of a particular object type into different region types; To segment the different region types, the image data is segmented into multiple regions using a predetermined subset of the wavelength bands specified in the stored segmentation profile data, wherein the regions of the region types correspond to the types of plastics, additives, or contaminants. A system that provides output data indicating the plurality of regions and the region type of each of the plurality of regions, comprising providing a set of image data for each region type, wherein each set of image data separates the region of the corresponding region type.

12. The system according to claim 11, wherein the predetermined subset of different wavelength bands includes different combinations of the wavelength bands, and each of the different combinations includes two or more of the wavelength bands.

13. The system according to claim 11, wherein the different predetermined subsets of wavelength bands include different pairs of wavelength bands.

14. The system according to claim 11, wherein the accessed data specifies a combination of two wavelength bands for at least one of the region types, which represents the difference between the image data for the two wavelength bands divided by the sum of the image data for the two wavelength bands.

15. For each of the different region types, accessing data indicating one or more operations to be performed on image data for a predetermined subset of the wavelength band corresponding to the region type, For each of the different region types, a modified set of image data is generated by performing one or more operations corresponding to the region type on a predetermined subset of the wavelength band specified for the region type. Includes, The system according to claim 11, wherein segmenting the image data into multiple regions includes segmenting the regions of the corresponding region type using the modified set of image data for each region type.

16. The system according to claim 15, wherein at least one of the sets of image data includes image data derived from one or more wavelength bands different from the predetermined subset of wavelength bands used for segmentation of the region type.

17. The system according to claim 11, wherein the region type corresponds to different materials.

18. One or more non-temporary computer-readable media that, when executed by one or more computers, store instructions that can be operated to cause a system to perform an action, wherein the action is Acquiring image data of a hyperspectral image using one or more computers, wherein the image data includes image data for each of a plurality of wavelength bands. Accessing stored segmentation profile data for a particular object type, which indicates a predetermined subset of the wavelength bands designated for segmenting different region types for images of an object of a particular object type, by one or more computers; The process involves one or more computers segmenting the image data into multiple regions using a predetermined subset of the wavelength bands specified in the stored segmentation profile data, wherein the regions of the region type correspond to the type of material, additive, or contaminant, and the image data is segmented into multiple regions. A non-temporary computer-readable medium that provides output data indicating the plurality of regions and the region type of each of the plurality of regions by one or more computers, comprising providing a set of image data for each region type, wherein each set of image data isolates the region of the corresponding region type.

Images