Method for classifying highly resolved multidimensional data

Abstract

A computer-implemented method for classifying data using a machine learning model, comprising the steps of: Receiving (101) a set of multidimensional data measured by means of sensors; Determining (102, 103, 104) several different subsampled representations of the received data, each representation having a resolution lower than the received data in at least one dimension; Applying (105) a first machine learning engine to said subsampled representations so as to convert them into a reduced feature space; Applying at least one other machine learning engine to said features from among a segmentation engine (107) or a semantic detection engine (106), so as to segment the multidimensional data into several classes of objects and / or to detect the presence of a given object in the multidimensional data. Figure 1

Description

Title of the invention: Method for classifying highly resolved multidimensional data

[0001] The invention relates to the field of automatic classification or segmentation of large multidimensional data. The field of the invention also includes artificial intelligence methods based on one or more machine learning engines, for example artificial neural networks, for performing automatic classification or segmentation of large multidimensional data.

[0002] Multidimensional data include, for example, multispectral or hyperspectral images with high spatial resolution and a large number of wavelength channels. They may also include three-dimensional images or three-dimensional voxel models. More generally, multidimensional data may include more than three dimensions; for example, they may consist of video sequences comprising two or three spatial dimensions and a temporal dimension.

[0003] When dealing with multidimensional data that has at least three dimensions or more, and where each dimension is highly resolved, the size of the data to be processed and stored quickly becomes very large. To perform automatic classification or segmentation of such data, a problem arises concerning computing power and memory storage, and more generally, the hardware limitations of the computers used to implement this type of algorithm.

[0004] There is therefore a problem to be solved in order to find solutions enabling the processing of highly resolved multidimensional data with computers having limited capabilities and resources to perform classification and / or segmentation.

[0005] A known approach to applying automatic classification methods to high-dimensional multidimensional data involves preprocessing the data by means of principal component analysis to reduce the data size in at least one dimension. Machine learning methods such as neural networks, random forests, or support vector machines are then applied to the compressed data.

[0006] For example, in the case of hyperspectral images with a large number of wavelength channels (typically several hundred), a principal component analysis can be applied to reduce the number of channels to about 3 to 5 channels.

[0007] Similarly, an equivalent treatment can be applied to reduce a dimension temporal, for a video sequence for example by selecting certain keyframes in the temporal sequence in order to decrease the temporal resolution.

[0008] Such an approach works correctly for problems aimed at classifying objects belonging to a small number of classes, typically two or three known classes. However, when it comes to separating objects of a known element from unknown objects, it is not possible to ensure that, among the unknown objects, some will not have the same spectral response as the known object in the few selected channels. Conversely, the probability that an unknown object will have the same spectral response as a known object in a large number of channels is much lower. There is therefore an advantage to retaining all the channels of the hyperspectral data to perform the classification.

[0009] Furthermore, there is also an advantage to carrying out the classification from a highly spatially resolved image, or even coupled with a depth map, which makes it possible to identify precise geometric features to achieve efficient segmentation.

[0010] Similarly, there is an interest in maintaining high temporal resolution for video sequences because selecting keyframes to reduce temporal resolution is a difficult problem to solve.

[0011] The invention proposes a new classification method that allows for the processing of raw multidimensional data by integrating several subsamples of this data, the parameters of which may or may not be learned automatically. The invention thus makes it possible to limit the computational complexity of the processing while maintaining a priori several subsampling options that allow for optimal adaptation to each type of object to be classified or segmented in the initial data.

[0012] Unlike traditional approaches based on an a priori reduction of data dimensions through principal component analysis, the invention makes it possible to improve classification performance for multiple objects by not eliminating, a priori, a significant part of the resolution of the initial data while limiting the computational complexity through the use of undersampled data.

[0013] The invention relates to a computer-implemented method for classifying data using a machine learning model, comprising the steps of: - Receive a set of multidimensional data measured using sensors, - Determine several different subsampled representations of the received data, each representation having a resolution lower than data received on at least one dimension, - Apply a first machine learning engine to said subsampled representations in order to convert them into a reduced feature space, - Apply at least one other machine learning engine to said features from among a segmentation engine or a semantic detection engine, so as to segment the multidimensional data into several classes of objects and / or to detect the presence of a given object in the multidimensional data.

[0014] According to a particular aspect of the invention, the multidimensional data are of one of the following types taken alone or in combination with another type: multispectral or hyperspectral images, three-dimensional images, video sequences in two or three spatial dimensions, measurements of temperature, pressure, humidity.

[0015] In a particular embodiment, the method according to the invention further comprises a machine learning engine configured to learn subsampling functions enabling the transformation of the received data set into several subsampled representations.

[0016] In a particular embodiment, the method according to the invention further includes a classification step based on probabilistic modeling to generate a probability of belonging to a class.

[0017] In one embodiment of the invention, the machine learning model is trained in a supervised manner and the method further includes selecting, from among all the subsampled representations of the multidimensional data, a subset of the representations that contribute most to the classification task.

[0018] In one embodiment of the invention, at least one of the machine learning engines is a convolutional neural network.

[0019] In one embodiment of the invention, the machine learning model is trained using segmentation masks associated with multidimensional training data, the segmentation masks being capable of segmenting the data into different classes.

[0020] In one embodiment of the invention, the segmentation engine is a convolutional neural network (202) configured to receive as input the features produced as output of the first machine learning engine and to deliver as output a multidimensional segmentation mask of dimensions identical to the received data, capable of classifying different objects contained in the received data.

[0021] In one embodiment of the invention, the detection engine is a convolutional neural network configured to receive as input the features produced as output of the first machine learning engine and to deliver as output semantic information of the presence, in the received data, of an object belonging to a given class.

[0022] In one embodiment of the invention, the multidimensional data comprise at least three different dimensions.

[0023] The invention also relates to a computer program comprising code instructions for implementing the method according to the invention when said program is executed on a computer and a computer-readable recording medium on which the computer program according to the invention is recorded.

[0024] Other features and advantages of the present invention will become more apparent from the following description in relation to the following accompanying drawings.

[0025] [Fig. 1] represents a flowchart detailing the steps for implementing a classification method according to an embodiment of the invention,

[0026] [Fig.2] represents a diagram of an example of an artificial neural network configured to implement the invention,

[0027] [Fig.3] represents a first example of object classification obtained by performing the invention,

[0028] [Fig.4] represents a second example of object classification obtained by carrying out the invention.

[0029] Fig. 1 illustrates, on a flowchart, the steps for implementing a method of classifying objects according to an embodiment of the invention.

[0030] The first step consists of acquiring multidimensional data. This data preferably has at least three dimensions. It may consist of hyperspectral or multispectral images with two spatial dimensions and one dimension corresponding to a number of spectral wavelengths. It may also consist of three-dimensional data, for example, 2D images accompanied by depth information. The multidimensional data may also be voxels characterizing a three-dimensional scene.

[0031] The imaging data can be supplemented by one or more additional dimensions corresponding to measurements from sensors such as temperature, pressure, and humidity measurements.

[0032] The data considered may also be video sequences, in which case a temporal dimension is added to the image data.

[0033] This may also refer to medical imaging data for which an additional dimension corresponds to a spatial dimension, according to which several MRI imaging acquisitions are performed in order to produce several cross-sectional images of a volume.

[0034] Generally, any data obtained from sensors and comprising several dimensions (preferably at least three) can be used as input to the method according to the invention insofar as there is a problem of detection, segmentation and / or classification of objects within this data.

[0035] A general objective of the method is to detect and / or categorize objects from the information captured by the measured data. For example, if the data are image data, the objects to be categorized are objects identified in the captured scene.

[0036] Another objective is to limit the size of the data to be processed by the classification algorithm while retaining the maximum amount of information present in the content of the data so as to allow classification of any type of object and not only of specific predefined objects.

[0037] Multidimensional data have dimensions exceeding several hundred elements. For example, they may be spectral images of size 512 by 512 pixels with 200 channels per pixel or 3D voxels of dimensions 1000 by 1000 by 1000 or even video sequences of size 1920x1090x100 images.

[0038] For this purpose, several subsampling steps 102,103,104 are applied to the acquired multidimensional data 101. Although three steps are shown in [Fig.1], the invention is not limited to this number and may include a higher number of subsampling steps.

[0039] Each subsampling step 102,103,104 is intended to subsample at least one dimension of the data with a predefined subsampling factor.

[0040] One advantage of using several subsamplers applied independently to each dimension is that it allows for a higher resolution to be maintained for some of the dimensions in the output of each subsampling step. In this way, the semantic content of the data can be exploited at full resolution at least in the output of one subsampling step, but the overall size of the subsampled data is reduced compared to the original data.

[0041] Considering as input hyperspectral images of dimensions N,P,Q, that is to say images of size N by P having Q channels of wavelengths, examples of possible subsampling are {N / 8,P / 8,Q}, {N / 4,P / 4,Q / 2}, {N / 2,P / 2,Q / 4}, {N,P,Q / 8}.

[0042] For example, for a spectral image of dimensions 512x512x200, an example of possible subsamplings consists of generating images of dimensions 512x512x25, 256x256x50, 128x128x100, 64x64x200.

[0043] Thus, advantageously, each dimension can be affected by a subsampling factor equal to a power of 2.

[0044] In this example, the input data volume is reduced from 512x512x200 to 512x512x200x(1 / 8 + 1 / 16 + 1 / 32 + 1 / 64). In this example, the volume of data to be processed is reduced, but the automatic learning classification method has access to the full spatial and spectral resolution according to the subsampler outputs.

[0045] The number of subsampling operations and the associated subsampling factors are predefined parameters of the invention.

[0046] In one embodiment of the invention, the subsampling functions are predefined; they correspond, for example, to common geometric functions such as means or median functions on a sliding window.

[0047] In another embodiment of the invention, the subsampling functions are performed by a convolutional neural network and are learned via training of this network.

[0048] At the output of steps 102, 103, 104, several subsampled versions of the acquired data are obtained.

[0049] Next, a processing step 105 is applied to convert the subsampled data into a relevant representation space. For this, step 105 can be carried out using a convolutional neural network trained to extract relevant features in a reduced-dimensional space according to an intermediate representation.

[0050] The intermediate representation provided as output from step 105 is then exploited via a semantic detection step 106 or a segmentation step 107, each of these two steps being able to be executed alone or in addition to the other.

[0051] The segmentation step 107 can be carried out using a convolutional neural network which takes as input the characteristics of the data in the intermediate representation and provides as output multidimensional data of identical dimensions to the acquired data 101 and which represent a data segmentation mask in which certain objects belonging to classes are identified.

[0052] The semantic detection step 106 can also be performed using a convolutional neural network that takes as input the characteristics of the data in the intermediate representation and provides as output binary data in the form of a segmentation indicator that indicates whether an object of a given category is present in the acquired data. For example, this indicator is a vector whose component indicates, by a binary value, whether an object belonging to a class is present in the acquired data.

[0053] Fig. 2 schematically represents an example of implementation of the method of Fig. 1 in the form of an artificial neural network.

[0054] In this example, the artificial neural network consists of a first neural network 201 configured to execute processing step 105, which takes as input the undersampled versions of the initial data and produces as output features in an intermediate space. In the example in [Fig. 2], the neural network 201 receives four inputs corresponding to four undersampled versions of the data with different undersampling rates.

[0055] A second neural network 202 takes as input the outputs of the first network 201 and provides a full-resolution output, i.e. having the same resolution as the initial data on all dimensions and corresponding to a classification mask.

[0056] A third neural network 203 takes as input the outputs of the first network 201 and provides an output corresponding to an indicator of the presence of an object of one or more given classes in the input data.

[0057] Each of the networks 202,203 can be used alone or the two networks can be used jointly.

[0058] In an alternative embodiment not shown in [Fig.2], the subsampling functions for obtaining the subsampled data according to four different subsampling rates as input to network 201 can be learned and executed by means of another artificial neural network which takes as input the fully resolved data acquired in step 101 and provides as output the subsampled data.

[0059] The training of the subsampling neural network can be done independently by configuring its cost function so as to minimize the information contained in the input data and lost by subsampling.

[0060] Alternatively, the entire neural network described in [Fig.2], including the subsampler network, can be fully learned in a supervised manner to perform the desired segmentation or classification task.

[0061] In one embodiment of the invention, during the training of the neural network of [Fig.2], the inputs of network 201 that contribute most to the outputs of networks 202 or 203 are identified for a type of data corresponding to the training data and a type of objects to be classified in this data. If some inputs do not contribute or contribute very little to the outputs, in an alternative embodiment, when using the neural network to perform the intended classification task, these inputs are removed so as to reduce the computational complexity.

[0062] Thus, an advantage of the invention is that it makes it possible to identify which subsampled data, and in particular according to which dimension, contribute best to the classification of a given object. Indeed, different semantic information can be contained in the different dimensions of the data.

[0063] For example, in some images, it can be observed that semantic information will always be present when only one spatial dimension is subsampled.

[0064] In another example, certain objects can be better identified in certain spectral channels of spectral images.

[0065] Without departing from the scope of the invention, the neural networks described in [Fig.2] can be replaced by other artificial intelligence or machine learning engines such as support vector machines or random forest algorithms.

[0066] In one embodiment, a particular classification algorithm can be implemented in the last layer of networks 202 and 203 to perform probabilistic modeling of the data and provide a probability of belonging to a class of objects.

[0067] A possible probabilistic modeling classification method is described in the scientific publication Li, C.-L. et al, “CutPaste: self supervised learning for anomaly detection and localization”, 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition.

[0068] The representation provided in [Fig. 2] shows a neural network architecture that takes the form of a grid, with communication between neurons in the network at the same resolution and at different resolutions (lower or higher). This can be seen in the representation of network 201.

[0069] This architecture can be modified and parameterized after learning by adding residual short-circuit type connections or by removing certain parts of the connections that do not contribute to the task we wish to solve in order to save on memory used and computation time.

[0070] Figure 3 illustrates, by way of example, a classification result obtained by means of the invention. The data considered for the example are 2D images. Figure 3 shows an image 301 of a table on which several objects are arranged, including batteries 303, a black plate 304 and a key 305.

[0071] The image 302 on the right represents the output of the neural network 202 which provides a segmentation mask of the same dimensions as the original image 301 and in which the object classes "stack", "black plate" and "key" are identified and spatially located.

[0072] Figure 4 illustrates another application of the invention relating to the detection of defects in tissues. The two images 401 and 402 represent two different tissue images. Each of these images contains a defect that is identified on the corresponding segmentation masks 411 and 412.

[0073] Images 403, 413, 404, 414 correspond to zooms on the area of ​​interest respectively of the original images 401, 411, 402, 412.

[0074] Unlike state-of-the-art methods, the invention makes it possible, through the use of different subsampled versions of the original multidimensional data, to retain the semantic information of the data while limiting the computational complexity.

[0075] Generally, the invention can be implemented using hardware and / or software components. The software components may be available as a computer program product on a computer-readable medium, which may be electronic, magnetic, optical, or electromagnetic. The hardware components may be available in whole or in part, including, for example, as dedicated integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs) and / or as neural circuits according to the invention, or as a digital signal processor (DSP) and / or as a graphics processing unit (GPU), and / or as a microcontroller, and / or as a general-purpose processor.

[0076] The data produced as input to the method according to the invention can be acquired by an image sensor, a camera, a photographic device, a satellite imager or any other image capture device or more generally any sensor enabling the measurement of physical data.

Claims

Claims

1. A computer-implemented method for classifying data using a machine learning model, comprising the steps of: - Receiving (101) a set of multidimensional data measured by means of sensors, - Determining (102,103,104) several different sub-sampled representations of the received data, each representation having a lower resolution than the received data on at least one dimension, each representation being obtained by applying a sub-sampling rate different from the other representations for at least one non-spatial dimension, - Applying (105) a first machine learning engine to said sub-sampled representations so as to convert them into a reduced-size feature space, - Applying at least one other machine learning engine to said features from a segmentation engine (107) or a semantic detection engine (106),in order to segment multidimensional data into several object classes and / or to detect the presence of a given object in the multidimensional data.,

2. Data classification method according to claim 1 in which the multidimensional data are of one of the following types taken alone or in combination with another type: multispectral or hyperspectral images, three-dimensional images, video sequences in two or three spatial dimensions, temperature, pressure, hygrometry measurements.

3. A method of classifying data according to any preceding claim further comprising a machine learning engine configured to learn subsampling functions for transforming the received data set into multiple subsampled representations.

4. A method of classifying data according to any preceding claim further comprising a classification step based on probabilistic modeling to generate a probability of belonging to a class.

5. A method of classifying data according to any preceding claim wherein the machine learning model is trained in a supervised manner and the method further comprises selecting, from among all sub-sampled representations of the multidimensional data, a subset of the representations that contribute most to the classification task.

6. A method of classifying data according to any one of the preceding claims wherein at least one of the machine learning engines is a convolutional neural network (201,202,203).

7. A method of classifying data according to any one of the preceding claims, wherein the machine learning model is trained using segmentation masks associated with multidimensional training data, the segmentation masks being capable of segmenting the data into different classes.

8. A method of classifying data according to any one of the preceding claims, wherein the segmentation engine is a convolutional neural network (202) configured to receive as input the characteristics produced as output from the first machine learning engine and to deliver as output a multidimensional segmentation mask of dimensions identical to the received data, capable of classifying different objects contained in the received data.

9. Method for classifying data according to any one of the preceding claims, in which the detection engine is a convolutional neural network (203) configured to receive as input the characteristics produced as output from the first machine learning engine and to deliver as output semantic information on the presence, in the received data, of an object belonging to a given class.

10. A method of classifying data according to any preceding claim wherein the multidimensional data comprises at least three different dimensions.

11. A computer program comprising code instructions for implementing the method according to any one of the preceding claims when said program is executed on a computer.

12. A computer-readable recording medium on which the computer program according to claim 11 is recorded.

Images