A method for analysing a cultural property

Abstract

A method for analysing a cultural property (1), comprising: capturing with a portable camera (2) a plurality of images of the cultural property successively illuminated in p different spectral bands, storing for each pixel or set of NxM pixels a vector (r) with at least p values, each of said p values being representative of the reflectance of a corresponding portion of the cultural property when illuminated in one of said spectral bands, classifying said vector in at least one class corresponding to a colouring material such as a pigment, dye or ink, used for each set, determining a plurality of colouring materials used in the cultural property.

Description

A method for analysing a cultural propertyTechnical domain

[0001] The present invention concerns the field of analysis of cultural properties.Related art

[0002] Classifying and authentifying cultural properties is a common yet challenging task.

[0003] In the present application, classifying a cultural property may include without limitation authentifying (i.e., classifying as authentic or non-authentic), dating (classifying among several periods), or determining the author or style of the cultural property.

[0004] A multitude of technologies have been developed and employed to ascertain the provenance and authenticity of cultural properties such as paintings. The foundational approach to this endeavor involves the verification of provenance, wherein the history of the painting, inclusive of ownership records, exhibition history, and literary mentions, is meticulously researched. This method requires access to documents or other evidences which are not always available or could be forged.

[0005] More recently, this process has been significantly enhanced through the application of new analysis methods that could be used for automated inspection of cultural properties, and storing a digital fingerprint of each piece into digital databases.

[0006] Infrared Reflectography (IRR) represents a technological advance in the analysis process, allowing for the examination of modifications madeduring the painting's creation process. This technique unveils the artist's unique methods or alterations hidden from the naked eye, providing key insights into the artwork's authenticity.

[0007] X-Ray Fluorescence (XRF) Spectroscopy and Raman Spectroscopy are instrumental techniques employed to identify the molecular composition of materials used in the artwork. XRF analysis aids in determining the artwork's age and verifying if the materials used align with a claimed period of creation by analyzing the elemental composition of the materials. Conversely, Raman spectroscopy, a non-destructive technique, excels in identifying organic materials by analyzing their molecular composition, offering a granular understanding of the substance without necessitating physical samples from the painting. Both approach however require complex equipment and know-how.

[0008] Artificial Intelligence (Al) and Machine Learning (ML) algorithms have emerged as new tools in the authentication process, capable of analyzing paintings for stylistic patterns and brushwork that may be indicative of a specific artist. These technologies leverage vast databases to conduct comparisons, identifying similarities or discrepancies that may influence the authentication verdict. An example of such method is described in US20240037914A1 .

[0009] There is a need for another method for analysing cultural properties that is easy to carry out and reliable for classification of the cultural property.Short disclosure of the invention

[0010] According to the invention, these aims are attained by the object of the attached claims, and notably by a method for analysing a cultural property, comprising the steps of: capturing with a portable camera a plurality of images of the cultural property successively illuminated in p different spectral bands,storing for each set of NxM pixels, where N and M are greater than 1, a vector with at least p values, each of said p values being representative of the reflectance of a corresponding portion of the cultural property when illuminated in one of said spectral bands, classifying said vector in at least one class corresponding to a colouring material (such as a pigment, dye or ink) used for each set, determining a plurality of colouring materials used in the cultural property

[0011] In the present application, the expression "cultural property" designates any object that hold cultural significance, including ancient artifacts, fine arts such as paintings, decorative arts, etc. The present invention is more specifically related to the classification of cultural properties at least partially covered with pigments, dies or inks.

[0012] In the present application, a value is said to be "representative" of the reflectance if it is an approximation of the reflectance.

[0013] The method may be considered as a measurement of physical properties of the cultural property. The method may be used for measuring or determining the composition, age, or other physical properties of the cultural property, etc,.The value representative of the reflectance may be defined as the amount of light that it reflects divided by the amount of light impinging on it.

[0014] This value exists for any given wavelength and is typically determined for a discrete set of wavelengths.

[0015] The vector may store reflectance values for each pixel of the set and for each illumination wavelength, i.e., values representative of the reflectance of portions of the cultural property corresponding to each pixels of said sets when illuminated in each of said spectral bands.

[0016] The number of values in each said vector is preferably at least equal to NxMxp. For example, if the number of pixels in each set is equal to 4 and the number of illumination wavelengths is equal to 15, the number of values in each vector is preferably at least equal to 60.

[0017] In one embodiment, the vector additionally stores, for each pixel of the set, values representative of the reflectance of a corresponding portion under a white illumination, i.e., simultaneously illuminated under a plurality of said wavelengths. The number of values in each said vector is then at least equal to NxMx(p+1). For example, if the number of pixels in each set is equal to 4 and the number of illumination wavelengths is equal to 15 plus the white illumination, the number of values in each vector is preferably at least equal to 64.

[0018] In another embodiment, the number of reflectance values in each vector is downscaled. In one example, the method comprises a step of computing for each wavelength the average reflectance value of all pixels in the NxM set. For example, if the number of pixels in each set is equal to 4 and the number of illumination wavelengths is equal to 16 (including a white illumination), the number of values in each vector could be 16, or any number lower than 64.

[0019] Each vector is then classified in u classes each corresponding to a colouring material used for each set. This amounts to determining a list of u pigments that may have been used to produce the portion of the artwork corresponding to the vector.

[0020] It is important to note that, although the vector may store reflectance values for each pixel in the corresponding set under each illumination wavelength, the classification is applied to the set as a whole. In other words, all pixels within the set are assigned the same u classes. This reduces the amount of memory required to store the class information and minimizes the impact of sensor noise, unwanted reflections and / or other local errors in the classification.

[0021] The classification could be performed with a machine learning system such as a first neural network.

[0022] Since it may be difficult to unambiguously determine the class of a coloring material based solely on the reflectance of an MxN pixel set, a plurality of u possible classes — typically between 2 and 10 — is assigned to each set.

[0023] The selection of the colouring material class actually assigned to a set is preferably carried out in a second step, distinct from the initial classification step. At the end of this step, a single class is assigned to each set if the corresponding portion is painted with a single pigment; multiple classes are assigned if the portion is painted with a mixture of several pigments.

[0024] The selection is based on information — such as the classes — relating to other sets. Indeed, it is unlikely that an artist would switch pigments to paint two regions of the same artwork with the same color, especially if those regions are close together. Therefore, the selection follows a probabilistic approach based on information drawn from multiple sets.

[0025] Again, this probabilistic approach among multiple sets, each comprising multiple pixels, minimizes the impact of sensor noise, unwanted reflections and / or other local errors in the classification.

[0026] The method may thus comprise a step of selecting a subset of v classes among said u classes based on classes corresponding to other sets of NxM pixels. In other words, the method includes a step of selecting v colouring material classes from among the u colouring material classes identified in the previous step, where v is a number greater than or equal to 1 and less than u.

[0027] This selection of a subset of v classes may be performed with a classifier such as a second neural network.

[0028] The u classes corresponding to a plurality of said sets may be simultaneously input to the classifier. In this case, the selection of the v classes assigned to each vector depends on the classes of other sets, such as sets corresponding to neighboring portions or portions with similar classes.

[0029] The method may comprise a step of compensating for the impact of ambient light. The compensation may involve capturing an ambient light image used for this compensation. The reflectance values in each said vector are preferably ambient-light compensated reflectance values.

[0030] The method may comprise : capturing an ambient light image of said cultural property under ambient light illumination only; substracting to said values representative of the reflectance when illuminated in one of said spectral bands a value (rambient) of the reflectance of the same portion in ambient light, so as to determine ambient light compensated values representative of the reflectance.

[0031] The method may comprise a step of normalizing the reflectance values. The reflectance values in each said vector are preferably normalized reflectance values.

[0032] This normalized reflectance in one spectral band may be determined by dividing the amount the light that it reflects when illuminated in one spectral band by the amount of light that a white reference reflects when illuminated under the same conditions.

[0033] The method can be used for classifying the cultural property.

[0034]

[0035] The machine learning system may use an attention mechanism for the classification of each vector, such that the classification of this vector depends more on other vectors that may have an impact on the classification of each vector, such as for example neighbor vectors or vectors of similar colour or luminance, than on other less focused vectors, such as remote vectors.

[0036] The machine learning system may simultaneously output classes corresponding to a plurality of vectors.

[0037] The method may comprise capturing an image of the cultural property under a white illumination.

[0038] For the purposes of this patent application, "white illumination " is defined as a lighting condition wherein the emitted light encompasses a broad spectrum of wavelengths within the visible range, resulting in a perception of white or near-white color by an observer. The spectral composition of white illumination may vary, and it does not necessitate an equal distribution of wavelengths across the visible spectrum. The spectral power distribution of white illumination may exhibit peaks and troughs across the visible spectrum, resulting in deviations from a perfectly flat spectrum. White illumination may exhibit slight deviations from perfect white, allowing for flexibility in colour temperature and rendering properties. The "white illumination" may for example correspond to a 5000K colour temperature.

[0039] This image under white illumination may be used to retrieve at least one additional value, or a set of RGB values in each vector.

[0040] Each said vector comprises at least one value, or a set of values, such as RGB values, representative of the reflectance of the corresponding portion of the cultural property under said white illumination. Those additional values may be used along with the previously indicated p values for the classification of the vector.

[0041] The image of the cultural property under white illumination can be displayed on a display, as an accurate representation of the cultural property as seen by an observer.

[0042] The representation may include the image under white illumination, and additional representations depending on the images captured under other illuminations.

[0043] A colouring material map may be generated that indicates a prediction of the colouring material used for each portion of the cultural property. Since colouring materials are assigned to sets of NxM pixels, and not to individual pixels, the resolution of the colouring material map is lower than the resolution of the pixel map.

[0044] The colouring material map may be displayed.

[0045] The colouring material map may be superimposed over the image under white illumination. The image under white illumination may be faded for this superimposition.

[0046] An image retrieved from the analysis may be superimposed over said image under white illumination. For example, an underlying pattern or painting may be superimposed and made more visible over the image under white illumination.

[0047] The method may comprise: displaying an image of said cultural property under said white illumination; superimposing said map or an image retrieved from said map over said image.

[0048] The method may comprise: capturing a plurality of reference images of items colouredwith known colouring materials; determining at least one reference vector with at least p values from each said plurality of reference images; associating a plurality of said reference vectors with one colouring material.

[0049] A machine learning system used for classifying the vectors may be trained with said reference vectors.

[0050] It is usually not possible to create reference images of all possible combinations of pigments, dies or inks. As pigments, inks and dies are often mixed, the method may comprise: defining additional classes of colouring materials corresponding to mixes of colouring materials; computing reference vectors associated to said additional classes.

[0051] The method may comprise: creating additional classes of colouring materials corresponding to altered, oxidated or aged colouring materials.; computing the reference vectors associated to said additional classes.

[0052] The method may comprise: capturing an image of said cultural property under an infrared illumination.

[0053] Each vector may comprise at least one value representative of the reflectance of the corresponding portion of the cultural property under the infrared illumination.

[0054] The method may comprise a step of revealing a subjacent pattern in said cultural property.

[0055] The subjacent pattern may correspond to patterns in the canvas, or to masked drawings or paintings.

[0056] The method may comprise a step of retrieving an electronic signature representing said map of the cultural property. The electronic signature may be encrypted and stored in a database. The electronic signature this forms a fingerprint of the cultural property and can be used for authentication.

[0057] The method may comprise a step of storing the signature or a hash of the signature in a blockchain.

[0058] The classifying step may comprise determining an epoque of production of said cultural property.

[0059] The epoque may be determined based on the choice of colouring materials.

[0060] The determination of an epoque of production of the cultural property may be linked to the output of the machine learning system trained to recognized colouring materials.

[0061] Alternatively, the determination of an epoque of production of the cultural property may be performed directly by the machine learning system trained to recognize colouring materials.

[0062] The classifying step may comprise determining the author of said cultural property.

[0063] The author may be determined based on the choice of colouring materials more frequently associated with an author.

[0064] The determination of an author of the cultural property may be linked to the output of the machine learning system trained to recognized colouring materials

[0065] Alternatively, the determination of an author of the cultural property may be performed directly by the machine learning system.

[0066] The illuminating may be performed in at least 12, preferably at least 15, spectral bands, but preferably less than 24 spectral bands. This compromise proved to be reliable enough for classifying a vast majority of known colouring materials, and still reasonably small for limiting the storage and computational power requirements.

[0067] The light used for illuminating in at least one of the spectral band may be polarized.

[0068] The cultural property may be illuminated under several time under the same spectral band but different polarizations.Short description of the drawings

[0069] Exemplar embodiments of the invention are disclosed in the description and illustrated by the drawings in which:Figure 1 schematically illustrates a system according to an embodiment of the invention;Figure 2 schematically illustrates various methods that could be considered for determining the reflectance of one object;Figure 3 schematically illustrates a camera and lighting system that could be used for illuminating and capturing images of a cultural property;Figure 4 schematically illustrates the determination of the reflectance of a cultural property successively illuminated in different spectral bands;Figure 5 schematically illustrates the reflectance spectrum of one portion of the image in different spectral bands;Figure 6 schematically illustrates a matrix of vector values;Figure 7 illustrates a machine learning system with input and output values.Examples of embodiments of the present invention

[0070] Figure 1 schematically illustrates a system according to one possible embodiment of the invention. It comprises in this example a digital handheld camera 2, such as a DSLR camera, a mirrorless camera or a smartphone with an integrated camera. The camera comprises or is connected with an illuminating system 20.

[0071] The camera 2 is suitable for capturing images of cultural properties 1, such as paintings, documents, etc. It can preferably generate images in a raw format

[0072] The images captured by the camera 2 may be transmitted to a processing system 3, such as a personal computer, a workstation, etc. The processing system 3 and the camera 2 may also be integrated into a single device.

[0073] The processing system 3 is connected to a remote server 5 or set of servers via a network, such as the Internet 4.

[0074] The method steps of the present invention are mainly performed by software programs stored and executed in the processing system 3 and / or in the remote server 5. Some programs may also be executed by the digital camera 2. The execution of some method steps by the processing system 3 or by the remote server 5 is a matter of design choice, although for some other method steps local execution in the processing system or remote execution within the server 5 may be preferred.

[0075] The analysis of the cultural property according to the invention is based on determining values representative of the reflectance of different surfaces (portions) of a cultural property as a function of wavelength. Different methods can be considered, as shown in Figure 2.

[0076] Diagram a on the left side of Figure 2 illustrates the reflectance of a portion of the image as a continuous function of the illumination wavelength. This plot provides accurate details of material reflectance, but generates a large amount of spectral data that must be processed. In addition, data acquisition is slow because the illumination wavelength must be continuously varied across the spectrum. The high spectral resolution usually has to be compensated for by low spatial resolution, i.e. large areas. This high spectral resolution is usually redundant and not required for accurate classification of the colouring materials used to produce the cultural property.

[0077] Diagram c to the right of Figure 2 shows the reflectance of a portion of the image in three relatively large and strongly overlapping wavelength bands R, G, B. In this case, the spectral resolution is low, but the spatial resolution could be very high, for example corresponding to the pixel resolution of the image sensor of the camera. In addition, the acquisition time could be very fast as the RGB values could be provided directly from the image sensor by taking one single picture.

[0078] Diagram b in the centre of Figure 2 illustrates the compromise chosen for the invention. A spectral signature of the cultural property is determined by capturing images of the cultural property illuminated sequentially in a limited number p of spectral bands, such as between 10 and 25 spectral bands, preferably ranging from the near UV to the near infrared. In one embodiment, the number p of spectral band is at least 12, for example 15, including one or two spectral bands for near UV, and one or two for near infrared, the other bands being used for visible light. The p spectral bands can be relatively narrow with only little overlap between neighbour bands. In other words, the spectral resolution is intermediate, but the spatial resolution can be as high as in scenario c or only slightly reduced.

[0079] The values obtained are only an approximation of the reflectance, an approximation as good as the emitting LEDs are narrow spectrally. The quality of the approximation may be improved with a white reference normalization. Alternatively, or in addition, it might be useful to simulate and correct the deviation between the reflectance measured with the image sensor and the "real" reflectance. In most situation, an approximation of the reflectance is a value sufficient since the machine learning system 9 described below has been trained with values corresponding to the same approximation of the reflectance.

[0080] As the colour emitted by the LEDs changes with temperature, the system may comprise a temperature sensor for measuring ambient temperature close to the emitting LEDs. The reflectance may be adjusted to account for the actual temperature of the LEDs.

[0081] Figure 3 illustrates an example of digital camera 2 and an illumination system that could be used within the invention. The camera 2 comprises a lens. The illuminating system 20 comprises a controller 200 which may be mounted on and triggered by the flash socket or the camera, and which controls the illuminating system.

[0082] A set of LEDs on an annular ring 201 around the lens 203 is controlled by the controller 200 via the control cable 21 to provide the desired amount of light 28 at the desired wavelength and at the desired time. Power for the LEDs may be provided by the camera 2 via the cable 22 22, or by batteries within the controller 20 and within the part of the illuminating system around the lens. A tether cable 22 may be provided between the camera 2 and the annular ring 201 to control the LEDs directly from the camera.

[0083] The controller 200 is adapted to sequentially activate the LEDs in the p different spectral bands during a sequence of image acquisitions by the camera, so as to acquire a series of p+1 images 7i, 72, ..., 1\, ... 7P+7Wof the cultural property illuminated in p different spectral bands, plus one image 7Win white light illumination. The acquisition of each image 1\ is synchronised with the activation of the LEDs in such a way that each image 7i is acquired by the camera during the emission of only one of the spectral bands, with the exception of the image 7Win white light illumination. An additional image 7 ambient of the cultural property illuminated in ambient light only is also captured, and will be used for compensation of ambient light, as will be described.

[0084] The illuminating system 20 further comprises an optional diffuser 202, and one or a plurality of optional linear or circular polarizers 204 for controlling the polarisation of the illumination light and / or of the reflected light. The polarizers may be manually controlled, for example using a control element on the illuminating system. Alternatively, they may be fixed or preferably motorised and controlled by the controller 20.

[0085] The polarisation filtering may be used in some spectral bands, and not used in other spectral bands. For example, visible light may be unpolarised, whereas infrared light could be polarised, for example to remove by polarisation filtering the direct reflection of the infrared light from the visible pigments. It is also possible to take two images of the cultural property with the same illumination but a different polarisation.

[0086] Polarising filters may also be applied to the camera lens to filter reflected light based on its polarisation. Thus, polarisation can be implemented both by the illumination system — affecting the emitted light — and / or by polarising filters on the camera lens — affecting the reflected light.The illumination system 20 may also include a control element, such as a button, for selecting infrared-only illumination. This mode can be used to continuously illuminate the cultural property and observe the resulting image on the camera display.

[0087] The illumination system 20 may also include a nother control element, such as a button, for selecting UV-only illumination. This mode can be used to continuously illuminate the cultural property and observe the resulting image on the camera display.

[0088] The illuminating system 20 further comprises a distance measuring sensor 23, preferably based on time-of-flight measurement, for measuring the distance 27 between the camera 2 and the cultural property 1.

[0089] The distance sensor may be used to know which white reference is to be taken for the normalization.

[0090] The distance sensor could also be used for straightening an image acquired at an angle . A plurality of distance sensors could be used to determine then angle between the sensor plane and the plane of the cultural property.

[0091] The distance sensors could also be used to adjust the illumination power of each band depending on the distance between the distance sensor and the cultural property.

[0092] The distance sensor could also be used to adjust the illumination power of each band depending on the cultural property, to optimize theamount of light reflected in each band. For example, a mainly blue painting would only reflect red light weakly, so that the power of the red LEDs could be increased without saturating the sensor.

[0093] This distance sensor could be used to prevent the illuminating system from triggering when the camera is outside a prescribed distance range.

[0094] Figure 4 schematically illustrates a reflectance measurement method that can be implemented in the context of the invention with the system described above.

[0095] A plurality of images 7i, 72,... 7P+7Wof the same cultural property 1 are taken and stored successively, as previously described. The spectral range of the illumination changes between each image. The image 7Wis taken with white illumination, at least one of the images among 7i to 7Pis taken with infrared illumination, and at least one of the images among 7i to 7Pis taken with ultraviolet illumination. The other images 7i among 7i to 7Pare taken in several successive visible spectral bands. One additional image 7ambient is taken in ambient light and used for compensating the effect of ambient light, as will be described. One additional image of the cultural property may be captured under white illumination.

[0096] Each of the p+2 images 7 comprises the same number of pixels.

[0097] The camera 2 is advantageously mounted on a tripod at a predefined distance from the cultural property, which reduces camera shake between successive images and ensures that the images are captured from the exact same location.

[0098] The p+2 images 7 in the series are transmitted to the computing system 3 and / or to the remote server 5. Pre-processing may be applied toeach image where they may be cropped and where optical aberrations including chromatic aberrations and / or geometrical deformations (e.g. perspective) may be corrected.

[0099] Optionally, sensor noise may be attenuated using existing noise compensation schemes, for example directly by the processor of the camera.

[0100] The computing system and / or the remote server then determine the reflectance of each portion of the cultural property in each illumination, and uses this information to determine the colouring material used.

[0101] The resolution of image sensors in modern digital cameras is often very high, for example higher than 24 megapixels, and can reach 60 megapixels or more. However, for most applications, it is not necessary to analyse the composition of the cultural property at such high resolution. This is because artistic works produced with brushes, for example, usually have a relatively low spatial frequency, and pigments or other colouring materials are typically used on portions of the work that are covered by more than one pixel. It is therefore appropriate to group adjacent pixels into larger sets Ski, so that one set Ski can correspond to one portion (subsurface) of the image covered by NxM pixels, where N and M are greater than or equal to 1, as illustrated in Figure 6. In this example, each set Ski contains 2x2 pixels p. The number of pixels in each set is preferably between 4 and 256.

[0102] One or more reflectance values r are calculated for each set, as will be described. However, as will be seen later, the classification of the image portion corresponding to that set, for the purpose of determining the colouring material used, may depend on the reflectance values of all pixels in this set as well as other sets.

[0103] The reflectance r of a portion of a cultural property 1 is defined as the amount of light that it reflects divided by the amount of light impinging on it. This reflectance r is calculated under each of the p illuminations, i.e., for each image 7i to 7Pof the series. Optionally, it may also be determined for the image 7Wtaken under white illumination.

[0104] In one embodiment, the effect of ambient light is compensated. Preferably, the corresponding reflectance value for each pixel i is obtained by first subtracting the brightness value of the corresponding pixel Pjj in ambient light from the brightness value of the same pixel Pjj illuminated within this spectral band i. The result is an ambient light compensated value rij of the reflectance of each pixel, for each illumination.

[0105] Additionally, or alternatively, this ambient light compensated value of the reflectance may be normalised by dividing it by the difference between the brightness value of the same portion of an image of a white reference 70 and the brightness of the same portion of an image 71 of the same white reference in ambient light or in total darkness.

[0106] The white references images are preferably captured and stored in advance, for example in a laboratory, prior to the capture of the cultural property. The images 70 of the white reference are preferably captured in advance under the same conditions (illuminations, exposure time, aperture, ISO setting, distance from the painting and LEDs temperatures) as the image 7j. The normalization by the image 71 of the white reference in ambient light ensures that the influences of the spatial inhomogeneity of the illumination, the camera sensor response and optical components light transmissions are compensated.

[0107] A (preferably ambient light compensated and / or normalized) reflectance value rij is determined for each pixel Pjj and for each spectral band i. The pixels are grouped in sets of MxN pixels, for example 2X2 pixels, corresponding to different portions {k,l} of the image 7i. For each set ofMxN pixel, a vector r is then formed that includes the reflectance values under each p illumination of all the pixels in the set.

[0108] In one example, each set Ski comprises 4 pixels and an image of the cultural property is captured, compensated and normalized under p=15 illuminations, plus one image under a white illumination. The number MxN of pixels in each set and the number of illuminations could be different. The vector r corresponding to the set Skithus comprises 4x(p+1)= 64 reflectance values.

[0109] This process is repeated for each portion of the image in spectral band i. It produces a grey level reflectance map n where each value indicates the reflectance rki,i of one pixel under illumination i.

[0110] Figure 5 schematically illustrates a reflectance spectrum r 8 of one pixel of the image. This spectrum can be calculated for each pixel of the image, and indicates its reflectance in different spectral bands.

[0111] Figure 6 schematically illustrates a vector rki determined for each pixel of a set Ski and includes the reflectance values of each pixel of the set under each of the p illuminations as well as, preferably, under a white illumination.

[0112] The analysis method further comprises a step of classifying each such vector rki to determine the colouring material that was used for the corresponding portion of the cultural property.

[0113] In one embodiment, the classification is performed by assigning each vector rki to one colouring material.

[0114] A clustering algorithm may be applied to generate clusters associated with known colouring materials. The clustering algorithm may use reference vectors and identify vectors that are closer to each other andthat correspond to one specific colouring material. Examples of clustering algorithms include K-means, hierarchical clustering, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), etc. This generates a number of clusters, where each cluster may be assigned to one material.

[0115] A distance metric is then used to quantify the similarity between a measured vector rki and all the clusters. Possible metrics include Euclidean distance, Manhattan distance, etc. The vector is then assigned to at least one of the identified clusters based on its similarity to the other points in the cluster. The result is a classification of the corresponding portion of the image of the cultural property that indicates the colouring material(s) used for that portion.

[0116] In another embodiment, the classification of vectors rid is performed using a machine learning system 9, such as a first neural network, trained with reference vectors, extracted from reflectance maps of images made with known colouring materials.

[0117] The machine learning system 9 is preferably trained with values representative of the reflectance obtained in the same conditions as the ones used for the capture, i.e., with the same illumination and image sensor.

[0118] The machine learning system 9 is preferably trained with values representative of the (normalized and ambient light corrected) reflectance. Class labels are assigned to the training reflectance values.

[0119] The machine learning system may be trained with noisy reflectance values, and learn how to compensate for noise.

[0120] The trained network classifies each vectors in u classes corresponding to a colouring material used for each set, u being an integer equal or larger thanl, preferably equal or larger than 2.

[0121] This amounts to determining a list of u possible colouring materials, such as pigments, that may have been used to produce the portion of the artwork corresponding to the vector rki and to the associated set Ski.

[0122] The number of classes assigned to each vector u is preferably an integer between 1 and 20, for example 5. For instance, in the case of an NxM set of green-coloured pixels, the classification may identify three classes corresponding to possible yellow pigments and two classes corresponding to possible blue pigments that could have been used to paint this set.

[0123] Another classifier, for example a second neural network, may be used for selecting, among this list of u classes of colouring materials possibly used for each set, the v ones that are the most likely to have been used, v is an integer greater than or equal to 1 and which can be equal or less than u. In the given example, this step may involve selecting the most likely blue pigment and the most likely yellow pigment from among the u previously identified pigments.

[0124] This selection of v classes of colouring materials considers the u classes assigned to other vectors. In other words, the classification of each set Ski depends on the u values of other vectors Smn, such as neighboring sets or sets with similar classes.

[0125] Such a classification of sets considering neighbor sets also reduces the impact of sensor noise, unwanted reflections and / or other local errors in the classification.

[0126] The values corresponding to a plurality of said sets may be simultaneously input to the second neural network.

[0127] It is indeed unlikely that an artist would use different pigments to produce similar colours on a work, particularly on adjacent portions. The number of colourants used in a work is often limited. So, for example, if an artist has used the pigment "cobalt blue" for many other sets of his work, then the probability is high that the same pigment has also been used to produce blue for another set.

[0128] The classification of each set may depend not only on the colour of each portion, but also on the strokes, patterns and lines of the image and the shape or dimensions of the different coloured surfaces. For example, fine, straight lines are often made with pencils or Indian ink, while some pigments are more often associated with larger surfaces. By simultaneously considering several portions of the image, a suitably trained machine learning system can recognise some strokes, patterns, and lines of the cultural property and deduce with greater certainty the pigment used.

[0129] The machine learning system may use an attention mechanism for the classification of each vector, such that the classification of this vector depends on other focused vectors that may have an impact on the classification of each vector, such as for example neighbour vectors or vectors of similar colour or luminance.

[0130] Other indication or metadata related to the cultural property may be used for the classification. Some input may include metadata related to the cultural property. Some inputs may be related to the likely period or author or style of the cultural property. Some inputs may relate to other similar cultural properties, and / or o their classification. Some inputs may be entered as text prompts for the machine learning system.

[0131] Additional colouring materials may be simulated, and additional reference vectors may be calculated and used for the clustering algorithm and / or for training the machine learning system 9. For example, one new colouring material may be defined that correspond to a mix of different colouring materials. Reference vectors assigned to the mix may becalculated based on properties of the reference vectors corresponding to the basic materials.

[0132] New reference vectors may also be calculated by simulating the effect of aging, oxidation or UV altering on known colouring materials. Those new synthetic reference vectors may be associated with the existing colouring material, or a new colouring material may be defined and associated with this vector, such as "altered colouring material x" for example.

[0133] As already mentioned, at least one of the reflectance map n corresponds to the reflectance of the cultural property under one near infrared illumination. The reflectance under an infrared illumination is used for classifying the different colouring materials. It can also be used for revealing a subjacent pattern in the cultural property, i.e., a pattern that cannot be seen under visible light. The subjacent pattern may correspond to hidden painting layers, hidden drawings, or to patterns in the canvas.

[0134] A map of colouring materials,, i.e. a representation of which colouring materials are used in the image of the cultural property, may then be generated and displayed by the computing system 3. The colouring materials may be indicated with colours, text, symbols, or any other suitable indication. It may also be possible to filter the colouring materials, for example by colour, period, etc. The map may be superimposed over an image the cultural property under white illumination.

[0135] The method may further comprise a step of displaying an image of the cultural property under said white illumination. Additional information obtained by the method, such as identification of colouring materials, subjacent patterns, etc, may be superimposed over that image.

[0136] The analysis of the cultural property 1 thus produces a number of other representations of the cultural property, including a plurality of reflectance maps n, and one colouring material map. These maps form aunique representation of the cultural property, similar to a fingerprint or unique electronic signature of the property.

[0137] This electronic signature may be stored and used for later authentication of the cultural property, or to detect any alteration or modification. It is preferably stored in a database on the remote server 7.

[0138] In a preferred embodiment, it is associated in a record with an identification (e.g. author, name) of the cultural property, and is encrypted prior to transmission and storage. In an embodiment, a hash of an electronic signature is computed and stored.

[0139] In an embodiment, the electronic signature, or a hash of the electronic signature, is stored in a blockchain as an immutable repository.

[0140] The method could be used to verify the authenticity of the cultural property, by comparing its electronic signature with a previously stored reference electronic signature.

[0141] The method could be used to verify the authenticity of the cultural property, by comparing the list of colouring materials with a list of possible colouring materials that could have been used or could not have been used.

[0142] The method could be used for determining an epoque of production of the cultural property, depending on the list of colouring materials used for the production.

[0143] The method could be used for determining the likely author of the cultural property, depending on the list of colouring materials used for the production.

[0144] The method and computer program product as described and claimed may also be used for forensic classification of substances such as urine, blood, explosives, gunpowder, and similar materials.Additional Features and Terminology

[0145] As used herein, the term "processing system," in addition to having its ordinary meaning, can refer to a device or set of interconnected devices that may process executable instructions to perform operations or may be configured after manufacturing to perform different operations responsive to processing the same inputs to the component.

[0146] Many other variations than those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain acts, events, or functions of any of the algorithms described herein can be performed in a different sequence, can be added, merged, or left out altogether (for example, not all described acts or events are necessary for the practice of the algorithms). Moreover, in certain embodiments, acts or events can be performed concurrently, for instance, through multithreaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially. In addition, different tasks or processes can be performed by different machines or computing systems that can function together.

[0147] Unless otherwise specified, the various illustrative logical blocks, modules, and algorithm steps described herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. The described functionality can be implemented in varying ways for eachparticular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosure.

[0148] Unless otherwise specified, the steps of a method, process, or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module stored in one or more memory devices and executed by one or more processors, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of non-transitory computer-readable storage medium, media, or physical computer storage known in the art. An example storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The storage medium can be volatile or non-volatile. The processor and the storage medium can reside in an ASIC.

[0149] Conditional language used herein, such as, among others, "can," "might," "may," "e.g.," and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements or states. Thus, such conditional language is not generally intended to imply that features, elements or states are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements or states are included or are to be performed in any particular embodiment. The terms "comprising," "including," "having," and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Further, the term "each," as used herein,in addition to having its ordinary meaning, can mean any subset of a set of elements to which the term "each" is applied.

Claims

Claims1. A method for analysing a cultural property (1), comprising: capturing with a portable camera (2) a plurality of images (7) of the cultural property successively illuminated in p different spectral bands, storing for each set of NxM pixels, where N and M are greater than 1, a vector (r) with at least p values (H), each of said p values (nJ being representative of the reflectance of a corresponding portion of the cultural property when illuminated in one of said spectral bands, classifying said vector (r) in at least one class corresponding to a colouring material used for each set, determining a plurality of colouring materials used in the cultural property.

2. The method of claim 1, wherein the number of values in each said vector is at least equal to NxMxp.

3. The method of one of the claims 1 to 2, wherein said step of classifying said vectors is performed with a machine learning system (9) such as a first neural network.

4. The method of one of the claims 1 to 3, wherein said step of classifying said vectors comprises classifying said vectors in u classes, u being an integer equal or larger than 2.

5. The method of claim 4, comprising a step of selecting a subset of v classes among said u classes based on other vectors corresponding to other sets of NxM pixels.

6. The method of claim 5, wherein said step of selecting a subset of v classes is performed with a classifier such as a second neural network.

7. The method of claim 3, wherein the p values corresponding to a plurality of said sets (s) are simultaneously input to said classifier.

8. The method of one of the claims 1 to 7, wherein the selection of the v classes assigned to one vector depend on classes assigned to other sets such as neighboring sets.

9. The method of one of the claims 1 to 8, comprising : capturing an ambient light image (7ambient) of said cultural property under ambient light illumination only; substracting to each said value representative of the reflectance when illuminated in one of said spectral bands a value (rambient) of the reflectance of the same portion in ambient light, so as to determine ambient light compensated values representative of the reflectance.

10. The method of one of the claims 1 to 9, said reflectance in one spectral band being normalized by dividing the amount the light that it reflects when illuminated in one spectral band by the amount of light that a white reference image (70) reflects when illuminated under the same conditions.11.The method of one of the preceding claims, comprising: capturing an image (7W) of said cultural property (1) under a white illumination.

12. The method of claim 11, wherein each said vector (r) comprises at least one value (rw) representative of the reflectance of the corresponding portion of the cultural property under said white illumination.

13. The method of one of the claims 11 to 12, comprising: displaying an image (7W) of said cultural property under said white illumination; superimposing said map or an image retrieved from said map over said image.

14. The method of one of the claim 1 to 13, comprising: capturing a plurality of reference images of items coloured with known colouring materials; determining at least one reference vector with at least p values from each said plurality of reference images; assigning a plurality of said reference vectors to each class.

15. The method of claims 3 and 14, wherein said machine learning system is trained with said reference vectors.

16. The method of claim 14, comprising: creating additional classes corresponding to mixes of colouring materials; computing the reference vectors associated to said additional classes.

17. The method of one of the claims 14 to 16, comprising: creating additional classes corresponding to altered, oxidated or aged colouring materials.; computing the reference vectors associated to said additional classes.

18. The method of one of the claims 1 to 17, comprising: capturing an image of said cultural property (1) under an infrared illumination, wherein each said vector (r) comprises at least one value representative of the reflectance of the corresponding portion of the cultural property under said infrared illumination.

19. The method of claim 18, comprising a step of revealing a subjacent pattern in said cultural property.

20. The method of one of the claims 1 to 19, comprising a step of computing an electronic signature representing said map of said cultural property, said electronic signature being encrypted and stored in a database and / or in a blockchain.

21. The method of one of the claims 1 to 19, wherein the step of illuminating is performed in at least 12, preferably 15, spectral bands plus one white illumination.

22. The method of one of the claims 1 to 20, wherein the light used for illuminating in at least one of the spectral band is polarized.

23. A computer product storing a computer program arranged for causing a computing system to perform the method of one of the preceding claims.

Images