Pigment classification method and related device

Abstract

The invention provides a pigment classification method and a related device, and the method comprises the steps: carrying out the decompression operation of an initial spectral image of a target picture collected by a spectrum collection device, obtaining a first spectral image, carrying out the correction operation of the first spectral image based on the aging degree of the target picture, obtaining a second spectral image, and carrying out the classification of pigment. Determining a difference image of the first spectral image and the second spectral image, performing splicing operation on the difference image and the first spectral image to obtain a target spectral image, and performing classification operation on the pigments in the target image based on the target spectral image to realize classification of the pigments. Besides, the first spectral image is corrected based on the aging degree of the target picture, the influence of physical characteristics such as aging and air drying of the pigment on pigment classification is considered, and the pigment classification precision is further improved. In addition, the hardware parameters and the software parameters are subjected to joint optimization operation and then local optimization operation, and the final pigment classification precision is improved.

Description

Technical Field

[0001] The present application relates to the field of pigment classification, and more specifically, to a pigment classification method and related devices. Background Art

[0002] In the fields of cultural relic conservation and art analysis, the precise classification and composition analysis of pigments is crucial for developing restoration plans and studying their historical evolution. Misclassification of pigments can lead to errors in the subsequent restoration plan.

[0003] Therefore, how to accurately classify pigments is a technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0004] In view of this, the present application provides a pigment classification method and related devices to solve the problem of the urgent need for accurate and fine classification of pigments.

[0005] To solve the above technical problems, this application adopts the following technical solutions:

[0006] A pigment classification method comprising:

[0007] Acquire an initial spectrum image of the target painting collected by a spectrum collection device;

[0008] performing a decompression operation on the initial spectral image to obtain a first spectral image;

[0009] performing a correction operation on the first spectral image based on the aging degree of the target painting to obtain a second spectral image;

[0010] determining a difference image between the first spectral image and the second spectral image, and performing a splicing operation on the difference image and the first spectral image to obtain a target spectral image;

[0011] Based on the target spectral image, a classification operation is performed on the pigments in the target painting to obtain a pigment classification result.

[0012] Optionally, performing a decompression operation on the initial spectral image to obtain a first spectral image includes:

[0013] Extracting a first feature map of the initial spectral image using an input convolutional layer;

[0014] Using multiple convolutional coding layers to gradually compress the size of the first feature map to obtain a second feature map; the number of feature channels in different convolutional coding layers increases layer by layer;

[0015] Recovering spatial details of the second feature map layer by layer using multiple convolutional decoding layers to obtain a third feature map;

[0016] The output layer is used to determine the first spectral image corresponding to the third feature map.

[0017] Optionally, based on the aging degree of the target painting, performing a correction operation on the first spectral image to obtain a second spectral image includes:

[0018] Based on reflectance difference information of different pixels in the first spectral image, performing a block operation on the first spectral image to obtain a plurality of sub-images;

[0019] Using the wavelength variation information of the reflectivity and the aging degree of the target painting, the reflectivity in the sub-image is corrected to obtain a corrected reflectivity of each sub-image;

[0020] A second spectral image is determined according to the corrected reflectivity of each sub-image.

[0021] Optionally, determining a difference image between the first spectral image and the second spectral image, and performing a splicing operation on the difference image and the first spectral image to obtain a target spectral image includes:

[0022] Calculating a residual between the first spectral image and the second spectral image to obtain a difference image;

[0023] A convolutional layer is used to perform a splicing operation on the difference image and the first spectral image to obtain a target spectral image.

[0024] Optionally, the process of determining the modulation height distribution data of the optical unit in the spectrum acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation includes:

[0025] During training, reconstruction loss and classification loss are calculated;

[0026] Performing weighted calculation on the reconstruction loss and the classification loss to obtain a total loss;

[0027] Determining initial modulation height distribution data of an optical unit in the spectrum acquisition device, initial decompression parameters used in a decompression operation, and / or initial classification parameters used in a classification operation using the total loss;

[0028] Optimizing the initial modulation height distribution data and / or the initial decompression parameters using the reconstruction loss to obtain target modulation height distribution data and / or target decompression parameters;

[0029] The initial classification parameters are optimized using the classification loss to obtain target classification parameters.

[0030] Optionally, based on the target spectral image, performing a classification operation on the pigments in the target painting to obtain a pigment classification result includes:

[0031] Processing the target spectral image using a pigment classification network to obtain a pigment classification result of the target painting;

[0032] The pigment classification network includes an input layer, a multi-layer fully connected hidden layer and an output classification layer;

[0033] The multi-layer fully connected hidden layer includes multiple levels of fully connected hidden layers; the number of neurons in different fully connected hidden layers is arranged in an exponentially decreasing order; and a target activation function is configured in the fully connected hidden layer.

[0034] Optionally, after acquiring the initial spectrum image of the target painting acquired by the spectrum acquisition device, the method further includes:

[0035] An image preprocessing operation is performed on the initial spectral image, where the image preprocessing includes at least one of dark current suppression, white balance adjustment, and illumination compensation.

[0036] A pigment classification device, comprising:

[0037] An image acquisition module is used to acquire an initial spectrum image of a target image acquired by a spectrum acquisition device;

[0038] a decompression module, configured to perform a decompression operation on the initial spectral image to obtain a first spectral image;

[0039] a correction module, configured to perform a correction operation on the first spectral image based on the aging degree of the target painting to obtain a second spectral image;

[0040] an image processing module, configured to determine a difference image between the first spectral image and the second spectral image, and perform a splicing operation on the difference image and the first spectral image to obtain a target spectral image;

[0041] The classification module is used to perform a classification operation on the pigments in the target painting based on the target spectral image to obtain a pigment classification result.

[0042] An electronic device comprising at least one processor and a memory connected to the processor, wherein:

[0043] The memory is used to store computer programs;

[0044] The processor is configured to execute the computer program so as to enable the electronic device to implement the above-mentioned pigment classification method.

[0045] A computer storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the above-mentioned pigment classification method.

[0046] The present application provides a method and related apparatus for pigment classification. In the present application, an initial spectral image of a target painting captured by a spectral acquisition device is obtained, the initial spectral image is decompressed to obtain a first spectral image, the first spectral image is corrected based on the aging degree of the target painting to obtain a second spectral image, a difference image between the first spectral image and the second spectral image is determined, the difference image and the first spectral image are spliced ​​to obtain a target spectral image, and the pigments in the target painting are classified based on the target spectral image, thereby achieving pigment classification. Furthermore, in the present application, the first spectral image is corrected based on the aging degree of the target painting, taking into account the influence of physical properties such as pigment aging and drying on pigment classification, thereby further improving the accuracy of the final pigment classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0048] Figure 1 A flowchart of a pigment classification method provided in an embodiment of the present application;

[0049] Figure 2 A flow chart of a process for determining a first spectral image provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a DOE spectrum decompression network provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of a DOE spectral image oil paint classification network provided in an embodiment of the present application;

[0052] Figure 5 A pigment sample and oil painting schematic diagram provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of pigment classification results provided in an embodiment of the present application;

[0054] Figure 7A flow chart of a process for determining a second spectral image provided in an embodiment of the present application;

[0055] Figure 8 A parameter optimization flow chart provided in an embodiment of the present application;

[0056] Figure 9 A flowchart of another pigment classification method provided in an embodiment of the present application;

[0057] Figure 10 A schematic structural diagram of a pigment classification device provided in an embodiment of the present application;

[0058] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] In the fields of cultural relic conservation and art analysis, the precise classification and compositional analysis of pigments, such as mineral pigments, are crucial for developing restoration plans and studying their historical evolution. Misclassification of pigments can lead to errors in subsequent restoration plans.

[0061] Therefore, how to accurately classify pigments is a technical problem that those skilled in the art urgently need to solve.

[0062] In actual scenarios, hyperspectral imaging technology can non-destructively obtain pigment spectral fingerprints and accurately distinguish multiple types of pigments. Therefore, in the embodiments of this application, hyperspectral imaging technology can be used to classify pigments.

[0063] To this end, the present application provides a pigment classification method and related devices. In the present application, an initial spectral image of a target painting collected by a spectral acquisition device is obtained, the initial spectral image is decompressed to obtain a first spectral image, and based on the aging degree of the target painting, a correction operation is performed on the first spectral image to obtain a second spectral image. A difference image between the first spectral image and the second spectral image is determined, and a splicing operation is performed on the difference image and the first spectral image to obtain a target spectral image. Based on the target spectral image, a classification operation is performed on the pigments in the target painting, thereby realizing the classification of the pigments.

[0064] In addition, in this application, based on the aging degree of the target painting, the first spectral image is corrected, taking into account the influence of physical properties such as pigment aging and air drying on pigment classification, thereby further improving the accuracy of the final pigment classification.

[0065] In addition, in the present application, the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation and / or the classification parameters used in the classification operation are first jointly optimized based on the loss information during training and then locally optimized, which fully utilizes the advantages of process linkage and improves the final pigment classification accuracy.

[0066] Based on the above, refer to Figure 1 , a pigment classification method may include:

[0067] S11. Acquire an initial spectrum image of the target painting collected by a spectrum collection device.

[0068] Among them, the target painting in the embodiment of the present application can be an oil painting, mural or other painting using pigments (such as mineral pigments). The subsequent embodiments are illustrated by taking the target painting as an oil painting.

[0069] Spectral imaging technology, such as hyperspectral imaging technology, can be used to collect a spectral image of the target image. The collected spectral image can be called an initial spectral image.

[0070] Hyperspectral imaging technology, among other things, acquires continuous spectral information across visible light and its adjacent wavelengths, forming a three-dimensional data cube consisting of two spatial dimensions and one spectral dimension. This allows for precise characterization of an object's spectral characteristics, material composition, and spatial structure. This technology has broad and in-depth application value in fields such as cultural relic restoration.

[0071] Traditional hyperspectral imaging systems often use push-broom or point-by-point scanning methods to acquire images. While these systems offer high spectral resolution, they suffer from slow acquisition speeds, complex systems, sensitivity to target motion, and difficulty in portable deployment. Driven by the demand for fast response and highly integrated applications, snapshot hyperspectral imaging systems, such as the Coded Aperture Snapshot Spectral Imaging (CASSI) system, have emerged. Using spatial encoding and spectral dispersion, CASSI systems capture information from multiple spectral bands of a target scene in a single exposure, significantly reducing imaging time.

[0072] However, CASSI systems and similar structures generally rely on complex refractive or reflective geometric optical devices (such as prisms, lens arrays, and mirrors). This not only results in large and expensive systems but also poor stability and maintainability over long periods of operation. Consequently, recent attempts have begun to replace traditional optical pathways with DOEs (Diffractive Optical Elements) to achieve miniaturized, lightweight, and cost-effective snapshot hyperspectral acquisition systems. DOEs offer advantages such as flexible wavefront modulation, mature fabrication processes, and strong designability, making them a key development direction for highly integrated spectral imaging systems.

[0073] To this end, the spectral acquisition device used in the embodiments of the present application, such as the hyperspectral acquisition device, is a self-developed DOE device, and the self-developed DOE device is a snapshot DOE hyperspectral imaging system. The snapshot DOE hyperspectral imaging system has a simple structure and can include a DOE trainable diffraction code and sensor.

[0074] Among them, in order to improve the quality and visual effects of the image and enhance the realism and three-dimensionality of the image, the sensor needs to support HDR (High Dynamic Range) and long exposure (reducing dark current noise) functions. The sensor used in the embodiment of the present application can be a monochrome CMOS (Complementary Metal-Oxide-Semiconductor) / CCD (Charge-coupled Device) sensor.

[0075] The DOE trainable diffraction encoding system resides within the DOE imaging chip, employing a trainable array of diffractive optical micro-units. The height parameter hi of each optical micro-unit (or optical unit) determines its phase modulation characteristics. The initial value of the height parameter is determined by random initialization and optimized through backpropagation during training. After DOE encoding, the input light field forms a single-channel compressed hybrid image, which is captured by a monochrome CMOS / CCD sensor.

[0076] When performing spectral imaging, such as hyperspectral imaging (spectral resolution of 10 -2 When collecting spectral images in the range of the order of λ, artificial light sources such as wide-spectrum LED (Light Emitting Diode) arrays (400-1000nm) or natural light sources are used to ensure coverage of the visible light to near-infrared band.

[0077] In addition, dark current calibration is also required during spectral image acquisition. Specifically, before each spectral image acquisition, a dark frame is captured to remove noise.

[0078] In the embodiment of the present application, the snapshot DOE hyperspectral imaging system is used to capture images that require pigment classification to obtain an initial spectral image, wherein the initial spectral image is a hyperspectral image.

[0079] In one implementation, an image preprocessing operation may be further performed on the initial spectral image, where the image preprocessing includes at least one of dark current suppression, white balance adjustment, and illumination compensation.

[0080] Among them, the dark current calibration operation can use the captured all-black frame to remove noise.

[0081] White balance adjustment is used to ensure that white objects in the picture appear truly white, so that the colors of the entire scene are accurately reproduced.

[0082] Illumination compensation can be achieved through the Retinex theory, a technique used by the retina and cerebral cortex. The core concept of Retinex theory is that the color (visual effect) perceived by the human eye is determined by both the light reflected from the object (the reflection component) and the ambient light (the illumination component). Our constant perception of color is the result of the brain automatically eliminating the effects of illumination. Therefore, Retinex theory can be used for illumination compensation.

[0083] In one implementation, only one of dark current suppression, white balance adjustment, and illumination compensation may be performed, or two of them may be performed, or each of them may be performed.

[0084] S12. Decompress the initial spectral image to obtain a first spectral image.

[0085] In real-world scenarios, the initial spectral image collected above is a single-channel compressed mixed image. It needs to be decompressed to restore the full content of the initial spectral image, resulting in a first spectral image. This first spectral image is also known as a hyperspectral image. The size of the first spectral image is H × W × N, where H, W, and N represent the length, width, and height, respectively.

[0086] In an embodiment of the present application, a DOE spectral decompression network, such as a DOE hyperspectral decompression network, is used to perform a decompression operation on the initial spectral image.

[0087] In one implementation, referring to Figure 2 , step S12 may include:

[0088] S21. Utilize the input convolutional layer to extract the first feature map of the initial spectral image.

[0089] In one implementation, the DOE hyperspectral decompression network is a deep convolutional network structure, such as a U-Net network. In one example, the structural diagram of the DOE hyperspectral decompression network can be referred to Figure 3 As shown, it contains the input convolution layer (referring to Figure 3 The first 3×3 convolution in the upper left corner + a nonlinear activation function (such as ReLU (Rectified Linear Unit) activation function), several convolutional coding layers (usually multiple convolutional coding layers), several convolutional decoding layers (usually multiple convolutional decoding layers) and the output layer.

[0090] Among them, the input convolution layer can adopt a convolution structure to receive the above-mentioned initial spectral image, perform feature extraction, and output a primary feature map, which can be called a first feature map.

[0091] S22. Use multiple convolutional coding layers to gradually compress the size of the first feature map to obtain a second feature map.

[0092] Among them, the number of feature channels in different convolutional coding layers increases layer by layer.

[0093] In the specific implementation, in the convolutional coding layer, each layer includes 3×3 convolution + non-linear activation function (such as ReLU activation function). In addition, it is also combined with 2×2 maximum pooling to achieve gradual compression of the feature map size. The number of feature channels of different convolutional coding layers increases layer by layer, and the deep feature expression can be enhanced with the help of residual connection.

[0094] Using a 3×3 convolution kernel for feature extraction captures local spatial information while maintaining efficient parameter count. Each convolutional layer generates multiple feature maps (number of channels). For example, the first layer may generate 64 channels, with subsequent layers doubling this number (128, 256, and so on).

[0095] The ReLU activation function introduces nonlinear characteristics to the network, enabling the network to learn complex function mappings, with efficient computation and alleviating the vanishing gradient problem (when the input is positive, the gradient is always 1).

[0096] Use a 2×2 window for max pooling with a stride of 2 to reduce the feature map size by half (e.g. 512×512 → 256×256). The purpose of the pooling operation is:

[0097] Reduce spatial resolution to reduce computational complexity; introduce translation invariance to enhance model robustness; expand the receptive field to capture more global features.

[0098] The first feature map is input into multiple convolutional coding layers to obtain the second feature map.

[0099] It should be noted that both the convolutional coding layer and the input convolutional layer use 3×3 convolution + nonlinear activation function (such as ReLU activation function). Figure 3 In the

[15] , except for the 3×3 convolution + non-linear activation function (such as ReLU activation function) used in the input convolution layer, other 3×3 convolution + non-linear activation functions (such as ReLU activation function) are located in the convolutional coding layer.

[0100] S23. Use multiple convolutional decoding layers to restore the spatial details of the second feature map layer by layer to obtain a third feature map.

[0101] In specific implementation, multiple convolutional decoding layers restore the spatial resolution of the second feature map layer by layer through transposed convolution or interpolation upsampling operations, and fuse the features of the corresponding encoding layer through skip connections at each layer to maintain spatial details.

[0102] Among them, the convolutional decoding layer restores the spatial resolution layer by layer through 2×2 deconvolution, and at the same time establishes jump connections through replication and cropping, fusing the features of the corresponding convolutional coding layer to retain spatial details, restore high spectral values, and obtain the third feature map.

[0103] S24. Determine the first spectral image corresponding to the third feature map using the output layer.

[0104] The output layer is a 1×1 convolution, which is used to restore the spectral image and output a high-fidelity decompressed spectral image. The high-fidelity decompressed spectral image is the first spectral image in the embodiment of the present application. The first spectral image is also a hyperspectral image.

[0105] In this embodiment, feature extraction, encoding compression, decoding and restoration are performed to finally output a first spectrum image.

[0106] S13. Based on the aging degree of the target painting, perform a correction operation on the first spectral image to obtain a second spectral image.

[0107] Among them, the color of the pigment of the oil painting will change after years of aging and drying. In the embodiment of the present application, the aging degree of the oil painting is taken into consideration and the first spectral image is corrected so that the second spectral image obtained after correction can take into account the aging degree of the oil painting.

[0108] S14: Determine a difference image between the first spectral image and the second spectral image, and perform a splicing operation on the difference image and the first spectral image to obtain a target spectral image.

[0109] In the embodiment of the present application, the difference image between the first spectral image and the second spectral image can reflect the influence of the aging degree of the oil painting on the spectral image.

[0110] In one implementation, when determining the difference image between the first spectral image and the second spectral image, a residual between the first spectral image and the second spectral image may be calculated to obtain the difference image.

[0111] In a specific implementation, for each pixel point in the first spectral image, a corresponding pixel point is selected from the second spectral image, and the residual of the reflectivity of the two pixel points is calculated, where the residual can be an absolute difference or a relative difference, depending on the actual configuration.

[0112] After the residuals of each pixel are calculated, the residuals of each pixel are combined in the order of the pixels to obtain a difference image. The spatial resolution and spectral dimension of the difference image are consistent with the first spectral image.

[0113] Then, a convolution layer is used to perform a concatenation operation on the difference image and the first spectral image to obtain a concatenated fusion feature map. The concatenated fusion feature map is a feature map obtained after image enhancement, and the feature map is the target spectral image in the embodiment of the present application.

[0114] In the embodiment of the present application, the accuracy of the final oil painting pigment classification is further improved by taking into account the physical properties of the pigment such as aging and drying.

[0115] S15. Based on the target spectral image, a classification operation is performed on the pigments in the target painting to obtain a pigment classification result.

[0116] In actual scenarios, step S15 can be implemented by a DOE spectral image oil paint classification network, such as a DOE hyperspectral image oil paint classification network.

[0117] The above-mentioned target spectral image is used as the input of the DOE spectral image oil painting pigment classification network. After being processed by the DOE spectral image oil painting pigment classification network, the pigment classification result of the oil painting can be obtained.

[0118] In one implementation, step S15 may include:

[0119] The target spectral image is processed using the pigment classification network to obtain the pigment classification result of the target painting.

[0120] The pigment classification network may be the above-mentioned DOE spectral image oil painting pigment classification network, or may be an MLP (Multi-Layer Perceptron) or the like.

[0121] In one implementation, Figure 4 As shown, the pigment classification network includes an input layer, a multi-layer fully connected hidden layer (hidden layer for short, Figure 4 There are two hidden layers drawn in the figure) and the output classification layer (referred to as the output layer).

[0122] Among them, the input layer is used to receive the target spectral image.

[0123] Multi-layer fully connected hidden layers consist of multiple levels of fully connected hidden layers. The number of neurons in different fully connected hidden layers decreases exponentially. As the number of neurons decreases exponentially, the feature space is gradually compressed, forcing the network to capture high-order statistical features (such as data cluster centers and category boundaries). The fully connected hidden layers are configured with a target activation function (such as the ReLU activation function).

[0124] The output classification layer is a fully connected layer with the same number of nodes as the number of pigment types to be identified, and the Softmax activation function is used for probabilistic output.

[0125] The pigment classification network described above yields a pigment classification result for the oil painting. This pigment classification result can be a semantic segmentation map, where each pixel independently predicts the pigment category, generating a pigment distribution map and annotating key mineral components (e.g., cinnabar, malachite).

[0126] In one implementation, the modulation height distribution data of the optical unit in the spectrum acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation are first jointly optimized based on loss information during training, and then locally optimized. The modulation height distribution data of the optical unit in the spectrum acquisition device is a hardware parameter, while the decompression parameters used in the decompression operation and the classification parameters used in the classification operation are software parameters.

[0127] In specific implementation, the total loss can be used to first jointly optimize the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation and / or the classification parameters used in the classification operation, and then the individual losses can be used to separately optimize the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation and the classification parameters used in the classification operation to obtain the optimal data.

[0128] In one example, in order to verify the effectiveness of the algorithm, Figure 5 and Figure 6 As shown in the figure, a mineral pigment sample collected by a self-developed hyperspectral camera and a traditional Chinese freehand oil painting were used for pigment classification test. The pigment sample and the oil painting are as follows: Figure 5 As shown, the classification results of gamboge pigment are as follows Figure 6 As shown. Figure 5 and Figure 6 By comparison, it can be seen that the classification result of the gamboge pigment in the examples of the present application is accurate.

[0129] Furthermore, during the experimental phase, the present invention collected and tested standard mineral pigment samples and real mural images. Using a proprietary DOE hyperspectral chip, the system can capture 31 spectral bands in a single pass, achieving an average relative error of less than 10% in spectral reconstruction. The combined reconstruction and classification model achieved an overall classification accuracy exceeding 95% in pigment identification experiments.

[0130] In this embodiment, an initial spectral image of a target painting captured by a spectral acquisition device is obtained, the initial spectral image is decompressed to obtain a first spectral image, the first spectral image is corrected based on the aging degree of the target painting to obtain a second spectral image, a difference image between the first spectral image and the second spectral image is determined, the difference image and the first spectral image are spliced ​​to obtain a target spectral image, and based on the target spectral image, the pigments in the target painting are classified, thereby achieving pigment classification. In addition, in this application, the first spectral image is corrected based on the aging degree of the target painting, taking into account the influence of physical properties such as pigment aging and air drying on pigment classification, further improving the accuracy of the final pigment classification. In addition, in this application, the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation are first jointly optimized based on the loss information during training and then locally optimized, fully utilizing the advantages of process linkage, improving the final pigment classification accuracy, and enabling accurate identification of pigments in complex artworks.

[0131] Based on any of the above embodiments, refer to Figure 7 , based on the aging degree of the target painting, performing a correction operation on the first spectral image to obtain a second spectral image may include:

[0132] S31 . Based on reflectivity difference information of different pixels in the first spectral image, perform a block operation on the first spectral image to obtain a plurality of sub-images.

[0133] In the embodiment of the present application, since the reflectivity data volume of the first spectral image is relatively large, direct processing of the first spectral image may result in low efficiency. Therefore, in the embodiment of the present application, the first spectral image can be segmented to obtain multiple sub-images. The amount of reflectivity data of the sub-images obtained after segmentation is smaller than that of the first spectral image, resulting in higher efficiency in processing the sub-images.

[0134] When performing block division, the block division operation can be performed manually or automatically.

[0135] When performing automatic block operations, a physical knowledge base embedding unit can be used. The physical knowledge base embedding unit can be a lightweight, trainable embedding layer that stores a set of learnable parameters representing the basic pigment optical properties (such as parameterized reflectance model coefficients and offsets that characterize the degree of aging). These parameters can be initialized using a small number of known pigment sample spectra and fine-tuned through gradient descent during the entire system training process to adapt to actual application scenarios.

[0136] The physical knowledge base embedding unit can be divided into blocks according to color, with different color areas divided into different sub-images, and areas with similar colors are located in the same sub-image.

[0137] Color can be represented by reflectivity, and different colors have different reflectivities. Therefore, in the embodiment of the present application, based on the reflectivity difference information of different pixels in the first spectral image, the portion of the image with similar reflectivity can be divided into one sub-image, and the portion with a larger reflectivity difference can be divided into another sub-image. Through this segmentation method, multiple sub-images can be obtained.

[0138] S32 , using the wavelength variation information of the reflectivity and the aging degree of the target image, correct the reflectivity in the sub-images to obtain the corrected reflectivity of each sub-image.

[0139] In actual scenarios, the aging degree of the target painting is a known parameter, such as the aging degree is aged for ten years, aged for twenty years, etc.

[0140] By analyzing the color aging of multiple oil paintings, we can find the common characteristics of color aging. For example, the common characteristics can be:

[0141] After a period of aging, such as ten years, the reflectivity of a certain band (such as 700) in the hyperspectral image will change, such as from one shape to another. In one example, it will change from a straight-line-like shape to a convex or concave shape. Therefore, the aging degree of the target image can be used to correct the wavelength change information of the reflectivity, such as correcting a convex or concave shape to a straight-line-like shape.

[0142] Through this correction operation, the reflectivity of each pixel of the oil painting before aging can be obtained, and this reflectivity is also the corrected reflectivity of the sub-image.

[0143] In one implementation, to improve correction efficiency, a physical model prediction unit can be constructed to perform the correction operation. The physical model prediction unit can be a lightweight and fully differentiable (i.e., learnable) computing structure, such as a small neural network. The physical model prediction unit can be trained using samples. These samples can include samples of the painting's aging degree, wavelength-dependent reflectivity, and the aforementioned basic pigment optical properties, enabling the physical model prediction unit to learn the effects of aging on reflectivity. Subsequently, the aging degree of the target painting and the wavelength-dependent reflectivity information are input into the physical model prediction unit to obtain the corrected reflectivity for each sub-image.

[0144] S33 : Determine a second spectral image according to the corrected reflectivity of each sub-image.

[0145] In this embodiment, the corrected reflectivity of each sub-image is combined according to the position of the sub-image in the first spectral image to obtain the second spectral image.

[0146] It should be noted that when segmenting and combining sub-images, it is necessary to combine spatial position information, which can be image coordinates. Each pixel at different spatial coordinates has a reflectivity. After the subsequent dynamic correction of the reflectivity, it is possible to know which pixel point at which spatial coordinate is corrected, and obtain the expected spectral reflectivity of the pigment at the current spatial coordinate in the "theoretical state (i.e., not aged)" or after a specific correction.

[0147] In practical scenarios, steps S31-S33 can be implemented using a physically guided spectral residual correction network. This network uses known physical optical properties of pigments (such as a basic reflectance model and typical aging or degradation patterns) to construct a differentiable physical prediction model. By inputting the first spectral image into the physical prediction model, the final second spectral image can be obtained.

[0148] In this embodiment, the prediction information of the physical prediction model can be deeply fused with the classification features of the above-mentioned first spectral image in the form of residuals. By taking into account the influence of physical properties such as pigment aging and drying on pigment classification, the accuracy of the final oil painting pigment classification can be further improved.

[0149] Based on any of the above embodiments, refer to Figure 8 The process of determining the modulation height distribution data of the optical unit in the spectrum acquisition device, the decompression parameters used in the decompression operation, and the classification parameters used in the classification operation may include:

[0150] S41. During the training process, the reconstruction loss and classification loss are calculated.

[0151] In this embodiment, the modulation height distribution data of the optical unit in the DOE imaging chip, the decompression parameters in the DOE hyperspectral decompression network, and / or the classification parameters in the DOE spectral image oil paint classification network are integrated into a single training closed loop for collaborative optimization, forming a unified end-to-end hyperspectral task processing framework. Joint optimization is performed during training by sharing optimization objectives, with the common goal of minimizing loss.

[0152] Specifically, the DOE trainable diffraction code is composed of multiple optical units, and the phase modulation of each optical unit is determined by its height, and the height parameter is a trainable variable. The DOE trainable diffraction code compresses and modulates the input scene to generate a single-channel compressed mixed image (i.e., the initial spectral image mentioned above). The DOE hyperspectral decompression network reconstructs it to obtain the first spectral image. The physically guided spectral residual correction network corrects the first spectral image based on the aging degree of the target painting to obtain the second spectral image, determines the difference image between the first spectral image and the second spectral image, and splices the difference image and the first spectral image to obtain the target spectral image. The DOE spectral image oil painting pigment classification network classifies the target spectral image into pigments to obtain the pigment classification results. The overall process is as follows: Figure 9 shown.

[0153] The system composed of multiple networks sets two loss functions during the training phase: the first is the reconstruction loss (Loss_recon), which calculates the mean squared error (MSE) between the reconstructed spectral image and the real spectral image in the dataset to obtain the reconstruction loss; the second is the classification loss (Loss_class), which is used to measure the cross entropy error between the classification result and the real pigment label. The final total loss function is defined as:

[0154] ;

[0155] in, is the total loss, and is a tunable hyperparameter, To rebuild the losses, is the classification loss.

[0156] The calculation formula is:

[0157]

[0158] in, is the batch size, is the sample index, is the real spectral image, To reconstruct the spectral image, that is, the image output by the DOE hyperspectral decompression network.

[0159] The calculation formula is:

[0160]

[0161] in, is the category index, is the total number of pigment categories, is the one-hot encoding of the true label, is the probability predicted by the model.

[0162] S42. Perform weighted calculation on the reconstruction loss and the classification loss to obtain the total loss.

[0163] The calculation formula for total loss refers to the corresponding description above.

[0164] S43. Determine initial modulation height distribution data of the optical unit in the spectrum acquisition device, initial decompression parameters used in a decompression operation, and / or initial classification parameters used in a classification operation using the total loss.

[0165] For specific optimization, the Adam (Adaptive Moment Estimation) optimizer can be used with an initial learning rate of 1×10 −4 , which can be adjusted dynamically later.

[0166] In specific implementation, Driven by the joint efforts of , the system can simultaneously optimize at least one of the following:

[0167] 1) The modulation height distribution of each optical unit in the DOE coding structure achieves the most suitable spectral coding for the oil painting scene;

[0168] 2) Decompression parameter configuration of DOE hyperspectral decompression network to improve the accuracy of spectral image restoration;

[0169] 3) Configuring the classification parameters of the DOE spectral image oil paint classification network to enhance the recognition ability of fine-grained pigments.

[0170] In one embodiment, the above three items can be optimized simultaneously. If all three items are optimized simultaneously, this joint optimization mechanism breaks down the barriers between traditional front-end acquisition and back-end recognition, achieving the coupling and synergy between the hardware physical structure and the deep neural network, allowing the system to maintain stable recognition performance under complex conditions such as different lighting, degradation, and noise. Compared to traditional decoding models, this module takes color fidelity and spatial consistency into consideration in its structural design, making it particularly suitable for images of cultural relics with rich spectral layers and significant noise interference.

[0171] Through the above-mentioned joint optimization, the initial modulation height distribution data of the optical unit in the spectrum acquisition device, the initial decompression parameters used in the decompression operation and / or the initial classification parameters used in the classification operation can be obtained.

[0172] S44 , optimizing the initial modulation height distribution data and / or the initial decompression parameters using the reconstruction loss to obtain target modulation height distribution data and / or target decompression parameters.

[0173] In specific implementation, the three parts are based on the overall In addition to jointly optimizing parameters, each component also performs local parameter re-optimization based on its own loss. The DOE imaging chip's parameters are locally re-optimized based on loss_recon, with the goal of reducing loss_recon. The DOE hyperspectral decompression network is also locally re-optimized based on loss_recon, with the goal of reducing loss_recon.

[0174] After the local optimization operation in this step, the target modulation height distribution data and target decompression parameters can be obtained.

[0175] S45. Optimize the initial classification parameters using the classification loss to obtain the target classification parameters.

[0176] Specifically, the DOE spectral image oil paint classification network will perform local re-optimization based on loss_class, with the goal of reducing loss_class and obtaining the target classification parameters.

[0177] It should be noted that the parameters in the physical guided spectral residual correction network are pre-trained and do not participate in the above joint optimization and local optimization operations.

[0178] After experimental analysis, it was found that the joint network structure improved the recognition accuracy by about 12% compared with the traditional reconstruction-classification serial process, while compressing the overall processing time to less than 5 seconds. The system volume and power consumption were greatly reduced, showing good deployability and practical value.

[0179] In this embodiment, the DOE imaging chip microstructure parameters, the DOE spectral image decompression algorithm parameters, and the DOE spectral image oil painting pigment classification algorithm parameters are first jointly optimized and then locally optimized. The same-return network joint fully utilizes the advantages of hardware-software process linkage and feature linkage, and improves the network's perception of subtle pigment changes. It is particularly suitable for situations where the image color fades severely, the pigment coverage is uneven, or there is little residue, which greatly improves the final oil painting pigment classification effect.

[0180] In summary, the embodiments of the present application provide an integrated deep neural network system for spectral image reconstruction and pigment classification. Through the self-developed DOE imaging chip and end-to-end integrated network architecture, unified modeling and optimization from original image acquisition, spectral reconstruction, correction to pigment classification are completed, significantly improving the pigment recognition accuracy and system practicality.

[0181] In one implementation, the integrated deep neural network system for spectral image reconstruction and pigment classification needs to be trained using a training dataset.

[0182] When constructing a training dataset, one can collect hyperspectral data (real or simulated) of artworks such as oil paintings and murals, covering different pigments (such as ochre, ultramarine, and lead white). This hyperspectral data is annotated with pixel-level pigment category labels, and the actual reflectance spectra are recorded (for reconstruction loss calculation).

[0183] In order to increase the data volume of the training data set, data enhancement technology can be used to simulate different lighting conditions (color temperature 2500K-10000K), noise levels (Gaussian + Poisson noise), and degradation (blurring, occlusion) through data enhancement to increase the data volume of the data set.

[0184] Based on the above embodiment of the pigment classification method, another embodiment of the present application provides a pigment classification device, referring to Figure 10 , which may include:

[0185] An image acquisition module 11 is used to acquire an initial spectrum image of a target painting acquired by a spectrum acquisition device;

[0186] a decompression module 12, configured to perform a decompression operation on the initial spectral image to obtain a first spectral image;

[0187] A correction module 13 is configured to perform a correction operation on the first spectral image based on the aging degree of the target painting to obtain a second spectral image;

[0188] An image processing module 14 is configured to determine a difference image between the first spectral image and the second spectral image, and perform a splicing operation on the difference image and the first spectral image to obtain a target spectral image;

[0189] A classification module 15 is used to classify the pigments in the target painting based on the target spectral image to obtain a pigment classification result;

[0190] In one implementation, the decompression module 12 includes:

[0191] an extraction submodule, configured to extract a first feature map of the initial spectral image using an input convolutional layer;

[0192] A compression submodule, configured to gradually compress the size of the first feature map using multiple convolutional coding layers to obtain a second feature map; the number of feature channels in different convolutional coding layers increases layer by layer;

[0193] A recovery submodule, configured to recover the spatial details of the second feature map layer by layer using multiple convolutional decoding layers to obtain a third feature map;

[0194] The image determination submodule is used to determine the first spectral image corresponding to the third feature map using the output layer.

[0195] In one implementation, the correction module 13 includes:

[0196] A block submodule, configured to perform a block operation on the first spectral image based on reflectance difference information of different pixel points in the first spectral image to obtain a plurality of sub-images;

[0197] A correction submodule is used to correct the reflectivity in the sub-images by using the wavelength variation information of the reflectivity and the aging degree of the target image, so as to obtain the corrected reflectivity of each sub-image;

[0198] The image processing submodule is used to determine a second spectral image according to the corrected reflectivity of each sub-image.

[0199] In one implementation, the image processing module 14 is specifically configured to:

[0200] The residual of the first spectral image and the second spectral image is calculated to obtain a difference image, and the difference image and the first spectral image are spliced ​​using a convolutional layer to obtain a target spectral image.

[0201] In one implementation, the method further includes:

[0202] The parameter determination module is used to determine the modulation height distribution data of the optical unit in the spectrum acquisition device, the decompression parameters used in the decompression operation and / or the classification parameters used in the classification operation.

[0203] The parameter determination module includes:

[0204] The loss calculation submodule is used to calculate the reconstruction loss and classification loss during the training process;

[0205] The weighted calculation submodule is used to perform weighted calculation on the reconstruction loss and classification loss to obtain the total loss;

[0206] a joint optimization submodule, configured to determine, by using the total loss, initial modulation height distribution data of the optical unit in the spectrum acquisition device, initial decompression parameters used in a decompression operation, and / or initial classification parameters used in a classification operation;

[0207] A first optimization submodule is configured to optimize the initial modulation height distribution data and / or the initial decompression parameters using the reconstruction loss to obtain target modulation height distribution data and / or target decompression parameters;

[0208] The second optimization submodule is used to optimize the initial classification parameters using the classification loss to obtain the target classification parameters.

[0209] In one implementation, the classification module 15 is specifically configured to:

[0210] The target spectral image is processed using the pigment classification network to obtain the pigment classification result of the target painting;

[0211] The pigment classification network includes an input layer, multiple fully connected hidden layers, and an output classification layer;

[0212] The multi-layer fully connected hidden layer includes multiple levels of fully connected hidden layers; the number of neurons in different fully connected hidden layers is arranged in an exponentially decreasing order; and a target activation function is configured in the fully connected hidden layer.

[0213] In one implementation, the method further includes:

[0214] The preprocessing module is used to perform image preprocessing operations on the initial spectral image, where the image preprocessing includes at least one of dark current suppression, white balance adjustment, and illumination compensation.

[0215] In this embodiment, an initial spectral image of a target painting captured by a spectral acquisition device is obtained, the initial spectral image is decompressed to obtain a first spectral image, the first spectral image is corrected based on the aging degree of the target painting to obtain a second spectral image, a difference image between the first spectral image and the second spectral image is determined, the difference image and the first spectral image are spliced ​​to obtain a target spectral image, and based on the target spectral image, a classification operation is performed on the pigments in the target painting, thereby achieving pigment classification. In addition, in this application, the first spectral image is corrected based on the aging degree of the target painting, taking into account the influence of physical properties such as pigment aging and air drying on pigment classification, thereby further improving the accuracy of the final pigment classification. In addition, in this application, the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation are first jointly optimized based on the loss information during training and then locally optimized, fully utilizing the advantages of process linkage and improving the final pigment classification accuracy.

[0216] It should be noted that for the working process of each module and sub-module in the embodiment of the present application, please refer to the corresponding description in the above embodiment and will not be repeated here.

[0217] An embodiment of the present application further provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0218] Memory is used to store computer programs;

[0219] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned pigment classification method.

[0220] refer to Figure 11 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 11 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0221] like Figure 11As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0222] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 11 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0223] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the pigment classification methods provided in the embodiments of the present application.

[0224] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any pigment classification method provided in the embodiment of the present application.

[0225] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pigment classification method, characterized in that: include: Acquire an initial spectrum image of the target painting collected by a spectrum collection device; performing a decompression operation on the initial spectral image to obtain a first spectral image; performing a correction operation on the first spectral image based on the aging degree of the target painting to obtain a second spectral image; determining a difference image between the first spectral image and the second spectral image, and performing a splicing operation on the difference image and the first spectral image to obtain a target spectral image; Based on the target spectral image, a classification operation is performed on the pigments in the target painting to obtain a pigment classification result.

2. The pigment classification method according to claim 1, characterized in that: Decompressing the initial spectral image to obtain a first spectral image includes: Extracting a first feature map of the initial spectral image using an input convolutional layer; Using multiple convolutional coding layers to gradually compress the size of the first feature map to obtain a second feature map; the number of feature channels in different convolutional coding layers increases layer by layer; Recovering spatial details of the second feature map layer by layer using multiple convolutional decoding layers to obtain a third feature map; The output layer is used to determine the first spectral image corresponding to the third feature map.

3. The pigment classification method according to claim 1, characterized in that: Based on the aging degree of the target painting, a correction operation is performed on the first spectral image to obtain a second spectral image, including: Based on reflectance difference information of different pixels in the first spectral image, performing a block operation on the first spectral image to obtain a plurality of sub-images; Using the wavelength variation information of the reflectivity and the aging degree of the target painting, the reflectivity in the sub-image is corrected to obtain a corrected reflectivity of each sub-image; A second spectral image is determined according to the corrected reflectivity of each sub-image.

4. The pigment classification method according to claim 1, characterized in that: Determining a difference image between the first spectral image and the second spectral image, and performing a splicing operation on the difference image and the first spectral image to obtain a target spectral image, including: Calculating a residual between the first spectral image and the second spectral image to obtain a difference image; A convolutional layer is used to perform a splicing operation on the difference image and the first spectral image to obtain a target spectral image.

5. The pigment classification method according to claim 1, characterized in that: The process of determining the modulation height distribution data of the optical unit in the spectrum acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation includes: During training, reconstruction loss and classification loss are calculated; Performing weighted calculation on the reconstruction loss and the classification loss to obtain a total loss; Determining initial modulation height distribution data of an optical unit in the spectrum acquisition device, initial decompression parameters used in a decompression operation, and / or initial classification parameters used in a classification operation using the total loss; Optimizing the initial modulation height distribution data and / or the initial decompression parameters using the reconstruction loss to obtain target modulation height distribution data and / or target decompression parameters; The initial classification parameters are optimized using the classification loss to obtain target classification parameters.

6. The pigment classification method according to claim 1, characterized in that: Based on the target spectral image, a classification operation is performed on the pigments in the target painting to obtain a pigment classification result, including: Processing the target spectral image using a pigment classification network to obtain a pigment classification result of the target painting; The pigment classification network includes an input layer, a multi-layer fully connected hidden layer and an output classification layer; The multi-layer fully connected hidden layer includes multiple levels of fully connected hidden layers; the number of neurons in different fully connected hidden layers is arranged in an exponentially decreasing order; and a target activation function is configured in the fully connected hidden layer.

7. The pigment classification method according to claim 1, characterized in that: After obtaining the initial spectrum image of the target image acquired by the spectrum acquisition device, the method further includes: An image preprocessing operation is performed on the initial spectral image, where the image preprocessing includes at least one of dark current suppression, white balance adjustment, and illumination compensation.

8. A pigment classification device, characterized in that: include: An image acquisition module is used to acquire an initial spectrum image of a target image acquired by a spectrum acquisition device; a decompression module, configured to perform a decompression operation on the initial spectral image to obtain a first spectral image; a correction module, configured to perform a correction operation on the first spectral image based on the aging degree of the target painting to obtain a second spectral image; an image processing module, configured to determine a difference image between the first spectral image and the second spectral image, and perform a splicing operation on the difference image and the first spectral image to obtain a target spectral image; The classification module is used to perform a classification operation on the pigments in the target painting based on the target spectral image to obtain a pigment classification result.

9. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so as to enable the electronic device to implement the pigment classification method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the pigment classification method according to any one of claims 1 to 7.

Images