A method and related apparatus for classifying pigments

Abstract

This application provides a pigment classification method and related apparatus. The method involves decompressing an initial spectral image of a target painting acquired by a spectral acquisition device to obtain a first spectral image. Based on the aging degree of the target painting, the first spectral image is corrected to obtain a second spectral image. A difference image between the first and second spectral images is determined, and the difference image and the first spectral image are stitched together to obtain the target spectral image. Based on the target spectral image, pigments in the target painting are classified, thus achieving pigment classification. Furthermore, the correction operation on the first spectral image based on the aging degree of the target painting considers the influence of physical characteristics such as pigment aging and drying on pigment classification, further improving the accuracy of pigment classification. Additionally, joint optimization of hardware and software parameters followed by local optimization further improves the final pigment classification accuracy.

Description

Technical Field

[0001] This application relates to the field of pigment classification, and more specifically, to a pigment classification method and related apparatus. Background Technology

[0002] In the fields of cultural relic conservation and art analysis, the precise classification and compositional analysis of pigments are crucial for developing restoration plans and studying their historical evolution. Incorrect pigment classification can lead to incorrect restoration plans.

[0003] Therefore, how to accurately classify pigments is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, this application provides a pigment classification method and related apparatus to solve the problem of the urgent need for accurate and detailed classification of pigments.

[0005] To solve the above-mentioned technical problems, this application adopts the following technical solution:

[0006] A method for classifying pigments, comprising:

[0007] Acquire the initial spectral image of the target image acquired by the spectral acquisition device;

[0008] The initial spectral image is decompressed to obtain the first spectral image;

[0009] Based on the aging degree of the target image, the first spectral image is corrected to obtain the second spectral image;

[0010] Determine the difference image between the first spectral image and the second spectral image, and perform a stitching operation on the difference image and the first spectral image to obtain the target spectral image;

[0011] Based on the target spectral image, the pigments in the target painting are classified to obtain the pigment classification results.

[0012] Optionally, the initial spectral image is decompressed to obtain a first spectral image, including:

[0013] The first feature map of the initial spectral image is extracted using the input convolutional layer;

[0014] The size of the first feature map is gradually compressed using multiple convolutional coding layers to obtain the second feature map; the number of feature channels in different convolutional coding layers increases layer by layer.

[0015] The spatial details of the second feature map are recovered layer by layer by using multiple convolutional decoding layers to obtain the third feature map;

[0016] The first spectral image corresponding to the third feature map is determined using the output layer.

[0017] Optionally, based on the aging degree of the target image, a correction operation is performed on the first spectral image to obtain a second spectral image, including:

[0018] Based on the reflectance difference information of different pixels in the first spectral image, the first spectral image is divided into blocks to obtain multiple sub-images;

[0019] Using the wavelength change information of reflectance and the aging degree of the target image, the reflectance in the sub-image is corrected to obtain the corrected reflectance of each sub-image.

[0020] The second spectral image is determined based on the corrected reflectance of each sub-image.

[0021] Optionally, a difference image is determined between the first spectral image and the second spectral image, and a stitching operation is performed on the difference image and the first spectral image to obtain a target spectral image, including:

[0022] Calculate the residual between the first spectral image and the second spectral image to obtain the difference image;

[0023] The target spectral image is obtained by stitching the difference image and the first spectral image together using a convolutional layer.

[0024] Optionally, the process for determining the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation includes:

[0025] During training, reconstruction loss and classification loss are calculated;

[0026] The total loss is obtained by weighting the reconstruction loss and the classification loss.

[0027] Using the total loss, the initial modulation height distribution data of the optical unit in the spectral acquisition device, the initial decompression parameters used in the decompression operation, and / or the initial classification parameters used in the classification operation are determined;

[0028] Using the reconstruction loss, the initial modulation height distribution data and / or the initial decompression parameters are optimized to obtain the target modulation height distribution data and / or the target decompression parameters;

[0029] The initial classification parameters are optimized using the classification loss to obtain the target classification parameters.

[0030] Optionally, based on the target spectral image, the pigments in the target painting are classified to obtain pigment classification results, including:

[0031] The target spectral image is processed using a pigment classification network to obtain the pigment classification result of the target painting;

[0032] The pigment classification network includes an input layer, multiple fully connected hidden layers, and an output classification layer.

[0033] The multi-layer fully connected hidden layer includes multiple levels of fully connected hidden layers; the number of neurons in different fully connected hidden layers decreases exponentially; and a target activation function is configured in each fully connected hidden layer.

[0034] Optionally, after acquiring the initial spectral image of the target image collected by the spectral acquisition device, the method further includes:

[0035] The initial spectral image is subjected to image preprocessing operations, the image preprocessing including at least one of dark current suppression, white balance adjustment and illumination compensation.

[0036] A pigment sorting device, comprising:

[0037] The image acquisition module is used to acquire the initial spectral image of the target image collected by the spectral acquisition device;

[0038] The decompression module is used to decompress the initial spectral image to obtain the first spectral image;

[0039] The correction module is used to perform a correction operation on the first spectral image based on the aging degree of the target image to obtain a second spectral image;

[0040] The image processing module is used to determine the difference image between the first spectral image and the second spectral image, and to perform a stitching operation on the difference image and the first spectral image to obtain the target spectral image;

[0041] The classification module is used to classify the pigments in the target painting based on the target spectral image to obtain the pigment classification results.

[0042] An electronic device includes at least one processor and a memory connected to the processor, wherein:

[0043] The memory is used to store computer programs;

[0044] The processor is used to execute the computer program so that the electronic device can implement the above-described pigment classification method.

[0045] A computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the above-described pigment classification method.

[0046] This application provides a pigment classification method and related apparatus. In this application, an initial spectral image of a target painting is acquired by a spectral acquisition device. The initial spectral image is decompressed to obtain a first spectral image. Based on the aging degree of the target painting, the first spectral image is corrected to obtain a second spectral image. A difference image between the first and second spectral images is determined. The difference image and the first spectral image are then stitched together to obtain a target spectral image. Based on the target spectral image, the pigments in the target painting are classified, thus achieving pigment classification. Furthermore, in this application, the correction operation on the first spectral image based on the aging degree of the target painting considers the influence of physical properties such as pigment aging and drying on pigment classification, further improving the accuracy of the final pigment classification. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a pigment classification method provided in this application embodiment;

[0049] Figure 2 A flowchart illustrating the process of determining a first spectral image as provided in an embodiment of this application;

[0050] Figure 3 A schematic diagram of the structure of a DOE spectral decompression network provided in an embodiment of this application;

[0051] Figure 4 A schematic diagram of the structure of a DOE spectral image oil paint classification network provided in this application embodiment;

[0052] Figure 5 A pigment sample and an oil painting schematic diagram provided for embodiments of this application;

[0053] Figure 6 This is a schematic diagram of pigment classification results provided in an embodiment of this application;

[0054] Figure 7A flowchart illustrating the process of determining a second spectral image as provided in an embodiment of this application;

[0055] Figure 8 A parameter optimization flowchart provided for an embodiment of this application;

[0056] Figure 9 A flowchart illustrating another pigment classification method provided in this application embodiment;

[0057] Figure 10 This is a schematic diagram of the structure of a pigment sorting device provided in an embodiment of this application;

[0058] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] In the fields of cultural relic conservation and art analysis, the detailed classification and compositional analysis of pigments, such as mineral pigments, are crucial for the formulation of restoration plans and the study of their historical evolution. Incorrect pigment classification can lead to incorrect restoration plans.

[0061] Therefore, how to accurately classify pigments is a technical problem that urgently needs to be solved by those skilled in the art.

[0062] In practical applications, hyperspectral imaging technology can non-destructively acquire the spectral fingerprints of pigments, enabling accurate differentiation among various pigment types. Therefore, in this embodiment, hyperspectral imaging technology can be used for pigment classification.

[0063] To this end, this application provides a pigment classification method and related apparatus. In this application, an initial spectral image of a target painting is acquired by a spectral acquisition device. The initial spectral image is decompressed to obtain a first spectral image. Based on the aging degree of the target painting, the first spectral image is corrected to obtain a second spectral image. The difference between the first and second spectral images is determined. The difference image and the first spectral image are stitched together to obtain a target spectral image. Based on the target spectral image, the pigments in the target painting are classified, thus realizing the classification of pigments.

[0064] In addition, in this application, the first spectral image is corrected based on the aging degree of the target painting, taking into account the influence of physical properties such as pigment aging and drying on pigment classification, thereby further improving the accuracy of the final pigment classification.

[0065] In addition, in this application, the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation are first jointly optimized based on the loss information during training, and then local optimization is performed. This fully utilizes the advantages of process linkage and improves the final pigment classification accuracy.

[0066] Based on the above, refer to Figure 1 A method for classifying pigments may include:

[0067] S11. Obtain the initial spectral image of the target image acquired by the spectral acquisition device.

[0068] In this embodiment, the target painting can be an oil painting, mural, or other painting that uses pigments (such as mineral pigments). Subsequent embodiments will use an oil painting as an example for illustration.

[0069] Spectral imaging techniques, such as hyperspectral imaging, can be used to acquire spectral images of the target image. The acquired spectral image can be called the initial spectral image.

[0070] Hyperspectral imaging technology, by acquiring continuous spectral information spanning the visible light spectrum and its adjacent bands, forms a three-dimensional data cube consisting of two spatial dimensions and one spectral dimension, thereby accurately characterizing the spectral features, material composition, and spatial structure of an object. This technology has broad and in-depth application value in fields such as cultural relic restoration.

[0071] Traditional hyperspectral imaging systems typically employ pushbroom or point-by-point scanning methods for image acquisition. While offering high spectral resolution, these methods generally suffer from drawbacks such as slow acquisition speed, system complexity, sensitivity to target motion, and difficulty in portable deployment. Driven by the demand for rapid response and highly integrated applications, snapshot hyperspectral imaging systems have emerged, such as the CASSI (Coded Aperture Snapshot Spectral Imaging) system. The CASSI system, through spatial coding and spectral dispersion, acquires information from multiple spectral bands of a target scene in a single exposure, significantly reducing imaging time.

[0072] However, CASSI systems and similar structures generally rely on complex refractive or reflective geometric optical components (such as prisms, lens arrays, and mirrors), which not only results in large system size and high cost but also poor stability and maintainability during long-term operation. Therefore, in recent years, attempts have begun to use DOEs (Diffractive Optical Elements) to replace traditional optical paths in order to achieve miniaturized, lightweight, and cost-effective snapshot hyperspectral acquisition systems. DOEs have advantages such as flexible wavefront modulation, mature fabrication processes, and strong designability, making them an important direction for highly integrated spectral imaging systems.

[0073] Therefore, the spectral acquisition device used in the embodiments of this application, such as the hyperspectral acquisition device, is a self-developed DOE device. The self-developed DOE device is a snapshot DOE hyperspectral imaging system. The snapshot DOE hyperspectral imaging system has a simple structure and may include DOE trainable diffraction coding and a sensor.

[0074] In order to improve image quality and visual effects, and enhance the realism and three-dimensionality of the image, the sensor needs to support HDR (High Dynamic Range) and long exposure (to reduce dark current noise). The sensor used in this application embodiment can be a monochrome CMOS (Complementary Metal-Oxide-Semiconductor) / CCD (Charge-coupled Device) sensor.

[0075] The DOE trainable diffraction code is located in the DOE imaging chip and employs a trainable diffractive optical micro-unit array. The height parameter hi of each optical micro-unit (referred to as an optical unit) determines its phase modulation characteristics. The initial value of the height parameter is determined by random initialization and optimized through backpropagation during training. After the input light field is encoded by DOE, it forms a single-channel compressed hybrid image, which is captured by a monochrome CMOS / CCD sensor.

[0076] When processing spectral images, such as hyperspectral images (spectral resolution of 10... -2 When acquiring spectral images in the λ-order range, artificial or natural light sources with a wide-spectrum LED (Light Emitting Diode) array (400-1000nm) are used to ensure coverage of the visible to near-infrared band.

[0077] In addition, dark current calibration is required during spectral image acquisition. Specifically, a dark frame is captured before each spectral image acquisition to remove noise.

[0078] In this embodiment, the aforementioned snapshot-type DOE hyperspectral imaging system is used to acquire images of the pigments to be classified, thereby obtaining an initial spectral image. The initial spectral image is a hyperspectral image.

[0079] In one implementation, the initial spectral image may also undergo image preprocessing operations, including at least one of dark current suppression, white balance adjustment, and illumination compensation.

[0080] Among them, the dark current calibration operation can use the captured completely black frames for noise removal.

[0081] White balance adjustment is used to ensure that white objects in the image appear true white, thereby ensuring accurate color reproduction throughout the scene.

[0082] Illumination compensation can be performed using the Retinex theory of retinal-cortical light compensation. The core idea of ​​the Retinex theory is that the color of an object perceived by the human eye (visual effect) is jointly determined by the light reflected from the object (reflection component) and the ambient light (illuminance component). Our constant perception of color is the result of the brain automatically filtering out the influence of illumination. Therefore, this Retinex theory can be used for illumination compensation.

[0083] In one implementation, only one of dark current suppression, white balance adjustment, and illumination compensation may be performed, or two of dark current suppression, white balance adjustment, and illumination compensation may be performed, or each of dark current suppression, white balance adjustment, and illumination compensation may be performed.

[0084] S12. Decompress the initial spectral image to obtain the first spectral image.

[0085] In practical scenarios, the initial spectral image acquired above is a single-channel compressed and mixed image, which needs to be decompressed to recover the complete content of the initial spectral image and obtain the first spectral image. The first spectral image is also a hyperspectral image. The size of the first spectral image is H×W×N, where H, W, and N represent length, width, and height, respectively.

[0086] In this embodiment, a DOE spectral decompression network, such as a DOE hyperspectral decompression network, is used to decompress the initial spectral image.

[0087] In one implementation, refer to Figure 2 Step S12 may include:

[0088] S21. Extract the first feature map of the initial spectral image using the input convolutional layer.

[0089] In one implementation, the DOE hyperspectral decompression network is a deep convolutional network structure, such as the U-Net network. A schematic diagram of the DOE hyperspectral decompression network structure can be found in one example. Figure 3 As shown, it includes an input convolutional layer (referring to...) Figure 3 The top left corner consists of a first 3×3 convolution with a non-linear activation function (such as ReLU (Rectified Linear Unit) activation function), several convolutional coding layers (usually multiple convolutional coding layers), several convolutional decoding layers (usually multiple convolutional decoding layers), and an output layer.

[0090] The input convolutional layer can adopt a convolutional structure to receive the initial spectral image mentioned above, perform feature extraction, and output a primary feature map, which can be called the first feature map.

[0091] S22. The size of the first feature map is gradually compressed using multiple convolutional coding layers to obtain the second feature map.

[0092] The number of feature channels in different convolutional coding layers increases progressively.

[0093] In the specific implementation, each convolutional coding layer includes a 3×3 convolution plus a non-linear activation function (such as the ReLU activation function). In addition, 2×2 max pooling is used to gradually compress the feature map size. The number of feature channels in different convolutional coding layers increases layer by layer, and residual connections can be used to enhance the expression of deep features.

[0094] The feature extraction process uses 3×3 convolutional kernels, which captures local spatial information while maintaining high efficiency in terms of parameter count. Each convolutional layer generates multiple feature maps (number of channels), for example, the first layer may generate 64 channels, and subsequent layers will double that number (128, 256, etc.).

[0095] The ReLU activation function introduces nonlinearity into the network, enabling it to learn complex function mappings, making it computationally efficient, and mitigating the gradient vanishing problem (the gradient is always 1 when the input is positive).

[0096] Max pooling is performed using a 2×2 window with a stride of 2, reducing the feature map size by half (e.g., 512×512 → 256×256). The purpose of the pooling operation is:

[0097] Reduce spatial resolution to decrease computational cost; introduce translation invariance to enhance model robustness; expand the receptive field to capture more global features.

[0098] The second feature map can be obtained by inputting the first feature map into multiple convolutional coding layers.

[0099] It should be noted that both the convolutional coding layer and the input convolutional layer use 3×3 convolution with a non-linear activation function (such as the ReLU activation function). Figure 3 In the convolutional coding layer, other 3×3 convolutions plus non-linear activation functions (such as ReLU activation functions) besides those used in the input convolutional layer are located in the convolutional coding layer.

[0100] S23. The spatial details of the second feature map are recovered layer by layer by using multiple convolutional decoding layers to obtain the third feature map.

[0101] In practice, multiple convolutional decoding layers restore the spatial resolution of the second feature map layer by layer through transposed convolution or interpolation upsampling operations, and fuse the features of the corresponding encoding layer through skip connections in each layer to preserve spatial details.

[0102] The convolutional decoding layer restores spatial resolution layer by layer through 2×2 deconvolution. At the same time, it establishes skip connections through copying and cropping, and fuses the features of the corresponding convolutional coding layer to retain spatial details, restore the hyperspectral number, and obtain the third feature map.

[0103] S24. Use the output layer to determine the first spectral image corresponding to the third feature map.

[0104] The output layer is a 1×1 convolution, which is used to recover the spectral image and output a high-fidelity decompressed spectral image. This high-fidelity decompressed spectral image is the first spectral image in this embodiment. The first spectral image is also a hyperspectral image.

[0105] In this embodiment, the process involves feature extraction, encoding and compression, decoding and recovery, and finally outputting the first spectral image.

[0106] S13. Based on the aging degree of the target image, perform a correction operation on the first spectral image to obtain the second spectral image.

[0107] In this application embodiment, the color of the pigments in an oil painting will change due to the aging and drying process over many years. The aging degree of the oil painting is taken into consideration, and the first spectral image is corrected so that the corrected second spectral image can take into account the aging degree of the oil painting.

[0108] S14. Determine the difference image between the first spectral image and the second spectral image, and perform a stitching operation on the difference image and the first spectral image to obtain the target spectral image.

[0109] In this embodiment of the application, the difference between the first spectral image and the second spectral image can reflect the influence of the aging degree of the oil painting on the spectral image.

[0110] In one implementation, when determining the difference image between the first spectral image and the second spectral image, the residual between the first spectral image and the second spectral image can be calculated to obtain the difference image.

[0111] In practice, for each pixel in the first spectral image, a corresponding pixel is selected from the second spectral image, and the residual difference in reflectance between the two pixels is calculated. The residual can be an absolute difference or a relative difference, depending on the actual configuration.

[0112] After calculating the residuals of each pixel, the residuals of each pixel are combined according to the pixel order to obtain the difference image. The spatial resolution and spectral dimension of this difference image are consistent with those of the first spectral image.

[0113] Then, a concatenation operation is performed on the difference image and the first spectral image using a convolutional layer to obtain a fused feature map. This fused feature map is the feature map obtained after image enhancement, and this feature map is the target spectral image in this embodiment.

[0114] In this embodiment of the application, the accuracy of the final oil paint classification is further improved by taking into account the physical properties of pigment aging and drying.

[0115] S15. Based on the target spectral image, perform a classification operation on the pigments in the target painting to obtain the pigment classification results.

[0116] In real-world scenarios, step S15 can be achieved using a DOE spectral image oil paint classification network, such as a DOE hyperspectral image oil paint classification network.

[0117] The target spectral image mentioned above is used as input to the DOE spectral image oil paint pigment classification network. After processing by the DOE spectral image oil paint pigment classification network, the pigment classification result of the oil painting can be obtained.

[0118] In one implementation, step S15 may include:

[0119] The target spectral image is processed using a pigment classification network to obtain the pigment classification results of the target painting.

[0120] Among them, the pigment classification network can be the DOE spectral image oil painting pigment classification network mentioned above, or it can be MLP (Multi-Layer Perceptron), etc.

[0121] In one implementation, such as Figure 4 As shown, the pigment classification network includes an input layer and multiple fully connected hidden layers (referred to as hidden layers). Figure 4 The diagram shows two hidden layers and an output classification layer (referred to as the output layer).

[0122] The input layer is used to receive the target spectral image.

[0123] A multi-layer fully connected hidden layer consists of multiple levels of fully connected hidden layers. The number of neurons in different fully connected hidden layers decreases exponentially. As the number of neurons decreases exponentially, the feature space is gradually compressed, forcing the network to capture higher-order statistical features (such as data cluster centers and class boundaries). A target activation function (such as ReLU activation) is configured in the fully connected hidden layer.

[0124] The output classification layer is a fully connected layer with the same number of nodes as the number of pigment types to be identified, and the Softmax activation function is used for probabilistic output.

[0125] After passing through the pigment classification network described above, the pigment classification results for oil paintings can be obtained. These results can be semantic segmentation maps, where each pixel independently predicts the pigment category, generating a pigment distribution map and labeling key mineral components (such as cinnabar and malachite).

[0126] In one implementation, the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation are first jointly optimized based on the loss information during training, and then local optimization is performed. Here, the modulation height distribution data of the optical unit in the spectral acquisition device are hardware parameters, while the decompression parameters used in the decompression operation and the classification parameters used in the classification operation are software parameters.

[0127] In practice, the total loss can be used to jointly optimize the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation. Subsequently, each loss can be used to optimize the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and the classification parameters used in the classification operation separately to obtain the optimal data.

[0128] In one example, to verify the effectiveness of the algorithm, such as Figure 5 and Figure 6 As shown, mineral pigment samples collected using a self-developed hyperspectral camera and a traditional Chinese freehand oil painting were used for pigment classification testing. The pigment samples and the oil painting are shown in the image. Figure 5 As shown, the classification results of gamboge pigment are as follows: Figure 6 As shown. By Figure 5 and Figure 6 The comparison shows that the classification result of gamboge pigment in the embodiments of this application is accurate.

[0129] In addition, this application embodiment collected and tested standard samples of mineral pigments and real mural images during the experimental phase. Using a self-developed DOE hyperspectral chip, the system can acquire 31 spectral bands in a single acquisition, with an average relative error of less than 10% in spectral reconstruction. The joint reconstruction and classification model achieved an overall classification accuracy of over 95% in pigment identification experiments.

[0130] In this embodiment, an initial spectral image of the target painting is acquired by a spectral acquisition device. This initial spectral image is then decompressed to obtain a first spectral image. Based on the aging degree of the target painting, the first spectral image is corrected to obtain a second spectral image. The difference between the first and second spectral images is determined, and the difference image and the first spectral image are stitched together to obtain the target spectral image. Based on the target spectral image, the pigments in the target painting are classified, thus achieving pigment classification. Furthermore, in this application, the correction operation on the first spectral image based on the aging degree of the target painting considers the influence of physical characteristics such as pigment aging and drying on pigment classification, further improving the accuracy of the final pigment classification. Additionally, in this application, the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation are first jointly optimized based on the loss information during training, and then locally optimized. This fully utilizes the advantages of process linkage, improving the final pigment classification accuracy and enabling accurate identification of pigments in complex artworks.

[0131] Based on any of the above embodiments, refer to Figure 7 Based on the aging degree of the target image, a correction operation is performed on the first spectral image to obtain a second spectral image, which may include:

[0132] S31. Based on the reflectance difference information of different pixels in the first spectral image, the first spectral image is divided into blocks to obtain multiple sub-images.

[0133] In this embodiment, since the reflectance data of the first spectral image is large, directly processing the first spectral image may result in low efficiency. Therefore, in this embodiment, the first spectral image can be divided into multiple sub-images. The reflectance data of the sub-images obtained after division is smaller than that of the first spectral image, resulting in higher efficiency in processing the sub-images.

[0134] When performing block segmentation, it can be done manually or automatically.

[0135] When performing automatic block segmentation, a physical knowledge base embedding unit can be used. The physical knowledge base embedding unit can be a lightweight, trainable embedding layer. The embedding layer is used to store a set of learnable parameters representing basic pigment optical properties (such as parameterized reflectance model coefficients, offsets characterizing the degree of aging). These parameters can be initialized using a small number of known pigment sample spectra and fine-tuned by gradient descent during the entire system training process to adapt to the actual application scenario.

[0136] The physical knowledge base embedding unit can be divided into blocks according to color, with different color regions divided into different sub-images, and regions with similar colors located within the same sub-image.

[0137] Color can be represented by reflectance, and different colors have different reflectance. Therefore, in this embodiment, based on the reflectance difference information of different pixels in the first spectral image, the parts of the image with similar reflectance can be divided into one sub-image, and the parts with large reflectance differences can be divided into another sub-image. Multiple sub-images can be obtained through this segmentation method.

[0138] S32. Using the wavelength change information of reflectance and the aging degree of the target image, the reflectance in the sub-image is corrected to obtain the corrected reflectance of each sub-image.

[0139] In real-world scenarios, the aging degree of the target drawing is a known parameter, such as aging for ten years, aging for twenty years, etc.

[0140] By analyzing the color aging of multiple oil paintings, commonalities in color aging can be identified. For example, these commonalities might include:

[0141] After a period of aging, such as ten years, the reflectance of a certain band (such as 700) in a hyperspectral image will change, such as changing from one shape to another. In one example, it may change from a straight line shape to a convex or concave shape. Therefore, the wavelength change information of reflectance can be corrected by using the aging degree of the target image, such as correcting a convex or concave shape to a straight line shape.

[0142] This correction operation yields the reflectance of each pixel in the painting before it ages, which is also the corrected reflectance of the sub-image.

[0143] In one implementation, to improve correction efficiency, a physical model prediction unit (PMU) can be constructed to perform the correction operation. The PMU can be a lightweight and fully differentiable (i.e., learnable) computational structure, such as a small neural network. The PMU can be trained using samples, which may include the aging degree of the oil painting, wavelength variations in reflectance, and the aforementioned basic pigment optical properties, enabling the PMU to learn the effect of aging on reflectance. Subsequently, by inputting the aging degree of the target painting and the wavelength variation information of reflectance into the PMU, the corrected reflectance of each sub-image can be obtained.

[0144] S33. Determine the second spectral image based on the corrected reflectance of each sub-image.

[0145] In this embodiment, the corrected reflectance of each sub-image is combined according to its position in the first spectral image to obtain the second spectral image.

[0146] It should be noted that when segmenting and combining sub-images, spatial location information needs to be combined. This spatial location information can be image coordinates. Each pixel in different spatial coordinates has a reflectance. After subsequent dynamic correction of reflectance, it is possible to know which spatial coordinate pixel has been corrected, and obtain the expected spectral reflectance of the pigment in the current spatial coordinate as either in the "theoretical state (i.e., not aged)" or after specific correction.

[0147] In practical scenarios, steps S31-S33 can be implemented using a physically guided spectral residual correction network. This network utilizes the known physical optical properties of the pigment (such as the basic reflectance model and typical aging or degradation modes) to construct a differentiable physical prediction model. By inputting the first spectral image into this model, the final second spectral image can be obtained.

[0148] In this embodiment, the prediction information of the physical prediction model can be deeply fused with the classification features of the first spectral image in the form of residuals. By taking into account the influence of physical properties such as pigment aging and drying on pigment classification, the accuracy of the final oil painting pigment classification is further improved.

[0149] Based on any of the above embodiments, refer to Figure 8 The process of determining the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and the classification parameters used in the classification operation may include:

[0150] S41. During training, calculate reconstruction loss and classification loss.

[0151] In this embodiment, the modulation height distribution data of the optical units in the DOE imaging chip, the decompression parameters in the DOE hyperspectral decompression network, and / or the classification parameters in the DOE hyperspectral image oil paint classification network are incorporated into the same training loop for collaborative optimization, forming a unified end-to-end hyperspectral task processing framework. During training, joint optimization is performed through a shared optimization objective, with the common goal of minimizing the loss.

[0152] Specifically, the DOE trainable diffraction code consists of multiple optical units, each whose phase modulation is determined by its height, which is a trainable variable. The DOE trainable diffraction code compresses and modulates the input scene to generate a single-channel compressed hybrid image (i.e., the initial spectral image mentioned above). The DOE hyperspectral decompression network reconstructs this image to obtain the first spectral image. The physically guided spectral residual correction network corrects the first spectral image based on the aging degree of the target painting to obtain the second spectral image. The difference between the first and second spectral images is determined, and the difference image and the first spectral image are stitched together to obtain the target spectral image. The DOE spectral image oil painting pigment classification network classifies the pigments in the target spectral image to obtain the pigment classification results. The overall process is as follows: Figure 9 As shown.

[0153] The system, composed of multiple networks, uses two loss functions during the training phase: The first is the reconstruction loss (Loss_recon), which calculates the mean squared error (MSE) between the reconstructed spectral image and the true spectral image in the dataset; the second is the classification loss (Loss_class), which measures the cross-entropy error between the classification result and the true pigment label. The final total loss function is defined as follows:

[0154] ;

[0155] in, For the total loss, and For adjustable hyperparameters, To rebuild the losses, For classification loss.

[0156] The calculation formula is:

[0157]

[0158] in, For batch size, For sample index, For true spectral images, To reconstruct the spectral image, i.e. the image output by the DOE hyperspectral decompression network.

[0159] The calculation formula is:

[0160]

[0161] in, For category indexing, This represents the total number of pigment categories. One-hot encoding for the real label, This represents the probability predicted by the model.

[0162] S42. Weight the reconstruction loss and classification loss to obtain the total loss.

[0163] The formula for calculating the total loss is explained in the above description.

[0164] S43. Using the total loss, determine the initial modulation height distribution data of the optical units in the spectral acquisition device, the initial decompression parameters used in the decompression operation, and / or the initial classification parameters used in the classification operation.

[0165] In specific optimization, the Adam (Adaptive Moment Estimation) optimizer can be used with an initial learning rate of 1×10⁻⁶. −4 This can be dynamically adjusted later.

[0166] In specific implementation, Under the combined drive of [various factors], the system can simultaneously optimize at least one of the following:

[0167] 1) The modulation height distribution of each optical unit in the DOE encoding structure achieves the most suitable spectral encoding for the oil painting scene;

[0168] 2) Configure the decompression parameters of the DOE hyperspectral decompression network to improve the accuracy of spectral image reconstruction;

[0169] 3) The classification parameters of the DOE spectral image oil paint classification network are configured to enhance the ability to identify fine-grained pigments.

[0170] In one embodiment, all three aspects mentioned above can be optimized simultaneously. This joint optimization mechanism breaks down the traditional barriers between front-end acquisition and back-end recognition, achieving coupling and collaboration between the hardware physical structure and the deep neural network. This allows the system to maintain stable recognition performance under complex conditions such as varying lighting, degradation, and noise. Compared to traditional decoding models, this module's structural design considers color fidelity and spatial consistency, making it particularly suitable for images of cultural relics with rich spectral levels and significant noise interference.

[0171] Through the above joint optimization, the initial modulation height distribution data of the optical unit in the spectral acquisition device, the initial decompression parameters used in the decompression operation, and / or the initial classification parameters used in the classification operation can be obtained.

[0172] S44. Using reconstruction loss, optimize the initial modulation height distribution data and / or initial decompression parameters to obtain the target modulation height distribution data and / or target decompression parameters.

[0173] In practical implementation, the three parts, in addition to being based on the overall... In addition to jointly optimizing parameters, each part also performs local parameter re-optimization based on its own loss. The DOE imaging chip's parameters are locally re-optimized based on the loss_recon, with the goal of reducing the loss_recon. The DOE hyperspectral decompression network also performs local re-optimization based on the loss_recon, with the goal of reducing the loss_recon.

[0174] After this local optimization operation, the target modulation height distribution data and target decompression parameters can be obtained.

[0175] S45. Optimize the initial classification parameters using the classification loss to obtain the target classification parameters.

[0176] Specifically, the DOE spectral image oil paint classification network will perform local re-optimization based on loss_class, with the goal of reducing loss_class and obtaining the target classification parameters.

[0177] It should be noted that the parameters in the physical-guided spectral residual correction network are pre-trained and do not participate in the joint optimization and local optimization operations mentioned above.

[0178] Experimental analysis shows that, compared with the traditional reconstruction-classification concatenation process, the joint network structure improves the recognition accuracy by about 12% while reducing the overall processing time to less than 5 seconds. The system size and power consumption are also significantly reduced, demonstrating good deployability and practical value.

[0179] In this embodiment, the DOE imaging chip microstructure parameters, DOE spectral image decompression algorithm parameters, and DOE spectral image oil paint classification algorithm parameters are first jointly optimized, and then locally optimized. This process, which utilizes the advantages of hardware-software process linkage and feature linkage, enhances the network's ability to perceive subtle pigment changes. It is particularly suitable for situations where the image color is severely faded, the pigment coverage is uneven, or there is little residue, and greatly improves the final oil paint classification effect.

[0180] In summary, the embodiments of this application provide an integrated deep neural network system for spectral image reconstruction and pigment classification. Through a self-developed DOE imaging chip and an end-to-end integrated network architecture, it completes unified modeling and optimization from original image acquisition, spectral reconstruction, correction to pigment classification, significantly improving the accuracy of pigment recognition and the practicality of the system.

[0181] In one implementation, the integrated deep neural network system for spectral image reconstruction and pigment classification needs to be trained using a training dataset.

[0182] When constructing the training dataset, hyperspectral data (real or simulated) of artworks such as oil paintings and murals can be collected, covering different pigments (such as ochre, ultramarine, lead white, etc.). Pixel-level pigment category labels are annotated on the hyperspectral data, and the true reflectance spectrum is recorded (for reconstruction loss calculation).

[0183] To increase the amount of data in the training dataset, data augmentation techniques can be used. These techniques simulate different lighting conditions (color temperature 2500K-10000K), noise levels (Gaussian + Poisson noise), and degradation (blurring, occlusion), thereby increasing the amount of data in the dataset.

[0184] Based on the embodiments of the above-described pigment classification method, another embodiment of this application provides a pigment classification device, referring to... Figure 10 It can include:

[0185] Image acquisition module 11 is used to acquire the initial spectral image of the target image acquired by the spectral acquisition device;

[0186] The decompression module 12 is used to decompress the initial spectral image to obtain the first spectral image;

[0187] The correction module 13 is used to perform a correction operation on the first spectral image based on the aging degree of the target image to obtain a second spectral image;

[0188] Image processing module 14 is used to determine the difference image between the first spectral image and the second spectral image, and to perform a stitching operation on the difference image and the first spectral image to obtain the target spectral image;

[0189] Classification module 15 is used to classify the pigments in the target painting based on the target spectral image to obtain the pigment classification results;

[0190] In one implementation, the decompression module 12 includes:

[0191] The extraction submodule is used to extract the first feature map of the initial spectral image using the input convolutional layer;

[0192] The compression submodule is used to progressively compress the size of the first feature map using multiple convolutional coding layers to obtain the second feature map; the number of feature channels in different convolutional coding layers increases layer by layer;

[0193] The recovery submodule is used to recover the spatial details of the second feature map layer by layer using multiple convolutional decoding layers to obtain the third feature map;

[0194] The image determination submodule is used to determine the first spectral image corresponding to the third feature map using the output layer.

[0195] In one implementation, the correction module 13 includes:

[0196] The block segmentation submodule is used to perform block segmentation on the first spectral image based on the reflectance difference information of different pixels in the first spectral image to obtain multiple sub-images;

[0197] The correction submodule is used to correct the reflectance in the sub-image by using the wavelength change information of reflectance and the aging degree of the target image, so as to obtain the corrected reflectance of each sub-image.

[0198] The image processing submodule is used to determine the second spectral image based on the corrected reflectance of each sub-image.

[0199] In one implementation, the image processing module 14 is specifically used for:

[0200] The residual between the first and second spectral images is calculated to obtain a difference image. A convolutional layer is then used to stitch the difference image and the first spectral image together to obtain the target spectral image.

[0201] One implementation also includes:

[0202] The parameter determination module is used to determine the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation.

[0203] The parameter determination module includes:

[0204] The loss calculation submodule is used to calculate the reconstruction loss and classification loss during training.

[0205] The weighted calculation submodule is used to calculate the total loss by weighting the reconstruction loss and the classification loss.

[0206] The joint optimization submodule is used to determine the initial modulation height distribution data of the optical unit in the spectral acquisition device, the initial decompression parameters used in the decompression operation, and / or the initial classification parameters used in the classification operation using the total loss.

[0207] The first optimization submodule is used to optimize the initial modulation height distribution data and / or the initial decompression parameters using the reconstruction loss to obtain target modulation height distribution data and / or target decompression parameters.

[0208] The second optimization submodule is used to optimize the initial classification parameters using the classification loss to obtain the target classification parameters.

[0209] In one implementation, the classification module 15 is specifically used for:

[0210] The target spectral image is processed using a pigment classification network to obtain the pigment classification results of the target painting;

[0211] The pigment classification network consists of an input layer, multiple fully connected hidden layers, and an output classification layer.

[0212] A multi-layer fully connected hidden layer consists of multiple levels of fully connected hidden layers; the number of neurons in different fully connected hidden layers decreases exponentially; and a target activation function is configured in the fully connected hidden layer.

[0213] One implementation also includes:

[0214] The preprocessing module is used to perform image preprocessing operations on the initial spectral image. The image preprocessing includes at least one of dark current suppression, white balance adjustment, and illumination compensation.

[0215] In this embodiment, an initial spectral image of the target painting is acquired by a spectral acquisition device. This initial spectral image is decompressed to obtain a first spectral image. Based on the aging degree of the target painting, the first spectral image is corrected to obtain a second spectral image. The difference between the first and second spectral images is determined, and the difference image and the first spectral image are stitched together to obtain the target spectral image. Based on the target spectral image, the pigments in the target painting are classified, thus achieving pigment classification. Furthermore, in this application, the correction operation on the first spectral image based on the aging degree of the target painting considers the influence of physical characteristics such as pigment aging and drying on pigment classification, further improving the accuracy of the final pigment classification. Additionally, in this application, the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation are first jointly optimized based on the loss information during training, and then locally optimized. This fully utilizes the advantages of process linkage, improving the final pigment classification accuracy.

[0216] It should be noted that the working process of each module and sub-module in the embodiments of this application is described in the corresponding descriptions in the above embodiments, and will not be repeated here.

[0217] This application also provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0218] Memory is used to store computer programs;

[0219] The processor is used to execute computer programs so that the electronic device can implement the pigment classification method described above.

[0220] refer to Figure 11 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0221] like Figure 11As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0222] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 11 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0223] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the pigment classification methods provided in this application.

[0224] This application also provides a computer-readable storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the pigment classification methods provided in this application.

[0225] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for classifying pigments, characterized in that, include: Acquire the initial spectral image of the target image acquired by the spectral acquisition device; The initial spectral image is decompressed to obtain the first spectral image; Based on the aging degree of the target image, the first spectral image is corrected to obtain the second spectral image; Determine the difference image between the first spectral image and the second spectral image, and perform a stitching operation on the difference image and the first spectral image to obtain the target spectral image; Based on the target spectral image, the pigments in the target painting are classified to obtain the pigment classification results; The step of correcting the first spectral image based on the aging degree of the target image to obtain the second spectral image includes: dividing the first spectral image into multiple sub-images based on the reflectance difference information of different pixels in the first spectral image; correcting the reflectance in the sub-images using the wavelength change information of reflectance and the aging degree of the target image to obtain the corrected reflectance of each sub-image; and determining the second spectral image based on the corrected reflectance of each sub-image. The step of determining the difference image between the first spectral image and the second spectral image includes: for each pixel in the first spectral image, selecting a corresponding pixel in the second spectral image, calculating the residual of the reflectance of the two pixels, and after calculating the residual of each pixel, combining the residuals of each pixel in the order of the pixels to obtain the difference image.

2. The pigment classification method according to claim 1, characterized in that, The initial spectral image is decompressed to obtain a first spectral image, including: The first feature map of the initial spectral image is extracted using the input convolutional layer; The size of the first feature map is gradually compressed using multiple convolutional coding layers to obtain the second feature map; the number of feature channels in different convolutional coding layers increases layer by layer. The spatial details of the second feature map are recovered layer by layer by using multiple convolutional decoding layers to obtain the third feature map; The first spectral image corresponding to the third feature map is determined using the output layer.

3. The pigment classification method according to claim 1, characterized in that, The target spectral image is obtained by stitching the difference image and the first spectral image together, including: The target spectral image is obtained by stitching the difference image and the first spectral image together using a convolutional layer.

4. The pigment classification method according to claim 1, characterized in that, The process of determining the modulation height distribution data of the optical unit in the spectral acquisition device, the decompression parameters used in the decompression operation, and / or the classification parameters used in the classification operation includes: During training, reconstruction loss and classification loss are calculated; The total loss is obtained by weighting the reconstruction loss and the classification loss. Using the total loss, the initial modulation height distribution data of the optical unit in the spectral acquisition device, the initial decompression parameters used in the decompression operation, and / or the initial classification parameters used in the classification operation are determined; Using the reconstruction loss, the initial modulation height distribution data and / or the initial decompression parameters are optimized to obtain the target modulation height distribution data and / or the target decompression parameters; The initial classification parameters are optimized using the classification loss to obtain the target classification parameters.

5. The pigment classification method according to claim 1, characterized in that, Based on the target spectral image, the pigments in the target painting are classified to obtain pigment classification results, including: The target spectral image is processed using a pigment classification network to obtain the pigment classification result of the target painting; The pigment classification network includes an input layer, multiple fully connected hidden layers, and an output classification layer. The multi-layer fully connected hidden layer includes multiple levels of fully connected hidden layers; the number of neurons in different fully connected hidden layers decreases exponentially; and a target activation function is configured in each fully connected hidden layer.

6. The pigment classification method according to claim 1, characterized in that, After acquiring the initial spectral image of the target image from the spectral acquisition device, the process also includes: The initial spectral image is subjected to image preprocessing operations, the image preprocessing including at least one of dark current suppression, white balance adjustment and illumination compensation.

7. A pigment sorting device, characterized in that, include: The image acquisition module is used to acquire the initial spectral image of the target image collected by the spectral acquisition device; The decompression module is used to decompress the initial spectral image to obtain the first spectral image; The correction module is used to perform a correction operation on the first spectral image based on the aging degree of the target image to obtain a second spectral image; The image processing module is used to determine the difference image between the first spectral image and the second spectral image, and to perform a stitching operation on the difference image and the first spectral image to obtain the target spectral image; The classification module is used to classify the pigments in the target painting based on the target spectral image to obtain the pigment classification result; Specifically, the correction module is used to divide the first spectral image into multiple sub-images based on the reflectance difference information of different pixels in the first spectral image; to correct the reflectance in the sub-images using the wavelength change information of reflectance and the aging degree of the target image, so as to obtain the corrected reflectance of each sub-image; and to determine the second spectral image based on the corrected reflectance of each sub-image. The step of determining the difference image between the first spectral image and the second spectral image includes: for each pixel in the first spectral image, selecting a corresponding pixel in the second spectral image, calculating the residual of the reflectance of the two pixels, and after calculating the residual of each pixel, combining the residuals of each pixel in the order of the pixels to obtain the difference image.

8. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the pigment classification method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the pigment classification method as described in any one of claims 1 to 6.

Images