Hyperspectral image reconstruction method and model training method

Abstract

The invention is suitable for the technical field of image processing, and provides a hyperspectral image reconstruction method and a model training method, and the method comprises the steps: obtaining a to-be-processed image which is an RGB image; performing image reconstruction on the to-be-processed image according to a detection model to obtain a reconstructed hyperspectral image; wherein the detection model comprises an encoder and a decoder, the encoder is used for extracting feature information of an image to be processed, and the decoder is used for performing feature fusion processing according to the feature information extracted by the encoder and reconstructing the image according to the fused feature information; the encoder comprises a first data processing unit constructed according to a spectral attention module and a space attention module; the decoder comprises a second data processing unit constructed according to a spectral attention module and a spatial attention module. The RGB image is reconstructed into the hyperspectral image by using the detection model in the application, and weak features in the image can be clearly presented.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a hyperspectral image reconstruction method and a model training method. Background Art

[0002] With the development of image processing technology, its application scope is becoming more and more extensive. For example, in the medical field, RGB images of human tissue can be taken to assist doctors in detecting or identifying lesions. However, for some weak features of human tissue, such as weak blood vessels, RGB images often cannot clearly present these human tissue information, which is not conducive to doctors observing these weak features. Summary of the invention

[0003] The embodiments of the present application provide a hyperspectral image reconstruction method and a model training method, and the obtained hyperspectral reconstructed image can present the weak features in the image more clearly.

[0004] In a first aspect, an embodiment of the present application provides a hyperspectral image reconstruction method, comprising:

[0005] Acquire an image to be processed, wherein the image to be processed is an RGB image;

[0006] Reconstructing the image to be processed according to the detection model to obtain a reconstructed hyperspectral image;

[0007] The detection model includes an encoder and a decoder, wherein the encoder is used to extract feature information of the image to be processed, and the decoder is used to perform feature fusion processing according to the feature information extracted by the encoder, and reconstruct the image according to the fused feature information;

[0008] The encoder includes a first data processing unit constructed according to a spectral attention module and a spatial attention module;

[0009] The decoder comprises second data processing units corresponding to the first data processing units in the encoder, each of the second data processing units comprising a spectral attention module and a spatial attention module;

[0010] The spectral attention module is used to process the spectral information in the image based on the attention mechanism, and the spatial attention module is used to process the spatial information in the image based on the attention mechanism.

[0011] In the embodiment of the present application, the detection model can better reconstruct the spectral curve of the image through learning in both spatial and spectral dimensions, which is beneficial to improving the reconstruction ability of the detection model; the decoder in the detection model can fuse the features of different visual field scales in the image, and make full use of the information of each level in the image, which is beneficial to improving the reconstruction ability of the detection model. The detection model in the present application has a high image reconstruction ability. By using the detection model in the present application to reconstruct the RGB image into a hyperspectral image, a clearer spectral reconstructed image can be obtained.

[0012] In a possible implementation manner of the first aspect, reconstructing the image to be processed according to the detection model to obtain a reconstructed hyperspectral image includes:

[0013] Performing feature extraction processing on the image to be processed by the first data processing unit to obtain first feature information of the image to be processed;

[0014] The second data processing unit performs feature fusion processing according to the first feature information to obtain a first fusion feature;

[0015] Image reconstruction is performed according to the first fusion feature to obtain a reconstructed hyperspectral image.

[0016] In the embodiment of the present application, during the decoder image reconstruction process, the feature information output by the encoder is taken into account, which can fully utilize the features in the image and help improve the reconstruction capability of the detection model.

[0017] In a possible implementation of the first aspect, the detection model further includes a bottleneck block, wherein the bottleneck block includes a third data processing unit constructed according to the spectral attention module and the spatial attention module;

[0018] The performing feature fusion processing according to the first feature information by the second data processing unit to obtain a first fusion feature includes:

[0019] Performing feature processing on the first feature information by the third data processing unit to obtain second feature information;

[0020] The second data processing unit performs feature fusion processing according to the first feature information and the second feature information to obtain the first fusion feature.

[0021] In the embodiment of the present application, by inserting a bottleneck block between the encoder and the decoder, the network is deepened (the number of network layers is increased), which can effectively reduce the network parameters and help improve the training efficiency; and in this way, the computational complexity of the large convolutional layer can be effectively reduced, thereby helping to improve the feature extraction capability of the model.

[0022] In a possible implementation manner of the first aspect, performing image reconstruction according to the first fusion feature to obtain a reconstructed hyperspectral image includes:

[0023] Image reconstruction is performed according to at least one feature of the first fusion features, the second feature information, and the image to be processed to obtain a reconstructed hyperspectral image.

[0024] In an embodiment of the present application, the decoder in the detection model can fuse features of different visual field scales in the image, make full use of information at all levels in the image, and help improve the reconstruction capability of the detection model.

[0025] In a possible implementation of the first aspect, the detection model is a trained model obtained by adjusting model parameters based on a reconstructed spectral image and an actual spectral image in sample data; the reconstructed spectral image is obtained by the second data processing unit performing image reconstruction according to a second fusion feature; the second fusion feature is obtained by the second data processing unit performing feature fusion processing according to third feature information; the third feature information is obtained by the first data processing unit performing feature extraction processing on an RGB image in the sample data; the RGB image and the actual spectral image are images obtained by photographing the same object by an RGB camera and a hyperspectral camera, respectively.

[0026] In a second aspect, an embodiment of the present application provides a model training method, which is applied to a detection model, wherein the detection model includes an encoder and a decoder, wherein the encoder includes a first data processing unit constructed according to a spectral attention module and a spatial attention module, and the decoder includes a second data processing unit corresponding to the first data processing unit in the encoder;

[0027] The method comprises:

[0028] Acquire sample data, where the sample data includes an RGB image and an actual spectrum image obtained by photographing the same object using an RGB camera and a hyperspectral camera respectively;

[0029] Performing feature extraction processing on the RGB image in the sample data by the first data processing unit to obtain third feature information;

[0030] Performing feature fusion processing according to the third feature information by the second data processing unit to obtain a second fusion feature;

[0031] Perform image reconstruction according to the second fusion feature to obtain a reconstructed spectral image;

[0032] The model parameters of the detection model are adjusted according to the reconstructed spectral image and the actual spectral image in the sample data until the trained detection model is obtained.

[0033] In the embodiment of the present application, the detection model can better reconstruct the spectral curve of the image through learning in both spatial and spectral dimensions, which is conducive to improving the reconstruction ability of the detection model. In addition, the decoder in the detection model can fuse the features of different visual field scales in the image, make full use of the information of each level in the image, and help improve the reconstruction ability of the detection model. Therefore, the detection model in the embodiment of the present application has a higher image reconstruction ability and can obtain a clearer spectral reconstructed image.

[0034] In a possible implementation manner of the second aspect, adjusting the model parameters of the detection model according to the reconstructed spectral image and the actual spectral image in the sample data until the trained detection model is obtained includes:

[0035] Calculating a target loss value, wherein the target loss value includes a first loss value and a second loss value, wherein the second loss value represents a loss value after amplifying an image difference corresponding to a target portion, and the image difference corresponding to the target portion is greater than the image difference corresponding to a non-target portion;

[0036] If the training condition is met, the current detection model is determined as the detection model after training;

[0037] If the training conditions are not met, the model parameters of the detection model are adjusted according to the target loss value, and the detection model is continuously trained according to the next sample data until the trained detection model is obtained.

[0038] In an embodiment of the present application, when calculating the loss value, the image error of the faint part of the image that is not easy to present clearly is magnified, so that the detection model can better learn the feature information of the faint part of the image, which is conducive to improving the image reconstruction effect.

[0039] In a possible implementation manner of the second aspect, calculating the target loss value includes:

[0040] Calculating a relative error between the reconstructed spectral image and the actual spectral image to obtain a first loss value;

[0041] According to the weight coefficient corresponding to each pixel block in the image, the error between the reconstructed spectral image and the actual spectral image is calculated to obtain a second loss value; wherein the weight coefficient corresponding to each pixel block in the image is positively correlated with the difference in the pixel value corresponding to the pixel block in the reconstructed spectral image and the actual spectral image;

[0042] The target loss value is calculated according to the first loss value and the second loss value.

[0043] In the embodiment of the present application, the first loss value can reflect the difference between the predicted spectral image and the real spectral image, and the second loss value can enhance the difference between the weak parts of the two. Therefore, the target loss value obtained according to the first loss value and the second loss value can not only reflect the overall credibility of the predicted spectral image, but also reflect the credibility of the weak parts of the predicted spectral image, which is conducive to improving the training effect of the model.

[0044] In a possible implementation manner of the second aspect, calculating the error between the reconstructed spectral image and the actual spectral image according to the weight coefficient corresponding to each pixel block in the image to obtain the second loss value includes:

[0045] According to the weight coefficients corresponding to the pixel blocks in the image, a weighted mean square error between the reconstructed spectral image and the actual spectral image is calculated to obtain the second loss value.

[0046] In the embodiment of the present application, the traditional mean square error is improved by adding weight coefficients of each pixel block to enhance the differences in weak parts, so that the detection model can better learn the feature information of weak parts in the image, which is beneficial to improving the image reconstruction effect.

[0047] In a possible implementation manner of the second aspect, the reconstructed spectral image includes N first pixel blocks, the actual spectral image includes N second pixel blocks, the i-th first pixel block corresponds to the i-th second pixel block, and N is an integer greater than 1;

[0048] The step of calculating the weighted mean square error between the reconstructed spectral image and the actual spectral image according to the weight coefficients corresponding to the respective pixel blocks in the image to obtain the second loss value includes:

[0049] Calculate the pth power of the difference between the ith first image block and the ith second image block to obtain a first value, where p is a positive integer;

[0050] Calculate the p-th power sum of the N first image blocks and the second image blocks corresponding to each of the N first image blocks to obtain a second value;

[0051] Calculate, according to the first value and the second value, a weight between the ith first image block and the ith second image block;

[0052] A weighted mean square error between the reconstructed spectral image and the actual spectral image is calculated according to a weight between the ith first image block and the ith second image block.

[0053] The above calculation method is equivalent to amplifying the difference of the weak part and increasing the weight of the difference of the weak part accordingly. In this way, the difference of the weak part is strengthened, so that the detection model can better learn the feature information of the weak part in the image, which is conducive to improving the image reconstruction effect.

[0054] In a possible implementation of the second aspect, the detection model further includes a bottleneck block, wherein the bottleneck block includes a third data processing unit constructed according to the spectral attention module and the spatial attention module;

[0055] The performing feature fusion processing according to the third feature information by the second data processing unit to obtain a second fusion feature includes:

[0056] Performing feature processing on the third feature information by the third data processing unit to obtain fourth feature information;

[0057] The second data processing unit performs feature fusion processing according to the third feature information and the fourth feature information to obtain the second fusion feature.

[0058] In the embodiment of the present application, during the image reconstruction process performed by the decoder, the feature information output by the encoder is taken into consideration, which can fully utilize the features in the image and help improve the reconstruction capability of the detection model.

[0059] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspect or the method described in any one of the second aspect is implemented.

[0060] In a fourth aspect, an embodiment of the present application provides an endoscope system, comprising a light source host, an image processing device as described in the third aspect, and a computer device as described in the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 It is a schematic diagram of the structure of the detection model provided in the embodiment of the present application;

[0063] Figure 2 is a structural schematic diagram of a detection model provided by another embodiment of the present application;

[0064] Figure 3 is a structural schematic diagram of a detection model provided by another embodiment of the present application;

[0065] Figure 4 is a structural schematic diagram of a detection model provided by another embodiment of the present application;

[0066] Figure 5 It is a flowchart of the model training method provided in the embodiment of the present application;

[0067] Figure 6 is a schematic diagram of a spectral attention module provided in an embodiment of the present application;

[0068] Figure 7 is a schematic diagram of a spatial attention module provided in an embodiment of the present application;

[0069] Figure 8 It is a flowchart of a hyperspectral image reconstruction method provided in an embodiment of the present application;

[0070] Fig. 9 is a schematic diagram of a reconstructed image provided by an embodiment of the present application;

[0071] Fig.10 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0073] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0074] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0075] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0076] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0077] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0078] With the development of image processing technology, its application scope is becoming more and more extensive. For example, in the medical field, RGB images of human tissue can be taken to assist doctors in detecting or identifying lesions. However, for some weak features of human tissue, such as weak blood vessels, RGB images often cannot clearly present these human tissue information, which is not conducive to doctors observing these weak features.

[0079] Based on this, an embodiment of the present application provides a hyperspectral image reconstruction method. In the embodiment of the present application, the detection model can better reconstruct the spectral curve of the image through learning in both spatial and spectral dimensions, which is beneficial to improving the reconstruction ability of the detection model; the decoder in the detection model can fuse the features of different visual field scales in the image, and make full use of the information at all levels in the image, which is beneficial to improving the reconstruction ability of the detection model. The detection model in the present application has a high image reconstruction capability. By using the detection model in the present application to reconstruct the RGB image into a hyperspectral image, a clearer spectral reconstructed image can be obtained.

[0080] For ease of explanation, the detection model involved in the embodiments of the present application is first introduced.

[0081] In the embodiment of the present application, the detection model includes an encoder and a decoder. The encoder is used to extract feature information of the image to be processed, and the decoder is used to perform feature fusion processing based on the feature information extracted by the encoder and reconstruct the image based on the fused feature information.

[0082] The encoder includes a first data processing unit consisting of a spectral attention module and a spatial attention module. The decoder includes second data processing units corresponding to the first data processing units in the encoder, and each second data processing unit includes a spectral attention module and a spatial attention module.

[0083] As an example, see Figure 1 , is a schematic diagram of the structure of the detection model provided in the embodiment of the present application. Figure 1 As shown, the detection model includes an encoder 11 and a decoder 12. The encoder 11 includes a first data processing unit 111, and the decoder includes a second data processing unit 121 and a fusion unit 123. The input end of the first data processing unit 111 is the input end of the detection model, and the output end of the first data processing unit 111 is connected to the input end of the second data processing unit 121. The input end of the detection model, the output end of the first data processing unit 111, and the output end of the second data processing unit 121 are respectively connected to the input end of the fusion unit 123, and the output end of the fusion unit 123 is the output end of the detection model.

[0084] The fusion unit 123 is used to fuse the input image to be processed, the features output by the first data processing unit 111 and the features output by the second data processing unit 121, reconstruct the image according to the fused feature information, and output a hyperspectral reconstructed image.

[0085] As another example, see Figure 2 , is a schematic diagram of the structure of a detection model provided by another embodiment of the present application. Figure 2 As shown, the detection model includes an encoder 11 and a decoder 12. The encoder 11 includes a first data processing unit 111 and a first data processing unit 112. The decoder 12 includes a second data processing unit 121 corresponding to the first data processing unit 111, a second data processing unit 122 corresponding to the first data processing unit 112, and a fusion unit 123.

[0086] The input end of the first data processing unit 111 is the input end of the detection model, and the output end of the first data processing unit 111 is connected to the input end of the first data processing unit 112 and the input end of the second data processing unit 121. The output end of the first data processing unit 112 is connected to the input end of the second data processing unit 122. The output end of the second data processing unit 122 is connected to the input end of the second data processing unit 121. The input end of the detection model, the output end of the first data processing unit 112, the output end of the second data processing unit 121, and the output end of the second data processing unit 122 are respectively connected to the input end of the fusion unit 123. The output end of the fusion unit 123 is the output end of the detection model.

[0087] The fusion unit 123 is used to fuse the input image to be processed, the features output by the first data processing unit 112, the features output by the second data processing unit 121, and the features output by the second data processing unit 122, and reconstruct the image according to the fused feature information to output a hyperspectral reconstructed image.

[0088] In the embodiment of the present application, the spectral attention module is used to process the spectral information in the image based on the attention mechanism, and the spatial attention module is used to process the spatial information in the image based on the attention mechanism. The detection model can better reconstruct the spectral curve of the image through learning in both spatial and spectral dimensions, which is beneficial to improving the reconstruction ability of the detection model. In addition, the decoder in the detection model can fuse the features of different visual field scales in the image, make full use of the information at all levels in the image, and help improve the reconstruction ability of the detection model. Therefore, the detection model in the embodiment of the present application has a higher image reconstruction ability and can obtain a clearer spectral reconstructed image.

[0089] In one embodiment, the detection model further includes a bottleneck block, which includes a third data processing unit constructed according to the spectral attention module and the spatial attention module.

[0090] As an example, see Figure 3 , is a schematic diagram of the structure of a detection model provided by another embodiment of the present application. Figure 3As shown, the detection model includes an encoder 11, a decoder 12 and a bottleneck block 13. The encoder 11 includes a first data processing unit 111, the decoder includes a second data processing unit 121 and a fusion unit 123, and the bottleneck block 13 includes a third data processing unit 131. The input end of the first data processing unit 111 is the input end of the detection model, and the output end of the first data processing unit 111 is connected to the input end of the third data processing unit 131 and the input end of the second data processing unit 121 respectively. The output end of the third data processing unit 131 is connected to the input end of the second data processing unit 121. The input end of the detection model, the output end of the second data processing unit 121 and the output end of the third data processing unit 131 are respectively connected to the input end of the fusion unit 123, and the output end of the fusion unit 123 is the output end of the detection model.

[0091] The fusion unit 123 is used to fuse the input image to be processed, the features output by the second data processing unit 121 and the features output by the third data processing unit 131, reconstruct the image according to the fused feature information, and output a hyperspectral reconstructed image.

[0092] As another example, see Figure 4 , is a schematic diagram of the structure of a detection model provided by another embodiment of the present application. Figure 4 As shown, the detection model includes an encoder 11, a decoder 12 and a bottleneck block 13. The encoder 11 includes a first data processing unit 111 and a first data processing unit 112. The decoder 12 includes a second data processing unit 121 corresponding to the first data processing unit 111, a second data processing unit 122 corresponding to the first data processing unit 112, and a fusion unit 123. The bottleneck block 13 includes a third data processing unit 131.

[0093] The input end of the first data processing unit 111 is the input end of the detection model, and the output end of the first data processing unit 111 is connected to the input end of the first data processing unit 112 and the input end of the second data processing unit 121 respectively. The output end of the first data processing unit 112 is connected to the input end of the second data processing unit 122 and the input end of the third data processing unit 131 respectively. The output end of the third data processing unit 131 is connected to the input end of the second data processing unit 122. The output end of the second data processing unit 122 is connected to the input end of the second data processing unit 121. The input end of the detection model, the output end of the second data processing unit 121, the output end of the second data processing unit 122, and the output end of the third data processing unit 131 are connected to the input end of the fusion unit 123 respectively. The output end of the fusion unit 123 is the output end of the detection model.

[0094] The fusion unit 123 is used to fuse the input image to be processed, the features output by the second data processing unit 121, the features output by the second data processing unit 122, and the features output by the third data processing unit 131, and reconstruct the image according to the fused feature information to output a hyperspectral reconstructed image.

[0095] In the embodiment of the present application, by inserting a bottleneck block between the encoder and the decoder, the network is deepened (the number of network layers is increased), which can effectively reduce the network parameters and help improve the training efficiency; and in this way, the computational complexity of the large convolutional layer can be effectively reduced, thereby helping to improve the feature extraction capability of the model.

[0096] It should be noted that the embodiment of the present application does not specifically limit the number of the first data processing unit, the second data processing unit, and the third data processing unit.

[0097] Based on the above detection model, the following introduces the model training method of the detection model. Figure 5 , is a flow chart of the model training method provided in the embodiment of the present application. As an example but not a limitation, the method may include the following steps:

[0098] S501, obtaining sample data.

[0099] The sample data includes RGB images and actual spectral images obtained by photographing the same object with an RGB camera and a hyperspectral camera respectively.

[0100] In some implementations, multiple sets of collected data may be obtained, each set of collected data including an RGB image and an actual spectral image obtained by photographing the same object using an RGB camera and a hyperspectral camera, respectively. The multiple sets of collected data are then divided into a training set and a test set. The collected data in the training set are the sample data described above, and the collected data in the test set are the test data. The sample data are used to train the detection model, and the test data are used to test the reconstruction capability of the trained detection model.

[0101] To facilitate training, in some implementations, the RGB images and actual spectral images in the sample data may be cropped to a uniform size, such as N×N.

[0102] S502: Perform feature extraction processing on the RGB image in the sample data by the first data processing unit to obtain third feature information.

[0103] In some embodiments, see Figure 6 , is a schematic diagram of a spectral attention module provided in an embodiment of the present application. Figure 6 As shown, the data processing steps of the spectral attention module may include:

[0104] 6-1. Reshape operation: Reshape the input size to X∈R H×W×C Reshape the feature cube into a shape of X∈R (HW)×C The feature cube of . H, W, and C are the length, width, and number of channels of the image, respectively. The Reshape operation is used to change the dimension and shape of the array without changing the elements themselves.

[0105] 6-2. Linear mapping transformation linear: transform the size X∈R (HW)×C The characteristic cube map of is the matrix Q∈R (HW)×C , K∈R (HW)×C and V∈R (HW)×C Where: Q = XW Q , K = XW K 、V=XW V , W Q , W K and W V To learn the parameters.

[0106] 6-3. Divide the matrices Q, K, and V into multi-head self-attention: Q = [Q1, Q2…Q t ], K=[K1,K2…K t ]、V=[V1,V2…V t ]. Where t is the number of attention heads.

[0107] 6-4. Calculate spectral attention: Among them, softmax is the normalization function, is the learning parameter, and each head attention is Head j =V j Attention j .

[0108] 6-5. Multi-head attention fusion Attention:

[0109] Attention=concat(Head1,Head2…Head j )W+f(V). Among them, concat is the fusion operation, W is the learning parameter, and f is the convolution operation.

[0110] After the multi-head attention fusion is obtained, the Attention can be reshaped and the output size is X∈R H×W×C The characteristic cube.

[0111] It can be seen that the above spectral attention module divides the input image into multiple attention heads, calculates the respective spectral attention heads, and fuses the multiple attention heads. In this way, the spectral details of the image can be better preserved while suppressing redundant information.

[0112] For some implementations, see Figure 7 , is a schematic diagram of a spatial attention module provided in an embodiment of the present application. Figure 7 As shown, the data processing steps of the spatial attention module may include:

[0113] 7-1. Pooling operation: for input size X∈R H×W×C The feature cube of is averaged pooled along the channel dimension to obtain spatial attention MeanPool∈R H×W×1 , the maximum pooling along the channel dimension obtains spatial attention AvgPool∈R H×W×1 .

[0114] 7-2. Attention splicing: Splice the spatial attention after average pooling and maximum pooling to get Spa_ Att =[MeanPool,AvgPool]∈R H×W×2 .

[0115] 7-3. Convolution operation Conv: Perform a convolution operation on the concatenated attention to obtain Spa_ Att =f(Spa_ Att )∈R H×W×1 . Where f is the convolution operation.

[0116] 7-4. Activation processing Sigmoid: Use the sigmoid function to calculate the weight and get the spatial weight value Spa_ w =sigmoid(Spa_ Att )∈R H×W×1 .

[0117] 7-5. Fusion processing: Spatial weight value Spa_ w With the original feature (size X∈R H×W×C The characteristic cube of out =Spa_ w *X∈R H×W×C .

[0118] It can be seen that the above-mentioned spatial attention module assigns different spatial weights to different positions of the input image, which is equivalent to adjusting the attention of each position in the feature map, thereby focusing on areas that deserve more attention and weakening useless features. In this way, the representation ability of local features in the image can be enhanced.

[0119] It should be noted that the spectral attention module and the spatial attention module in the first data processing unit can be processed in parallel or in series. If parallel processing is performed, the first data processing unit fuses the data output by the spectral attention module and the spatial attention module to obtain the third feature information. If serial processing is performed, such as the spatial attention module is connected in series after the spectral attention, the data output by the spatial attention module is the third feature information; such as the spectral attention module is connected in series after the spatial attention module, the data output by the spectral attention module is the third feature information.

[0120] In addition, the data processing steps of the spectral attention module in the second data processing unit and the third data processing unit are the same as those of the spectral attention module in the first data processing unit, and the data processing steps of the spatial attention module in the second data processing unit and the third data processing unit are the same as those of the spatial attention module in the first data processing unit. The spectral attention module and the spatial attention module in the second data processing unit and the third data processing unit will not be described in detail below.

[0121] In some implementations, two adjacent first data processing units may be connected via a downsampling operation. Figure 4 Taking the detection model shown as an example, the output data of the first data processing unit 111 is downsampled, and the processed data is input into the first data processing unit 112. In this way, different first data processing units correspond to different image scales, which is equivalent to extracting feature information of different scales in the image, which is conducive to improving the image reconstruction effect.

[0122] S503: Perform feature fusion processing according to the third feature information by the second data processing unit to obtain a second fusion feature.

[0123] In some embodiments, the detection model does not include a bottleneck block. Accordingly, S503 may be implemented as follows: the second data processing unit performs feature fusion processing according to the third feature information output by the corresponding first data processing unit to obtain a second fusion feature.

[0124] by Figure 1 Taking the detection model shown as an example, the second data processing unit 121 performs feature fusion processing according to the feature information E1 (third feature information) output by the first data processing unit 111 to obtain feature E2 (second fusion feature).

[0125] by Figure 2Taking the detection model shown as an example, the second data processing unit 122 performs feature fusion processing based on the feature information E1 output by the first data processing unit 112 to obtain feature information E4 (second fused feature); the second data processing unit 121 then performs feature fusion processing based on the feature information E1 output by the first data processing unit 111 and the feature information E4 output by the second data processing unit 122, and outputs feature information E2 (second fused feature).

[0126] In some other embodiments, the detection model includes a bottleneck block, and accordingly, S503 may be implemented by: performing feature processing on the third feature information by a third data processing unit to obtain fourth feature information; and performing feature fusion processing on the third feature information and the fourth feature information by a second data processing unit to obtain a second fusion feature.

[0127] by Figure 3 Taking the detection model shown as an example, the third data processing unit 131 performs feature processing on the feature information E1 (third feature information) output by the first data processing unit 111 to obtain feature information E5 (fourth feature information); the second data processing unit 121 performs feature fusion processing based on the feature information E1 and the feature information E5 to obtain feature information E2 (second fusion feature).

[0128] by Figure 4 Taking the detection model shown as an example, the third data processing unit 131 performs feature processing on the feature information E2 (third feature information) output by the first data processing unit 121 to obtain feature information E5 (fourth feature information); the second data processing unit 122 performs feature fusion processing based on the feature information E2 and the feature information E5 to obtain feature information E4 (second fused feature); the second data processing unit 121 performs feature fusion processing based on the feature information E4 and the feature information E1 to obtain feature information E2 (second fused feature).

[0129] In some implementations, two adjacent second data processing units may be connected via an upsampling operation. Figure 4 Taking the detection model shown as an example, the output data E4 of the second data processing unit 122 is upsampled, and the processed data is input into the second data processing unit 121. In this way, different second data processing units correspond to different image scales, and each second data processing unit has the same image scale as its corresponding first data processing unit, which is equivalent to reconstructing the image using feature information of different scales in the image, which is conducive to improving the image reconstruction effect.

[0130] S504: Reconstruct the image according to the second fusion feature to obtain a reconstructed spectral image.

[0131] In the embodiment of the present application, S504 may be executed by a fusion unit in the decoder.

[0132] In some embodiments, the detection model does not include a bottleneck block. Accordingly, S504 may include: performing image reconstruction according to at least one feature third feature information in the second fusion feature and an RGB image in the sample image to obtain a reconstructed hyperspectral image.

[0133] by Figure 1 Taking the detection model shown in FIG. 1 as an example, the fusion unit 123 performs image reconstruction based on the feature information E1 (third feature information) output by the first data processing unit 111, the feature information E2 (second fusion feature) output by the second data processing unit 121, and the RGB image in the input sample image to obtain a reconstructed hyperspectral image. Specifically, the feature information E1 can be upsampled to the same size as the input image, and then the upsampled image is superimposed with the feature information E2 and the RGB image in the sample image, and then a convolution operation is performed to obtain a reconstructed hyperspectral image.

[0134] by Figure 2 Taking the detection model shown as an example, the fusion unit 123 performs image reconstruction based on at least one of the feature information E2 (second fusion feature) output by the second data processing unit 121 and the feature information E4 (second fusion feature) output by the second data processing unit 122, the feature information E3 (third feature information) output by the first data processing unit 112, and the RGB image in the sample image to obtain a reconstructed hyperspectral image. Specifically, the feature information E3 and the feature information E4 can be upsampled to the same size as the input image, and then the upsampled image is superimposed with the feature information E2 and the RGB image in the sample image, and then the reconstructed hyperspectral image is obtained through a convolution operation.

[0135] In some other embodiments, the detection model includes a bottleneck block, and accordingly, S504 may include: performing image reconstruction according to at least one feature in the second fusion feature, the fourth feature information, and the RGB image in the sample image to obtain a reconstructed hyperspectral image.

[0136] by Figure 3 Taking the detection model shown in FIG. 1 as an example, the fusion unit 123 performs image reconstruction based on the feature information E5 (fourth feature information) output by the third data processing unit 131, the RGB image in the sample image, and the feature information E2 (second fusion feature) output by the second data processing unit 121 to obtain a reconstructed hyperspectral image. Specifically, the feature information E5 can be upsampled to the same size as the input image, and then the upsampled image is superimposed with the feature information E2 and the RGB image in the sample image, and then a convolution operation is performed to obtain a reconstructed hyperspectral image.

[0137] by Figure 4 Taking the detection model shown in FIG. 1 as an example, the fusion unit 123 performs image reconstruction based on any one of the feature information E2 (second fusion feature) output by the second data processing unit 121 and the feature information E4 output by the second data processing unit 122, the feature information E5 (fourth feature information) output by the third data processing unit 131, and the RGB image in the sample image to obtain a reconstructed hyperspectral image. Specifically, the feature information E4 and the feature information E5 can be upsampled to the same size as the input image, and then the upsampled image is superimposed with the feature information E2 and the RGB image in the sample image, and then a convolution operation is performed to obtain a reconstructed hyperspectral image.

[0138] In the embodiment of the present application, different weights may be assigned to different feature information during the superposition process. For example, feature information that has a greater impact on the final imaging may be assigned a greater weight.

[0139] S505, adjusting the model parameters of the detection model according to the reconstructed spectral image and the actual spectral image in the sample data until a trained detection model is obtained.

[0140] In some embodiments, S505 may include:

[0141] Calculating a target loss value, wherein the target loss value includes a first loss value and a second loss value, wherein the second loss value represents a loss value after amplifying an image difference corresponding to a target portion, and the image difference corresponding to the target portion is greater than the image difference corresponding to a non-target portion;

[0142] If the training condition is met, the current detection model is determined as the detection model after training;

[0143] If the training conditions are not met, the model parameters of the detection model are adjusted according to the target loss value, and the detection model is continuously trained according to the next sample data until the trained detection model is obtained.

[0144] For example, for RGB images of brain tissue, the weak blood vessels may not be clearly presented. Therefore, in the early stage of model training, the image difference corresponding to the weak blood vessels in the RGB image may be large, and the weak blood vessels in the RGB image are recorded as the target part, and the part other than the weak blood vessels in the RGB image is the non-target part.

[0145] In an embodiment of the present application, when calculating the loss value, the image error of the faint part of the image that is not easy to present clearly is magnified, so that the detection model can better learn the feature information of the faint part of the image, which is conducive to improving the image reconstruction effect.

[0146] One way to calculate the target loss value includes:

[0147] 1. Calculate the relative error between the reconstructed spectral image and the actual spectral image to obtain the first loss value.

[0148] The relative error is calculated by calculating the ratio of the difference between the predicted result and the true result to the true result, which can reflect the credibility of the reconstructed spectral image.

[0149] For example, the formula Calculate the first loss value. Where L r Represents the first loss value, y i represents the actual spectral value in the actual spectral image, represents the predicted spectral value in the reconstructed spectral image, and N is the number of pixel blocks.

[0150] 2. According to the weight coefficients corresponding to each pixel block in the image, the error between the reconstructed spectral image and the actual spectral image is calculated to obtain the second loss value.

[0151] The weight coefficient corresponding to each pixel block in the image is positively correlated with the difference in pixel values ​​corresponding to the pixel block in the reconstructed spectral image and the actual spectral image.

[0152] The second loss value can reflect the image difference of the faint part in the image.

[0153] In some implementations, the second loss value is calculated by calculating a weighted mean square error between the reconstructed spectral image and the actual spectral image according to weight coefficients corresponding to each pixel block in the image to obtain the second loss value.

[0154] The mean square error is a method of averaging the entire data set by summing the squares of the differences between the predicted value and the actual value. The smaller the mean square error, the more accurate the prediction result.

[0155] In the embodiment of the present application, the traditional mean square error is improved by adding weight coefficients of each pixel block to enhance the differences in weak parts, so that the detection model can better learn the feature information of weak parts in the image, which is beneficial to improving the image reconstruction effect.

[0156] As described above, the sample is divided into a size of N×N, then the reconstructed spectrum image includes N first pixel blocks, the actual spectrum image includes N second pixel blocks, the i-th first pixel block corresponds to the i-th second pixel block, and N is an integer greater than 1.

[0157] Specifically, the weighted mean square error is calculated as follows:

[0158] Calculate the p-th power of the difference between the i-th first image block and the i-th second image block to obtain a first value, where p is a positive integer;

[0159] Calculate the p-th power sum of the N first image blocks and the second image blocks corresponding to each other to obtain a second value;

[0160] Calculate the weight between the i-th first image block and the i-th second image block according to the first value and the second value;

[0161] According to the weight between the i-th first image block and the i-th second image block, a weighted mean square error between the reconstructed spectral image and the actual spectral image is calculated.

[0162] For example, by formula Calculate the weighted mean square error. The weight between the i-th first image block and the i-th second image block is P is greater than 1.

[0163] The above calculation method is equivalent to amplifying the difference of the weak part and increasing the weight of the difference of the weak part accordingly. In this way, the difference of the weak part is strengthened, so that the detection model can better learn the feature information of the weak part in the image, which is conducive to improving the image reconstruction effect.

[0164] 3. Calculate the target loss value according to the first loss value and the second loss value.

[0165] In some embodiments, the first loss value and the second loss value may be directly added or weightedly summed to obtain a target loss value.

[0166] For example, the target loss value is:

[0167]

[0168] In the embodiment of the present application, the first loss value can reflect the difference between the predicted spectral image and the real spectral image, and the second loss value can enhance the difference between the weak parts of the two. Therefore, the target loss value obtained according to the first loss value and the second loss value can not only reflect the overall credibility of the predicted spectral image, but also reflect the credibility of the weak parts of the predicted spectral image, which is conducive to improving the training effect of the model.

[0169] It is understandable that in the early stage of training, the predicted spectral image may be quite different from the real spectral image as a whole. At this time, the second loss value tends to reflect the difference in the non-faint part of the image. As the number of training iterations increases, the overall difference between the predicted spectral image and the real spectral image becomes smaller and smaller. At this time, the second loss value tends to reflect the difference in the faint part of the image. Therefore, as the number of training iterations increases, the spectral image predicted by the detection model can be closer and closer to the real spectral image, both in terms of the overall image and the faint details.

[0170] Based on the above detection model and its training method, the reconstruction method of hyperspectral image is introduced below. Figure 8 , is a flow chart of a hyperspectral image reconstruction method provided in an embodiment of the present application. As an example but not a limitation, the method may include the following steps:

[0171] S801, obtaining an image to be processed, wherein the image to be processed is an RGB image.

[0172] S802: reconstruct the image to be processed according to the detection model to obtain a reconstructed hyperspectral image.

[0173] In some embodiments, S802 may include:

[0174] Performing feature extraction processing on the image to be processed by the first data processing unit to obtain first feature information of the image to be processed;

[0175] Performing feature fusion processing according to the first feature information by the second data processing unit to obtain a first fusion feature;

[0176] The image is reconstructed according to the first fusion feature to obtain a reconstructed hyperspectral image.

[0177] In some implementations, the detection model does not include a bottleneck block. Accordingly, a method for acquiring the first fusion feature includes: a second data processing unit performs feature fusion processing according to first feature information output by a first data processing unit corresponding thereto to obtain a first fusion feature.

[0178] This implementation is similar to the implementation in S503. Figure 1 and Figure 2 The fusion process described as an example is the same, and the details can be found in the description in the aforementioned embodiment, which will not be repeated here.

[0179] In some implementations, the detection model includes a bottleneck block, and accordingly, the method for obtaining the first fusion feature includes: performing feature processing on the first feature information through the third data processing unit to obtain the second feature information; performing feature fusion processing according to the first feature information and the second feature information through the second data processing unit to obtain the first fusion feature.

[0180] This implementation is similar to the implementation in S503. Figure 3 and Figure 4 The fusion process described as an example is the same, and the details can be found in the description in the aforementioned embodiment, which will not be repeated here.

[0181] In some implementations, the step of reconstructing the image according to the first fusion feature may include:

[0182] Image reconstruction is performed according to at least one feature of the first fusion features, the second feature information, and the image to be processed to obtain a reconstructed hyperspectral image.

[0183] This implementation method is the same as the method of performing image reconstruction according to the second fusion feature in S504. For details, please refer to the description in the S504 embodiment, which will not be repeated here.

[0184] Below Figure 4 The detection model shown in FIG. 1 is used as an example to illustrate the reconstruction process of the hyperspectral image. The image to be processed (RGB image) is input into the detection model: first, it passes through the first data processing unit 111 to obtain the feature information E1∈R H×W×C ; Downsample the feature information E1 and input it into the first data processing unit 112 to obtain the feature information The characteristic information E3 is downsampled and input into the third data processing unit 131 to obtain the characteristic information The characteristic information E5 is combined with the characteristic information E2 and input into the second data processing unit 122 to obtain the characteristic information The characteristic information E4 is merged with the characteristic information E1 and input into the second data unit 121 to obtain the characteristic information E2∈R H×W×C ; Upsample the feature information E4 and E5 to the size of the image to be processed, and then superimpose them with E2 and the image to be processed to obtain a superimposed image; perform a convolution operation on the superimposed image to obtain a reconstructed hyperspectral image.

[0185] For example, see Fig. 9 , is a schematic diagram of a reconstructed image provided in an embodiment of the present application. Fig. 9 As shown in (a) in , it is the image to be processed; Fig. 9 (b) in the figure shows the reconstructed hyperspectral image. Fig. 9It can be seen that the faint blood vessel part 92 in the reconstructed hyperspectral image is clearer than the faint blood vessel part 91 in the image to be processed.

[0186] In the embodiment of the present application, the detection model can better reconstruct the spectral curve of the image through learning in both spatial and spectral dimensions, which is conducive to improving the reconstruction ability of the detection model. In addition, the decoder in the detection model can fuse the features of different visual field scales in the image, make full use of the information of each level in the image, and help improve the reconstruction ability of the detection model. Therefore, the detection model in the embodiment of the present application has a higher image reconstruction ability and can obtain a clearer spectral reconstructed image.

[0187] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0188] Fig.10 Schematic diagram of the structure of the computer device provided in the embodiment of the present application. Fig.10 As shown, the computer device 10 of this embodiment includes: at least one processor 100 ( Fig.10 Only one is shown in the figure) a processor, a memory 101, and a computer program 102 stored in the memory 101 and executable on the at least one processor 100, when the processor 100 executes the computer program 102, the steps in the above-mentioned hyperspectral image reconstruction method embodiment or the steps in the above-mentioned model training method embodiment are implemented.

[0189] The computer device 10 may be a desktop computer, a notebook, a PDA, a cloud server or other computing device. The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Fig.10 This is only an example of the computer device 10 and does not constitute a limitation on the computer device 10 . The computer device 10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0190] It is understandable that when the computer device 10 is used to implement the steps in the above-mentioned hyperspectral image reconstruction method embodiment, the computer device 10 may be an image processing device in an endoscope system. When the computer device 10 is used to implement the steps in the above-mentioned model training method embodiment, the computer device 10 may be a server for communicating with an image processing device in an endoscope system.

[0191] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0192] In some embodiments, the memory 101 may be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 101 may also be an external storage device of the computer device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 10. Further, the memory 101 may also include both an internal storage unit of the computer device 10 and an external storage device. The memory 101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0193] An embodiment of the present application also provides an endoscope system, including a light source host, an endoscope, and the above-mentioned computer device.

[0194] It is understandable that, in practical applications, the above-mentioned image processing device and computer device may be separately provided or integrated into a terminal device, which is not specifically limited in the embodiments of the present application.

[0195] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0196] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0198] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0199] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0200] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A hyperspectral image reconstruction method, It is characterized in that include: Acquire an image to be processed, wherein the image to be processed is an RGB image; Reconstructing the image to be processed according to the detection model to obtain a reconstructed hyperspectral image; The detection model includes an encoder and a decoder, wherein the encoder is used to extract feature information of the image to be processed, and the decoder is used to perform feature fusion processing according to the feature information extracted by the encoder, and reconstruct the image according to the fused feature information; The encoder includes a first data processing unit constructed according to a spectral attention module and a spatial attention module; The decoder comprises second data processing units corresponding to the first data processing units in the encoder, each of the second data processing units comprising a spectral attention module and a spatial attention module; The spectral attention module is used to process the spectral information in the image based on the attention mechanism, and the spatial attention module is used to process the spatial information in the image based on the attention mechanism.

2. The hyperspectral image reconstruction method according to claim 1, It is characterized in that The step of reconstructing the image to be processed according to the detection model to obtain a reconstructed hyperspectral image includes: Performing feature extraction processing on the image to be processed by the first data processing unit to obtain first feature information of the image to be processed; The second data processing unit performs feature fusion processing according to the first feature information to obtain a first fusion feature; Image reconstruction is performed according to the first fusion feature to obtain a reconstructed hyperspectral image.

3. The hyperspectral image reconstruction method according to claim 2, It is characterized in that The detection model also includes a bottleneck block, which includes a third data processing unit constructed according to the spectral attention module and the spatial attention module; The performing feature fusion processing according to the first feature information by the second data processing unit to obtain a first fusion feature includes: Performing feature processing on the first feature information by the third data processing unit to obtain second feature information; The second data processing unit performs feature fusion processing according to the first feature information and the second feature information to obtain the first fusion feature.

4. The hyperspectral image reconstruction method according to claim 3, It is characterized in that The performing image reconstruction according to the first fusion feature to obtain a reconstructed hyperspectral image includes: Image reconstruction is performed according to at least one feature of the first fusion features, the second feature information, and the image to be processed to obtain a reconstructed hyperspectral image.

5. The hyperspectral image reconstruction method according to claim 1, It is characterized in that The detection model is a trained model obtained by adjusting model parameters based on the reconstructed spectral image and the actual spectral image in the sample data; the reconstructed spectral image is obtained by the second data processing unit performing image reconstruction according to the second fusion feature; the second fusion feature is obtained by the second data processing unit performing feature fusion processing according to the third feature information; the third feature information is obtained by the first data processing unit performing feature extraction processing on the RGB image in the sample data; the RGB image and the actual spectral image are images obtained by photographing the same object by an RGB camera and a hyperspectral camera respectively.

6. A model training method, It is characterized in that Applied to a detection model, the detection model includes an encoder and a decoder, the encoder includes a first data processing unit constructed according to a spectral attention module and a spatial attention module, and the decoder includes a second data processing unit corresponding to the first data processing unit in the encoder; The method comprises: Acquire sample data, where the sample data includes an RGB image and an actual spectrum image obtained by photographing the same object using an RGB camera and a hyperspectral camera respectively; Performing feature extraction processing on the RGB image in the sample data by the first data processing unit to obtain third feature information; Performing feature fusion processing according to the third feature information by the second data processing unit to obtain a second fusion feature; Perform image reconstruction according to the second fusion feature to obtain a reconstructed spectral image; The model parameters of the detection model are adjusted according to the reconstructed spectral image and the actual spectral image in the sample data until the trained detection model is obtained.

7. The method according to claim 6, It is characterized in that The adjusting the model parameters of the detection model according to the reconstructed spectral image and the actual spectral image in the sample data until the trained detection model is obtained comprises: Calculating a target loss value, wherein the target loss value includes a first loss value and a second loss value, wherein the second loss value represents a loss value after amplifying an image difference corresponding to a target portion, and the image difference corresponding to the target portion is greater than the image difference corresponding to a non-target portion; If the training condition is met, the current detection model is determined as the detection model after training; If the training conditions are not met, the model parameters of the detection model are adjusted according to the target loss value, and the detection model is continuously trained according to the next sample data until the trained detection model is obtained.

8. The method according to claim 7, It is characterized in that The calculating target loss value comprises: Calculating a relative error between the reconstructed spectral image and the actual spectral image to obtain a first loss value; According to the weight coefficient corresponding to each pixel block in the image, the error between the reconstructed spectral image and the actual spectral image is calculated to obtain a second loss value; wherein the weight coefficient corresponding to each pixel block in the image is positively correlated with the difference in the pixel value corresponding to the pixel block in the reconstructed spectral image and the actual spectral image; The target loss value is calculated according to the first loss value and the second loss value.

9. The method according to claim 8, It is characterized in that The step of calculating the error between the reconstructed spectral image and the actual spectral image according to the weight coefficients corresponding to the respective pixel blocks in the image to obtain the second loss value includes: According to the weight coefficients corresponding to the pixel blocks in the image, a weighted mean square error between the reconstructed spectral image and the actual spectral image is calculated to obtain the second loss value.

10. The method according to claim 6, It is characterized in that The detection model also includes a bottleneck block, which includes a third data processing unit constructed according to the spectral attention module and the spatial attention module; The performing feature fusion processing according to the third feature information by the second data processing unit to obtain a second fusion feature includes: Performing feature processing on the third feature information by the third data processing unit to obtain fourth feature information; The second data processing unit performs feature fusion processing according to the third feature information and the fourth feature information to obtain the second fusion feature.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented, or the method according to any one of claims 6 to 11 is implemented.

12. An endoscope system, comprising a light source host, an endoscope and the computer device according to claim 11.

Images