Wheat grain microstructure change observation method and device based on multi-modal fusion imaging

Abstract

The invention discloses a wheat grain microstructure change observation method and device based on multi-modal fusion imaging, and the method comprises the following steps: S1, inputting the X-ray and infrared modal data of wheat grains into a fusion backbone network, and generating a fusion feature map; s2, establishing a comparison optimization learning model for foreground targets and background details of wheat grain imaging; and S3, constructing an adaptive weight loss constraint function to realize wheat multi-modal image enhancement representation. According to the invention, a multi-modal fusion imaging method is utilized to overcome the defects of single-modal imaging, such as low imaging quality and single imaging channel. And meanwhile, redundant information brought by fusion is reduced by adopting a multi-level feature integration coupling contrast learning method, so that multi-modal image enhancement representation of the wheat grains is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of observation of microstructure changes of wheat grains, and in particular to a method and device for observing microstructure changes of wheat grains based on multimodal fusion imaging. Background Art

[0002] Storage of barley after harvest is usually essential before consumption. The storage life of barley can be affected by both biological factors (insects and fungi) and abiotic factors (temperature, humidity and moisture), resulting in losses ranging from 1% to 50%. The intrinsic structure and characteristics of seeds may contribute to their defense mechanism against corruption under adverse conditions. X-ray imaging technology is currently used to study the internal structure of seeds. However, traditional imaging methods using X-rays are affected by imaging resolution and non-correlated sources, resulting in poor imaging results. Considering that the degree of heat sensitivity inside the seeds is different, infrared imaging can be used, but single-channel imaging causes the image to lose some details. Summary of the invention

[0003] The present invention proposes a method and device for observing changes in the microstructure of wheat grains based on multimodal fusion imaging, which involves related issues of observing the microstructure, nutrition and biochemical changes of barley grains during barley storage, and can solve at least one of the technical problems in the background technology.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for observing changes in wheat grain microstructure based on multimodal fusion imaging comprises the following steps:

[0006] S1, input the X-ray and infrared modality data of wheat grains into the fusion backbone network and generate a fusion feature map;

[0007] S2, establish a contrast optimization learning model for the foreground target and background details of the wheat grain fusion feature map;

[0008] S3. Construct an adaptive weight loss constraint function to optimize the contrast optimization learning model in S2 to achieve enhanced multimodal image representation of wheat grains.

[0009] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0010] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0011] It can be seen from the above technical solutions that the method and device for observing changes in the microstructure of wheat grains based on multimodal fusion imaging of the present invention realizes the complementarity of information of two modalities by image fusion, and reduces the redundant information brought by fusion by a multi-level feature integration coupled contrast learning method to enhance the imaging effect. The present invention can realize the complementary characteristic information of the two modalities and enhance the common information between the two modalities, thereby realizing detailed imaging of the microstructure of barley, so as to realize the study of the influence of its structure on the storage of barley.

[0012] Specifically, the present invention overcomes the shortcomings of existing methods, makes full use of the high penetration of X-rays, can image thicker or denser wheat grains, and uses infrared radiation to perceive the heat source of wheat grains, and can achieve detailed imaging according to the distribution of the target heat source. By using the imaging and fusion of the two modes, it is possible to achieve an enhanced imaging representation effect on wheat grains, thereby achieving effective observation of the microstructure of wheat grains. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a network structure diagram of an embodiment of the present invention;

[0014] Figure 2 Schematic diagram of CA channel attention according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0016] like Figure 1 As shown, the method for observing changes in wheat grain microstructure based on multimodal fusion imaging described in this embodiment includes the following steps:

[0017] S1, input the X-ray and infrared modality data of wheat grains into the fusion backbone network and generate a fusion feature map;

[0018] S2, establish a contrast optimization learning model for the foreground target and background details of wheat grain imaging;

[0019] S3. Construct an adaptive weight loss constraint function to realize the multimodal image enhancement representation of wheat grains.

[0020] Furthermore, the above step S1: inputting the X-ray and infrared modality data of the wheat grains into the fusion backbone network and generating a fusion feature map specifically includes the following subdivision steps S11 to S14:

[0021] S11: X-ray imaging F X and infrared imaging FIR Input into the backbone network to extract fusion features, which are expressed as follows:

[0022]

[0023] in, is the 0th layer fusion feature map, specifically F IR and F X Channel fusion graph before inputting the backbone network. is the fusion feature map in the i-th layer of the backbone network, i=1, 2, 3 represents the number of layers in the backbone network, CONV is a combination of two 3×3 convolutions, BN is batch normalization, and LR is the activation function. It should be noted that the number of output channels of each layer is 32, 64, 128, and 256.

[0024] S12: X-ray imaging F X and infrared imaging F IR They are input into a single backbone network respectively, as shown below:

[0025]

[0026] in, and They represent the feature maps of the i-th layer, i = {1, 2}; VGG is the pre-trained VGG-19 method. It should be noted that the number of channels output by each layer is 64, 128, and 256.

[0027] S13: Set the F with the number of channels to 64, 128, 256 X 、F IR and F U They are input into the multi-level attention module respectively, which are specifically expressed as follows:

[0028]

[0029] Among them, CA represents channel attention; and They represent the feature maps after channel attention respectively; Concat represents Concatenate on the channel; Conv represents 3×3 convolution; represents the fused attention feature map, i = {1, 2, 3}.

[0030] S14: Deconvolution enhanced fusion feature map, represented as follows:

[0031]

[0032] Among them, VConv is the dilated convolution; and They represent upsampled fusion feature maps with 256, 128, 64, and 32 channels respectively.

[0033] Furthermore, the above step S2: establishing a contrast optimization learning model for the foreground target and background details of the grain imaging includes the following subdivision steps S21 to S22:

[0034] S21: Construct the foreground target optimization target, which is expressed as follows:

[0035]

[0036] Where N and M represent the number of VGG layers and the number of negative samples of X-ray imaging selected as positive samples; μ i express The foreground target features in represents negative samples; represents a positive sample; ‖.‖ 1 represents the L1 norm.

[0037] S22: Construct the background target optimization target, which is expressed as follows:

[0038]

[0039] Where N and M represent the number of VGG layers and the number of negative samples of infrared imaging selected as positive samples, respectively; x i express Background target features in ; represents negative samples; represents a positive sample; ‖.‖ 1 represents the L1 norm.

[0040] It should be noted that, in obtaining VGG will be used to implement contrastive learning, and the contrastive learning strategy is described in S2.

[0041] Furthermore, the above step S3: constructing an adaptive weight loss constraint function to realize the multimodal image enhancement representation of wheat grains specifically includes the following subdivision steps S31 to S33:

[0042] S31: Construct an adaptive loss constraint function, which is expressed as follows:

[0043]

[0044]

[0045] in, and Represent similarity loss, structural similarity loss and strength similarity loss respectively; α is a hyperparameter; S(.) represents SSIM calculation; ‖.‖2 is the mean square error; L represents the pixel value range; p x Indicates the probability of the pixel value in the image.

[0046] S32: Finally, the network continuously optimizes the following objectives to achieve the effect of wheat grain multimodal image enhancement.

[0047]

[0048] In summary, the method for observing the microstructure changes of wheat grains based on multimodal fusion imaging in the embodiment of the present invention utilizes the method of multimodal fusion imaging to complement the defects of single-modal imaging, such as low imaging quality and single imaging channel. At the same time, the method of multi-level feature integration coupled contrast learning is used to reduce the redundant information brought by fusion, so as to realize the enhanced representation of multimodal images of wheat grains.

[0049] In specific applications, the present invention overcomes the shortcomings of existing methods, fully utilizes the high penetration of X-rays, can image thicker or denser wheat grain targets, and uses infrared radiation to perceive the heat source of wheat grains, and can achieve detailed imaging according to the distribution of the target heat source. By using the imaging and fusion of the two modes, it is possible to achieve an enhanced imaging representation effect for wheat grains, thereby achieving effective observation of the microscopic structure of wheat grains.

[0050] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0051] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0052] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods for observing changes in the microstructure of wheat grains based on multimodal fusion imaging in the above-mentioned embodiments.

[0053] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above methods.

[0054] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0055] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0056] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0057] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for observing changes in wheat grain microstructure based on multimodal fusion imaging, characterized in that: The following steps are included: S1, input the X-ray and infrared modality data of wheat grains into the fusion backbone network and generate a fusion feature map; S2, establish a contrast optimization learning model for the foreground target and background details of the wheat grain fusion feature map; S3. Construct an adaptive weight loss constraint function to optimize the contrast optimization learning model in S2 to achieve enhanced representation of multimodal images of wheat grains.

2. The method for observing changes in wheat grain microstructure based on multimodal fusion imaging according to claim 1, characterized in that: Step S1 specifically includes: S11: X-ray imaging F X and infrared imaging F IR Input into the backbone network to extract fusion features, which are expressed as follows: in, is the 0th layer fusion feature map, specifically F IR and F X Channel fusion graph before inputting into the backbone network; is the fused feature map in the i-th layer of the backbone network, i=1, 2, 3 represents the number of layers in the backbone network, CONV is a combination of two 3×3 convolutions, BN is batch normalization, and LR is the activation function; S12: X-ray imaging F X and infrared imaging F IR They are input into a single backbone network respectively, as shown below: in, and They represent the feature maps of the i-th layer, i = {1, 2}; VGG is the pre-trained VGG-19 method; S13: Set the F with the number of channels to 64, 128, 256 X 、F IR and F U They are input into the multi-level attention module respectively, which are specifically expressed as follows: Among them, CA represents channel attention; and They represent the feature maps after channel attention respectively; Concat represents Concatenate on the channel; Conv represents 3×3 convolution; represents the fused attention feature map, i = {1, 2, 3}; S14: Deconvolution enhanced fusion feature map, represented as follows: Among them, VConv is the dilated convolution; and They represent upsampled fusion feature maps with 256, 128, 64, and 32 channels respectively.

3. The method for observing changes in wheat grain microstructure based on multimodal fusion imaging according to claim 2, characterized in that: The specific steps of step S2 are as follows: S21: Constructing the foreground target optimization target for the wheat grain fusion feature map, which is expressed as follows: Where N and M represent the number of VGG layers and the number of negative samples of X-ray imaging selected as positive samples; μ i express The foreground target features in represents negative samples; represents a positive sample; ‖.‖1 represents the L1 norm; S22: Construct the background target optimization target in wheat grain fusion imaging, which is expressed as follows: Where N and M represent the number of VGG layers and the number of negative samples of infrared imaging selected as positive samples, respectively; x i express Background target features in ; represents negative samples; represents a positive sample; ‖.‖1 represents the L1 norm.

4. The method for observing changes in wheat grain microstructure based on multimodal fusion imaging according to claim 3, characterized in that: Step S3 specifically includes: S31: Construct an adaptive loss constraint function, which is expressed as follows: in, and Respectively represent similarity loss, structural similarity loss and intensity similarity loss; α is a hyperparameter; S(.) represents SSIM calculation; ‖.‖2 is the mean square error; L represents the pixel value range; p x Represents the probability of the pixel value in the image; S32: Finally, the network continuously optimizes the following objectives to achieve the effect of wheat grain multimodal image enhancement; 5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 4.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 4.

Images