Method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation

Abstract

The present application discloses a method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation, which relates to the field of agricultural intelligent technology, and aims to solve the problems that existing methods using ordinary linear array cameras cannot detect nearly white foreign fibers such as plastic films, and methods using multispectral cameras cannot detect foreign fibers under poor lighting conditions. The method includes: collecting a full-color image of raw cotton and a spectral image of raw cotton, wherein the full-color image of raw cotton is collected by a color linear array CCD camera, and the spectral image of raw cotton is collected by a multispectral spectrometer; inputting the full-color image of raw cotton and the spectral image of raw cotton into a network structure for fusion and data augmentation of bimodal images to perform fusion, so as to obtain a first multi-feature image of raw cotton; inputting the first multi-feature image of raw cotton into a pre-trained foreign fiber detection model for raw cotton to perform detection, so as to obtain a first foreign fiber detection result of raw cotton.

Description

Technical Field

[0001] This application relates to the field of agricultural intelligent technologies, and particularly to a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data augmentation. Background Art

[0002] Cotton plays a crucial and indispensable role in people's lives and is intricately intertwined with all aspects of human existence, including daily life, medical applications, military uses, and agriculture. China is an important producer of raw cotton, and the cotton output in the past five years has exceeded 16 million tons. With the expansion of the planting area of machine-picked cotton, the content of foreign fibers in cotton has increased significantly. Among them, the main foreign fibers existing in raw cotton include chemical fibers, residual films, feathers, hair, and fabrics. In the subsequent processing of raw cotton, if foreign fibers in raw cotton cannot be detected and removed in a timely manner, they will break into many tiny particles, which are intertwined with raw cotton yarns, possibly leading to problems such as easy breakage of the yarn and uneven dyeing, thus affecting the quality of textiles.

[0003] Currently, the sorting and removal of foreign fibers in raw cotton usually use optical detection technologies. The detection principle of optical detection technologies is as follows: Raw cotton containing foreign fibers enters from the cotton inlet under the action of air flow and flows along the cotton conveying pipeline. The image acquisition system performs image acquisition, and through image processing algorithms, feature extraction and other processing are carried out in real time, feedback information such as whether there are foreign fibers and their positions, and the information is transmitted to the control unit. If there are foreign fibers in the raw cotton, the industrial control computer drives the solenoid valve at the corresponding position to eject high-pressure gas to remove the foreign fibers. It can be foreseen that the image processing algorithm for foreign fibers is the key technology among them. Currently, the main image processing algorithms adopted are detection using the U-Net network, detection using polarization imaging and enhanced YOLOV5, and detection using the combination of near-infrared spectroscopy and convolutional neural network. Among them, using the combination of near-infrared spectroscopy and convolutional neural network to detect foreign fibers in raw cotton has a great advantage in terms of detection accuracy compared with other methods, but it still cannot achieve precise detection. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, this application provides a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data augmentation to solve the problems that existing ordinary linear array cameras cannot detect nearly white foreign fibers such as plastic films and multi-spectral cameras cannot detect foreign fibers under poor lighting conditions, so as to synthesize a fusion image containing prominent targets and rich texture details under ordinary lighting conditions, achieving the purpose of accurately detecting foreign fibers in raw cotton.

[0005] To this end, the embodiments of this application provide a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data augmentation, including:

[0006] Collect the full-color image and spectral image of raw cotton. Among them, the full-color image of raw cotton is collected by a color linear array CCD camera, and the spectral image of raw cotton is collected by a multispectrometer;

[0007] Input the full-color image and spectral image of raw cotton into the network structure for the fusion and data enhancement of bimodal images for fusion to obtain the first multi-feature image of raw cotton;

[0008] Input the first multi-feature image of raw cotton into a pre-trained raw cotton foreign fiber detection model for detection to obtain the first raw cotton foreign fiber detection result.

[0009] According to the above-mentioned method for detecting foreign fibers in raw cotton based on bimodal image fusion and data enhancement, in the step of inputting the full-color image and spectral image of raw cotton into the network structure for the fusion and data enhancement of bimodal images for fusion to obtain the first multi-feature image of raw cotton, the network structure specifically includes:

[0010] A feature extraction module, which is used to extract features from the full-color image and spectral image of raw cotton;

[0011] A feature fusion module, whose input end is connected to the output end of the feature extraction module, and the feature fusion module is used to fuse the features of the full-color image and spectral image of raw cotton;

[0012] A feature enhancement module, whose input end is connected to the output end of the feature fusion module, and is used to perform enhancement processing on the fused features to obtain the first multi-feature image of raw cotton.

[0013] According to the above-mentioned method for detecting foreign fibers in raw cotton based on bimodal image fusion and data enhancement, in the step of inputting the full-color image and spectral image of raw cotton into the network structure for the fusion and data enhancement of bimodal images for fusion to obtain the first multi-feature image of raw cotton, the feature extraction module is specifically formed by two dual-channel modules. Each dual-channel module includes a first channel and a second channel. Each first channel includes two first structures composed of a 3×3 convolutional layer and a ReLU activation layer connected in sequence and two second structures composed of a 3×3 convolutional layer and a ReLU activation layer based on residual connection; the second channel includes three third structures composed of a 5×5 convolutional layer and a ReLU activation layer connected in sequence, and the two second channels are connected by a layer of Batch Norm regularization layer.

[0014] According to the above-mentioned method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation, in the step of inputting the raw cotton full-color image and the raw cotton spectral image into the network structure for fusion and data augmentation of bimodal images to obtain the first multi-feature image of raw cotton, the feature fusion module specifically includes three ResBlock modules connected in sequence and two fourth structures each composed of a 3×3 convolutional layer and a ReLU activation layer. The input end of the first ResBlock module among the three ResBlock modules is connected to the output end of the feature extraction module.

[0015] According to the above-mentioned method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation, in the step of inputting the raw cotton full-color image and the raw cotton spectral image into the network structure for fusion and data augmentation of bimodal images to obtain the first multi-feature image of raw cotton, the feature enhancement module is specifically constructed by three channels. All three channels are connected to the second fourth structure. The first channel is directly connected to the output; the second channel includes a 3×3 convolutional layer, a Batch Norm regularization layer, a ReLU activation layer, and a 3×3 convolutional layer connected in sequence; the third channel includes a Real FFT2d layer, a 1×1 convolutional layer, a Batch Norm regularization layer, a ReLU activation layer, a 1×1 convolutional layer, and an Inv Real FFT2d layer connected in sequence. The output ends of the three channels are connected and output the first multi-feature image of raw cotton.

[0016] According to the above-mentioned method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation, in the step of inputting the first multi-feature image of raw cotton into the pre-trained foreign fiber detection model of raw cotton for detection to obtain the detection result of foreign fibers in raw cotton, the pre-trained foreign fiber detection model of raw cotton is jointly determined by the training data set and the foreign fiber detection network of raw cotton;

[0017] Among them, the training data set is multiple second multi-feature images of raw cotton and their corresponding target foreign fiber detection results of raw cotton. The second multi-feature images of raw cotton are obtained by inputting the raw cotton full-color image and the raw cotton spectral image into the network structure for fusion and data augmentation of bimodal images. The target foreign fiber detection results of raw cotton are obtained by manually annotating the second multi-feature images of raw cotton.

[0018] According to the above-mentioned method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation, the step in which the pre-trained foreign fiber detection model of raw cotton is jointly determined by the training data set and the foreign fiber detection network of raw cotton is specifically as follows:

[0019] Obtain multiple second multi-feature images of raw cotton and their corresponding target raw cotton foreign fiber detection results;

[0020] Input the multiple second multi-feature images of raw cotton into a pre-built raw cotton foreign fiber detection network for detection to obtain multiple second raw cotton foreign fiber detection results;

[0021] Compare the multiple second raw cotton foreign fiber detection results with their corresponding target raw cotton foreign fiber detection results until the second raw cotton foreign fiber detection results meet the preset conditions, and determine the raw cotton foreign fiber detection network as a trained raw cotton foreign fiber detection model.

[0022] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data augmentation, in the step of inputting the multiple second multi-feature images of raw cotton and the target multi-feature images of raw cotton into a pre-built raw cotton foreign fiber detection network for detection to obtain multiple second raw cotton foreign fiber detection results, the pre-built raw cotton foreign fiber detection network specifically includes:

[0023] A backbone module, which is used to extract foreign fiber features in raw cotton;

[0024] A neck module, whose input end is connected to the output end of the backbone module and is used to fuse and enhance foreign fiber features in raw cotton;

[0025] A prediction head module, whose input end is connected to the output end of the neck module and is used to output raw cotton foreign fiber detection results.

[0026] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data augmentation, in the step of inputting the multiple second multi-feature images of raw cotton and the target multi-feature images of raw cotton into a pre-built raw cotton foreign fiber detection network for detection to obtain multiple second raw cotton foreign fiber detection results, the backbone module specifically includes: two CBH convolution structures connected in sequence, one layer of CSP structure, one layer of CBH structure, one layer of inception structure, one layer of CSP structure, and a CBH convolution structure;

[0027] The neck module is specifically constructed by three channels. The first channel includes an upsampling layer. The output end of the upsampling layer and the output end of the first CSP structure in the backbone module are connected to the first feature splicing ring. The output end of the first feature splicing ring is sequentially connected to a CSP structure, a CBH convolutional structure, and an upsampling layer. The output end of the latter upsampling layer and the output end of the first CBH structure in the backbone module are both connected to the input end of the second feature splicing ring. The second feature splicing ring is connected to a CSP structure; The second channel is that the output end of the first feature splicing ring and the output end of the first channel are connected to a CBH convolutional structure and then both are connected to the third feature splicing ring. The output end of the third feature splicing ring is connected to a CSP structure; The third channel includes the output end of the backbone module and the output end of the second channel. After connecting to a CBH convolutional structure, both are connected to the fourth feature splicing ring. The output end of the fourth feature splicing ring is connected to a CSP structure;

[0028] The prediction head module specifically includes three Reshape layers. The input ends of the three Reshape layers are respectively connected to the output ends of the three channels of the neck module. The output ends of the three Reshape layers are connected to the same feature splicing ring. The output end of this feature splicing ring is sequentially connected to two CBH convolutional structures and two Full connections.

[0029] According to the above-mentioned raw cotton foreign fiber detection method based on bimodal image fusion and data augmentation, in the step of inputting multiple second raw cotton multi-feature images and target raw cotton multi-feature images into a pre-built raw cotton foreign fiber detection network for detection to obtain multiple second raw cotton foreign fiber detection results, the CBH structure specifically includes: a 3×3 convolutional layer, a Batch Norm regularization layer, and a Hard-Swish activation layer connected in sequence;

[0030] The CSP structure specifically includes: two CBH convolutional structures, a 3×3 convolutional layer, and a 3×3 convolutional layer connected based on a residual network and connected to a feature splicing ring in sequence. The output end of the feature splicing ring is sequentially connected to a Batch Norm regularization layer, a leakyrelu activation layer, and a CBH convolutional structure;

[0031] The specific Inception structure is constructed by four channels. The first channel directly connects the output end of a 1×1 convolutional layer to the feature splicing loop; the second channel includes a 1×1 convolutional layer, a 3×3 convolutional layer, and the feature splicing loop connected in sequence; the third channel includes a 3×3 max-pooling layer, a 3×3 convolutional layer, and the feature splicing loop connected in sequence; the fourth channel includes a 1×1 convolutional layer and a 3×3 convolutional layer, and this 3×3 convolutional layer is respectively connected to a 1×3 convolutional layer and a 3×1 convolutional layer, and finally the output result is connected to the feature splicing loop again.

[0032] The beneficial effects of the raw cotton foreign fiber detection method based on dual-modal image fusion and data augmentation provided by this application are at least as follows:

[0033] By setting the network structure of dual-modal image fusion and data augmentation, this application fuses the raw cotton full-color image and the raw cotton spectral image to obtain the first multi-feature raw cotton image that contains both rich texture and other detailed structures and rich visible light details. By inputting the first multi-feature raw cotton image into the pre-trained raw cotton foreign fiber detection model for detection, it can accurately detect foreign fibers such as colored cloth strips clearly visible under visible light conditions and foreign fibers such as white feathers, hairs, and transparent films that are easy to identify under multi-spectral conditions, thus realizing the accurate detection of raw cotton foreign fibers. Description of the Drawings

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0035] Figure 1 It is a flowchart of the raw cotton foreign fiber detection method based on dual-modal image fusion and data augmentation provided by the embodiments of this application.

[0036] Figure 2 It is a schematic diagram of the composition of the image acquisition device involved in the raw cotton foreign fiber detection method based on dual-modal image fusion and data augmentation provided by the embodiments of this application.

[0037] Figure 3 It is a schematic diagram of the composition of the network structure involved in the raw cotton foreign fiber detection method based on dual-modal image fusion and data augmentation provided by the embodiments of this application.

[0038] Figure 4Schematic diagram of the composition of the ResBlock module in the raw cotton foreign fiber detection method based on bimodal image fusion and data augmentation provided by the embodiments of the present application.

[0039] Figure 5 Schematic diagram of the composition of the feature enhancement module in the raw cotton foreign fiber detection method based on bimodal image fusion and data augmentation provided by the embodiments of the present application.

[0040] Figure 6 Schematic diagram of the composition of the raw cotton foreign fiber detection network in the raw cotton foreign fiber detection method based on bimodal image fusion and data augmentation provided by the embodiments of the present application.

[0041] Figure 7 Schematic diagram of the composition of the CBH convolutional structure in the raw cotton foreign fiber detection method based on bimodal image fusion and data augmentation provided by the embodiments of the present application.

[0042] Figure 8 Schematic diagram of the composition of the CSP structure in the raw cotton foreign fiber detection method based on bimodal image fusion and data augmentation provided by the embodiments of the present application.

[0043] Figure 9 Schematic diagram of the composition of the inception structure in the raw cotton foreign fiber detection method based on bimodal image fusion and data augmentation provided by the embodiments of the present application. Detailed implementation manners

[0044] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] Cotton plays a crucial and indispensable role in people's lives and is intricately intertwined with all aspects of human survival, including daily life, medical applications, military uses and agriculture. China is an important raw cotton producer, and the cotton output in the past five years has exceeded 5 million tons. With the expansion of the planting area of machine-picked cotton, the content of foreign fibers in cotton has increased significantly. Among them, the main foreign fibers existing in raw cotton include chemical fibers, residual films, feathers, hair and fabrics. In the subsequent processing of raw cotton, if the foreign fibers in raw cotton cannot be detected and removed in time, they will break into many tiny particles, which are intertwined with raw cotton yarns, may cause problems such as easy breakage of the yarn and uneven dyeing, thus affecting the quality of textiles.

[0046] Currently, the sorting and removal of foreign fibers in raw cotton are usually carried out using optical detection technology. The detection principle of optical detection technology is as follows: Raw cotton containing foreign fibers enters from the cotton inlet under the action of air flow and flows along the cotton conveying pipeline. The image acquisition system performs image acquisition, and through image processing algorithms, feature extraction and other processing are carried out in real time, and information such as whether there are foreign fibers and their positions is fed back, and the information is transmitted to the control unit. If there are foreign fibers in the raw cotton, the industrial control computer drives the solenoid valve at the corresponding position to eject high-pressure gas to remove the foreign fibers. It can be foreseen that the image processing algorithm for foreign fibers is the key technology among them. Currently, the main image processing algorithms adopted are detection using the U-Net network, detection using polarization imaging and enhanced YOLOV5, and detection using the combination of near-infrared spectroscopy and convolutional neural network. Among them, using the combination of near-infrared spectroscopy and convolutional neural network to detect foreign fibers in raw cotton has great advantages in terms of detection accuracy compared with other methods, but it still cannot achieve precise detection.

[0047] After research by the inventor, it is found that in the existing image processing algorithms for detecting foreign fibers in raw cotton using the combination of near-infrared spectroscopy and convolutional neural network, if a color linear array CCD camera is used to collect images, the full-color images collected have the characteristics of high spatial resolution, contain rich texture structures, and have good detection effects on colored foreign fibers, but the collection of feature information of foreign fibers that are not easily distinguishable such as white feathers, hairs, and films is not prominent enough; if a multispectrometer is used to collect images, the multispectral images collected have the characteristics of high spectral resolution, and have good detection effects on white feathers, hairs, films, and foreign fibers with fluorescence characteristics, but lack detailed feature information, and the recognition of feature information of colored and easily distinguishable foreign fibers is not accurate enough. However, the foreign fibers in raw cotton include not only foreign fibers such as white feathers, hairs, and films, but also colored foreign fibers. How to simultaneously utilize the advantages of high spatial resolution and high spectral resolution to detect foreign fibers in raw cotton and achieve precise detection of foreign fibers in raw cotton is the problem to be solved in this application.

[0048] This application collects the full-color image of raw cotton and the spectral image of raw cotton. Among them, the full-color image of raw cotton is collected by a color linear array CCD camera, and the spectral image of raw cotton is collected by a multispectrometer; the full-color image of raw cotton and the spectral image of raw cotton are input into a network structure for the fusion and data enhancement of bimodal images for fusion to obtain the first multi-feature image of raw cotton; the first multi-feature image of raw cotton is input into a pre-trained detection model for foreign fibers in raw cotton for detection to obtain the first detection result of foreign fibers in raw cotton. That is to say, in this embodiment, by setting a network structure for the fusion and data enhancement of bimodal images, the full-color image of raw cotton and the spectral image of raw cotton are fused to obtain the first multi-feature image of raw cotton that contains both rich texture and other detailed structures and rich visible light details. In this embodiment, by inputting the first multi-feature image of raw cotton into a pre-trained detection model for foreign fibers in raw cotton for detection, it is possible to accurately detect foreign fibers such as colored cloth strips clearly visible under visible light conditions and foreign fibers such as white feathers, hairs, and transparent films that are easy to identify under multi-spectral conditions, so as to achieve the precise detection of foreign fibers in raw cotton.

[0049] The following further illustrates the invention content by describing the embodiments in conjunction with the accompanying drawings.

[0050] Refer to Figure 1 , this embodiment provides a method for detecting foreign fibers in raw cotton based on the fusion and data enhancement of bimodal images. The method for detecting foreign fibers in raw cotton includes:

[0051] S10. Collect the full-color image of raw cotton and the spectral image of raw cotton;

[0052] S20. Input the full-color image of raw cotton and the spectral image of raw cotton into a network structure for the fusion and data enhancement of bimodal images for fusion to obtain the first multi-feature image of raw cotton;

[0053] S30. Input the first multi-feature image of raw cotton into a pre-trained detection model for foreign fibers in raw cotton for detection to obtain the first detection result of foreign fibers in raw cotton.

[0054] Specifically, the pre-trained detection model for foreign fibers in raw cotton is implemented on the Pytorch (which is an open-source Python machine learning library based on Torch and is used for applications such as natural language processing) framework. The full-color image of raw cotton is collected by a color linear array CCD camera, the spectral image of raw cotton is collected by a multispectrometer, and the first multi-feature image of raw cotton is obtained by fusing the collected full-color image of raw cotton and the spectral image of raw cotton into a network structure for the fusion and data enhancement of bimodal images. The first detection result of foreign fibers in raw cotton can be understood as a kind of information. For example, it can be considered whether there are foreign fibers in the raw cotton and the position information of the foreign fibers in the raw cotton.

[0055] Among them, referring to Figure 2 , the image acquisition device for the raw cotton full-color image and the raw cotton spectral image may include a cotton flow channel 1, a lamp tube 3, a lens 4, two color linear array CCD cameras 5, and two multispectrometers 6. The channel wall 2 of the cotton flow channel 1 may be made of a highly transparent glass panel to ensure the image quality. The cotton flow channel 1 is used to transport the cotton flow, and the lamp tube 3 is used to provide light source for the cotton flow. The cotton flow channel 1 is divided into an upper cotton flow channel and a lower cotton flow channel. One color linear array CCD camera 5 and one multispectrometer 6 are arranged in both the upper cotton flow channel and the lower cotton flow channel. To ensure the spatial consistency of the raw cotton full-color image and the raw cotton spectral image, the raw cotton full-color image and the raw cotton spectral image are collected from the cotton flow at the same position. The two cameras in each group are respectively placed at the upper cotton flow channel and the lower cotton flow channel. When the cotton flow passes through the cotton flow channel, the raw cotton full-color image and the raw cotton spectral image are collected by the color linear array CCD camera and the multispectrometer.

[0056] Among them, since the multispectrometer has higher requirements for images, in order to reduce the feature loss of the raw cotton spectral image, the multispectrometer is used to directly collect images at the acquisition point of the cotton flow channel, and the color linear array CCD camera collects images through the reflection of the lens 4. It should be noted that according to the classification of foreign fibers in raw cotton in different bands, the classification effect of the raw cotton spectral image is the best when the band is 780nm - 850nm. Therefore, when collecting images, the band of the raw cotton spectral image collected by the multispectrometer is set to 780nm - 850nm.

[0057] Optionally, referring to Figure 3 , in one embodiment, in the step of inputting the raw cotton full-color image and the raw cotton spectral image into the network structure for fusion and data enhancement of the dual-modal image to obtain the first raw cotton multi-feature image, the network structure specifically includes a feature extraction module, a feature fusion module, and a feature enhancement module. The feature extraction module is used to extract features from the raw cotton full-color image and the raw cotton spectral image. The input end of the feature fusion module is connected to the output end of the feature extraction module, and the feature fusion module is used to fuse the features of the raw cotton full-color image and the raw cotton spectral image. The input end of the feature enhancement module is connected to the output end of the feature fusion module and is used to enhance the fused features to obtain the first raw cotton multi-feature image.

[0058] Optionally, referring to Figure 3, in one embodiment, the feature extraction module is specifically formed by constructing two dual-channel modules, and the two dual-channel modules are used for multi-scale feature extraction to fully capture complementary features in the full-color image of raw cotton and the spectral image of raw cotton. Each of the two dual-channel modules includes a first channel and a second channel. Each first channel includes two first structures connected in sequence, each of which is composed of a 3×3 convolutional layer (abbreviation: Conv3×3) and a ReLU activation layer (abbreviation: ReLU), and two second structures based on residual connections, each of which is composed of a 3×3 convolutional layer and a ReLU activation layer; the second channel includes three third structures connected in sequence, each of which is composed of a 5×5 convolutional layer and a ReLU activation layer. The outputs of the two second channels are connected by a Batch Norm regularization layer (abbreviation: BN), aiming to change the number of channels so that the number of channels of each channel processing structure is the same, facilitating the next step of feature fusion.

[0059] Optionally, refer to Figure 3 , in one embodiment, the feature fusion module specifically includes three ResBlock modules (abbreviation: ResBlock) connected in sequence and two fourth structures composed of a 3×3 convolutional layer and a ReLU activation layer. The input end of the first ResBlock module among the three ResBlock modules is respectively connected to the output end of the feature extraction module;

[0060] Among them, refer to Figure 4 , the ResBlock module includes two fifth structures connected in sequence, each of which is composed of a 3×3 convolutional layer and a ReLU activation layer, and a 3×3 convolutional layer. After including a residual structure, it is connected to a ReLU activation layer.

[0061] Optionally, refer to Figure 5 , in one embodiment, the feature enhancement module is specifically constructed by three channels, including three-channel multi-scale feature enhancement based on residual structure connection. Among them, the three channels are all connected to the second fourth structure, and the first channel is directly connected to the output; the second channel includes a 3×3 convolutional layer, a Batch Norm regularization layer, a ReLU activation layer, and a 3×3 convolutional layer connected in sequence; the third channel includes a Real FFT2d layer (abbreviation: Real FFT2d), a 1×1 convolutional layer, a Batch Norm regularization layer, a ReLU activation layer, a 1×1 convolutional layer, and an Inv Real FFT2d layer (abbreviation: Inv Real FFT2d) connected in sequence. The output ends of the three channels are connected and output the first multi-feature image of raw cotton.

[0062] In this embodiment, a residual mechanism is introduced to address the problems of unclear edges and degraded feature information in the multi-feature image of raw cotton. By using the Real FFT2d layer and the Inv Real FFT2d layer, the multi-feature image of raw cotton is enhanced, achieving a significant acceleration in the stage of detecting foreign fibers in raw cotton, thereby efficiently improving the overall computational efficiency.

[0063] Optionally, in one embodiment, in the step of inputting the first multi-feature image of raw cotton into a pre-trained foreign fiber detection model of raw cotton for detection to obtain the foreign fiber detection result of raw cotton, the pre-trained foreign fiber detection model of raw cotton is jointly determined by a training dataset and a foreign fiber detection network of raw cotton;

[0064] Among them, the training dataset is multiple second multi-feature images of raw cotton and their corresponding target foreign fiber detection results of raw cotton. The second multi-feature image of raw cotton is obtained by fusing a full-color image of raw cotton and a spectral image of raw cotton into a network structure for fusion and data augmentation of dual-modal images. The target foreign fiber detection result of raw cotton is obtained by manually annotating the second multi-feature image of raw cotton.

[0065] Optionally, in one embodiment, the step of jointly determining the pre-trained foreign fiber detection model of raw cotton by a training dataset and a foreign fiber detection network of raw cotton is specifically as follows:

[0066] Obtain multiple second multi-feature images of raw cotton and their corresponding target foreign fiber detection results of raw cotton;

[0067] Input multiple second multi-feature images of raw cotton into a pre-built foreign fiber detection network of raw cotton for detection to obtain multiple second foreign fiber detection results of raw cotton;

[0068] Compare the multiple second foreign fiber detection results of raw cotton with their corresponding target foreign fiber detection results of raw cotton until the second foreign fiber detection results of raw cotton meet the preset conditions, and determine the foreign fiber detection network of raw cotton as the trained foreign fiber detection model of raw cotton.

[0069] Specifically, the foreign fiber detection network of raw cotton can be a convolutional neural network based on the YOLO series. The preset condition is: input multiple different second multi-feature images of raw cotton into the pre-built foreign fiber detection network of raw cotton for training until the output second foreign fiber detection results of raw cotton are continuously close to the target foreign fiber detection results of raw cotton, obtain the optimal weight parameters of the foreign fiber detection model of raw cotton, and load the optimal weight parameters of the foreign fiber detection model of raw cotton into the pre-built foreign fiber detection network of raw cotton to form the trained foreign fiber detection model of raw cotton.

[0070] Further, refer toFigure 6 , in one embodiment, the pre-built raw cotton foreign fiber detection network specifically includes a Backbone module, a Neck module, and a Prediction module. The input end of the Backbone module is connected to the output end of the feature enhancement module and is used to extract the foreign fiber features in the raw cotton. The input end of the Neck module is connected to the output end of the Backbone module and is used to fuse and enhance the foreign fiber features in the raw cotton. The input end of the Prediction module is connected to the output end of the Neck module and is used to output the raw cotton foreign fiber detection result.

[0071] Further, refer to Figure 6 , in one embodiment, the Backbone module specifically includes: two CBH convolutional structures (abbreviation: CBH) connected in sequence, one CSP structure (abbreviation: CSP), one CBH structure, one inception structure (abbreviation: inception), one CSP structure, and a CBH convolutional structure.

[0072] Further, refer to Figure 6 , in one embodiment, the Neck module adopts a bottom-up feature fusion structure to fully perform feature fusion. Specifically, it is constructed by three channels. The first channel includes an upsampling layer. The output end of the upsampling layer and the output end of the first CSP structure in the Backbone module are both connected to the first feature splicing ring. The output end of the first feature splicing ring is sequentially connected to one CSP structure, one CBH convolutional structure, and one upsampling layer. The output end of the subsequent upsampling layer and the output end of the first CBH structure in the Backbone module are both connected to the input end of the second feature splicing ring. The second feature splicing ring is connected to one CSP structure; The second channel is that the output end of the first feature splicing ring and the output end of the first channel are both connected to one CBH convolutional structure and then connected to the third feature splicing ring. The output end of the third feature splicing ring is connected to one CSP structure; The third channel includes the output end of the Backbone module and the output end of the second channel. After connecting one CBH convolutional structure, they are both connected to the fourth feature splicing ring. The output end of the fourth feature splicing ring is connected to one CSP structure;

[0073] Among them, refer to Figure 6 , the Prediction module specifically includes three Reshape layers (abbreviation: Reshape). The input ends of the three Reshape layers are respectively connected to the output ends of the three channels of the Neck module. The output ends of the three Reshape layers are connected to the same feature splicing ring. The output end of this feature splicing ring is sequentially connected to two CBH convolutional structures and two Fullconnection layers (abbreviation: Full connection) to output the raw cotton foreign fiber detection result.

[0074] Further, referring to Figure 7 , in one embodiment, the CBH structure specifically includes: a 3×3 convolutional layer, a Batch Norm regularization layer, and a Hard-Swish activation layer (abbreviation: Hard-Swish) connected in sequence.

[0075] Further, referring to Figure 8 , in one embodiment, the CSP structure specifically includes: two CBH convolutional structures connected in sequence, a 3×3 convolutional layer, and a 3×3 convolutional layer connected based on a residual network and connected to a feature splicing loop. The output end of the feature splicing loop is connected to a Batch Norm regularization layer, a leakyrelu activation layer (abbreviation: leakyrelu), and a CBH convolutional structure in sequence.

[0076] Further, referring to Figure 9 , in one embodiment, the inception structure is specifically constructed by four channels. The output end of a 1×1 convolutional layer in the first channel is directly connected to a feature splicing loop; the second channel includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a feature splicing loop connected in sequence; the third channel includes a 3×3 max pooling layer, a 3×3 convolutional layer, and a feature splicing loop connected in sequence; the fourth channel includes a 1×1 convolutional layer, a 3×3 convolutional layer, and this 3×3 convolutional layer is respectively connected to a 1×3 convolutional layer and a 3×1 convolutional layer, and finally the output result is connected to the feature splicing loop again.

[0077] In this embodiment, by means of dual-modal image fusion of the raw cotton full-color image and the raw cotton spectral image, the high spatial resolution of the raw cotton full-color image and the high spectral resolution advantage of the raw cotton spectral image are fully utilized, richer raw cotton image feature information is obtained, the detection accuracy of the raw cotton foreign fiber detection model for foreign fibers in raw cotton is improved, and data augmentation is performed on the raw cotton multi-feature image, further improving the extraction efficiency of the model for the raw cotton multi-feature image, thereby improving the detection accuracy of the raw cotton foreign fiber detection model for raw cotton foreign fibers, and ensuring the accuracy and rapidity of the detection of raw cotton foreign fibers.

[0078] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation, characterized in that, Including: Collecting the full-color image and spectral image of raw cotton. Among them, the full-color image of raw cotton is collected by a color linear array CCD camera, and the spectral image of raw cotton is collected by a multispectrometer; Inputting the full-color image and spectral image of raw cotton into a network structure for fusion and data enhancement of bimodal images to perform fusion, and obtaining a first multi-feature image of raw cotton; Inputting the first multi-feature image of raw cotton into a pre-trained foreign fiber detection model for raw cotton to perform detection, and obtaining a first foreign fiber detection result of raw cotton; The network structure specifically includes: A feature extraction module, which is used to extract features from the full-color image and spectral image of raw cotton; A feature fusion module, whose input end is connected to the output end of the feature extraction module, and the feature fusion module is used to fuse the features of the full-color image and spectral image of raw cotton; A feature enhancement module, whose input end is connected to the output end of the feature fusion module, and is used to perform enhancement processing on the fused features to obtain a first multi-feature image of raw cotton; The feature extraction module is composed of two dual-channel modules. One dual-channel module inputs the full-color image of raw cotton, and the other dual-channel inputs the spectral image of raw cotton. Each dual-channel module includes a first channel and a second channel. Each first channel includes two first structures composed of a 3×3 convolutional layer and a ReLU activation layer connected in sequence and two second structures composed of a 3×3 convolutional layer and a ReLU activation layer based on residual connection; the second channel includes three third structures composed of a 5×5 convolutional layer and a ReLU activation layer connected in sequence, and the two second channels are connected through a Batch Norm regularization layer.

2. The method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation according to claim 1, wherein The feature fusion module specifically includes three ResBlock modules connected in sequence and two fourth structures composed of a 3×3 convolutional layer and a ReLU activation layer. The input end of the first ResBlock module among the three ResBlock modules is connected to the output end of the feature extraction module.

3. The method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation according to claim 2, wherein The feature enhancement module is composed of three channels, and all three channels are connected to the second fourth structure; The input end of the first channel of the feature enhancement module is directly connected to its output end; The second channel of the feature enhancement module includes a 3×3 convolutional layer, a Batch Norm regularization layer, a ReLU activation layer, and a 3×3 convolutional layer connected in sequence; The third channel of the feature enhancement module includes a Real FFT2d layer, a 1×1 convolutional layer, a Batch Norm regularization layer, a ReLU activation layer, a 1×1 convolutional layer, and an Inv Real FFT2d layer connected in sequence; the output ends of the three channels are connected and output a first multi-feature image of raw cotton.

4. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data augmentation according to claim 1, wherein, The pre-trained foreign fiber detection model for raw cotton is jointly determined by a training data set and a foreign fiber detection network for raw cotton; Among them, the training data set is multiple multi-feature images of raw cotton and their corresponding detection results of foreign fibers in the target raw cotton. The second multi-feature image of raw cotton is obtained by fusing a full-color image of raw cotton and a spectral image of raw cotton into a network structure for fusion and data enhancement of dual-modal images. The detection result of foreign fibers in the target raw cotton is obtained by manually annotating the second multi-feature image of raw cotton.

5. The method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation according to claim 4, wherein The specific steps for jointly determining the pre-trained foreign fiber detection model of raw cotton through the training data set and the foreign fiber detection network of raw cotton are as follows: Obtain multiple second multi-feature images of raw cotton and their corresponding detection results of foreign fibers in the target raw cotton; Input multiple second multi-feature images of raw cotton into the pre-built foreign fiber detection network of raw cotton for detection to obtain multiple second foreign fiber detection results of raw cotton; Compare multiple second foreign fiber detection results of raw cotton with their corresponding detection results of foreign fibers in the target raw cotton until the second foreign fiber detection results of raw cotton meet the preset conditions, and determine the foreign fiber detection network of raw cotton as the trained foreign fiber detection model of raw cotton.

6. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data augmentation according to claim 5, wherein The pre-built foreign fiber detection network of raw cotton specifically includes: A backbone module, which is used to extract the foreign fiber features in raw cotton; A neck module, whose input end is connected to the output end of the backbone module and is used to fuse and enhance the foreign fiber features in raw cotton; A prediction head module, whose input end is connected to the output end of the neck module and is used to output the detection result of foreign fibers in raw cotton.

7. The method for detecting foreign fibers in raw cotton based on bimodal image fusion and data augmentation according to claim 6, wherein The backbone module specifically includes: two layers of CBH convolution structures connected in sequence, one layer of CSP structure, one layer of CBH convolution structure, one layer of inception structure, one layer of CSP structure, and one layer of CBH convolution structure; The neck module consists of three channels, specifically including: The first channel of the neck module includes a first upsampling layer. The output end of the first upsampling layer and the output end of the first CSP structure in the backbone module are both connected to the input end of the first feature splicing ring. After the output end of the first feature splicing ring, one layer of CSP structure, one layer of CBH convolution structure, and a second upsampling layer are connected in sequence. The output end of the second upsampling layer and the output end of the first CBH convolution structure in the backbone module are both connected to the input end of the second feature splicing ring. The output end of one layer of CSP structure connected after the second feature splicing ring is used as the output end of the first channel of the neck module; The input end of the second channel of the neck module is the output end of the first feature splicing ring and is connected to the input end of the third feature splicing ring. The output end of the first channel of the neck module is connected to one layer of CBH convolution structure and then connected to the input end of the third feature splicing ring. The output end of one layer of CSP structure connected after the third feature splicing ring is used as the output end of the second channel of the neck module; The input end of the third channel of the neck module is the output end of the backbone module and is connected to the input end of the fourth feature splicing ring. The output end of the second channel of the neck module is connected to a CBH convolution structure and then connected to the input end of the fourth feature splicing ring. The output end of a CSP structure connected after the fourth feature splicing ring serves as the output end of the third channel of the neck module; The prediction head module specifically includes three Reshape layers. The input ends of the three Reshape layers are respectively connected to the output ends of the three channels of the neck module. The output ends of the three Reshape layers are all connected to a fifth feature splicing ring, and two CBH convolution structures and two Full connection layers are sequentially connected after the fifth feature splicing ring.

8. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data augmentation according to claim 7, characterized in that, The CBH convolution structure specifically includes: a 3×3 convolution layer, a Batch Norm regularization layer, and a Hard-Swish activation layer connected in sequence; The CSP structure specifically includes: after a 3×3 convolution layer and a fifth structure connected based on a residual network are connected in parallel, they are sequentially connected to a sixth feature splicing ring, a Batch Norm regularization layer, a leakyrelu activation layer, and a CBH convolution structure. The fifth structure includes two CBH convolution structures and a 3×3 convolution layer connected in sequence; The inception structure consists of four channels. The first channel of the inception structure includes a 1×1 convolution layer; The second channel of the inception structure includes a 1×1 convolution layer and a 3×3 convolution layer connected in sequence; The third channel of the inception structure includes a 3×3 max pooling layer and a 3×3 convolution layer connected in sequence; The fourth channel of the inception structure includes a 1×1 convolution layer, a 3×3 convolution layer, and a sixth structure formed by a 1×3 convolution layer and a 3×1 convolution layer connected in parallel; Finally, the output ends of the four channels are all connected to the seventh feature splicing ring.

Images