Raw cotton foreign fiber detection method based on bimodal image fusion and data enhancement

Abstract

The invention discloses a raw cotton foreign fiber detection method based on bimodal image fusion and data enhancement, relates to the technical field of agricultural intelligence, and aims to solve the problems that near-white foreign fibers such as mulching films and the like cannot be detected by using a common linear array camera and foreign fibers under poor illumination conditions cannot be detected by using a multispectral camera in the prior art. The method comprises the steps that a raw cotton full-color image and a raw cotton spectral image are collected, the raw cotton full-color image is collected through a color linear array CCD camera, and the raw cotton spectral image is collected through a multi-spectrometer; inputting the raw cotton full-color image and the raw cotton spectral image into the bimodal image fusion and data enhancement network structure for fusion to obtain a first raw cotton multi-feature image; and inputting the first raw cotton multi-feature image into a pre-trained raw cotton foreign fiber detection model for detection to obtain a first raw cotton foreign fiber detection result.

Description

Technical Field

[0001] The present application relates to the field of agricultural intelligent technology, and in particular to a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement. Background Art

[0002] Cotton plays a vital and indispensable role in national life and is intricately intertwined with all aspects of human existence, including daily life, medical applications, military use, and agriculture. my country is an important producer of raw cotton, with cotton production exceeding 16 million tons in the past five years. With the expansion of machine-picked cotton planting areas, the content of foreign fibers in cotton has increased significantly. Among them, the main foreign fibers present in raw cotton include chemical fibers, residual films, feathers, hair, and fabrics. In the subsequent processing of raw cotton, if the foreign fibers in the raw cotton cannot be detected and removed in time, they will cause them to break into many tiny particles. These particles are intertwined with the raw cotton yarn, which may cause problems such as easy breakage of the yarn and uneven dyeing, thereby affecting the quality of textiles.

[0003] At present, optical detection technology is usually used to sort and remove foreign fibers in raw cotton. The detection principle of optical detection technology is: raw cotton containing foreign fibers enters from the cotton inlet under the action of airflow and flows along the cotton conveying pipeline. The image acquisition system performs image acquisition, and the image processing algorithm performs feature extraction and other processing in real time, and feedbacks whether there are foreign fibers and their positions and other information, and transmits the information to the control unit. If there are foreign fibers in the raw cotton, the industrial computer drives the corresponding position solenoid valve to spray high-pressure gas to remove the foreign fibers. It can be foreseen that the image processing algorithm for foreign fibers is one of the key technologies. At present, the main image processing algorithms used are detection using U-Net network, detection using polarization imaging and enhanced YOLOV5, and detection using a combination of near-infrared spectroscopy and convolutional neural network. Among them, the combination of near-infrared spectroscopy and convolutional neural network to detect foreign fibers in raw cotton has a greater advantage in detection accuracy compared to other methods, but it still cannot achieve accurate detection. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present application provides a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement to solve the problem that ordinary linear array cameras cannot detect nearly white foreign fibers such as ground film and that multispectral cameras cannot detect foreign fibers under poor lighting conditions. This can synthesize a fused image containing prominent targets and rich texture details under normal lighting conditions, thereby achieving the purpose of accurately detecting foreign fibers in raw cotton.

[0005] To this end, the present application embodiment provides a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement, including:

[0006] Collecting a full-color image of raw cotton and a spectral image of raw cotton, wherein the full-color image of raw cotton is collected by a color linear array CCD camera, and the spectral image of raw cotton is collected by a multi-spectrometer;

[0007] The raw cotton full-color image and the raw cotton spectral image are input into the dual-modal image fusion and data enhancement network structure for fusion, so as to obtain a first raw cotton multi-feature image;

[0008] The first raw cotton multi-feature image is input into a pre-trained raw cotton foreign fiber detection model for detection to obtain the first raw cotton foreign fiber detection result.

[0009] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement, in the step of inputting the raw cotton full-color image and the raw cotton spectral image into the dual-modal image fusion and data enhancement network structure for fusion to obtain the first raw cotton multi-feature image, the network structure specifically includes:

[0010] A feature extraction module, which is used to extract features from the raw cotton full-color image and the raw cotton spectral image;

[0011] A feature fusion module, whose input end is connected to the output end of the feature extraction module, and the feature fusion module is used to fuse the features of the raw cotton full-color image and the raw cotton spectral image;

[0012] The feature enhancement module has an input end connected to the output end of the feature fusion module and is used to enhance the fused features to obtain a first raw cotton multi-feature image.

[0013] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement, in the step of inputting the raw cotton full-color image and the raw cotton spectral image into the dual-modal image fusion and data enhancement network structure for fusion to obtain the first raw cotton multi-feature image, the feature extraction module is specifically constructed by two dual-channel modules, each dual-channel module includes a first channel and a second channel, each first channel includes two first structures consisting of a 3×3 convolution layer and a ReLU activation layer connected in sequence and two second structures consisting of a 3×3 convolution layer and a ReLU activation layer based on residual connection; the second channel includes three third structures consisting of a 5×5 convolution layer and a ReLU activation layer connected in sequence, and the two second channels are connected through a Batch Norm regularization layer.

[0014] According to the above-mentioned raw cotton foreign fiber detection method based on bimodal image fusion and data enhancement, in the step of inputting the raw cotton full-color image and the raw cotton spectral image into the bimodal image fusion and data enhancement network structure for fusion to obtain a first raw cotton multi-feature image, the feature fusion module specifically includes three ResBlock modules connected in sequence and two fourth structures consisting of a 3×3 convolution layer and a ReLU activation layer, and the input end of the first ResBlock module among the three ResBlock modules is connected to the output end of the feature extraction module.

[0015] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement, in the step of inputting the raw cotton full-color image and the raw cotton spectral image into the dual-modal image fusion and data enhancement network structure for fusion to obtain the first raw cotton multi-feature image, the feature enhancement module is specifically constructed with three channels, the three channels are all connected to the second fourth structure, and the first channel is directly connected to the output; the second channel includes a 3×3 convolution layer, a Batch Norm regularization layer, a ReLU activation layer and a 3×3 convolution layer connected in sequence; the third channel includes a Real FFT2d layer, a 1×1 convolution layer, a Batch Norm regularization layer, a ReLU activation layer, a 1×1 convolution layer and an Inv Real FFT2d layer connected in sequence, the output ends of the three channels are connected, and the first raw cotton multi-feature image is output.

[0016] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement, in the step of inputting the first raw cotton multi-feature image into a pre-trained raw cotton foreign fiber detection model for detection to obtain the raw cotton foreign fiber detection result, the pre-trained raw cotton foreign fiber detection model is jointly determined by a training data set and a raw cotton foreign fiber detection network;

[0017] Among them, the training data set is a plurality of second raw cotton multi-feature images and their corresponding target raw cotton foreign fiber detection results, the second raw cotton multi-feature images are obtained by fusing the raw cotton full-color image and the raw cotton spectral image into a dual-modal image fusion and data enhancement network structure, and the target raw cotton foreign fiber detection results are obtained based on manual annotation of the second raw cotton multi-feature images.

[0018] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement, the steps of determining the pre-trained raw cotton foreign fiber detection model by the training data set and the raw cotton foreign fiber detection network are as follows:

[0019] Acquire a plurality of second raw cotton multi-feature images and their corresponding target raw cotton foreign fiber detection results;

[0020] Inputting a plurality of second raw cotton multi-feature images into a pre-built raw cotton foreign fiber detection network for detection, and obtaining a plurality of second raw cotton foreign fiber detection results;

[0021] The plurality of second raw cotton foreign fiber detection results are compared with their corresponding target raw cotton foreign fiber detection results until the second raw cotton foreign fiber detection results meet the preset conditions, and the raw cotton foreign fiber detection network is determined as the trained raw cotton foreign fiber detection model.

[0022] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement, in the step of inputting a plurality of second raw cotton multi-feature images and a target raw cotton multi-feature image into a pre-built raw cotton foreign fiber detection network for detection to obtain a plurality of second raw cotton foreign fiber detection results, the pre-built raw cotton foreign fiber detection network specifically includes:

[0023] A backbone module, which is used to extract foreign fiber features from raw cotton;

[0024] A neck module, whose input end is connected to the output end of the backbone module and is used to integrate and enhance the characteristics of foreign fibers in the raw cotton;

[0025] The prediction head module has an input end connected to the output end of the neck module and is used to output the detection result of the foreign fiber of the raw cotton.

[0026] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement, in the step of inputting a plurality of second raw cotton multi-feature images and a target raw cotton multi-feature image into a pre-built raw cotton foreign fiber detection network for detection to obtain a plurality of second raw cotton foreign fiber detection results, the backbone module specifically includes: two CBH convolution structures, a layer of CSP structure, a layer of CBH structure, a layer of inception structure, a layer of CSP structure and a CBH convolution structure connected in sequence;

[0027] The neck module is specifically constructed of three channels, the first channel includes an upsampling layer, the output end of the upsampling layer and the output end of the first layer of CSP structure in the backbone module are connected to the first feature splicing ring, the output end of the first feature splicing ring is connected to a layer of CSP structure, a layer of CBH convolution structure and a layer of upsampling layer in sequence, the output end of the latter upsampling layer and the output end of the first layer of CBH structure in the backbone module are both connected to the input end of the second feature splicing ring, and the second feature splicing ring is connected to a layer of CSP structure; the second channel is the output end of the first feature splicing ring, and the output end of the first channel is connected to a layer of CBH convolution structure and then connected to the third feature splicing ring, and the output end of the third feature splicing ring is connected to a layer of CSP structure; the third channel includes the output end of the backbone module, the output end of the second channel is connected to a layer of CBH convolution structure and then connected to a fourth feature splicing ring, and the output end of the fourth feature splicing ring is connected to a layer of CSP structure;

[0028] The prediction head module specifically includes three Reshape layers, the input ends of the three Reshape layers are respectively connected to the output ends of the three channels of the neck module, the output ends of the three Reshape layers are connected to the same feature splicing ring, and the output end of the feature splicing ring is connected to two CBH convolution structures and two Full connections in sequence.

[0029] According to the above-mentioned raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement, in the step of inputting a plurality of second raw cotton multi-feature images and a target raw cotton multi-feature image into a pre-built raw cotton foreign fiber detection network for detection to obtain a plurality of second raw cotton foreign fiber detection results, the CBH structure specifically includes: a 3×3 convolution layer, a Batch Norm regularization layer and a Hard-Swish activation layer connected in sequence;

[0030] The CSP structure specifically includes: two CBH convolutional structures connected in sequence, a 3×3 convolutional layer and a 3×3 convolutional layer based on residual network connection are connected to a feature splicing ring, and the output end of the feature splicing ring is connected to a Batch Norm regularization layer, a leakyrelu activation layer and a CBH convolutional structure in sequence;

[0031] The inception structure is specifically constructed with four channels. The first channel includes a 1×1 convolution layer, the output end of which is directly connected to the feature splicing ring; the second channel includes a 1×1 convolution layer, a 3×3 convolution layer and a feature splicing ring connected in sequence; the third channel includes a 3×3 maximum pooling layer, a 3×3 convolution and a feature splicing ring connected in sequence; the fourth channel includes a 1×1 convolution layer and a 3×3 convolution layer connected in sequence, and the 3×3 convolution layer is respectively connected to a 1×3 convolution layer and a 3×1 convolution layer, and the output result is finally connected to the feature splicing ring.

[0032] The beneficial effects of the raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement provided by the present application are at least:

[0033] The present application sets up a network structure for dual-modal image fusion and data enhancement to fuse the full-color image of raw cotton and the spectral image of raw cotton, thereby obtaining a first raw cotton multi-feature image that contains both rich detail structures such as textures and rich visible light details. The present application inputs the first raw cotton multi-feature image into a pre-trained raw cotton foreign fiber detection model for detection, and can accurately detect foreign fibers such as colored cloth strips that are clearly visible under visible light conditions and foreign fibers such as white feathers, hair and transparent films that are easily identifiable under multi-spectral conditions, thereby achieving accurate detection of foreign fibers in raw cotton. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A flow chart of a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement provided in an embodiment of the present application.

[0036] Figure 2 A schematic diagram of the composition of an image acquisition device involved in a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement provided in an embodiment of the present application.

[0037] Figure 3 A schematic diagram of the composition of the network structure involved in the raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement provided in an embodiment of the present application.

[0038] Figure 4A schematic diagram of the composition of the ResBlock module in the raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement provided in an embodiment of the present application.

[0039] Figure 5 A schematic diagram of the composition of a feature enhancement module in a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement provided in an embodiment of the present application.

[0040] Figure 6 A schematic diagram of the composition of a raw cotton foreign fiber detection network in a raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement provided in an embodiment of the present application.

[0041] Figure 7 A schematic diagram of the composition of the CBH convolution structure in the raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement provided in an embodiment of the present application.

[0042] Figure 8 A schematic diagram of the composition of the CSP structure in the raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement provided in an embodiment of the present application.

[0043] Fig. 9 A schematic diagram of the composition of the inception structure in the raw cotton foreign fiber detection method based on dual-modal image fusion and data enhancement provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] Cotton plays a vital and indispensable role in national life and is intricately intertwined with all aspects of human existence, including daily life, medical applications, military use, and agriculture. my country is an important producer of raw cotton, with cotton production exceeding 5 million tons in the past five years. With the expansion of machine-picked cotton planting areas, the content of foreign fibers in cotton has increased significantly. Among them, the main foreign fibers present in raw cotton include chemical fibers, residual films, feathers, hair, and fabrics. In the subsequent processing of raw cotton, if the foreign fibers in the raw cotton cannot be detected and removed in time, they will cause them to break into many tiny particles. These particles are intertwined with the raw cotton yarn, which may cause problems such as easy breakage of the yarn and uneven dyeing, thereby affecting the quality of textiles.

[0046] At present, optical detection technology is usually used to sort and remove foreign fibers in raw cotton. The detection principle of optical detection technology is: raw cotton containing foreign fibers enters from the cotton inlet under the action of airflow and flows along the cotton conveying pipeline. The image acquisition system performs image acquisition, and the image processing algorithm performs feature extraction and other processing in real time, and feedbacks whether there are foreign fibers and their positions and other information, and transmits the information to the control unit. If there are foreign fibers in the raw cotton, the industrial computer drives the corresponding position solenoid valve to spray high-pressure gas to remove the foreign fibers. It can be foreseen that the image processing algorithm for foreign fibers is one of the key technologies. At present, the main image processing algorithms used are detection using U-Net network, detection using polarization imaging and enhanced YOLOV5, and detection using a combination of near-infrared spectroscopy and convolutional neural network. Among them, the combination of near-infrared spectroscopy and convolutional neural network to detect foreign fibers in raw cotton has a greater advantage in detection accuracy compared to other methods, but it still cannot achieve accurate detection.

[0047] The inventors have found that in the existing image processing algorithm for detecting foreign fibers in raw cotton by combining near-infrared spectroscopy with convolutional neural networks, if a color linear array CCD camera is used to collect images, the collected full-color image has the characteristics of high spatial resolution, contains rich texture structures, and has a good detection effect on foreign fibers with color, but the collection of characteristic information of foreign fibers that are not easy to distinguish, such as white feathers, hair and films, is not prominent enough; if a multi-spectrometer is used to collect images, the collected multi-spectral image has the characteristics of high spectral resolution, and has a good detection effect on white feathers, hair, films and foreign fibers with fluorescent characteristics, but lacks detailed characteristic information, and the recognition of characteristic information of foreign fibers with easy color distinction is not accurate enough. However, the foreign fibers in raw cotton include not only foreign fibers such as white feathers, hair and films, but also foreign fibers with color. How to simultaneously use the advantages of high spatial resolution and high spectral resolution to detect foreign fibers in raw cotton and achieve accurate detection of foreign fibers in raw cotton is a problem to be solved in this application.

[0048] The present application collects a full-color image of raw cotton and a spectral image of raw cotton, wherein the full-color image of raw cotton is collected by a color linear array CCD camera, and the spectral image of raw cotton is collected by a multi-spectrometer; the full-color image of raw cotton and the spectral image of raw cotton are input into a network structure of dual-modal image fusion and data enhancement for fusion, so as to obtain a first raw cotton multi-feature image; the first raw cotton multi-feature image is input into a pre-trained raw cotton foreign fiber detection model for detection, so as to obtain a first raw cotton foreign fiber detection result. That is to say, the present embodiment fuses the full-color image of raw cotton and the spectral image of raw cotton by setting a network structure of dual-modal image fusion and data enhancement, so as to obtain a first raw cotton multi-feature image containing both rich detail structures such as texture and rich visible light details. The present embodiment inputs the first raw cotton multi-feature image into a pre-trained raw cotton foreign fiber detection model for detection, so as to accurately detect foreign fibers such as colored cloth strips that are clearly visible under visible light conditions and foreign fibers such as white feathers, hair and transparent films that are easily identifiable under multi-spectral conditions, thereby realizing accurate detection of foreign fibers in raw cotton.

[0049] The invention is further described below by describing embodiments in conjunction with the accompanying drawings.

[0050] See also Figure 1 This embodiment provides a method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement, and the method for detecting foreign fibers in raw cotton includes:

[0051] S10, collecting full-color images and spectral images of raw cotton;

[0052] S20, inputting the raw cotton full-color image and the raw cotton spectral image into a dual-modal image fusion and data enhancement network structure for fusion, to obtain a first raw cotton multi-feature image;

[0053] S30, inputting the first raw cotton multi-feature image into a pre-trained raw cotton foreign fiber detection model for detection, and obtaining a first raw cotton foreign fiber detection result.

[0054] Specifically, the pre-trained raw cotton foreign fiber detection models are all implemented on the Pytorch (which is an open source Python machine learning library based on Torch and used for applications such as natural language processing) framework, the raw cotton full-color image is collected by a color linear array CCD camera, the raw cotton spectral image is collected by a multi-spectrometer, and the first raw cotton multi-feature image is obtained by inputting the collected raw cotton full-color image and raw cotton spectral image into a dual-modal image fusion and data enhancement network structure. The first raw cotton foreign fiber detection result can be understood as a kind of information, for example: it can be considered whether there are foreign fibers in the raw cotton and the position information of the foreign fibers in the raw cotton.

[0055] Among them, see Figure 2 The image acquisition device for the full-color image and the spectral image of raw cotton may include a cotton flow channel 1, a lamp tube 3, a lens 4, two color linear array CCD cameras 5 and two multi-spectrometers 6. The channel wall 2 of the cotton flow channel 1 may be made of a high-transmittance glass panel to ensure image quality. The cotton flow channel 1 is used to transmit the cotton flow. The lamp tube 3 is used to provide a light source for the cotton flow. The cotton flow channel 1 is divided into an upper cotton flow channel and a lower cotton flow channel. Both the upper cotton flow channel and the lower cotton flow channel are provided with a color linear array CCD camera 5 and a multi-spectrometer 6. To ensure the spatial consistency of the full-color image and the spectral image of raw cotton, the full-color image and the spectral image of raw cotton are respectively acquired at the same position. The two cameras of each group are respectively placed at the upper cotton flow channel and the lower cotton flow channel. When the cotton flow passes through the cotton flow channel, the full-color image and the spectral image of raw cotton are acquired by the color linear array CCD camera and the multi-spectrometer.

[0056] Among them, since the multi-spectrometer has higher requirements for images, in order to reduce the feature loss of the raw cotton spectral image, the multi-spectrometer is used to directly collect images at the collection point of the cotton flow channel, and the color linear array CCD camera collects images through the reflection of lens 4. It is worth noting that according to the classification of foreign fibers in the raw cotton in different bands, the classification effect is best when the band of the raw cotton spectral image is 780nm-850nm. Therefore, when collecting images, the band of the raw cotton spectral image collected by the multi-spectrometer is set to 780nm-850nm.

[0057] Optional, see Figure 3 In one embodiment, in the step of inputting the raw cotton full-color image and the raw cotton spectral image into the network structure of dual-modal image fusion and data enhancement to obtain a first raw cotton multi-feature image, the network structure specifically includes a feature extraction module, a feature fusion module and a feature enhancement module. The feature extraction module is used to extract features from the raw cotton full-color image and the raw cotton spectral image. The input end of the feature fusion module is connected to the output end of the feature extraction module. The feature fusion module is used to fuse the features of the raw cotton full-color image and the raw cotton spectral image. The input end of the feature enhancement module is connected to the output end of the feature fusion module and is used to enhance the fused features to obtain the first raw cotton multi-feature image.

[0058] Optional, see Figure 3In one embodiment, the feature extraction module is specifically formed by two dual-channel modules, and the two dual-channel modules are used for multi-scale feature extraction to fully capture the complementary features in the raw cotton full-color image and the raw cotton spectral image. Each dual-channel module includes a first channel and a second channel, and each first channel includes two first structures consisting of a 3×3 convolution layer (abbreviated as: Conv3×3) and a ReLU activation layer (abbreviated as: ReLU) connected in sequence and two second structures consisting of a 3×3 convolution layer and a ReLU activation layer based on residual connection; the second channel includes three third structures consisting of a 5×5 convolution layer and a ReLU activation layer connected in sequence, and the output ends of the two second channels are connected by a Batch Norm regularization layer (abbreviated as: BN), aiming to change the number of channels so that the number of channels of each channel processing structure is the same, which is convenient for the next step of feature fusion.

[0059] Optional, see Figure 3 In one embodiment, the feature fusion module specifically includes three ResBlock modules (referred to as: ResBlock) connected in sequence and two fourth structures consisting of a 3×3 convolution layer and a ReLU activation layer, and the input end of the first ResBlock module of the three ResBlock modules is respectively connected to the output end of the feature extraction module;

[0060] Among them, see Figure 4 The ResBlock module includes two fifth structures consisting of a 3×3 convolution layer and a ReLU activation layer connected in sequence, and a 3×3 convolution layer, which includes a residual structure and is connected to a ReLU activation layer.

[0061] Optional, see Figure 5 In one embodiment, the feature enhancement module is specifically constructed with three channels, including a three-channel multi-scale feature enhancement constructed based on a residual structure connection, wherein the three channels are all connected to the second fourth structure, and the first channel is directly connected to the output; the second channel includes a 3×3 convolution layer, a Batch Norm regularization layer, a ReLU activation layer and a 3×3 convolution layer connected in sequence; the third channel includes a Real FFT2d layer (abbreviated as: Real FFT2d), a 1×1 convolution layer, a Batch Norm regularization layer, a ReLU activation layer, a 1×1 convolution layer and an Inv Real FFT2d layer (abbreviated as: Inv Real FFT2d) connected in sequence, the output ends of the three channels are connected, and the first raw cotton multi-feature image is output.

[0062] This embodiment introduces a residual mechanism to address the problems of unclear edges and degraded feature information of raw cotton multi-feature images, and uses the Real FFT2d layer and Inv Real FFT2d layer methods to enhance the raw cotton multi-feature images, achieving significant acceleration in the raw cotton foreign fiber detection stage, thereby effectively improving the overall computing efficiency.

[0063] Optionally, in one embodiment, in the step of inputting the first raw cotton multi-feature image into a pre-trained raw cotton foreign fiber detection model for detection to obtain the raw cotton foreign fiber detection result, the pre-trained raw cotton foreign fiber detection model is jointly determined by a training data set and a raw cotton foreign fiber detection network;

[0064] Among them, the training data set is a plurality of second raw cotton multi-feature images and their corresponding target raw cotton foreign fiber detection results, the second raw cotton multi-feature images are obtained by fusing the raw cotton full-color image and the raw cotton spectral image into a dual-modal image fusion and data enhancement network structure, and the target raw cotton foreign fiber detection results are obtained based on manual annotation of the second raw cotton multi-feature images.

[0065] Optionally, in one embodiment, the pre-trained raw cotton foreign fiber detection model is determined by the training data set and the raw cotton foreign fiber detection network in the following steps:

[0066] Acquire a plurality of second raw cotton multi-feature images and their corresponding target raw cotton foreign fiber detection results;

[0067] Inputting a plurality of second raw cotton multi-feature images into a pre-built raw cotton foreign fiber detection network for detection, and obtaining a plurality of second raw cotton foreign fiber detection results;

[0068] The plurality of second raw cotton foreign fiber detection results are compared with their corresponding target raw cotton foreign fiber detection results until the second raw cotton foreign fiber detection results meet the preset conditions, and the raw cotton foreign fiber detection network is determined as the trained raw cotton foreign fiber detection model.

[0069] Specifically, the raw cotton foreign fiber detection network can be a convolutional neural network based on the YOLO series, and the preset condition is: a plurality of different second raw cotton multi-feature images are input into a pre-built raw cotton foreign fiber detection network for training, until the output second raw cotton foreign fiber detection result is continuously close to the target raw cotton foreign fiber detection result, and the optimal weight parameters of the raw cotton foreign fiber detection model are obtained, and the optimal weight parameters of the raw cotton foreign fiber detection model are loaded into the pre-built raw cotton foreign fiber detection network to form a trained raw cotton foreign fiber detection model.

[0070] For further information, see Figure 6 In one embodiment, the pre-built raw cotton foreign fiber detection network specifically includes a backbone module, a neck module and a prediction module. The input end of the backbone module is connected to the output end of the feature enhancement module and is used to extract the foreign fiber features in the raw cotton. The input end of the neck module is connected to the output end of the backbone module and is used to fuse and enhance the foreign fiber features in the raw cotton. The input end of the prediction head module is connected to the output end of the neck module and is used to output the raw cotton foreign fiber detection result.

[0071] For further information, see Figure 6 In one embodiment, the backbone module specifically includes: two CBH convolution structures (abbreviated as: CBH) connected in sequence, a layer of CSP structure (abbreviated as: CSP), a layer of CBH structure, a layer of inception structure (abbreviated as: inception), a layer of CSP structure and a CBH convolution structure.

[0072] For further information, see Figure 6 In one embodiment, the neck module adopts a bottom-up feature fusion structure to fully perform feature fusion, which is specifically constructed by three channels, the first channel includes an upsampling layer, the output end of the upsampling layer and the output end of the first layer of CSP structure in the backbone module are both connected to the first feature splicing ring, the output end of the first feature splicing ring is connected to a layer of CSP structure, a layer of CBH convolution structure and a layer of upsampling layer in sequence, the output end of the latter upsampling layer and the output end of the first layer of CBH structure in the backbone module are both connected to the input end of the second feature splicing ring, and the second feature splicing ring is connected to a layer of CSP structure; the second channel is the output end of the first feature splicing ring, and the output end of the first channel is connected to a third feature splicing ring after connecting a layer of CBH convolution structure, and the output end of the third feature splicing ring is connected to a layer of CSP structure; the third channel includes the output end of the backbone module, the output end of the second channel is connected to a fourth feature splicing ring after connecting a layer of CBH convolution structure, and the output end of the fourth feature splicing ring is connected to a layer of CSP structure;

[0073] Among them, see Figure 6 The prediction head module specifically includes three Reshape layers (abbreviated as: Reshape), the input ends of the three Reshape layers are respectively connected to the output ends of the three channels of the neck module, the output ends of the three Reshape layers are connected to the same feature splicing ring, and the output end of the feature splicing ring is connected to two CBH convolution structures and two Fullconnection layers (abbreviated as: Full connection) in sequence to output the detection result of raw cotton foreign fiber.

[0074] For further information, see Figure 7 In one embodiment, the CBH structure specifically includes: a 3×3 convolutional layer, a Batch Norm regularization layer, and a Hard-Swish activation layer (abbreviated as: Hard-Swish) connected in sequence.

[0075] For further information, see Figure 8 In one embodiment, the CSP structure specifically includes: two CBH convolutional structures connected in sequence, a 3×3 convolutional layer and a 3×3 convolutional layer based on residual network connection are connected to a feature splicing ring, and the output end of the feature splicing ring is connected to a Batch Norm regularization layer, a leakyrelu activation layer (abbreviated as: leakyrelu) and a CBH convolutional structure in sequence.

[0076] For further information, see Fig. 9 In one embodiment, the inception structure is specifically constructed with four channels, the first channel includes a 1×1 convolution layer whose output end is directly connected to the feature splicing ring; the second channel includes a 1×1 convolution layer, a 3×3 convolution layer and a feature splicing ring connected in sequence; the third channel includes a 3×3 maximum pooling layer, a 3×3 convolution layer and a feature splicing ring connected in sequence; the fourth channel includes a 1×1 convolution layer and a 3×3 convolution layer connected in sequence, the 3×3 convolution layer is respectively connected to a 1×3 convolution layer and a 3×1 convolution layer, and the output result is finally connected to the feature splicing ring.

[0077] This embodiment fully utilizes the high spatial resolution of the raw cotton full-color image and the advantages of the raw cotton spectral image and high spectral resolution by performing dual-modal image fusion on the raw cotton full-color image and the raw cotton spectral image, obtains richer raw cotton image feature information, improves the detection accuracy of the raw cotton foreign fiber detection model for foreign fibers in the raw cotton, and performs data enhancement on the raw cotton multi-feature image, further improving the model's extraction efficiency for the raw cotton multi-feature image, thereby improving the detection accuracy of the raw cotton foreign fiber detection model for foreign fibers in the raw cotton, and ensuring the accuracy and speed of the raw cotton foreign fiber detection.

[0078] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement, characterized in that: include: Collecting a full-color image of raw cotton and a spectral image of raw cotton, wherein the full-color image of raw cotton is collected by a color linear array CCD camera, and the spectral image of raw cotton is collected by a multi-spectrometer; The raw cotton full-color image and the raw cotton spectral image are input into the dual-modal image fusion and data enhancement network structure for fusion, so as to obtain a first raw cotton multi-feature image; The first raw cotton multi-feature image is input into a pre-trained raw cotton foreign fiber detection model for detection to obtain the first raw cotton foreign fiber detection result.

2. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 1, characterized in that: In the step of inputting the raw cotton full-color image and the raw cotton spectral image into the dual-modal image fusion and data enhancement network structure for fusion to obtain the first raw cotton multi-feature image, the network structure specifically includes: A feature extraction module, which is used to extract features from the raw cotton full-color image and the raw cotton spectral image; A feature fusion module, whose input end is connected to the output end of the feature extraction module, and the feature fusion module is used to fuse the features of the raw cotton full-color image and the raw cotton spectral image; The feature enhancement module has an input end connected to the output end of the feature fusion module and is used to enhance the fused features to obtain a first raw cotton multi-feature image.

3. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 2, characterized in that: In the step of inputting the raw cotton full-color image and the raw cotton spectral image into the dual-modal image fusion and data enhancement network structure for fusion to obtain the first raw cotton multi-feature image, the feature extraction module is specifically constructed by two dual-channel modules, each dual-channel module includes a first channel and a second channel, each first channel includes two first structures consisting of a 3×3 convolution layer and a ReLU activation layer connected in sequence and two second structures consisting of a 3×3 convolution layer and a ReLU activation layer based on residual connection; the second channel includes three third structures consisting of a 5×5 convolution layer and a ReLU activation layer connected in sequence, and the two second channels are connected through a Batch Norm regularization layer.

4. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 3, characterized in that: In the step of inputting the raw cotton full-color image and the raw cotton spectral image into the network structure of dual-modal image fusion and data enhancement for fusion to obtain a first raw cotton multi-feature image, the feature fusion module specifically includes three ResBlock modules connected in sequence and two fourth structures consisting of a 3×3 convolution layer and a ReLU activation layer, and the input end of the first ResBlock module among the three ResBlock modules is connected to the output end of the feature extraction module.

5. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 4, characterized in that: In the step of inputting the raw cotton full-color image and the raw cotton spectral image into the network structure of dual-modal image fusion and data enhancement to obtain the first raw cotton multi-feature image, the feature enhancement module is specifically constructed with three channels, and the three channels are all connected to the second fourth structure, and the first channel is directly connected to the output; the second channel includes a 3×3 convolution layer, a Batch Norm regularization layer, a ReLU activation layer and a 3×3 convolution layer connected in sequence; the third channel includes a Real FFT2d layer, a 1×1 convolution layer, a Batch Norm regularization layer, a ReLU activation layer, a 1×1 convolution layer and an Inv Real FFT2d layer connected in sequence, the output ends of the three channels are connected, and the first raw cotton multi-feature image is output.

6. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 1, characterized in that: In the step of inputting the first raw cotton multi-feature image into a pre-trained raw cotton foreign fiber detection model for detection to obtain the raw cotton foreign fiber detection result, the pre-trained raw cotton foreign fiber detection model is jointly determined by a training data set and a raw cotton foreign fiber detection network; Among them, the training data set is a plurality of second raw cotton multi-feature images and their corresponding target raw cotton foreign fiber detection results, the second raw cotton multi-feature images are obtained by fusing the raw cotton full-color image and the raw cotton spectral image into a dual-modal image fusion and data enhancement network structure, and the target raw cotton foreign fiber detection results are obtained based on manual annotation of the second raw cotton multi-feature images.

7. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 6, characterized in that: The pre-trained raw cotton foreign fiber detection model is determined by the training data set and the raw cotton foreign fiber detection network in the following steps: Acquire a plurality of second raw cotton multi-feature images and their corresponding target raw cotton foreign fiber detection results; Inputting a plurality of second raw cotton multi-feature images into a pre-built raw cotton foreign fiber detection network for detection, and obtaining a plurality of second raw cotton foreign fiber detection results; The plurality of second raw cotton foreign fiber detection results are compared with their corresponding target raw cotton foreign fiber detection results until the second raw cotton foreign fiber detection results meet the preset conditions, and the raw cotton foreign fiber detection network is determined as the trained raw cotton foreign fiber detection model.

8. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 7, characterized in that: In the step of inputting the plurality of second raw cotton multi-feature images and the target raw cotton multi-feature image into a pre-built raw cotton foreign fiber detection network for detection to obtain a plurality of second raw cotton foreign fiber detection results, the pre-built raw cotton foreign fiber detection network specifically comprises: A backbone module, which is used to extract foreign fiber features from raw cotton; A neck module, whose input end is connected to the output end of the backbone module and is used to integrate and enhance the characteristics of foreign fibers in the raw cotton; The prediction head module has an input end connected to the output end of the neck module and is used to output the detection result of the foreign fiber of the raw cotton.

9. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 8, characterized in that: In the step of inputting the plurality of second raw cotton multi-feature images and the target raw cotton multi-feature images into a pre-built raw cotton foreign fiber detection network for detection to obtain the plurality of second raw cotton foreign fiber detection results, the backbone module specifically comprises: two CBH convolution structures, a layer of CSP structure, a layer of CBH structure, a layer of inception structure, a layer of CSP structure and a CBH convolution structure connected in sequence; The neck module is specifically constructed of three channels, the first channel includes an upsampling layer, the output end of the upsampling layer and the output end of the first layer of CSP structure in the backbone module are connected to the first feature splicing ring, the output end of the first feature splicing ring is connected to a layer of CSP structure, a layer of CBH convolution structure and a layer of upsampling layer in sequence, the output end of the latter upsampling layer and the output end of the first layer of CBH structure in the backbone module are both connected to the input end of the second feature splicing ring, and the second feature splicing ring is connected to a layer of CSP structure; the second channel is the output end of the first feature splicing ring, and the output end of the first channel is connected to a layer of CBH convolution structure and then connected to the third feature splicing ring, and the output end of the third feature splicing ring is connected to a layer of CSP structure; the third channel includes the output end of the backbone module, the output end of the second channel is connected to a layer of CBH convolution structure and then connected to a fourth feature splicing ring, and the output end of the fourth feature splicing ring is connected to a layer of CSP structure; The prediction head module specifically includes three Reshape layers, the input ends of the three Reshape layers are respectively connected to the output ends of the three channels of the neck module, the output ends of the three Reshape layers are connected to the same feature splicing ring, and the output end of the feature splicing ring is connected to two CBH convolution structures and two Full connections in sequence.

10. The method for detecting foreign fibers in raw cotton based on dual-modal image fusion and data enhancement according to claim 9, characterized in that: In the step of inputting the plurality of second raw cotton multi-feature images and the target raw cotton multi-feature image into a pre-built raw cotton foreign fiber detection network for detection to obtain the plurality of second raw cotton foreign fiber detection results, the CBH structure specifically comprises: a 3×3 convolution layer, a Batch Norm regularization layer and a Hard-Swish activation layer connected in sequence; The CSP structure specifically includes: two CBH convolutional structures connected in sequence, a 3×3 convolutional layer and a 3×3 convolutional layer based on residual network connection are connected to a feature splicing ring, and the output end of the feature splicing ring is connected to a Batch Norm regularization layer, a leakyrelu activation layer and a CBH convolutional structure in sequence; The inception structure is specifically constructed with four channels. The first channel includes a 1×1 convolution layer, the output end of which is directly connected to the feature splicing ring; the second channel includes a 1×1 convolution layer, a 3×3 convolution layer and a feature splicing ring connected in sequence; the third channel includes a 3×3 maximum pooling layer, a 3×3 convolution and a feature splicing ring connected in sequence; the fourth channel includes a 1×1 convolution layer and a 3×3 convolution layer connected in sequence, and the 3×3 convolution layer is respectively connected to a 1×3 convolution layer and a 3×1 convolution layer, and the output result is finally connected to the feature splicing ring.

Images