A method and system for detecting impurities in bird's nest based on polarization imaging

Abstract

The application discloses a kind of based on polarization imaging's bird's nest impurity detection method and system, belong to image processing and deep learning technical field.It includes: obtaining the polarization sub-image of multiple polarization directions of bird's nest, calculates intensity image and linear polarization degree image;Two kinds of images are respectively extracted features, generate intensity feature map and polarization degree feature map;Two kinds of feature maps are fused, generate fusion feature map;Fusion feature map is input into detection network and carries out multi-scale feature extraction, and the convolution layer of detection network utilizes linear polarization degree image to adjust depth convolution kernel weight;Multi-scale feature fusion is carried out to multi-scale feature map, and the class information of the boundary box position of bird's nest impurity is output.The above-mentioned scheme utilizes polarization imaging to distinguish bird's nest and impurity from physical dimension, realizes depth collaborative detection by polarization guided convolution weight adjustment mechanism, network keeps lightweight, and is suitable for industrial real-time detection scene.

Description

Technical Field

[0001] This invention relates to the fields of image processing and deep learning technology, and in particular to a method and system for detecting impurities in bird's nest based on polarization imaging. Background Technology

[0002] Bird's nest is a precious tonic food, mainly composed of the saliva secreted by swiftlets mixed with feathers, plant fibers, and other substances. During the harvesting, processing, and packaging of bird's nest, various impurities such as feathers, eggshell fragments, black spots, and grass clippings inevitably get mixed in. These impurities not only affect the product's appearance but may also cause food safety issues. Therefore, strict impurity testing and sorting must be carried out before bird's nest enters the market.

[0003] Traditional methods for detecting impurities in bird's nests primarily rely on manual visual sorting, which suffers from low efficiency, high cost, and susceptibility to subjective factors. With the development of machine vision technology, automated detection methods based on ordinary visible light images are gradually being introduced into bird's nest production lines. These methods use industrial cameras to capture color images of the bird's nests and employ image processing algorithms or deep learning models to identify impurity areas within the images.

[0004] However, detection methods based on ordinary visible light images have the following shortcomings: First, the bird's nest matrix itself is translucent milky white, and its contrast with light-colored impurities (such as white feathers, transparent eggshell fragments, etc.) in terms of color and grayscale is extremely low, making it difficult to distinguish with traditional image features, resulting in a high false negative rate; Second, the filamentous texture structure inside the bird's nest is complex, and it will produce a lot of light and dark changes under ordinary light, which can easily be misjudged as impurities, causing false positives; Third, some impurities are embedded inside the bird's nest or tightly combined with the matrix, and it is difficult to accurately identify their boundaries and categories based solely on the intensity information of light reflected from the surface.

[0005] Polarization imaging technology, by acquiring light intensity information of a target in different polarization directions, can reveal differences in material properties that ordinary intensity images cannot reflect. Different substances, due to differences in surface morphology, internal structure, and refractive index, alter the polarization state of incident light to varying degrees. This physical characteristic provides a new dimension for distinguishing bird's nest matrix from impurities. However, how to effectively integrate polarization information into deep learning detection networks to create a synergistic complement with intensity information, while maintaining the network's lightweight nature to meet the demands of real-time industrial detection, remains a pressing technical challenge. Summary of the Invention

[0006] To overcome the above shortcomings, this invention provides a method and system for detecting impurities in bird's nests based on polarization imaging, which solves the technical problems of high false positive rate of light-colored impurities and many false positives of bird's nest texture in ordinary visible light image detection, while ensuring the lightweight and real-time performance of the detection network.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for detecting impurities in bird's nest based on polarization imaging, comprising: A polariton image with multiple polarization directions is acquired for the bird's nest to be tested, and an intensity image and a linear polarization degree image are calculated based on the polariton image; Feature extraction is performed on the intensity image and the linear polarization image respectively to generate an intensity feature map and a polarization feature map; The intensity feature map is convolved pointwise and activated by a function to generate a first spatial gating map. The polarization feature map is then multiplied element-wise by the first spatial gating map, and the residuals are summed to obtain a modulated polarization feature map. The polarization feature map is then convolved pointwise and activated by a function to generate a second spatial gating map. The intensity feature map is then multiplied element-wise by the second spatial gating map, and the residuals are summed to obtain a modulated intensity feature map. The modulated polarization feature map and the modulated intensity feature map are then concatenated and their channels are compressed to obtain a fused feature map. The fused feature map is input into a detection network for multi-scale feature extraction. The detection network includes a convolutional layer with a depthwise convolutional kernel. The weights of the depthwise convolutional kernel are adjusted using the linear polarization image to obtain the multi-scale feature map. Multi-scale feature fusion is performed on the multi-scale feature map, and the bounding box location and category information of bird's nest impurities are output.

[0008] Optionally, acquiring polariton images of multiple polarization directions of the bird's nest to be tested, and calculating the intensity image and linear polarization degree image based on the polariton images, includes: Polariton images of the bird's nest under test in multiple polarization directions were obtained using polarization imaging. The total intensity component, the first linear polarization component, and the second linear polarization component in the Stokes parameters are calculated based on the polariton images of the multiple polarization directions. The image corresponding to the total light intensity component is used as the intensity image, and the linear polarization degree image is calculated based on the first linear polarization component and the second linear polarization component.

[0009] Optionally, the step of extracting features from the intensity image and the linear polarization image to generate an intensity feature map and a polarization feature map includes: The intensity image is input into the first convolutional layer for feature extraction to obtain the intensity feature map; The linear polarization image is input into the second convolutional layer for feature extraction to obtain the polarization feature map; The first convolutional layer and the second convolutional layer have the same structure, and the intensity feature map and the polarization feature map have the same spatial size and number of channels.

[0010] Optionally, obtaining the fused feature map includes: The intensity feature map is processed by pointwise convolution and activation function to generate a first spatial gating map; Multiply the first spatial gating map element by element with the polarization degree feature map, and then add the first spatial gating map element by element with the polarization degree feature map to obtain the modulated polarization degree feature map. The polarization feature map is processed by pointwise convolution and activation function to generate a second spatial gated map; The second spatial gating map is multiplied element-by-element by the intensity feature map, and then added element-by-element by the intensity feature map to obtain the modulated intensity feature map; The modulated polarization feature map and the modulated intensity feature map are concatenated along the channel dimension, and then the number of channels is compressed by pointwise convolution to output the fused feature map.

[0011] Optionally, adjusting the weights of the depthwise convolution kernel using the linear polarization image includes: The input feature map of the convolutional layer is divided into a first branch feature and a second branch feature according to the channel, and the first branch feature is retained as the first branch output. The linear polarization image is subjected to average pooling to the same spatial size as the second branch feature to obtain the pooled linear polarization image. The pooled linear polarization image is input into a prediction network consisting of two layers of pointwise convolution and activation functions, and the output is a convolution kernel perturbation residual tensor that matches the depth convolution kernel size. The perturbation residual tensor of the convolution kernel is reshaped and added to the standard depth convolution kernel to obtain a dynamic convolution kernel. The dynamic convolution kernel is then used to perform channel-wise depth convolution on the second branch feature to obtain the convolution-processed second branch feature. The second branch feature after convolution processing is then adjusted for the number of channels by pointwise convolution, concatenated with the first branch feature, and then output by pointwise convolution.

[0012] Optionally, the step of reshaping the perturbation residual tensor of the convolution kernel and adding it to the standard depth convolution kernel to obtain a dynamic convolution kernel, and using the dynamic convolution kernel to perform channel-wise depth convolution on the second branch features, includes: The convolution kernel perturbation residual tensor is defined as a spatial distribution that matches the standard depth convolution kernel size, wherein each spatial location in the spatial distribution corresponds to a set of depth convolution kernel perturbation values; For each spatial location in the second branch feature, the perturbation value of the depth convolution kernel corresponding to the spatial location is added to the standard depth convolution kernel to obtain the dynamic convolution kernel corresponding to the spatial location; Using the dynamic convolution kernel corresponding to each spatial location, perform channel-wise convolution operations on the second branch features in the neighborhood of that spatial location.

[0013] Optionally, the step of inputting the fused feature map into the detection network for multi-scale feature extraction to obtain a multi-scale feature map includes: The detection network adopts a multi-stage structure, which includes multiple stages connected in sequence. Each stage consists of several convolutional layers. The convolutional layers use the linear polarization image to adjust the depth convolution kernel weights. Each stage reduces spatial resolution and increases the number of channels through downsampling convolution, outputting feature maps of various scales; The feature maps output from multiple stages are selected as the multi-scale feature maps.

[0014] Secondly, the present invention provides a bird's nest impurity detection system based on polarization imaging, comprising: The image determination module is used to acquire polariton images of multiple polarization directions of the bird's nest to be detected, and to calculate an intensity image and a linear polarization degree image based on the polariton images; The feature map generation module is used to extract features from the intensity image and the linear polarization image respectively, and generate an intensity feature map and a polarization feature map. The fusion module is used to generate a first spatially gated map from the intensity feature map through pointwise convolution and activation functions; to multiply the polarization feature map element-wise using the first spatially gated map and add the residuals to obtain a modulated polarization feature map; to generate a second spatially gated map from the polarization feature map through pointwise convolution and activation functions; to multiply the intensity feature map element-wise using the second spatially gated map and add the residuals to obtain a modulated intensity feature map; and to concatenate the modulated polarization feature map and the modulated intensity feature map and compress the channels to obtain a fused feature map. A feature extraction module is used to input the fused feature map into a detection network for multi-scale feature extraction; wherein, the detection network includes a convolutional layer, the convolutional layer has a depthwise convolutional kernel, and the weights of the depthwise convolutional kernel are adjusted using the linear polarization image to obtain a multi-scale feature map; The detection output module is used to perform multi-scale feature fusion on the multi-scale feature map and output the bounding box position and category information of bird's nest impurities.

[0015] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the methods described in the first aspect.

[0016] Compared with the closest existing technology, the present invention has the following advantages: First, this invention utilizes polarization imaging technology to obtain linear polarization information of bird's nest from a physical dimension, providing a new feature dimension for impurity detection. Due to differences in material properties, the bird's nest matrix and impurities exhibit different degrees of change in the polarization state of light. This difference is clearly reflected in the polarization image, helping to distinguish light-colored impurities with extremely low contrast in ordinary intensity images, while also reducing false detections caused by bird's nest texture.

[0017] Second, this invention employs a bidirectional spatially gated feature fusion method, enabling intensity features and polarization features to modulate each other, achieving deep interaction and complementarity between the two modal information. The structural information in the intensity features can guide the focus of the polarization features, while the material property information in the polarization features can enhance areas that are difficult to distinguish in the intensity features; the two work synergistically to improve detection performance.

[0018] Third, this invention introduces a polarization-guided weight adjustment mechanism into the convolutional layers of the detection network, dynamically adjusting the weights of the deep convolutional kernels using linear polarization images. This mechanism enables the convolutional kernels to adaptively adjust their response modes according to the polarization characteristics of different regions, enhancing feature extraction in regions with significant polarization differences and maintaining normal processing in regions with uniform polarization, thus achieving deep integration of polarization information from the input level to the convolutional kernel level.

[0019] Fourth, the detection network of this invention adopts a multi-stage structure, extracting features of different scales step by step through downsampling convolution and feature enhancement. Combined with a multi-scale feature fusion mechanism, it can simultaneously capture the overall outline of large-scale impurities and the detailed features of small-scale impurities, adapting to the detection needs of impurities of different sizes and types.

[0020] Fifth, the overall network structure of this invention remains lightweight, enabling real-time inference on embedded computing platforms, and is suitable for high-speed detection scenarios on industrial production lines. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0022] Figure 1 This is a flowchart of a method for detecting impurities in bird's nest based on polarization imaging provided by the present invention; Figure 2 This is a schematic diagram of polarization imaging and polarization parameter calculation provided by the present invention; Figure 3 This is a schematic diagram of the dual-channel feature extraction and bidirectional gating fusion structure provided by the present invention; Figure 4 This is a schematic diagram of a bird's nest impurity detection system based on polarization imaging provided by the present invention; Figure 5 This is a schematic diagram of the electronic device structure used to implement the methods and system embodiments of the present invention. Detailed Implementation

[0023] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore merely examples, and should not be construed as limiting the scope of protection of the present invention.

[0024] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by one of ordinary skill in the art.

[0025] This invention provides a method and system for detecting impurities in bird's nest based on polarization imaging. The embodiments of this invention will be described below with reference to the accompanying drawings.

[0026] Example 1: As Figure 1 As shown, Embodiment 1 of the present invention provides a method for detecting impurities in bird's nest based on polarization imaging. The method specifically includes the following steps: S101. Obtain polariton images of the bird's nest to be tested in multiple polarization directions, and calculate the intensity image and the degree of linear polarization image based on the polariton images. like Figure 2 As shown, this step acquires polariton images of the bird's nest to be tested in multiple polarization directions using polarization imaging, and calculates the intensity image and linear polarization degree image based on the polariton images.

[0027] Polarization imaging acquires polarimetric images of the bird's nest under test in multiple polarization directions. Polarization imaging refers to using imaging equipment with polarization resolution to acquire images of the light intensity distribution of the target in different polarization directions.

[0028] In this embodiment, a pixel-level polarization camera is used for acquisition. Each pixel unit of the camera integrates four polarization microstructures in different directions, and sub-images in four polarization directions can be obtained simultaneously in a single exposure.

[0029] The total intensity component, the first linear polarization component, and the second linear polarization component in the Stokes parameters are calculated based on the polariton images of the multiple polarization directions. The Stokes parameters are a set of physical quantities describing the polarization state of light. The total intensity component represents the total intensity of the incident light, the first linear polarization component represents the difference in polarization intensity in the horizontal and vertical directions, and the second linear polarization component represents the difference in polarization intensity in the 45° and 135° directions.

[0030] The specific calculation method is as follows: Add the corresponding pixel values ​​of the two polariton images in the 0-degree and 90-degree directions to obtain the total light intensity component; subtract the pixel value of the 90-degree polariton image from the pixel value of the 0-degree polariton image to obtain the first linear polarization component; subtract the pixel value of the 135-degree polariton image from the pixel value of the 45-degree polariton image to obtain the second linear polarization component. These three components constitute three two-dimensional images.

[0031] The image corresponding to the total light intensity component is used as the intensity image, and the degree of linear polarization image is calculated based on the first and second linear polarization components. The intensity image reflects the normal light intensity distribution of the target, similar to a normal visible light image. The degree of linear polarization is a physical quantity describing the degree of linear polarization of light. It is calculated as follows: for each pixel location, first calculate the sum of the squares of the first and second linear polarization components, then take the square root of this sum, and finally divide the square root result by the total light intensity component to obtain the degree of linear polarization value for that pixel location. The degree of linear polarization values ​​for all pixel locations constitute the degree of linear polarization image.

[0032] Linear polarization images reflect the depolarization characteristics of different regions on the target surface. Different materials, due to differences in surface roughness, internal structure, and refractive index, alter the polarization state of incident light to varying degrees, thus exhibiting different grayscale distributions in the polarization image. The differences in material properties between the bird's nest matrix and impurities make them distinguishable in the polarization image, providing additional discrimination criteria for subsequent detection networks.

[0033] S102. Perform feature extraction on the intensity image and the linear polarization image respectively to generate an intensity feature map and a polarization feature map. This step extracts features from the intensity image and the linear polarization image respectively, generating an intensity feature map and a polarization feature map. The specific process is as follows: The intensity image is input into the first convolutional layer for feature extraction to obtain the intensity feature map. The first convolutional layer consists of several convolutional operation units, used to extract low-level visual features from the intensity image, such as edges, textures, gradients, etc.

[0034] like Figure 3 As shown, the linear polarization degree image is input into the second convolutional layer for feature extraction to obtain the polarization degree feature map. The second convolutional layer has the same structure as the first convolutional layer and is used to extract polarization-related feature information from the linear polarization degree image.

[0035] The first and second convolutional layers have identical structures, and the intensity feature map and the polarization feature map have the same spatial size and number of channels. The parameters of the two convolutional layers are not shared; each is learned independently to adapt to the different distribution characteristics of the intensity and polarization information. The identical spatial size ensures a one-to-one correspondence between the two feature maps in spatial location during subsequent fusion operations; the identical number of channels ensures that the feature representation capabilities of the two modalities are at a comparable level, preventing one modality's features from dominating the fusion result.

[0036] By using a dual-channel independent extraction method, intensity information and polarization information maintain their independence during the feature extraction stage. Their respective feature spaces are not disturbed by the other mode, which helps to preserve the unique attributes of the two types of information and provides a high-quality feature foundation for subsequent deep fusion.

[0037] S103. The intensity feature map is convolved pointwise and activated by a function to generate a first spatial gating map. The polarization feature map is multiplied element-wise by the first spatial gating map and the residuals are summed to obtain a modulated polarization feature map. The polarization feature map is convolved pointwise and activated by a function to generate a second spatial gating map. The intensity feature map is multiplied element-wise by the second spatial gating map and the residuals are summed to obtain a modulated intensity feature map. The modulated polarization feature map and the modulated intensity feature map are concatenated and their channels are compressed to obtain a fused feature map. This embodiment employs a bidirectional spatial gating fusion mechanism, enabling the features of the two modes to modulate and complement each other. The specific process is as follows: The intensity feature map is processed by pointwise convolution and an activation function to generate a first spatial gating map. Pointwise convolution is used to adjust the number of channels and learn the correlation between channels, while the activation function maps the gating values ​​to a suitable range. The value at each spatial location in the first spatial gating map represents the modulation weight of the intensity feature on the polarization feature at that location.

[0038] The first spatial gating map is multiplied element-wise by the polarization degree feature map, and then added element-wise to the polarization degree feature map to obtain the modulated polarization degree feature map. This residual connection modulation method introduces the guiding role of intensity features while preserving the original information of polarization features.

[0039] The polarization feature map is processed by pointwise convolution and activation function to generate a second spatial gating map. The value at each spatial location in the second spatial gating map represents the modulation weight of the polarization feature on the intensity feature at that location.

[0040] The second spatial gating map is multiplied element-by-element by the intensity feature map, and then added element-by-element by the intensity feature map to obtain the modulated intensity feature map.

[0041] The modulated polarization feature map and the modulated intensity feature map are concatenated along the channel dimension, and then the number of channels is compressed by pointwise convolution to output the fused feature map. Channel concatenation merges the features of the two modalities along the channel dimension, while pointwise convolution is used to learn the fusion relationship between channels, restoring the number of channels to the level before fusion, and outputting the final fused feature map.

[0042] Through the aforementioned bidirectional gating fusion mechanism, intensity features and polarization features achieve deep interaction: structural information in the intensity features can guide the polarization features to focus on specific spatial regions, while material property information in the polarization features can enhance regions that are difficult to distinguish in the intensity features. The information from the two modes complements and synergizes with each other, providing richer input for subsequent deep feature extraction.

[0043] S104. The fused feature map is input into a detection network for multi-scale feature extraction. The detection network includes convolutional layers with depthwise convolutional kernels, and the weights of these kernels are adjusted using the linear polarization image to obtain the multi-scale feature map. This step involves inputting the fused feature maps into the detection network for multi-scale feature extraction. The detection network is the core component of this invention, and its main feature is that the convolutional layers in the network have depthwise convolutional kernels, and the weights of the depthwise convolutional kernels can be adjusted using linear polarization images, allowing the convolutional kernels to exhibit different response modes in different polarization regions. The overall structure of the detection network and the convolutional layer weighting mechanism are described in detail below.

[0044] (1) Detection network multi-stage structure The detection network adopts a multi-stage structure, which includes multiple stages connected in sequence. Each stage consists of several convolutional layers, and the convolutional layers use the linear polarization image to adjust the depth convolution kernel weights.

[0045] Each stage reduces spatial resolution and increases the number of channels through downsampling convolutions, outputting feature maps at various scales. Downsampling convolutions are achieved by setting a stride greater than one, progressively reducing the spatial size of the feature maps. The increase in the number of channels maintains sufficient feature representation at lower spatial resolutions. After downsampling, the feature maps are enhanced by several convolutional layers in this stage. These layers operate using the polarization-guided weighting method described above, performing depth extraction and transformation on the features while maintaining the spatial resolution.

[0046] Since downsampling is performed at each stage, the feature maps output by each stage have different spatial scales and number of channels, corresponding to different receptive fields. The feature maps in the shallow stages have high spatial resolution, few channels, and small receptive fields, making them suitable for capturing the detailed features of small-scale targets; the feature maps in the deep stages have low spatial resolution, many channels, and large receptive fields, making them suitable for capturing the semantic features of large-scale targets.

[0047] Feature maps from multiple stages are selected as the multi-scale feature maps. These multi-scale feature maps provide feature inputs at different levels for subsequent multi-scale feature fusion and detection.

[0048] (2) The convolutional layer uses the linear polarization image to adjust the weights of the depth convolution kernel. The specific process of adjusting the weights of the depthwise convolutional kernels in the convolutional layers of the detection network using linear polarization images is as follows: The input feature map of the convolutional layer is divided into a first branch feature and a second branch feature according to each channel. The first branch feature is retained as the first branch output. The first branch forms a direct connection to retain the original feature information; the second branch enters the subsequent weighted convolutional processing to learn the polarization-guided dynamic features.

[0049] The linear polarization image is then subjected to average pooling to achieve the same spatial size as the second branch feature, resulting in a pooled linear polarization image. The average pooling operation matches the resolution of the polarization image with the resolution of the feature map processed by the current convolutional layer, ensuring that subsequent convolutional kernel adjustments correspond one-to-one with the spatial positions of the feature map.

[0050] The pooled linear polarization image is input into a prediction network consisting of two layers of pointwise convolutions and activation functions. The output is a convolution kernel perturbation residual tensor that matches the size of the depthwise convolution kernel. The prediction network takes a single-channel pooled polarization image as input and outputs a multidimensional tensor whose dimensions match the size of the depthwise convolution kernel. The convolution kernel perturbation residual tensor represents the adjustment amount that needs to be superimposed on the standard convolution kernel, and its value is learned from the polarization image through the network.

[0051] The reshaped residual tensor of the perturbation kernel is added to the standard depth convolution kernel to obtain the dynamic convolution kernel. The standard depth convolution kernel refers to the basic depth convolution kernel parameters obtained during network initialization and used in training updates.

[0052] The second branch features are subjected to channel-wise depthwise convolution using the dynamic convolution kernel to obtain the convolutionally processed second branch features. Channel-wise depthwise convolution means that each input channel is convolved using an independent convolution kernel, without mixing information between channels. Since the weights of the dynamic convolution kernel are modulated by the polarization image, the convolution operation at different spatial locations has different convolution kernel parameters, which can adaptively adjust the feature extraction method according to the local polarization characteristics.

[0053] The second branch feature, after convolution processing, is adjusted for the number of channels through pointwise convolution, then concatenated with the first branch feature, and finally output through pointwise convolution. Pointwise convolution is used to adjust the number of channels in the second branch feature to match the number of channels in the first branch feature; the concatenation operation merges the features of the two branches in the channel dimension; the final pointwise convolution is used for channel fusion and information integration, outputting the final result of the convolutional layer.

[0054] The core idea of ​​the aforementioned convolutional layer weighting mechanism is to dynamically adjust the weight distribution of the convolutional kernel using the material property information provided by the polarization image, enabling the convolution operation to have different response characteristics in different polarization regions. In regions with drastic changes in polarization (typically corresponding to impurity boundaries or areas of material difference), the convolutional kernel can enhance the extraction ability of edges and details; in regions with uniform polarization (typically corresponding to pure bird's nest matrix), the convolutional kernel maintains the conventional feature extraction mode. This adaptive mechanism allows the network to pay more attention to regions with abnormal polarization information, thereby improving the accuracy of impurity detection.

[0055] (3) Dynamic convolution kernel generation and convolution at spatial locations The process of generating and using the dynamic convolutional kernel described above can be further refined into a spatially location-specific implementation. The convolutional kernel perturbation residual tensor does not correspond to a single, uniform convolutional kernel, but rather to the perturbation value of the convolutional kernel at each spatial location on the feature map. The specific implementation process is as follows: The convolution kernel perturbation residual tensor is defined as a spatial distribution matching the size of the standard depthwise convolution kernel, where each spatial location corresponds to a set of depthwise convolution kernel perturbation values. In other words, the convolution kernel perturbation residual tensor includes not only the parameter dimension of the convolution kernel but also the spatial location dimension. Within the spatial distribution, each spatial location corresponds to an independent set of depthwise convolution kernel perturbation values, and the perturbation values ​​at different spatial locations can be different.

[0056] For each spatial location in the second branch feature, the perturbation value of the depthwise convolution kernel corresponding to that spatial location is added to the standard depthwise convolution kernel to obtain the dynamic convolution kernel corresponding to that spatial location. Since each spatial location has its own independent perturbation value, the dynamic convolution kernel generated at each location is different.

[0057] Using a dynamic convolution kernel corresponding to each spatial location, a channel-wise convolution operation is performed on the second branch features within the neighborhood of that spatial location. In other words, during the convolution operation, the convolution kernel parameters used at each pixel location on the feature map are different, and the convolution kernel weights are dynamically adjusted as the spatial location changes.

[0058] This position-by-position dynamic convolution method allows for precise adjustment of the convolution kernel down to each spatial location, maximizing the utilization of the spatial distribution information of the polarization degree image. The network can finely adjust the convolution response at each pixel location based on the polarization degree value, thereby achieving in-depth utilization of polarization information.

[0059] S105. Perform multi-scale feature fusion on the multi-scale feature map and output the bounding box position and category information of bird's nest impurities. This step performs multi-scale feature fusion on the multi-scale feature maps and outputs the bounding box location and category information of bird's nest impurities. The specific process is as follows: The multi-scale feature map is subjected to top-down upsampling fusion and bottom-up downsampling fusion sequentially to obtain the fused multi-scale feature map. The multi-scale feature fusion adopts a bidirectional fusion method. The top-down upsampling fusion process starts from the deepest feature map and propagates upwards level by level. After the deep feature map is expanded in spatial size through upsampling, it is fused with the feature map from the previous stage, passing the semantic information from the deep layer to the shallow layer and enhancing the semantic expressive power of the shallow features.

[0060] The bottom-up downsampling fusion process starts with the shallowest feature map and proceeds downwards level by level. After the shallow feature map is downsampled to reduce its spatial size, it is fused with the feature map of the next stage, passing the detailed information of the shallow layer to the deeper layers and enhancing the detailed representation ability of the deep features.

[0061] After bidirectional fusion, a multi-scale feature map is obtained. Each scale of the fused feature map contains both deep semantic information and shallow detailed information, resulting in stronger feature representation capabilities.

[0062] Based on the fused multi-scale feature maps, bounding box regression parameters, class probabilities, and confidence scores for each corresponding scale are predicted. Detection prediction is performed on the feature maps at each scale. The bounding box regression parameters are used to predict the location and size of the impurity target, the class probability is used to determine the impurity's category, and the confidence score is used to measure the reliability of the prediction results.

[0063] Non-maximum suppression (NMS) is applied to the prediction results at each scale to obtain the bounding box locations and category information of the bird's nest impurities. NMS is a commonly used post-processing method in object detection, used to remove duplicate detection boxes and retain the optimal detection results. In the processing, all detection boxes are first sorted according to their confidence scores. Then, the detection box with the highest confidence score is selected sequentially, and other detection boxes with an overlap exceeding a set threshold are removed until all detection boxes have been processed.

[0064] After the non-maximum suppression process is completed, the final bird's nest impurity detection results are obtained, including the bounding box location and category information of each impurity target.

[0065] In one embodiment, the detection network needs to undergo offline training to obtain the optimal parameter configuration. This training is part of the model preprocessing process, and the training process is not executed in the actual industrial detection inference stage. The training steps are as follows: A polarized image dataset containing annotations for various types of bird's nest impurities was constructed, and the samples in the dataset were divided into a training set and a validation set. Each image in the dataset corresponds to a bounding box annotation and a category annotation for the impurity target. The training set was used for learning network parameters, and the validation set was used to monitor performance changes during the training process.

[0066] The detection network is trained using the training set. A loss function is obtained by weighted summation of the bounding box regression loss, classification loss, and target confidence loss. The network parameters are updated using stochastic gradient descent based on this loss function. During training, the learning rate is gradually decreased from its initial value using cosine annealing. The bounding box regression loss measures the positional and size differences between the predicted and ground truth bounding boxes; the classification loss measures the difference between the predicted and ground truth classes; and the target confidence loss measures the difference between the predicted confidence and the existence of the true target. The three losses are weighted and summed to obtain the total loss function value. The learning rate is adjusted using cosine annealing, starting from an initial value and gradually decreasing with each training iteration, allowing for rapid convergence in the early stages of training and fine-tuning of parameters in the later stages.

[0067] A polarization-assisted loss is introduced during training. For each anchor box predicted as a positive sample, the average linear polarization degree within its corresponding region is calculated. When the average value is lower than a preset prior threshold, a penalty term is generated, which is included in the loss function for parameter updates. The design logic of this auxiliary loss is as follows: if the network predicts a bird's nest matrix region with low linear polarization degree as an impurity, a penalty is generated to suppress false positives.

[0068] In each training round, samples from the training set are input into the network in batches. Forward propagation calculates the loss function value, backpropagation calculates the parameter gradients, and then an optimization algorithm updates the network parameters. As the number of training rounds increases, the loss function value gradually decreases, and the network's detection performance gradually improves. During training, the detection performance is periodically evaluated on the validation set to monitor training progress and prevent overfitting.

[0069] After training, the final detection network model is obtained, which can be used for actual bird's nest impurity detection tasks.

[0070] Example 2: Based on the same technical concept, Example 2 of the present invention also provides a bird's nest impurity detection system based on polarization imaging, used to implement the detection method described in Example 1 above. Figure 4 As shown, the system adopts a modular design, including an image determination module 210, a feature map generation module 220, a fusion module 230, a feature extraction module 240, and a detection output module 250.

[0071] The image determination module 210 is used to acquire polariton images of the bird's nest to be tested in multiple polarization directions, and calculate the intensity image and linear polarization degree image based on the polariton images. This module is responsible for acquiring polarization images and calculating polarization parameters, providing basic data for subsequent processing.

[0072] The feature map generation module 220 is used to extract features from the intensity image and the linear polarization image respectively, generating intensity feature maps and polarization feature maps. This module extracts features of the two modes separately through two independent convolutional branches, maintaining the independence of each feature.

[0073] The fusion module 230 is used to generate a first spatially gated map from the intensity feature map through pointwise convolution and activation functions; to multiply the polarization feature map element-wise using the first spatially gated map and add the residuals to obtain a modulated polarization feature map; to generate a second spatially gated map from the polarization feature map through pointwise convolution and activation functions; to multiply the intensity feature map element-wise using the second spatially gated map and add the residuals to obtain a modulated intensity feature map; and to concatenate the modulated polarization feature map and the modulated intensity feature map and compress the channels to obtain a fused feature map. This module adopts a bidirectional spatially gated fusion mechanism to achieve deep interaction and complementarity between the two modal features.

[0074] The feature extraction module 240 is used to input the fused feature map into the detection network for multi-scale feature extraction. The detection network includes convolutional layers with depthwise convolutional kernels, and the weights of the depthwise convolutional kernels are adjusted using a linear polarization image to obtain multi-scale feature maps. This module is the core of the system and is used for polarization-guided depth feature extraction.

[0075] The detection output module 250 is used to perform multi-scale feature fusion on multi-scale feature maps and output the bounding box location and category information of bird's nest impurities. This module is responsible for multi-scale feature fusion, detection prediction, and post-processing, and finally outputs the detection results.

[0076] The modules described above are connected via data interfaces and work together to automatically detect impurities in bird's nests. The system can be deployed on an embedded computing platform and used in conjunction with polarization imaging equipment to achieve real-time detection on industrial production lines.

[0077] Example 3: This embodiment of the invention also provides an electronic device corresponding to Example 1. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it causes the processor to perform the steps of any one of the methods described in S101-S105.

[0078] like Figure 5 As shown, the electronic device may include: at least one processor 31, at least one network interface 35, a user interface 34, a memory 36, and at least one communication bus 32.

[0079] The communication bus 32 is used to enable communication between these components.

[0080] The user interface 34 may include a display screen and a camera. Optionally, the user interface 34 may also include a standard wired interface and a wireless interface.

[0081] The network interface 35 may optionally include a standard wired interface or a wireless interface (such as a WIFI interface).

[0082] The processor 31 may include one or more processing cores. The processor 31 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 36, and by calling data stored in the memory 36. Optionally, the processor 31 may employ Digital Signal Processing (DSP) or Field Programmable Gate Array (FPGA). The processor 31 can be implemented using at least one hardware form, such as a Programmable Gate Array (FPGA) or a Programmable Logic Array (PLA). The processor 31 can integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem can also be implemented as a separate chip without being integrated into the processor 31.

[0083] The memory 36 may include random access memory (RAM) or read-only memory. Optionally, the memory 36 may include a non-transitory computer-readable storage medium. The memory 36 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 36 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 36 may also be at least one storage device located remotely from the aforementioned processor 31. Figure 5 As shown, the memory 36, which serves as a computer storage medium, may include an operating system, a network communication module, and a user interface module.

[0084] exist Figure 5 In the electronic device shown, the user interface 34 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 31 can be used to call an application program of a bird's nest impurity detection method based on polarization imaging stored in the memory 36. When executed by one or more processors 31, the electronic device executes one or more of the bird's nest impurity detection methods based on polarization imaging as described in steps S101-S105 of the above embodiment.

[0085] Those skilled in the art will clearly understand that the technical solutions of the present invention can be implemented by means of software and / or hardware. In this specification, "module" refers to software and / or hardware capable of independently or in conjunction with other components to perform a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0086] Specifically, the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0087] The code of the computer program can be in the form of source code, object code, executable file, or some intermediate form.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for detecting impurities in bird's nest based on polarization imaging, characterized in that, include: A polariton image with multiple polarization directions is acquired for the bird's nest to be tested, and an intensity image and a linear polarization degree image are calculated based on the polariton image; Feature extraction is performed on the intensity image and the linear polarization image respectively to generate an intensity feature map and a polarization feature map; The intensity feature map is convolved pointwise and activated by a function to generate a first spatial gating map. The polarization feature map is then multiplied element-wise by the first spatial gating map, and the residuals are summed to obtain a modulated polarization feature map. The polarization feature map is then convolved pointwise and activated by a function to generate a second spatial gating map. The intensity feature map is then multiplied element-wise by the second spatial gating map, and the residuals are summed to obtain a modulated intensity feature map. The modulated polarization feature map and the modulated intensity feature map are then concatenated and their channels are compressed to obtain a fused feature map. The fused feature map is input into a detection network for multi-scale feature extraction. The detection network includes a convolutional layer with a depthwise convolutional kernel. The weights of the depthwise convolutional kernel are adjusted using the linear polarization image to obtain the multi-scale feature map. Multi-scale feature fusion is performed on the multi-scale feature map, and the bounding box location and category information of bird's nest impurities are output.

2. The method according to claim 1, characterized in that, The process of acquiring polariton images of multiple polarization directions of the bird's nest to be tested, and calculating the intensity image and the degree of linear polarization image based on the polariton images, includes: Polariton images of the bird's nest under test in multiple polarization directions were obtained using polarization imaging. The total intensity component, the first linear polarization component, and the second linear polarization component in the Stokes parameters are calculated based on the polariton images of the multiple polarization directions. The image corresponding to the total light intensity component is used as the intensity image, and the linear polarization degree image is calculated based on the first linear polarization component and the second linear polarization component.

3. The method according to claim 1, characterized in that, The step of extracting features from the intensity image and the linear polarization image to generate an intensity feature map and a polarization feature map includes: The intensity image is input into the first convolutional layer for feature extraction to obtain the intensity feature map; The linear polarization image is input into the second convolutional layer for feature extraction to obtain the polarization feature map; The first convolutional layer and the second convolutional layer have the same structure, and the intensity feature map and the polarization feature map have the same spatial size and number of channels.

4. The method according to claim 1, characterized in that, The obtained fused feature map includes: The intensity feature map is processed by pointwise convolution and activation function to generate a first spatial gating map; Multiply the first spatial gating map element by element with the polarization degree feature map, and then add the first spatial gating map element by element with the polarization degree feature map to obtain the modulated polarization degree feature map. The polarization feature map is processed by pointwise convolution and activation function to generate a second spatial gated map; The second spatial gating map is multiplied element-by-element by the intensity feature map, and then added element-by-element by the intensity feature map to obtain the modulated intensity feature map; The modulated polarization feature map and the modulated intensity feature map are concatenated along the channel dimension, and then the number of channels is compressed by pointwise convolution to output the fused feature map.

5. The method according to claim 1, characterized in that, The step of adjusting the weights of the depth convolution kernel using the linear polarization image includes: The input feature map of the convolutional layer is divided into a first branch feature and a second branch feature according to the channel, and the first branch feature is retained as the first branch output. The linear polarization image is subjected to average pooling to the same spatial size as the second branch feature to obtain the pooled linear polarization image. The pooled linear polarization image is input into a prediction network consisting of two layers of pointwise convolution and activation functions, and the output is a convolution kernel perturbation residual tensor that matches the depth convolution kernel size. The perturbation residual tensor of the convolution kernel is reshaped and added to the standard depth convolution kernel to obtain a dynamic convolution kernel. The dynamic convolution kernel is then used to perform channel-wise depth convolution on the second branch feature to obtain the convolution-processed second branch feature. The second branch feature after convolution processing is then adjusted for the number of channels by pointwise convolution, concatenated with the first branch feature, and then output by pointwise convolution.

6. The method according to claim 5, characterized in that, The step of reshaping the perturbation residual tensor of the convolution kernel and adding it to the standard depth convolution kernel to obtain a dynamic convolution kernel, and then using the dynamic convolution kernel to perform channel-wise depth convolution on the second branch features includes: The convolution kernel perturbation residual tensor is defined as a spatial distribution that matches the standard depth convolution kernel size, wherein each spatial location in the spatial distribution corresponds to a set of depth convolution kernel perturbation values; For each spatial location in the second branch feature, the perturbation value of the depth convolution kernel corresponding to the spatial location is added to the standard depth convolution kernel to obtain the dynamic convolution kernel corresponding to the spatial location; Using the dynamic convolution kernel corresponding to each spatial location, perform channel-wise convolution operations on the second branch features in the neighborhood of that spatial location.

7. The method according to claim 1, characterized in that, The step of inputting the fused feature map into the detection network for multi-scale feature extraction to obtain the multi-scale feature map includes: The detection network adopts a multi-stage structure, which includes multiple stages connected in sequence. Each stage consists of several convolutional layers. The convolutional layers use the linear polarization image to adjust the depth convolution kernel weights. Each stage reduces spatial resolution and increases the number of channels through downsampling convolution, outputting feature maps of various scales; Feature maps output from multiple stages are selected as the multi-scale feature maps.

8. The method according to claim 1, characterized in that, The step of performing multi-scale feature fusion on the multi-scale feature map and outputting the bounding box location and category information of bird's nest impurities includes: The multi-scale feature map is subjected to top-down upsampling fusion and bottom-up downsampling fusion in sequence to obtain the fused multi-scale feature map. Based on the fused multi-scale feature map, predict the bounding box regression parameters, class probability, and confidence score for the corresponding scale. Non-maximum suppression processing is applied to the prediction results at each scale to obtain the bounding box location and category information of the bird's nest impurities.

9. A bird's nest impurity detection system based on polarization imaging, characterized in that, include: The image determination module is used to acquire polariton images of multiple polarization directions of the bird's nest to be detected, and to calculate an intensity image and a linear polarization degree image based on the polariton images; The feature map generation module is used to extract features from the intensity image and the linear polarization image respectively, and generate an intensity feature map and a polarization feature map. The fusion module is used to generate a first spatially gated map from the intensity feature map through pointwise convolution and activation functions; to multiply the polarization feature map element-wise using the first spatially gated map and add the residuals to obtain a modulated polarization feature map; to generate a second spatially gated map from the polarization feature map through pointwise convolution and activation functions; to multiply the intensity feature map element-wise using the second spatially gated map and add the residuals to obtain a modulated intensity feature map; and to concatenate the modulated polarization feature map and the modulated intensity feature map and compress the channels to obtain a fused feature map. A feature extraction module is used to input the fused feature map into a detection network for multi-scale feature extraction; wherein, the detection network includes a convolutional layer, the convolutional layer has a depthwise convolutional kernel, and the weights of the depthwise convolutional kernel are adjusted using the linear polarization image to obtain a multi-scale feature map; The detection output module is used to perform multi-scale feature fusion on the multi-scale feature map and output the bounding box position and category information of bird's nest impurities.

10. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the method according to any one of claims 1-8.

Images