A multi-view fruit quality detection method, device, system, medium and product

Abstract

This invention discloses a multi-view fruit quality detection method, apparatus, system, medium, and product. The method includes acquiring a multi-view composite image of the fruit to be detected, wherein the composite image is obtained by stitching together sub-images from multiple perspectives. These sub-images include directly captured images of the actual fruit and virtual images resulting from one or more reflections by a plane mirror. A pre-trained target detection network model is used to detect the sub-images, obtaining detection results for each sub-image. These detection results include the predicted bounding box, confidence score, and single-view quality normalization score of the fruit in each sub-image. The single-view quality normalization scores from the detection results of each sub-image are then fused using a multi-view method to obtain a multi-view quality normalization score. This invention aims to achieve multi-view fruit quality detection to obtain consistent, multi-view fruit quality detection results.

Description

Technical Field

[0001] This application relates to fruit quality testing technology in the agricultural field, specifically to a multi-view fruit quality testing method, device, system, medium, and product. Background Technology

[0002] Currently, fruit quality mainly relies on manual sorting, which is not only inefficient but also easily influenced by human subjectivity and experience, making it difficult to meet the requirements of large-scale industrialization. The external quality of fruit is mainly reflected in its physical characteristics, including color, defect, shape, and size. Fruit quality needs to be graded based on these characteristics. At the beginning of fruit external quality detection research, machine vision methods were commonly used. This method does not require a large amount of labeled data for training. It analyzes quality characteristics by collecting image information. In specific scenarios, it can be manually defined, which has the advantages of simplicity and efficiency. However, the simple machine vision method relies too much on the creation of prior features by humans and cannot meet the requirements of various fruit quality detection.

[0003] With the maturity of artificial intelligence technology, various detection methods based on convolutional neural networks have emerged. Relying on the powerful feature extraction capabilities of convolutional neural networks, these methods effectively get rid of the subjectivity of manually designed features and greatly improve accuracy. However, mature target detection methods are all single-view, and it is often unreasonable to evaluate fruit quality from a single perspective. Therefore, designing a multi-view target detection network is necessary to meet the accuracy requirements of large-scale industrial detection. In addition, real-time performance is also an important requirement for designing this target detection network. Conventional high-precision target detectors are mostly two-stage, that is, first generating proposal boxes through the Region Proposal Network (RPN) and then detecting, which limits the detection efficiency. Therefore, it is necessary to design an efficient multi-view single-stage target detection network. Fruit quality is related to color, shape and defects, as well as fruit size. Existing detection methods still face the following problems: (1) one-sided detection, that is, only detecting part of the fruit surface rather than the entire surface information. (2) low detection accuracy and efficiency, especially in multi-view scenarios, it is difficult to complete the task using only a classification model. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multi-view fruit quality detection method, device, system, medium and product to address the above-mentioned problems of the prior art. The present invention aims to achieve multi-view fruit quality detection to obtain multi-view and consistent fruit quality detection results.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A multi-view fruit quality testing method includes the following steps:

[0007] S101, acquire a multi-view composite image of the fruit being tested. The multi-view composite image is obtained by stitching together sub-images from multiple perspectives. The sub-images from multiple perspectives include directly captured images of the actual object and virtual images after being reflected once or multiple times by a plane mirror.

[0008] S102, the target detection network model pre-trained by the multi-view synthetic image detects the detection results of each sub-image. The detection results include the prediction box, confidence score and single-view quality normalization score of the detected fruit in the sub-image.

[0009] S103, perform multi-view fusion of the single-view quality normalization scores in the detection results of each sub-image to obtain a multi-view quality normalization score:

[0010] ;

[0011] in, To normalize the quality score from multiple perspectives, The total number of viewpoints. From the perspective The weight, From the perspective Single-view quality normalized score, all views The sum of the weights is 1, and the weight of the viewpoint corresponding to the real image is greater than the weight of the viewpoint corresponding to the virtual image.

[0012] Optionally, step S103 further includes:

[0013] S201, Scaling and correcting the predicted bounding boxes in the detection results of each sub-image;

[0014] S202, Select the prediction box with the highest confidence level;

[0015] S203, calculate the estimated fruit diameter based on the prediction frame with the highest confidence level:

[0016] ;

[0017] in, To estimate the diameter of the fruit, The width of the predicted bounding box with the highest confidence level. For pixel size, The equivalent imaging distance is the viewpoint corresponding to the prediction box with the highest confidence level. This refers to the camera's focal length.

[0018] Optionally, the scaling correction of the predicted bounding boxes in the detection results of each sub-image in step S201 includes multiplying the width and height distribution of the predicted bounding boxes by a scaling factor. The calculation function expression of the scaling factor is as follows:

[0019] ;

[0020] in, This is the scaling factor. The number of times the sub-image is reflected by the mirror. The distance between the plane mirror and the fruit being tested. The imaging distance of a directly captured image of a real object. , This represents the maximum allowed reflection order.

[0021] Optionally, step S203 may further include correcting the multi-view quality normalization score based on the estimated fruit diameter:

[0022] ;

[0023] ;

[0024] in, The corrected multi-view quality normalized score, To normalize the quality score from multiple perspectives, This is a correction factor for the uniformity of estimated fruit diameter. To estimate the diameter of the fruit, and These are the minimum and maximum diameters of standard fruit, respectively. For standard fruit diameter, This is the penalty factor for size deviation.

[0025] Optionally, the target detection network model in step S102 is an improved YOLOv5 detection model. The improved YOLOv5 detection model includes a backbone network, a neck network, and a detection network connected in sequence. The backbone network includes a convolutional module, a four-level feature extraction unit, and a fast spatial pyramid pooling module connected in sequence. The feature extraction unit includes a cascaded convolutional module and a cross-scale residual stacked structure layer. The cross-scale residual stacked structure layer includes two branches, one of which includes a convolutional layer and the other includes a convolutional layer and a bottleneck layer. The output features of the two branches are connected by a connection module and then passed through a convolutional layer to obtain the final output features. The bottleneck layer includes two bottleneck branches, one of which includes a convolutional layer and the other includes two cascaded convolutional layers. The fast spatial pyramid pooling module includes a convolutional layer, a connection module, and a convolutional layer connected in sequence. The output features of the first convolutional layer are also connected by a parallel bypass branch. The bypass branch includes three cascaded max pooling layers, and the output features of the three max pooling layers are all used as input features of the connection module.

[0026] Optionally, the neck network comprises 17 layers connected in sequence: convolutional layers, upsampling modules, connection modules, cross-scale residual stacked structure layers, convolutional layers, multi-scale dilated convolutional channel attention modules, upsampling modules, connection modules, cross-scale residual stacked structure layers, convolutional layers, multi-scale dilated convolutional channel attention modules, connection modules, cross-scale residual stacked structure layers, convolutional layers, multi-scale dilated convolutional channel attention modules, connection modules, and cross-scale residual stacked structure layers. The detection network includes three detection heads. The output features of layers 9, 13, and 17 in the 17 layers are processed by an additional convolutional layer and used as input features for one detection head. The 2nd, 13th, and 17th layers in the backbone network are... The three multi-scale residual stacked structure layers and the fast spatial pyramid pooling module output features at three different scales, which are used as input features for layers 8, 3, and 1 of the 17-layer neck network. The output features of layers 1 and 6 of the 17-layer neck network are also used as input features for layers 12 and 16 of the 17-layer neck network, respectively. The multi-scale dilated convolutional channel attention module includes three two-dimensional convolutional branches with different dilation rates. The input features are processed by the three two-dimensional convolutional branches to extract features F1, F2, and F3 with different receptive fields. Then, features F1, F2, and F3 are channel compressed by the first 1×1 convolution and then concatenated to form a joint multi-scale feature tensor. Joint multi-scale feature tensor After a second 1×1 convolution, channel transformation is performed to project features with different hole rates onto a unified feature domain to obtain unified features. To unify features The response values ​​of each scale channel are obtained by global average pooling using the following formula:

[0027] ;

[0028] in, For scale channel Response volume This is the index number of the two-dimensional convolution branch. and To unify characteristics Height and width, To unify features Located at position on the r-th scale channel The eigenvalues; the attention weights for the three scale channels are obtained by normalizing the responses of each scale channel using the Softmax activation function:

[0029] ;

[0030] in, For scale channel Attention weights For scale channel Response volume The index number is the two-dimensional convolutional branch; features F1, F2, and F3 are multiplied by the attention weights of the corresponding two-dimensional convolutional branches and then summed to obtain the final output multi-scale augmentation features. .

[0031] A multi-view fruit quality detection device for applying the aforementioned multi-view fruit quality detection method includes a first conveyor belt, a second conveyor belt, a flipping mechanism, a first camera, a first plane mirror, a second camera, and a second plane mirror. The flipping mechanism is arranged between the first and second conveyor belts to flip the fruit to be detected on the first conveyor belt and place it on the second conveyor belt. The first camera is arranged on the infeed side of the first conveyor belt, and multiple first plane mirrors are arranged obliquely behind the target detection position of the first camera on the first conveyor belt. The second camera is arranged on the outfeed side of the second conveyor belt, and multiple second plane mirrors are arranged obliquely behind the target detection position of the second camera on the second conveyor belt. The height of the first conveyor belt is higher than that of the second conveyor belt. The flipping mechanism is a flexible silicone baffle arranged between the first and second conveyor belts, which allows the fruit to be detected to fall onto the flexible silicone baffle when it is output from the first conveyor belt. Under the guidance of the flexible silicone baffle, the fruit is flipped due to gravity and falls onto the second conveyor belt.

[0032] In addition, the present invention also provides a multi-view fruit quality detection system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the multi-view fruit quality detection method.

[0033] In addition, the present invention provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the multi-view fruit quality detection method by a processor.

[0034] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multi-view fruit quality detection method by a processor.

[0035] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The method of the present invention includes acquiring a multi-view composite image of the fruit to be tested, wherein the multi-view composite image is obtained by stitching together sub-images from multiple perspectives, and the sub-images from multiple perspectives include directly captured images of the actual object and virtual images after being reflected once or multiple times by a plane mirror; the target detection network model pre-trained on the multi-view composite image is used to detect the detection results of each sub-image, wherein the detection results include the predicted bounding box, confidence score, and single-view quality normalization score of the fruit to be tested in the sub-image; the single-view quality normalization scores in the detection results of each sub-image are fused from multiple perspectives to obtain a multi-view quality normalization score. The present invention can achieve multi-view fruit quality detection to obtain multi-view, consistent fruit quality detection results. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of a multi-view composite image of the fruit being tested in an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of the network structure of the improved YOLOv5 detection model in an embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram of the network structure of the cross-scale residual stacking structure layer (C3) in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of the network structure of the Fast Spatial Pyramid Pooling Module (SPPF) in an embodiment of the present invention.

[0041] Figure 6 This is a schematic diagram of the network structure of the multi-scale dilated convolutional channel attention module (CAM) in an embodiment of the present invention.

[0042] Figure 7 This is a top view schematic diagram of the multi-view fruit quality detection device in an embodiment of the present invention.

[0043] Figure 8 This is a side view of the first conveyor belt, the second conveyor belt, and the flipping mechanism in an embodiment of the present invention.

[0044] Legend: 1. First conveyor belt; 2. Second conveyor belt; 3. Tilting mechanism; 4. First camera; 5. First plane mirror; 6. Second camera; 7. Second plane mirror. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0046] like Figure 1 As shown, the multi-view fruit quality detection method in this embodiment includes the following steps:

[0047] S101, acquire a multi-view composite image of the fruit being inspected, wherein the multi-view composite image is obtained by stitching together sub-images from multiple perspectives, such as... Figure 2 As shown, the multi-view sub-images include directly captured real object images (a and b) and virtual image images (a#1 and a#2, b#1 and b#2) after being reflected once or multiple times by a plane mirror, where a is the upper view and b is the lower view.

[0048] S102, the target detection network model pre-trained by the multi-view synthetic image detects the detection results of each sub-image. The detection results include the prediction box, confidence score and single-view quality normalization score of the detected fruit in the sub-image.

[0049] S103, perform multi-view fusion of the single-view quality normalization scores in the detection results of each sub-image to obtain a multi-view quality normalization score:

[0050] ;

[0051] in, To normalize the quality score from multiple perspectives, The total number of viewpoints. From the perspective The weight, From the perspective Single-view quality normalized score, all views The sum of the weights is 1, and the weight of the viewpoint corresponding to the real image is greater than the weight of the viewpoint corresponding to the virtual image.

[0052] For the two composite images obtained after flipping them vertically, the post-processing module determines the fruit surface area corresponding to the detection box based on its spatial position and geometric distribution. It then performs fusion calculations on indicators such as color uniformity, defect area ratio, shape deviation, and smoothness consistency of the same fruit across multiple viewing angles. The calculation employs a weighted average method based on image sharpness, taking into account the real image... It has no reflected light loss and minimal distortion, and its characteristic confidence is the highest, while the virtual image varies with the number of reflections. The sharpness decreases with increasing perspective. Therefore, this embodiment constructs the aforementioned multi-view fusion functional expression model to calculate the multi-view quality normalization score. .in, The total number of effective viewpoints (in this embodiment) ), From the perspective The single-view quality normalized score (including weighted sub-items for color, defects, and shape). Weighting coefficients. Assigned based on viewpoint type: when viewpoint Real-time weighting coefficients for real-image images Values When perspective Weighting coefficients for a single-reflection virtual image Values And satisfy the normalization condition In this embodiment, real image weights are set. Greater than virtual image weight (For example, take) =0.25, =0.125), thereby reducing the interference of virtual image edge blurring on the final rating. Robust characterization of the entire surface quality is achieved through multi-view weighted complementarity. The aim is to ensure the comprehensive multi-view characterization of fruit appearance quality and avoid missed detection, false detection and rating distortion caused by a single view.

[0053] As an optional implementation, the multi-view synthesized image in this embodiment is 640×960 in size and is obtained by stitching together two sub-images. One sub-image includes a real object image a, virtual images a#1 and a#2; the other sub-image includes a real object image b, virtual images b#1 and b#2. In this embodiment, the multi-view synthesized image treats multiple targets of the detected fruit from multiple perspectives as a single target in multi-target detection, thereby enabling parallel detection of multiple perspectives.

[0054] Considering that target detection may not be able to detect all six viewpoints, it is necessary to determine the detected viewpoints and then calculate the fruit diameter by combining the camera's shooting distance, camera intrinsic parameters, and detection frame width. The difficulty in this process lies in determining whether the fruit detected by different detection frames originates from specular reflection or actual shooting. For actual shooting, the shooting distance is used; for specular reflection, an equivalent distance needs to be used. To solve the above technical problems, step S103 of this embodiment further includes:

[0055] S201, Scaling and correcting the predicted bounding boxes in the detection results of each sub-image;

[0056] S202, Select the prediction box with the highest confidence level;

[0057] S203, calculate the estimated fruit diameter based on the prediction frame with the highest confidence level:

[0058] ;

[0059] in, To estimate the diameter of the fruit, The width (in pixels) of the prediction box with the highest confidence level. For pixel size, The equivalent imaging distance is the viewpoint corresponding to the prediction box with the highest confidence level. This refers to the camera's focal length.

[0060] By utilizing the reflective imaging properties of a plane mirror, a monocular camera can obtain... Several projected images of fruit from different perspectives, including one real image and... There are virtual images. Let the imaging distance corresponding to the real image be . For those who have gone through the first The equivalent imaging distance of the virtual image formed by secondary reflection can be expressed as:

[0061] ;

[0062] In the formula, The distance between the first plane mirror 5 and the fruit being tested is [missing information]. The distance between the mirror and the fruit This can be adjusted according to specific working conditions. Therefore, the equivalent distance of the virtual image increases linearly with the number of reflections, resulting in a decrease in image size as the distance increases. Real image The imaging distance is , No. The equivalent imaging distance of the secondary reflection virtual image is Since the image size is inversely proportional to the distance, in order to correct the size of the detection box in the virtual image to the scale of the real image (i.e., to unify it to the standard observation distance), (Scale below), scaling factor Defined as the ratio of equivalent distances. In this embodiment, step S201, scaling and correcting the predicted bounding boxes in the detection results of each sub-image includes adjusting the width of the predicted bounding boxes. and height The distribution is multiplied by a scaling factor for scaling correction, resulting in a corrected normalized size that eliminates the visual size difference caused by optical path difference. The scaling factor is calculated using the following function:

[0063] ;

[0064] in, This is the scaling factor. The number of times the sub-image is reflected by the mirror. The distance between the plane mirror and the fruit being tested. The imaging distance of a directly captured image of a real object. , This represents the maximum allowed reflection order.

[0065] Fruit quality cannot be graded solely by appearance; fruit size plays a significant role in fruit quality classification. Therefore, as an optional implementation, this embodiment further includes correcting the multi-view quality normalization score based on the estimated fruit diameter after step S203.

[0066] ;

[0067] ;

[0068] in, The corrected multi-view quality normalized score, To normalize the quality score from multiple perspectives, This is a correction factor for the uniformity of estimated fruit diameter. To estimate the diameter of the fruit, and These are the minimum and maximum diameters of standard fruit, respectively. For standard fruit diameter, This is the size deviation penalty factor. The formula indicates that when the fruit diameter is within the standard range, the appearance score is retained; when the diameter is too large or too small, the appearance score is reduced in weight according to the degree of deviation. By obtaining the fruit diameter and then correcting the fruit grade obtained from the multi-view fruit comprehensive rating module, the fruit quality consistency rating result is obtained.

[0069] To address the issues of low detection accuracy, slow detection speed, and difficulty in simultaneously processing multi-scale surface features in multi-view synthetic image scenarios, the target detection network model in step S102 of this embodiment is an improved YOLOv5 detection model. By modifying the feature fusion path and attention mechanism in the backbone and neck networks of the improved YOLOv5 detection model specifically for the current fruit detection scenario, the model's ability to extract fruit color changes, surface defect textures, and multi-scale structural features is enhanced, significantly improving detection accuracy while maintaining real-time detection speed. For example... Figure 3 As shown, the improved YOLOv5 detection model includes a backbone network, a neck network, and a detection network connected in sequence. The backbone network includes a series of convolutional modules, a four-level feature extraction unit, and a fast spatial pyramid pooling (SPPF) module. The feature extraction unit includes cascaded convolutional modules and a cross-scale residual stacking structure layer (C3). The convolutional module is formed by a combination of convolution operations, batch normalization, and the SiLU activation function, and can be represented as:

[0070] ;

[0071] in, and These are the output features and input features of the convolutional module, respectively. The SiLU activation function is used. For batch normalization, For convolution kernel, This is a convolutional operator. The backbone network sequentially passes the input features through the first convolutional module, the second convolutional module, and the first cross-scale residual stacked structure layer (C3) to output the first-scale features. Then, the first-scale features are input into the fourth convolutional module and the second cross-scale residual stacked structure layer (C3) to obtain the second-scale features. The second-scale features are then input into the fifth convolutional module, the third cross-scale residual stacked structure layer (C3), and the fast spatial pyramid pooling module (SPPF) to extract semantic features with a large receptive field, resulting in the third-scale features. Finally, the three sets of scale features are input into the neck network to achieve cross-scale semantic integration. Figure 4 As shown, the cross-scale residual stacked structure layer (C3) includes two branches. One branch (direct mapping branch) includes a convolutional layer, and the other branch (bottleneck mapping branch with residual structure) includes a convolutional layer and a bottleneck layer. The output features of the two branches are connected by a connection module and then passed through a convolutional layer to obtain the final output features, which can be represented as:

[0072] ;

[0073] ;

[0074] in, For the convolution kernel of the last convolutional layer, For the input characteristics of the connection module, and These are the output features of the two branches, respectively. This is a bottleneck mapping branch with residual structure. As a direct mapping branch, the above structure can enhance multi-channel expressive power without significantly increasing model complexity. To address the issue of consistent background across multiple viewpoints in industrial scenarios, which can easily lead to overfitting, the bottleneck layer of the cross-scale residual stacked structure layer (C3) includes two bottleneck branches. One bottleneck branch includes a convolutional layer, and the other bottleneck branch includes two cascaded convolutional layers, which can be represented as:

[0075] ;

[0076] in, The input features for the bottleneck layer, The input features of the two cascaded convolutional layers to the bottleneck layer Extracted low-level detail features It is a 1×1 convolution kernel. For convolution operators, As the input features of the bottleneck layer, this structure establishes a controllable information pathway between deep semantic features and low-level detailed features by introducing 1×1 convolutional skip connections. It can adaptively and selectively retain low-level information according to the complexity of input features, significantly improving the generalization ability and stability in multi-view fruit detection tasks.

[0077] The Fast Spatial Pyramid Pooling (SPPF) module is used to quickly capture contextual information at different scales, introducing a large receptive field through the recursive stacking of multi-level pooling kernels. For example... Figure 5 As shown, the Fast Spatial Pyramid Pooling (SPPF) module includes sequentially connected convolutional layers, a connection module, and another convolutional layer. The output feature of the first convolutional layer is also connected to a parallel bypass branch, which includes three sequentially cascaded max-pooling layers. The output features of these three max-pooling layers all serve as input features to the connection module. Its output can be expressed as:

[0078] ;

[0079] in, For the output characteristics of the Fast Spatial Pyramid Pooling Module (SPPF), The output features of the first convolutional layer, , and The output features of the three cascaded max pooling layers are represented by a 5×5 pooling kernel, which enhances the module's ability to capture fruit targets of different sizes while ensuring high efficiency.

[0080] The feature fusion stage of the neck network employs a Feature Pyramid Network (FPN) structure, combined with a bottom-up Path Augmentation Network (PANet), to achieve bidirectional transfer of semantic and detailed features. For example... Figure 3As shown, the neck network consists of 17 layers connected in sequence: convolutional layers, upsampling modules, connection modules, cross-scale residual stacked structure layers (C3), convolutional layers, multi-scale dilated convolutional channel attention modules (CAM), upsampling modules, connection modules, cross-scale residual stacked structure layers (C3), convolutional layers, multi-scale dilated convolutional channel attention modules (CAM), connection modules, cross-scale residual stacked structure layers (C3), convolutional layers, multi-scale dilated convolutional channel attention modules (CAM), connection modules, and cross-scale residual stacked structure layers (C3). The detection network comprises three detection heads. The output features of layers 9, 13, and 17 of the 17-layer network are processed by an additional convolutional layer and then used as input features for one detection head. The second and third cross-scale residual stacking structure layers (C3) and the fast spatial pyramid pooling module (SPPF) in the backbone network output features at three different scales, which are then used as input features for layers 8, 3, and 1 of the 17-layer neck network. The output features of layers 1 and 6 of the 17-layer neck network are also used as input features for layers 12 and 16 of the 17-layer neck network, respectively. This structure ensures the consistency of multi-scale semantics, thereby improving detection accuracy.

[0081] Based on the improved feature extraction network described above, in this embodiment, to address the issue of uneven feature responses in multi-view synthesized images under different scales and reflection conditions, and to further enhance the model's selective representation of the real surface areas of the fruit, such as... Figure 6 As shown, the multi-scale dilated convolutional channel attention module (CAM) includes three two-dimensional convolutional branches with different dilation rates. The input features are processed by the three two-dimensional convolutional branches to extract features F1, F2, and F3 with different receptive fields. Then, features F1, F2, and F3 are respectively processed by the first 1×1 convolution for channel compression, and then channel concatenation is performed to form a joint multi-scale feature tensor. Joint multi-scale feature tensor After a second 1×1 convolution, channel transformation is performed to project features with different hole rates onto a unified feature domain to obtain unified features. To unify features The response values ​​of each scale channel are obtained by global average pooling using the following formula:

[0082] ;

[0083] in, For scale channel Response volume This is the index number of the two-dimensional convolution branch. and To unify features Height and width, To unify features Located at position on the r-th scale channel The eigenvalues; the attention weights for the three scale channels are obtained by normalizing the responses of each scale channel using the Softmax activation function:

[0084] ;

[0085] in, For scale channel Attention weights For scale channel Response volume The index number is the two-dimensional convolutional branch; features F1, F2, and F3 are multiplied by the attention weights of the corresponding two-dimensional convolutional branches and then summed to obtain the final output multi-scale augmentation features. The multi-scale dilated convolutional channel attention module (CAM) is based on a multi-scale dilated convolution and adaptive weighted fusion mechanism. It can automatically select the feature branch that is most suitable for the current fruit structure information from multiple different receptive field scales while maintaining the integrity of the original feature space structure, thus achieving cross-viewpoint and cross-scale consistency enhancement.

[0086] In this embodiment, the multi-scale dilated convolutional channel attention module (CAM) uses feature maps... As input, this feature map is fed into three two-dimensional convolutional branches with different dilation rates. All three convolutional kernels employ... The convolutional form, with its hole ratio set as follows: The dilated convolution operation can be represented as:

[0087] ;

[0088] in, For scale channel The dilated convolution operation at position Output feature value at, For scale channel The convolution kernel at position The value of , For feature map Middle position eigenvalues ​​at that location This is the index of the scale channel (in this embodiment, the value is the hole rate). The scale channel... The convolutional branches primarily extract high-frequency details such as local textures and minor defects; the scale channel... The convolutional branches have a larger receptive field, capable of capturing color distribution and morphological changes over a larger area of ​​the fruit surface; scale channels The convolutional branches have a further expanded coverage, making them more adaptable to lateral parallax variations in synthesized images from flipped viewpoints. Feature maps are obtained after passing through three dilated convolutional branches. , , They have the same spatial size but their receptive fields cover different scales, thus forming a multi-scale feature set:

[0089] ;

[0090] Subsequently, the above features are compressed through 1×1 convolution. The compression process can be represented as follows:

[0091] ;

[0092] in, For scale channel The characteristics of the channel after compression The convolution kernel is a 1×1 convolution. It is a convolution operator;

[0093] A joint multi-scale feature tensor is formed through channel splicing operations. :

[0094] ;

[0095] in, This is for channel splicing operations. Features are the channel-compressed characteristics of the three scale channels;

[0096] Building upon this, a second 1×1 convolution is used to perform channel transformation on the concatenated features, projecting features with different dilation rates onto a unified feature domain.

[0097] ;

[0098] To achieve adaptive scale selection, this module uses the Softmax activation function to normalize the responses of the three branches along the channel dimension, obtaining the attention weights for the three branches:

[0099] ;

[0100] in This represents the response of the corresponding feature branch after global statistics. The normalization property of Softmax ensures that:

[0101] ;

[0102] This results in a strict multi-scale adaptive weight allocation mechanism.

[0103] Obtain weight Then, these values ​​are applied to the corresponding branch feature maps to achieve channel-by-channel weighting and calibration.

[0104] ;

[0105] The final multi-scale enhanced features are obtained through element-wise weighted fusion:

[0106] ;

[0107] The above output multi-scale enhancement features While preserving the local texture, global morphological structure, and geometric consistency of multi-view imaging in the input image, it effectively suppresses redundant activation in virtual image regions caused by specular reflection and highlights real surface structure regions with higher quality discriminative significance. Dilated convolution expands the receptive field without increasing computational cost, enabling the Scale-Dilated Convolutional Channel Attention Module (CAM) to significantly improve the model's ability to represent differences in fruit structure from multiple perspectives while maintaining high inference speed. This enhances the reliability of subsequent detection modules in color anomaly recognition, defect distribution judgment, and morphological consistency analysis. Through the aforementioned Scale-Dilated Convolutional Channel Attention Module (CAM) configuration, the neck network in this embodiment can adaptively select the optimal combination of features from different perspectives at multiple scales during the feature fusion stage. This gives the model higher perspective robustness and structural consistency representation ability in high-speed industrial flow scenarios. After fusion output, the processed multi-scale features are input into the neck network for bidirectional information enhancement, giving the fused features both stronger semantic discriminative ability and accurate spatial localization capability.

[0108] After feature fusion, three sets of detection feature maps corresponding to high, medium, and low resolutions were obtained. These feature maps were then input into the YOLOv5 network for target detection inference. This module aligns with the dense prediction approach of YOLOv5. The detection network generates candidate boxes for each feature point based on the dense prediction mechanism and regresses the predicted box, confidence score, and single-view quality normalization score of the detected fruit. For example, the function expression for the regression of the predicted box position can be represented as:

[0109] ;

[0110] in, For the regression of the predicted box position, For the Sigmoid function, The output features of the neck network, As a scale factor, The parameters are the center or width and height of the predicted bounding box. During the training phase, the aforementioned data augmentation strategy is adopted to enable the model to learn robust feature representations under conditions of virtual images, small-sized fruit projections, and uneven lighting, ensuring stable and reproducible detection performance under industrial flow and high-speed acquisition conditions.

[0111] To verify the performance of the object detection network model proposed in this embodiment, the model was validated on existing public datasets. The comparison models used included YOLOv5 and YOLOv5+CAM, where YOLOv5+CAM is an enhanced version of YOLOv5, specifically a combination of YOLOv5 and the Contextual Attention Module (CAM). The metrics used included: Precision (B), representing the proportion of truly correct predictions among positive samples, reflecting the accuracy of detection; Recall (B), representing the proportion of all real targets successfully detected by the model, reflecting the completeness of detection; and the F1 score (B), which is the harmonic mean of precision and recall, comprehensively measuring the balance between the two. The mean average precision (mAP) is calculated using three methods: mAP50(B), mAP75(B), and mAP50-95(B). mAP50(B) is the mean average precision calculated for all categories with an IoU (Intersection over Union) threshold of 0.5, and is one of the most commonly used metrics in object detection. mAP75(B) is the mAP calculated under a more stringent IoU threshold of 0.75, requiring the predicted bounding box to highly overlap with the ground truth bounding box. mAP50-95(B) is the average mAP across multiple thresholds with IoU values ​​ranging from 0.5 to 0.95, in steps of 0.05, providing a more comprehensive reflection of the model's overall performance under different localization accuracy requirements. The final results are shown in Table 1.

[0112] Table 1: Comparison of detection results from multiple models

[0113]

[0114] As shown in Table 1, compared with the comparative models YOLOv5 and YOLOv5+CAM, the target detection network model (the model of this method) in this embodiment achieves performance similar to the optimal YOLOv5+CAM in terms of accuracy, while outperforming the comparative models YOLOv5 and YOLOv5+CAM in the other five indicators. Therefore, the target detection network model proposed in this embodiment can achieve multi-view fruit quality detection to obtain consistent fruit quality detection results from multiple perspectives.

[0115] like Figure 7 and Figure 8As shown, this embodiment also provides a multi-view fruit quality detection device for applying the aforementioned multi-view fruit quality detection method, including a first conveyor belt 1, a second conveyor belt 2, a flipping mechanism 3, a first camera 4, a first plane mirror 5, a second camera 6, and a second plane mirror 7. The flipping mechanism 3 is arranged between the first conveyor belt 1 and the second conveyor belt 2 to flip the fruit to be detected on the first conveyor belt 1 and place it on the second conveyor belt 2. The first camera 4 is arranged on the receiving side of the first conveyor belt 1. The number of first plane mirrors 5 is multiple and they are respectively arranged on the first camera 4. The target detection position is located diagonally behind the first conveyor belt 1; the second camera 6 is arranged on the discharge side of the second conveyor belt 2; the second plane reflector 7 consists of multiple pieces, which are respectively arranged diagonally behind the target detection position of the second camera 6 on the second conveyor belt 2; the height of the first conveyor belt 1 is higher than that of the second conveyor belt 2; the flipping mechanism 3 is a flexible silicone baffle arranged between the first conveyor belt 1 and the second conveyor belt 2, which is used to make the fruit being detected fall onto the flexible silicone baffle when it is output from the first conveyor belt 1, and flip over due to gravity under the guidance of the flexible silicone baffle and fall onto the second conveyor belt 2.

[0116] In this embodiment, the widths of the first conveyor belt 1 and the second conveyor belt 2 are 300mm. The first camera 4 and the second camera 6 are CCD cameras, and the CCD camera imaging parameters are: resolution... Pixel size ,focal length The first camera 4 and the second camera 6 are mounted above the fruit shooting point. Using a first plane mirror 5 and a second plane mirror 7 with silver-plated front surfaces can avoid ghosting caused by refraction shift of the glass medium.

[0117] In this embodiment, there are two first planar reflectors 5, which are respectively arranged obliquely behind the target detection position of the first camera 4 on the first conveyor belt 1, and the included angle between the two first planar reflectors 5 is 120 degrees. Taking the imaging of the first camera 4 as an example, the flipping mechanism 3 flips the sample by means of a drop and a flexible silicone baffle. When the fruit to be detected passes through the first conveyor belt 1 in front of the first camera 4, the two first planar reflectors 5 are placed on the side of the sample at an included angle of 120 degrees. A light source is installed around the first planar reflectors 5 so that the light shines evenly on the surface of the first planar reflectors 5. Due to the spherical reflection of some fruit, a polarizer can be added in front of the optical lens of the first camera 4 and rotated appropriately to filter out the reflective spots on the surface of smooth fruit to the greatest extent. With the help of the reflection of the first planar reflectors 5, the monocular camera (first camera 4) can capture the front and side image information of the fruit bunch. The intersection point of the imaging principal rays of the first camera 4, the fruit to be detected, and the two first planar reflectors 5 is on the same straight line, and the two planar reflectors are respectively at an angle They are symmetrically arranged on both sides of the imaging center line. In this embodiment, the included angle of the mirrors... Therefore, the deflection angle of each mirror towards the center line is... This arrangement ensures that the reflection path maintains strict geometric symmetry. With this configuration, utilizing the reflection imaging characteristics of the plane mirror, a monocular camera can obtain... Several projected images of fruit from different perspectives, including one real image and... There are virtual images. Let the imaging distance corresponding to the real image be . For those who have gone through the first The equivalent imaging distance of the virtual image formed by secondary reflection can be expressed as:

[0118] ;

[0119] In the formula, The distance between the first plane mirror 5 and the fruit being tested is [missing information]. The distance between the mirror and the fruit It can be adjusted according to specific working conditions. Therefore, the equivalent distance of the virtual image increases linearly with the number of reflections, resulting in a decrease in the image size as the distance increases. According to the pinhole imaging model, the size of the fruit on the imaging plane... Compared with the actual diameter of the object satisfy:

[0120] ;

[0121] in, Where is the focal length of the camera lens. In this embodiment, the trend of the virtual image size decreasing with increasing number of reflections can be written as: ; Describe the real image ( A plane mirror has the largest and clearest image size, while the sharpness of a virtual image decreases with increasing distance. Simultaneously, because reflections from a plane mirror introduce a need for field-of-view expansion, the width of the plane mirror increases with the number of reflections. Must meet:

[0122] ;

[0123] in This is the effective field of view of the camera. When... When the image size is large, the required mirror width increases dramatically, making the structure difficult to implement. Furthermore, the projection areas between multiple virtual images are prone to geometric overlap, significantly increasing the difficulty of object segmentation and negatively impacting the stability of subsequent online detection. Based on the above imaging geometry analysis, it can be seen that when… At this time, the number of images is three-view (real image + one virtual image on each side of the left and right), which can cover the entire surface of the fruit and avoid the decrease in clarity and increase in structural complexity caused by multiple reflections. Therefore, this embodiment adopts a three-view imaging scheme as the optimal design.

[0124] Similarly, there are two second plane mirrors 7, which are respectively arranged diagonally behind the target detection position on the second conveyor belt 2 by the second camera 6, and the included angle between the two second plane mirrors 7 is 120 degrees. The fruit to be detected first passes on the first conveyor belt 1, where the first camera 4 can simultaneously capture the real image a and the virtual images a#1 and a#2; then, the fruit to be detected is flipped up and down by the flipping mechanism 3 and enters the second conveyor belt 2. On the second conveyor belt 2, the second camera 6 can simultaneously capture the real image b and the virtual images b#1 and b#2. The two sets of photos are stitched together to obtain a 640×960 image, which can quickly stitch together a multi-view composite image of the fruit to be detected. In this way, only two cameras and one flipping mechanism 3 are needed to obtain the full surface information of the fruit. The device in this embodiment can operate at high speed, continuously and stably in industrial settings, enabling the collection of information from the entire surface of fruit, high-precision target detection, multi-view quality fusion and structural consistency assessment. It can effectively overcome the problems of traditional single-view methods, such as one-sided detection, insufficient detection accuracy, and inability to uniformly consider fruit size, providing a practical and scalable intelligent visual inspection solution for fruit grading.

[0125] Those skilled in the art will understand that the technical solutions provided by this invention can take the form of methods, systems, or computer program products. For example, this invention can provide a multi-view fruit quality detection system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the multi-view fruit quality detection method. This invention can provide a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the multi-view fruit quality detection method via a processor. This invention can provide a computer program product including a computer program or instructions programmed or configured to execute the multi-view fruit quality detection method via a processor. Furthermore, this invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this invention can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of a flowchart and / or block diagram, and combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-view fruit quality detection method, characterized in that, Includes the following steps: S101, acquire a multi-view composite image of the fruit being tested. The multi-view composite image is obtained by stitching together sub-images from multiple perspectives. The sub-images from multiple perspectives include directly captured images of the actual object and virtual images after being reflected once or multiple times by a plane mirror. S102, the target detection network model pre-trained by the multi-view synthetic image detects the detection results of each sub-image. The detection results include the prediction box, confidence score and single-view quality normalization score of the detected fruit in the sub-image. S103, perform multi-view fusion of the single-view quality normalization scores in the detection results of each sub-image to obtain a multi-view quality normalization score: ； in, To normalize the quality score from multiple perspectives, The total number of viewpoints. From the perspective The weight, From the perspective Single-view quality normalized score, all views The sum of the weights is 1, and the weight of the viewpoint corresponding to the real image is greater than the weight of the viewpoint corresponding to the virtual image; step S103 also includes: S201, Scaling and correcting the predicted bounding boxes in the detection results of each sub-image; S202, Select the prediction box with the highest confidence level; S203, calculate the estimated fruit diameter based on the prediction frame with the highest confidence level: ； in, To estimate the diameter of the fruit, The width of the predicted bounding box with the highest confidence level. For pixel size, The equivalent imaging distance is the viewpoint corresponding to the prediction box with the highest confidence level. The camera focal length is used; the multi-view quality normalization score is corrected based on the estimated fruit diameter. ； ； in, The corrected multi-view quality normalized score, To normalize the quality score from multiple perspectives, This is a correction factor for the uniformity of estimated fruit diameter. To estimate the diameter of the fruit, and These are the minimum and maximum diameters of standard fruit, respectively. For standard fruit diameter, This is the penalty factor for size deviation.

2. The multi-view fruit quality inspection method according to claim 1, characterized in that, Step S201 involves scaling and correcting the predicted bounding boxes in the detection results of each sub-image by multiplying the width and height distribution of the predicted bounding boxes by a scaling factor. The calculation function expression for the scaling factor is as follows: ； in, This is the scaling factor. The number of times the sub-image is reflected by the mirror. The distance between the plane mirror and the fruit being tested. The imaging distance of a directly captured image of a real object. , This represents the maximum allowed reflection order.

3. The multi-view fruit quality inspection method according to claim 1, characterized in that, The target detection network model in step S102 is an improved YOLOv5 detection model. The improved YOLOv5 detection model includes a backbone network, a neck network, and a detection network connected in sequence. The backbone network includes a convolutional module, a four-level feature extraction unit, and a fast spatial pyramid pooling module connected in sequence. The feature extraction unit includes a cascaded convolutional module and a cross-scale residual stacked structure layer. The cross-scale residual stacked structure layer includes two branches, one of which includes a convolutional layer and the other includes a convolutional layer and a bottleneck layer. The output features of the two branches are connected by a connection module and then passed through a convolutional layer to obtain the final output features. The bottleneck layer includes two bottleneck branches, one of which includes a convolutional layer and the other includes two cascaded convolutional layers. The fast spatial pyramid pooling module includes a convolutional layer, a connection module, and a convolutional layer connected in sequence. The output features of the first convolutional layer are also connected by a parallel bypass branch. The bypass branch includes three cascaded max pooling layers, and the output features of the three max pooling layers are all used as input features of the connection module.

4. The multi-view fruit quality inspection method according to claim 3, wherein, The neck network comprises 17 layers connected in sequence: convolutional layers, upsampling modules, connection modules, cross-scale residual stacked structure layers, convolutional layers, multi-scale dilated convolutional channel attention modules, connection modules, and cross-scale residual stacked structure layers. The detection network includes three detection heads. The output features of layers 9, 13, and 17 are processed by an additional convolutional layer and used as input features for one detection head. The second and third layers of the backbone network... The cross-scale residual stacking structure layer and the fast spatial pyramid pooling module output features at three different scales, which are used as input features for layers 8, 3, and 1 of the 17-layer neck network. The output features of layers 1 and 6 of the 17-layer neck network are also used as input features for layers 12 and 16 of the 17-layer neck network, respectively. The multi-scale dilated convolutional channel attention module includes three two-dimensional convolutional branches with different dilation rates. The input features are processed by the three two-dimensional convolutional branches to extract features F1, F2, and F3 with different receptive fields. Then, features F1, F2, and F3 are channel compressed by the first 1×1 convolution and then concatenated to form a joint multi-scale feature tensor. Joint multi-scale feature tensor After a second 1×1 convolution, channel transformation is performed to project features with different hole rates onto a unified feature domain to obtain unified features. To unify features The response values ​​of each scale channel are obtained by global average pooling using the following formula: ； in, For scale channel Response volume This is the index number of the two-dimensional convolution branch. and To unify characteristics Height and width, To unify characteristics Located at position on the r-th scale channel The eigenvalues; the attention weights for the three scale channels are obtained by normalizing the responses of each scale channel using the Softmax activation function: ； in, For scale channel Attention weights For scale channel Response volume The index number is the two-dimensional convolutional branch; features F1, F2, and F3 are multiplied by the attention weights of the corresponding two-dimensional convolutional branches and then summed to obtain the final output multi-scale augmentation features. .

5. A multi-view fruit quality detection apparatus for applying the multi-view fruit quality detection method according to any one of claims 1 to 4, characterized by The system includes a first conveyor belt (1), a second conveyor belt (2), a flipping mechanism (3), a first camera (4), a first plane mirror (5), a second camera (6), and a second plane mirror (7). The flipping mechanism (3) is arranged between the first conveyor belt (1) and the second conveyor belt (2) to flip the fruit to be detected on the first conveyor belt (1) and place it on the second conveyor belt (2). The first camera (4) is arranged on the receiving side of the first conveyor belt (1). The first plane mirror (5) consists of multiple mirrors and is arranged diagonally behind the target detection position of the first camera (4) on the first conveyor belt (1). The second camera (6) is arranged on the discharge side of the second conveyor belt (2). The second plane mirror (7) consists of multiple pieces and is arranged diagonally behind the target detection position of the second camera (6) on the second conveyor belt (2). The height of the first conveyor belt (1) is higher than that of the second conveyor belt (2). The flipping mechanism (3) is a flexible silicone baffle arranged between the first conveyor belt (1) and the second conveyor belt (2). It is used to make the fruit to be detected fall onto the flexible silicone baffle when it is output from the first conveyor belt (1). Under the guidance of the flexible silicone baffle, it flips due to gravity and falls onto the second conveyor belt (2).

6. A multi-view fruit quality inspection system comprising a microprocessor and a memory connected to each other, characterized by, The microprocessor is programmed or configured to perform the multi-view fruit quality detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium having stored therein a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the multi-view fruit quality detection method according to any one of claims 1 to 4 via a processor.

8. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the multi-view fruit quality detection method according to any one of claims 1 to 4 via a processor.

Images