Fruit defect detection device and method based on reflection imaging

Abstract

The invention provides a fruit defect detection device and method based on reflection imaging, and relates to the technical field of fruit detection.The device comprises a conveying mechanism, a trigger, a reflection imaging mechanism, a shooting mechanism and a fruit defect analysis system, the trigger sends shooting signals to the reflection imaging mechanism and the shooting mechanism when detecting the fruit, the reflection imaging mechanism adjusts the angle of the reflection lens and reflects the side surface and the back surface of the fruit to the visual field of the shooting mechanism, and the shooting mechanism shoots the front surface of the fruit and the side surface and the back surface reflected by the reflection lens to obtain a global fruit image set. The fruit defect analysis system inputs a test image and outputs a defect segmentation result through an image set training model, and judges the grade according to a preset rule; according to the device, single-camera shooting dead angles are eliminated through reflection, a multi-camera scheme is replaced, cost and maintenance are reduced, transmission, triggering and shooting linkage is adapted to industrial continuous production, and the detection efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of fruit detection technology, specifically to a fruit defect detection device and method based on reflection imaging. Background Technology

[0002] In existing technologies, when a single camera directly photographs fruit, there is a limited field of view, making it difficult to cover the entire surface of the fruit. This can easily create blind spots, causing local surface features to be missed and making it impossible to obtain a complete global image of the fruit's surface, thus affecting the comprehensiveness and accuracy of defect detection.

[0003] Meanwhile, traditional detection methods rely on manual labor or simple algorithms in feature extraction and defect identification, resulting in low accuracy in identifying complex and diverse surface blemishes on fruits. Furthermore, grading decisions are easily influenced by subjective factors, leading to poor consistency and failing to meet the demands of efficient and accurate detection in industrial production. In addition, while multi-camera shooting solutions can address the field-of-view problem to some extent, they increase hardware costs and equipment maintenance complexity, hindering practical application and promotion. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a fruit defect detection device and method based on reflection imaging, which addresses the shortcomings of the prior art.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: a fruit defect detection device based on reflection imaging, comprising a transmission mechanism, a trigger, a reflection imaging mechanism, an imaging mechanism, and a fruit defect analysis system; The conveying mechanism is used to convey the fruit to be analyzed to the shooting mechanism; The trigger is used to send a shooting signal to the reflection imaging mechanism and the shooting mechanism when the fruit to be analyzed is detected; The reflection imaging mechanism is used to adjust the angle of the reflective lens according to the shooting signal to align with the fruit to be analyzed, so that the surfaces of the fruit to be analyzed located on the side and back of the shooting mechanism are reflected into the field of view of the shooting mechanism. The imaging mechanism is used to take multiple pictures of the front of the fruit to be analyzed, as well as the side and back of the fruit to be analyzed through the reflecting lens, to obtain a global fruit image set; The fruit defect analysis system is used to train the constructed fruit defect analysis model based on the global fruit image set; The test fruit image is input into the trained fruit defect analysis model, which outputs the segmentation results corresponding to different types of defects. The segmentation results are then graded based on the preset grading rules to obtain the fruit grade grading results.

[0006] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A fruit defect detection method based on reflection imaging, employing the fruit defect detection device as described above, including a conveying mechanism, a trigger, a reflection imaging mechanism, an imaging mechanism, and a fruit defect analysis system; comprising the following steps: The conveying mechanism transports the fruit to be analyzed to the shooting mechanism; When the trigger detects the fruit to be analyzed, it sends a shooting signal to the reflection imaging mechanism and the shooting mechanism. The reflective imaging mechanism adjusts the angle of the reflective lens to align with the fruit to be analyzed according to the shooting signal, so that the surfaces of the fruit to be analyzed located on the side and back of the shooting mechanism are reflected into the field of view of the shooting mechanism; The imaging mechanism takes multiple images of the front of the fruit to be analyzed, as well as the side and back of the fruit through the reflecting lens, to obtain a global fruit image set; The fruit defect analysis system trains the constructed fruit defect analysis model based on the global fruit image set, inputs test fruit images into the trained fruit defect analysis model, outputs segmentation results corresponding to different types of defects, and determines the grade of the segmentation results based on preset grade grading rules to obtain fruit grade grading results.

[0007] The beneficial effects of this invention are: by adjusting the angle of the reflective lens through the reflective imaging mechanism, the side and back surfaces of the fruit are reflected to the field of view of the shooting mechanism, and combined with frontal shooting, a single industrial camera can acquire a global surface image of the fruit, eliminating the blind spots of traditional single-camera shooting and avoiding the omission of local features; A global detection system is achieved by using an imaging mechanism in conjunction with reflective lenses, replacing the multi-camera solution, reducing equipment investment costs, and also reducing the complexity of equipment installation and maintenance; By linking the conveying mechanism, trigger, and shooting mechanism, automatic shooting is triggered during the dynamic conveying of fruits, adapting to industrial continuous production scenarios and improving detection efficiency. Attached Figure Description

[0008] Figure 1 This is a block diagram showing the connection of various components of the fruit defect detection device provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the fruit defect detection device provided in an embodiment of the present invention; Figure 3 This is a structural diagram of the fruit defect analysis model provided in an embodiment of the present invention; Figure 4 A flowchart of a fruit defect detection method provided in an embodiment of the present invention.

[0009] In the attached diagram, the component names represented by each label are as follows: 1. Conveying mechanism, 2. Trigger, 3. Shooting mechanism, 4. Fruit to be analyzed, 5. Reflecting lens, 6. Drive motor, 7. Tripod. Detailed Implementation

[0010] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0011] like Figure 1-2 As shown, this embodiment of the invention provides a fruit defect detection device based on reflection imaging, including a conveying mechanism 1, a trigger 2, a reflection imaging mechanism, an imaging mechanism 3, and a fruit defect analysis system; The conveying mechanism 1 is used to convey the fruit 4 to be analyzed to the shooting mechanism 3; The trigger 2 is used to send a shooting signal to the reflection imaging mechanism and the shooting mechanism 3 when the fruit 4 to be analyzed is detected; The reflection imaging mechanism is used to adjust the angle of the reflective lens 5 to align with the fruit 4 to be analyzed according to the shooting signal, so that the surfaces of the fruit 4 to be analyzed located on the side and back of the shooting mechanism 3 are reflected into the field of view of the shooting mechanism 3. The shooting mechanism 3 is used to take multiple pictures of the front of the fruit to be analyzed 4 and the side and back of the fruit to be analyzed 4 through the reflecting lens 5 to obtain a global fruit image set. The fruit defect analysis system is used to train the constructed fruit defect analysis model based on the global fruit image set; The test fruit image is input into the trained fruit defect analysis model, which outputs the segmentation results corresponding to different types of defects. The segmentation results are then graded based on the preset grading rules to obtain the fruit grade grading results.

[0012] The imaging mechanism 3 is an industrial camera. The fruit moves from left to right on a conveyor belt. When the fruit passes trigger 2, trigger 2 transmits a signal to the industrial camera in imaging mechanism 3. The industrial camera takes a picture of the fruit. The two lenses are adjusted to the reflection angle by an adjustable and rotating tripod. The industrial camera can then capture an image of the entire surface of the fruit. The image is sent to the fruit defect analysis system for identification, and finally, the fruit is graded.

[0013] The reflection imaging mechanism is located on one side of the transmission mechanism 1, and the shooting mechanism 3 is located on the other side of the transmission mechanism 1 opposite to the reflection imaging mechanism.

[0014] Trigger 2 is connected to the reflective imaging mechanism and the shooting mechanism 3 wirelessly or via wire.

[0015] The fruit defect analysis system is wirelessly or wiredly connected to the shooting location 3.

[0016] In this embodiment, the angle of the reflective lens 5 is adjusted by the reflective imaging mechanism to reflect the side and back surfaces of the fruit to the field of view of the shooting mechanism 3. Combined with frontal shooting, a single industrial camera can acquire a global surface image of the fruit, eliminating the blind spots of traditional single-camera shooting and avoiding the omission of local features. The imaging mechanism 3, combined with the reflective lens 5, achieves global detection, replacing the multi-camera solution, reducing equipment investment costs, and also reducing the complexity of equipment installation and maintenance; By linking the conveying mechanism 1, the trigger 2, and the shooting mechanism 3, automatic shooting is triggered during the dynamic conveying of fruits, which is suitable for industrial continuous production scenarios and improves detection efficiency.

[0017] Preferably, in the fruit defect analysis system, training the constructed fruit defect analysis model based on the global fruit image set includes: Each fruit image in the global fruit image set is labeled, and the labeling includes the fruit category, the surface defect category, and the area where the defect is located; Specifically, LabelMe or other data annotation tools are used for labeling and segmentation. Through labeling and segmentation, the fruits to be identified and the types of defects on their surfaces are categorized, such as citrus fruits and their surface defects like ulcers and blemishes. The labeling also includes the area occupied by the fruits and their surface defects for subsequent network localization. Correctly labeling each category using annotation software results in higher accuracy, which in turn improves the accuracy of the network during training. This completes the annotation of the global fruit image set, which is then fed into the fruit defect analysis model for training.

[0018] like Figure 3 As shown, a fruit defect analysis model is constructed based on the DeepLabv3+ semantic image segmentation deep learning model. The fruit defect analysis model includes an encoder module and a decoder module. The encoder module includes a DCNN deep convolutional neural network, an ASPP (space pyramid pooling) module with dilated spatial pyramids, and a global pooling layer connected in sequence. Based on the labeled fruit images, the DCNN deep convolutional neural network is trained to extract features and output low-level semantic features of the fruit surface texture. The ASPP void spatial pyramid pooling module performs multi-scale fusion processing on the low-level semantic features to obtain multi-scale features of fruit surface blemish categories. The global pooling layer performs global pooling on the multi-scale features to obtain high-level semantic features, and then compresses the high-level semantic features to output compressed high-level semantic features. The decoder module includes a dimensionality reduction module, an upsampling module, a concat feature fusion module, and a restoration module. The input of the dimensionality reduction module is connected to the DCNN deep convolutional neural network. The input of the upsampling module is connected to the output of the global pooling layer. The outputs of both the dimensionality reduction module and the upsampling module are connected to the input of the concat feature fusion module. The output of the concat feature fusion module is connected to the restoration module. The dimensionality reduction module performs convolutional dimensionality reduction processing on the low-level semantic features output by the DCNN deep convolutional neural network, and inputs the dimensionality-reduced low-level semantic features into the concat feature fusion module; The upsampling module upsamples the compressed high-level semantic features output by the global pooling layer and inputs the upsampled high-level semantic features into the concat feature fusion module. The concat feature fusion module fuses the dimensionality-reduced low-level semantic features and the upsampled high-level semantic features to obtain fused semantic features. The restoration module performs bilinear interpolation upsampling on the fused semantic features to restore the size of the fused semantic features to the same size as the fruit images in the global fruit image set, thereby generating a segmentation result.

[0019] In the decoder module, the pooled high-level semantic features are upsampled by 4 times, the low-level semantic features extracted by the backbone network (encoder) are reduced in dimensionality by 1*1 convolution, and then the two processed semantic features are fused together (concat). Then, the image size is restored by using 3×3 convolution and 4 times bilinear interpolation upsampling, and finally the segmentation result is generated.

[0020] In this embodiment, the tool is used to label fruit categories, blemish categories, and regions to provide high-quality labeled samples for model training. The labeling accuracy directly improves the subsequent model recognition accuracy. The encoder and decoder modules have a clear division of labor. The encoder module extracts multi-scale features, while the decoder module fuses high- and low-level features and restores the image size, ensuring the semantic accuracy and spatial detail integrity of the segmentation results. An end-to-end training process based on a global fruit image set enables the model to adapt to different fruit surface features, improving its ability to generalize the recognition of complex defects.

[0021] Preferably, the step of training the DCNN deep convolutional neural network based on the labeled fruit images to extract features and output low-level semantic features of the fruit surface texture includes: The DCNN deep convolutional neural network uses the Xception model as its backbone network. The Xception model includes channel-wise convolutional layers and point-wise convolutional layers. The channel-wise convolutional layer convolves each channel of the labeled fruit image individually to obtain the convolutional features corresponding to each channel, represented as follows: , in, The image shows the labeled fruit. Let be the weight of the convolution kernel corresponding to the d-th channel at position (m, n), where m and n represent the spatial offset of the convolution kernel, and d is the channel index, corresponding to the d-th channel of the labeled fruit image. The labeled fruit image is located on the d-th channel. Pixel value at; The pointwise convolutional layer integrates the convolutional features corresponding to each channel and outputs low-level semantic features, represented as: , in, This represents the intermediate feature value at position (i,j) and channel d after channel-wise convolution. represents the weights of the pointwise convolution kernel on channel d.

[0022] Through 1×1 convolution kernel Integrate channel information, reduce parameters, and extract low- to high-level features (such as edges, textures, and semantic information).

[0023] In this embodiment, the Xception model is used to combine channel-wise convolution and point-wise convolution. Channel-wise convolution processes each channel separately to retain fine-grained features, while point-wise convolution integrates channel information through 1×1 convolution, thereby reducing the number of parameters and improving feature extraction efficiency. By utilizing the principle of depthwise separable convolution, low-level semantic features such as fruit surface texture and edges are effectively extracted, laying the foundation for subsequent multi-scale fusion and defect recognition. Compared to traditional convolution, depthwise separable convolution significantly reduces computation, making the model more suitable for real-time detection scenarios while maintaining accuracy.

[0024] Preferably, the ASPP hollow spatial pyramid pooling module performs multi-scale fusion processing on the low-level semantic features to obtain multi-scale features of fruit surface defect categories, including: The ASPP dilated spatial pyramid pooling module includes 1×1 standard convolution and 3×3 dilated convolution. The 1×1 standard convolution is used to perform feature fusion at different scales on the low-level semantic features in branches with different dilation rates, resulting in fused features at different scales. The 3×3 dilated convolution is used to expand the receptive field of the fused features at different scales, resulting in multi-scale features for fruit surface blemish categories, represented as follows: , in, For different scales of fused features at location Pixel value at that location, d represents the weight of the dilated convolution kernel at position (m,n), and d is the dilation rate.

[0025] The dilation rates of 3×3 dilated convolutions are 6, 12, and 18. Dilated convolutions introduce dilation into the convolution kernel, which can expand the receptive field without increasing the number of parameters, making it very useful for capturing features over longer distances. This is then combined with a global pooling operation, and all the results are concatenated, then subjected to channel reduction using a standard 1×1 convolution before being fed into the decoder module.

[0026] In this embodiment, the ASPP module achieves feature fusion of branches with different dilation rates through 1×1 standard convolution, and expands the receptive field by combining 3×3 dilated convolution, which can simultaneously capture multi-scale features of small blemishes (such as mottled skin) and large-area defects (such as ulcers) on the surface of fruit. Dilated convolution expands the receptive field by introducing a dilation rate without increasing the kernel size or the number of parameters, thus improving global context awareness while controlling model complexity. Multi-scale features are adapted to defects of different sizes and types, improving the accuracy of classifying and locating complex blemishes on the surface of fruits.

[0027] Preferably, the concat feature fusion module fuses the dimensionality-reduced low-level semantic features and the upsampled high-level semantic features to obtain fused semantic features, including: Let the low-level semantic features after dimensionality reduction be... , dimension Let the high-level semantic features after upsampling be... , dimension The dimensionality-reduced low-level semantic features and the upsampled high-level semantic features are fused using a feature fusion formula to obtain fused semantic features, expressed as follows: , Among them, the feature dimension after fusion is .

[0028] In this embodiment, the concat feature fusion module fuses the dimensionality-reduced low-level semantic features (including detailed information) with the upsampled high-level semantic features (including global semantics) to achieve complementarity between detailed and semantic information and improve the completeness of the segmentation results. The fused feature dimension is the sum of the number of low-level and high-level feature channels, which avoids dimensionality explosion while retaining key information and balances model accuracy and computational efficiency. By using channel stitching to directly integrate high- and low-level features, the original information of the two features is not diluted, thus providing support for the subsequent reconstruction module to output high-precision segmentation results.

[0029] Preferably, the step of determining the grade of the segmentation results based on preset grading rules to obtain fruit grade grading results includes: The segmentation results determine the type, area, and number of defects on the fruit surface. These defects are then matched with the preset grade grading rules for first-grade, second-grade, third-grade, and unqualified products to obtain the corresponding fruit grade grading results.

[0030] Specifically, based on the segmentation results output by the fruit defect analysis model, the defect information is first quantitatively extracted, including: Defect type identification: Based on the category labels of different pixel regions in the segmentation results, determine the type of defects present on the fruit surface (such as spots, rot, scratches, deformities, etc.). Defect area calculation: Count the number of pixels occupied by each type of defect area in the image, combine the size ratio of the image to the actual fruit, convert it into the actual defect area (unit: square centimeters), and calculate the percentage of the total defect area to the fruit surface area; Defect count: Count the independent defect regions in the segmentation results to obtain the number of each defect type and the total number of defects.

[0031] Subsequently, the extracted defect information is matched against the preset quality grading rules in multiple dimensions. The rules are set as follows: Grade 1 product criteria: No defects, or only very minor blemishes that do not affect edibility (such as a single spot with an area ≤0.5cm² and a quantity ≤1). Second-grade product judgment rules: There are minor defects, the total area of ​​defects accounts for ≤5%, and there are no serious defects such as rot or severe deformity. The area of ​​a single defect is ≤1cm², and the number of the same defect is ≤2. The criteria for judging Grade 3 products are as follows: obvious defects exist, the total defect area accounts for 5%-15%, a small amount of minor rot or deformity is allowed, but the area of ​​malignant defects is ≤3cm², and the total number of defects is ≤5. Rules for determining non-conforming products: The total area of ​​defects is greater than 15%, or there are defects such as large-area rot (area > 3cm²), severe deformity, etc. that affect food safety or commodity value, or the number of defects is greater than 5.

[0032] Finally, based on the matching results, the corresponding grade of the fruit is determined, and the judgment result of Grade 1, Grade 2, Grade 3 or unqualified product is output.

[0033] In this embodiment, quantitative indicators such as defect type, area (including proportion), and quantity are extracted from the segmentation results to achieve multi-dimensional evaluation of fruit surface defects, avoid grading bias caused by a single indicator, replace manual subjective grading, and ensure the consistency and objectivity of the judgment results.

[0034] like Figure 2 As shown, preferably, the reflective imaging mechanism includes at least two sets of reflective lens units, each set of reflective lens units including a reflective lens 5, a drive motor 6, and a tripod 7. The reflective lens 5 is mounted on the rotating shaft at the top of the triangular bracket 7, the drive motor 6 is mounted on the triangular bracket 7, and the output end of the drive motor 6 is connected to the rotating shaft; The drive motor 6 starts according to the shooting signal and receives the control command sent by the back end. It obtains the rotation angle data from the control command and controls the rotation shaft to rotate according to the preset rotation angle, thereby driving the reflective lens 5 to rotate together.

[0035] Specifically, after the drive motor 6 is started according to the shooting signal, the angle control of the reflective lens 5 adopts a dual-mode collaborative mechanism of "preset angle basic control + dynamic command adjustment", which is implemented as follows: In the pre-set rotation angle control mode, the system calibrates the optimal reflection angle parameters for different fruit types based on their variety (e.g., apples, citrus, pears), average size, and typical morphological characteristics through prior experiments. These parameters, including the starting angle, target angle, and rotation rate of the reflective lens, are pre-stored in the parameter database of the control module. When trigger 2 detects the fruit and sends a shooting signal, drive motor 6 first calls the pre-set rotation angle data matching the current fruit type, controlling the rotation shaft to rotate the reflective lens 5 to the target angle at a set rate, ensuring that key areas on the sides and back of the fruit can accurately fall into the field of view of the shooting mechanism 3 through lens reflection. This mode is suitable for batch fruit testing scenarios with high standardization and small size differences, enabling millisecond-level rapid angle adjustment and ensuring testing efficiency for continuous industrial production.

[0036] In dynamic command control mode, drive motor 6 receives dynamic control commands from the fruit defect analysis system in real time. After the initial shooting, the shooting mechanism 3 feeds back the first captured reflective image to the backend system in real time. The system uses algorithms such as image clarity analysis and reflective area integrity detection to determine whether there is a deviation in the current reflective lens angle (such as incomplete reflection of some side areas or distortion of the reflected image). If a deviation exists, the backend system generates an angle correction command based on the image analysis results. The command includes specific angle adjustment values ​​(such as "rotate clockwise by 2°" or "rotate counterclockwise by 1.5°"). After receiving the command, drive motor 6 immediately controls the rotation axis to fine-tune the angle of reflective lens 5 according to the correction value, and triggers shooting mechanism 3 to take a second shot after the adjustment is completed, until the reflected image covers the complete surface of the fruit's side and back. This mode can effectively adapt to individual differences in fruit (such as uneven size or irregular shape) or slight positional shifts during transportation, ensuring the integrity and clarity of the global image set through dynamic feedback adjustment.

[0037] In practical applications, the two control modes can be seamlessly switched: the system defaults to prioritizing the pre-set rotation angle to quickly respond to the shooting signal, while the background monitors the reflection effect in real time; when the reflected image is detected to be unsatisfactory, it automatically switches to the background command control mode for dynamic correction, which not only ensures the high efficiency of continuous industrial production, but also improves the detection accuracy in complex scenarios through flexible adjustment, ultimately achieving the camera to image the entire surface of the fruit without blind spots.

[0038] It should be understood that the core function of the reflecting lens 5 is to detect the entire surface of the fruit. Through the principle of specular reflection, it detects the fruit features in the blind spots of the industrial camera, thereby solving the problem of incomplete fruit feature capture by a single industrial camera. The camera doesn't directly capture a virtual image of the fruit within the reflecting lens 5. Instead, it uses the reflection from the lens 5 to allow the industrial camera to capture surface features of the fruit that were previously in blind spots, ultimately obtaining a true image of the fruit's entire surface. The reflecting lens 5 serves only as an optical aid, eliminating blind spots by altering the light path. The object of the industrial camera's imaging is essentially still the fruit itself; the mirror reflection simply expands the imaging range, ensuring that a single camera can acquire complete surface information of the fruit. This provides comprehensive image data support for subsequent defect identification and quality grading.

[0039] In this embodiment, the reflective imaging mechanism receives control commands through the drive motor 6 and precisely adjusts the angle of the reflective lens 5 according to the rotation angle data to ensure that the side and back surfaces of fruits of different sizes and shapes can be accurately reflected into the field of view of the imaging mechanism.

[0040] Example 2: As Figure 4 As shown, this embodiment of the invention provides a fruit defect detection method based on reflection imaging, employing the fruit defect detection device described above, including a conveying mechanism, a trigger, a reflection imaging mechanism, an imaging mechanism, and a fruit defect analysis system; the method includes the following steps: The conveying mechanism transports the fruit to be analyzed to the shooting mechanism; When the trigger detects the fruit to be analyzed, it sends a shooting signal to the reflection imaging mechanism and the shooting mechanism. The reflective imaging mechanism adjusts the angle of the reflective lens to align with the fruit to be analyzed according to the shooting signal, so that the surfaces of the fruit to be analyzed located on the side and back of the shooting mechanism are reflected into the field of view of the shooting mechanism; The imaging mechanism takes multiple images of the front of the fruit to be analyzed, as well as the side and back of the fruit through the reflecting lens, to obtain a global fruit image set; The fruit defect analysis system trains the constructed fruit defect analysis model based on the global fruit image set, inputs test fruit images into the trained fruit defect analysis model, outputs segmentation results corresponding to different types of defects, and determines the grade of the segmentation results based on preset grade grading rules to obtain fruit grade grading results.

[0041] Preferably, training the constructed fruit defect analysis model based on the global fruit image set includes: Each fruit image in the global fruit image set is labeled, and the labeling includes the fruit category, the surface defect category, and the area where the defect is located; A fruit defect analysis model is constructed, which includes an encoder module and a decoder module. The encoder module includes a DCNN deep convolutional neural network, an ASPP (space pyramid pooling) module with dilated spatial pyramids, and a global pooling layer connected in sequence. Based on the labeled fruit images, the DCNN deep convolutional neural network is trained to extract features and output low-level semantic features of the fruit surface texture. The ASPP void spatial pyramid pooling module performs multi-scale fusion processing on the low-level semantic features to obtain multi-scale features of fruit surface blemish categories. The global pooling layer performs global pooling on the multi-scale features to obtain high-level semantic features, and then compresses the high-level semantic features to output compressed high-level semantic features. The decoder module includes a dimensionality reduction module, an upsampling module, a concat feature fusion module, and a restoration module. The input of the dimensionality reduction module is connected to the DCNN deep convolutional neural network. The input of the upsampling module is connected to the output of the global pooling layer. The outputs of both the dimensionality reduction module and the upsampling module are connected to the input of the concat feature fusion module. The output of the concat feature fusion module is connected to the restoration module. The dimensionality reduction module performs convolutional dimensionality reduction processing on the low-level semantic features output by the DCNN deep convolutional neural network, and inputs the dimensionality-reduced low-level semantic features into the concat feature fusion module; The upsampling module upsamples the compressed high-level semantic features output by the global pooling layer and inputs the upsampled high-level semantic features into the concat feature fusion module. The concat feature fusion module fuses the dimensionality-reduced low-level semantic features and the upsampled high-level semantic features to obtain fused semantic features. The restoration module performs bilinear interpolation upsampling on the fused semantic features to restore the size of the fused semantic features to the same size as the fruit images in the global fruit image set, thereby generating a segmentation result.

[0042] Preferably, the step of training the DCNN deep convolutional neural network based on the labeled fruit images to extract features and output low-level semantic features of the fruit surface texture includes: The DCNN deep convolutional neural network uses the Xception model as its backbone network. The Xception model includes channel-wise convolutional layers and point-wise convolutional layers. The channel-wise convolutional layer convolves each channel of the labeled fruit image individually to obtain the convolutional features corresponding to each channel, represented as follows: , in, The image shows the labeled fruit. Let be the weight of the convolution kernel corresponding to the d-th channel at position (m, n), where m and n represent the spatial offset of the convolution kernel, and d is the channel index, corresponding to the d-th channel of the labeled fruit image. The labeled fruit image is located on the d-th channel. Pixel value at; The pointwise convolutional layer integrates the convolutional features corresponding to each channel and outputs low-level semantic features, represented as: , in, This represents the intermediate feature value at position (i,j) and channel d after channel-wise convolution. represents the weights of the pointwise convolution kernel on channel d.

[0043] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0044] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0045] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A fruit defect detection device based on reflection imaging, characterized in that, This includes a transmission mechanism, a trigger, a reflective imaging mechanism, a shooting mechanism, and a fruit defect analysis system; The conveying mechanism is used to convey the fruit to be analyzed to the shooting mechanism; The trigger is used to send a shooting signal to the reflection imaging mechanism and the shooting mechanism when the fruit to be analyzed is detected; The reflection imaging mechanism is used to adjust the angle of the reflective lens according to the shooting signal to align with the fruit to be analyzed, so that the surfaces of the fruit to be analyzed located on the side and back of the shooting mechanism are reflected into the field of view of the shooting mechanism. The imaging mechanism is used to take multiple pictures of the front of the fruit to be analyzed, as well as the side and back of the fruit to be analyzed through the reflecting lens, to obtain a global fruit image set; The fruit defect analysis system is used to train the constructed fruit defect analysis model based on the global fruit image set; The test fruit image is input into the trained fruit defect analysis model, which outputs the segmentation results corresponding to different types of defects. The segmentation results are then graded based on the preset grading rules to obtain the fruit grade grading results.

2. The fruit defect detection device according to claim 1, characterized in that, In the fruit defect analysis system, the constructed fruit defect analysis model is trained based on the global fruit image set, including: Each fruit image in the global fruit image set is labeled, and the labeling includes the fruit category, the surface defect category, and the area where the defect is located; A fruit defect analysis model is constructed, which includes an encoder module and a decoder module. The encoder module includes a DCNN deep convolutional neural network, an ASPP (space pyramid pooling) module with dilated spatial pyramids, and a global pooling layer connected in sequence. Based on the labeled fruit images, the DCNN deep convolutional neural network is trained to extract features and output low-level semantic features of the fruit surface texture. The ASPP void spatial pyramid pooling module performs multi-scale fusion processing on the low-level semantic features to obtain multi-scale features of fruit surface blemish categories. The global pooling layer performs global pooling on the multi-scale features to obtain high-level semantic features, and then compresses the high-level semantic features to output compressed high-level semantic features. The decoder module includes a dimensionality reduction module, an upsampling module, a concat feature fusion module, and a restoration module. The input of the dimensionality reduction module is connected to the DCNN deep convolutional neural network. The input of the upsampling module is connected to the output of the global pooling layer. The outputs of both the dimensionality reduction module and the upsampling module are connected to the input of the concat feature fusion module. The output of the concat feature fusion module is connected to the restoration module. The dimensionality reduction module performs convolutional dimensionality reduction processing on the low-level semantic features output by the DCNN deep convolutional neural network, and inputs the dimensionality-reduced low-level semantic features into the concat feature fusion module; The upsampling module upsamples the compressed high-level semantic features output by the global pooling layer and inputs the upsampled high-level semantic features into the concat feature fusion module. The concat feature fusion module fuses the dimensionality-reduced low-level semantic features and the upsampled high-level semantic features to obtain fused semantic features. The restoration module performs bilinear interpolation upsampling on the fused semantic features to restore the size of the fused semantic features to the same size as the fruit images in the global fruit image set, thereby generating a segmentation result.

3. The fruit defect detection device according to claim 2, characterized in that, The DCNN deep convolutional neural network is trained based on the labeled fruit images to extract features and output low-level semantic features of the fruit surface texture, including: The DCNN deep convolutional neural network uses the Xception model as its backbone network. The Xception model includes channel-wise convolutional layers and point-wise convolutional layers. The channel-wise convolutional layer convolves each channel of the labeled fruit image individually to obtain the convolutional features corresponding to each channel, represented as follows: ， in, The image shows the labeled fruit. Let be the weight of the convolution kernel corresponding to the d-th channel at position (m, n), where m and n represent the spatial offset of the convolution kernel, and d is the channel index, corresponding to the d-th channel of the labeled fruit image. The labeled fruit image is located on the d-th channel. Pixel value at; The pointwise convolutional layer integrates the convolutional features corresponding to each channel and outputs low-level semantic features, represented as: ， in, This represents the intermediate feature value at position (i,j) and channel d after channel-wise convolution. represents the weights of the pointwise convolution kernel on channel d.

4. The fruit defect detection device according to claim 2, characterized in that, The ASPP hollow spatial pyramid pooling module performs multi-scale fusion processing on the low-level semantic features to obtain multi-scale features of fruit surface blemish categories, including: The ASPP dilated spatial pyramid pooling module includes 1×1 standard convolution and 3×3 dilated convolution. The 1×1 standard convolution is used to perform feature fusion at different scales on the low-level semantic features in branches with different dilation rates, resulting in fused features at different scales. The 3×3 dilated convolution is used to expand the receptive field of the fused features at different scales, resulting in multi-scale features for fruit surface blemish categories, represented as follows: ， in, For different scales of fused features at location Pixel value at that location, d represents the weight of the dilated convolution kernel at position (m,n), and d is the dilation rate.

5. The fruit defect detection device according to claim 2, characterized in that, The concat feature fusion module fuses the dimensionality-reduced low-level semantic features and the upsampled high-level semantic features to obtain fused semantic features, including: Let the low-level semantic features after dimensionality reduction be... , dimension Let the high-level semantic features after upsampling be... , dimension The dimensionality-reduced low-level semantic features and the upsampled high-level semantic features are fused using a feature fusion formula to obtain fused semantic features, expressed as follows: ， Among them, the feature dimension after fusion is .

6. The fruit defect detection device according to claim 2, characterized in that, The process of determining the grade of the segmentation results based on preset grading rules to obtain fruit grade grading results includes: The segmentation results determine the type, area, and number of defects on the fruit surface. These defects are then matched with the preset grade grading rules for first-grade, second-grade, third-grade, and unqualified products to obtain the corresponding fruit grade grading results.

7. The fruit defect detection device according to any one of claims 1 to 6, characterized in that, The reflective imaging mechanism includes at least two sets of reflective mirror units, each set of reflective mirror units including a reflective mirror, a drive motor, and a tripod support. The reflective lens is mounted on the rotating shaft at the top of the triangular bracket, and the output end of the drive motor is connected to the rotating shaft; The drive motor starts according to the shooting signal, and controls the rotation shaft to rotate according to the preset rotation angle, thereby driving the reflective lens to rotate together.

8. A method for detecting fruit defects based on reflection imaging, employing the fruit defect detection device according to any one of claims 1 to 7, comprising a conveying mechanism, a trigger, a reflection imaging mechanism, an imaging mechanism, and a fruit defect analysis system; characterized in that, Includes the following steps: The conveying mechanism transports the fruit to be analyzed to the shooting mechanism; When the trigger detects the fruit to be analyzed, it sends a shooting signal to the reflection imaging mechanism and the shooting mechanism. The reflective imaging mechanism adjusts the angle of the reflective lens to align with the fruit to be analyzed according to the shooting signal, so that the surfaces of the fruit to be analyzed located on the side and back of the shooting mechanism are reflected into the field of view of the shooting mechanism; The imaging mechanism takes multiple images of the front of the fruit to be analyzed, as well as the side and back of the fruit through the reflecting lens, to obtain a global fruit image set; The fruit defect analysis system trains the constructed fruit defect analysis model based on the global fruit image set, inputs test fruit images into the trained fruit defect analysis model, outputs segmentation results corresponding to different types of defects, and determines the grade of the segmentation results based on preset grade grading rules to obtain fruit grade grading results.

9. The fruit defect detection method according to claim 8, characterized in that, The fruit defect analysis model is trained based on the global fruit image set, including: Each fruit image in the global fruit image set is labeled, and the labeling includes the fruit category, the surface defect category, and the area where the defect is located; A fruit defect analysis model is constructed, which includes an encoder module and a decoder module. The encoder module includes a DCNN deep convolutional neural network, an ASPP (space pyramid pooling) module with dilated spatial pyramids, and a global pooling layer connected in sequence. Based on the labeled fruit images, the DCNN deep convolutional neural network is trained to extract features and output low-level semantic features of the fruit surface texture. The ASPP void spatial pyramid pooling module performs multi-scale fusion processing on the low-level semantic features to obtain multi-scale features of fruit surface blemish categories. The global pooling layer performs global pooling on the multi-scale features to obtain high-level semantic features, and then compresses the high-level semantic features to output compressed high-level semantic features. The decoder module includes a dimensionality reduction module, an upsampling module, a concat feature fusion module, and a restoration module. The input of the dimensionality reduction module is connected to the DCNN deep convolutional neural network. The input of the upsampling module is connected to the output of the global pooling layer. The outputs of both the dimensionality reduction module and the upsampling module are connected to the input of the concat feature fusion module. The output of the concat feature fusion module is connected to the restoration module. The dimensionality reduction module performs convolutional dimensionality reduction processing on the low-level semantic features output by the DCNN deep convolutional neural network, and inputs the dimensionality-reduced low-level semantic features into the concat feature fusion module; The upsampling module upsamples the compressed high-level semantic features output by the global pooling layer and inputs the upsampled high-level semantic features into the concat feature fusion module. The concat feature fusion module fuses the dimensionality-reduced low-level semantic features and the upsampled high-level semantic features to obtain fused semantic features. The restoration module performs bilinear interpolation upsampling on the fused semantic features to restore the size of the fused semantic features to the same size as the fruit images in the global fruit image set, thereby generating a segmentation result.

10. The fruit defect detection method according to claim 9, characterized in that, The DCNN deep convolutional neural network is trained based on the labeled fruit images to extract features and output low-level semantic features of the fruit surface texture, including: The DCNN deep convolutional neural network uses the Xception model as its backbone network. The Xception model includes channel-wise convolutional layers and point-wise convolutional layers. The channel-wise convolutional layer convolves each channel of the labeled fruit image individually to obtain the convolutional features corresponding to each channel, represented as follows: ， in, The image shows the labeled fruit. Let be the weight of the convolution kernel corresponding to the d-th channel at position (m, n), where m and n represent the spatial offset of the convolution kernel, and d is the channel index, corresponding to the d-th channel of the labeled fruit image. The labeled fruit image is located on the d-th channel. Pixel value at; The pointwise convolutional layer integrates the convolutional features corresponding to each channel and outputs low-level semantic features, represented as: ， in, This represents the intermediate feature value at position (i,j) and channel d after channel-wise convolution. represents the weights of the pointwise convolution kernel on channel d.

Images