A method and system for semantic parsing of images of string fruits and training of difficult samples

Abstract

The application discloses a kind of string fruit image semantic analysis and difficult sample training method and system, comprising: obtaining training sample set;According to RGB image and depth map, generate RGB-D four-channel input tensor, and generate effective depth auxiliary information according to depth map;Determine whether sample data is difficult sample or whether there is difficult pixel area;In each iteration training, after the data enhancement of the sample data currently sampled, input semantic analysis network, optimize features by context enhancement and detail structure enhancement, output semantic structure analysis result and effective depth auxiliary prediction result according to the enhanced features;According to semantic annotation, effective depth auxiliary information, prediction result and determination result, the network is iteratively trained.The application improves the segmentation accuracy of target fruit and connection structure in string fruit image and model generalization ability by multi-modal fusion and multi-task learning.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method and system for semantic parsing and training of difficult samples for images of clustered fruits. Background Technology

[0002] Mechanized harvesting of clustered fruits (such as grapes, tomatoes, lychees, and longans) is an important research direction in the field of agricultural automation. Harvesting robots need to accurately identify target fruits, distinguish non-target fruits, and locate connecting structures such as fruit stalks and calyxes in the field environment before they can complete subsequent cutting and grasping operations. Therefore, accurate image semantic analysis of fruits and connecting structures is a core technical step in achieving automated harvesting.

[0003] In existing technologies, semantic segmentation methods based on RGB images have been applied to fruit recognition. These methods use convolutional neural networks to classify image pixels, which can distinguish fruits from the background to a certain extent. However, RGB images only provide color and texture information. When fruits and leaves are similar in color, fruits overlap, or lighting conditions are complex, the segmentation stability is insufficient. Especially for small connected structures with small pixel proportions and insignificant visual differences, such as fruit stalks and calyxes, the segmentation accuracy often fails to meet the requirements of harvesting operations.

[0004] Some existing techniques attempt to incorporate depth maps to assist segmentation. Depth maps provide spatial distance information, which helps distinguish visually close objects but with different spatial locations. However, these techniques typically only perform simple channel concatenation or feature addition between RGB images and depth maps, without fully considering contextual enhancement and detail structure enhancement of multi-scale features, resulting in limited segmentation accuracy for fruit boundaries and connection structures.

[0005] Regarding training strategies, existing methods employ uniform loss weights and sampling strategies for different samples and pixel regions. In real-world harvesting scenarios, small, distant fruits and intricate connections have very small pixel areas in images. Depth sensors often experience measurement failures on transparent surfaces, in occluded areas, or at long distances. Different acquisition devices and lighting conditions also lead to domain shift issues. Existing technologies lack differentiated training mechanisms for these challenging samples and pixel regions, limiting the model's generalization ability in complex scenarios.

[0006] Furthermore, existing semantic parsing networks typically only output pixel-level semantic classification results, without predicting or evaluating the reliability of depth information. Downstream 3D point cloud generation and grasping pose estimation modules rely on depth information for spatial localization; unreliable depth information directly impacts the success rate of subsequent operations. Current technologies lack online evaluation mechanisms for depth information quality, failing to provide downstream tasks with auxiliary quality cues beyond semantic categories.

[0007] In summary, the existing technology has the following problems: the fusion of RGB and depth maps is not fully utilized, there is a lack of contextual enhancement and detail structure enhancement of multi-scale features, and the segmentation accuracy is insufficient; there is a lack of differentiated training strategies for small targets, fine connection structures, low effective depth samples and cross-domain samples, which limits the generalization ability; the network output information is monotonous and does not include depth information reliability assessment, so it cannot provide richer perceptual information for downstream tasks. Summary of the Invention

[0008] To address at least one of the above technical problems, embodiments of the present invention provide a method and system for semantic parsing and training of difficult samples in images of clustered fruits.

[0009] On one hand, embodiments of the present invention provide a method for semantic parsing and hard sample training of images of clustered fruits, including:

[0010] Step S1: Obtain a training sample set, which includes sample data of multiple clusters of fruits. Each sample data includes at least an RGB image, a depth map aligned with the RGB image, a semantic annotation for identifying the category to which the pixel belongs, and a sample source identifier.

[0011] Step S2: For each sample data in the training sample set, generate an RGB-D four-channel input tensor based on the RGB image and the depth map, and generate effective depth auxiliary information based on the depth map;

[0012] Step S3: For each sample data in the training sample set, determine whether the sample data is a difficult sample or whether there is a difficult pixel region based on at least one of the corresponding semantic annotation, effective depth auxiliary information, and sample source identifier, and obtain the determination result.

[0013] Step S4: During each iteration of training, for the sample data sampled in the current iteration, its RGB-D four-channel input tensor is augmented and then input into the semantic parsing network. In the semantic parsing network, the features of the input tensor are enhanced and optimized through context enhancement and detail structure enhancement. Based on the enhanced and optimized features, the semantic parsing network performs category parsing and effective depth prediction on each pixel corresponding to the input tensor, and outputs the corresponding semantic structure parsing results and effective depth auxiliary prediction results. Based on the semantic annotations corresponding to the sample data, the effective depth auxiliary information, the semantic structure parsing results, the effective depth auxiliary prediction results, and the corresponding judgment results, the semantic parsing network is iteratively trained.

[0014] Preferably, step S2 includes performing the following operations for each sample data in the training sample set:

[0015] The RGB image is normalized to obtain normalized RGB three-channel data;

[0016] The depth map is transformed or normalized according to the depth scale to obtain normalized depth channel data.

[0017] Normalized RGB three-channel data and normalized depth channel data are concatenated along the channel dimension to generate an RGB-D four-channel input tensor;

[0018] Based on invalid values, outliers, void regions, or depth boundaries in the depth map, effective depth region indication information or depth confidence information is generated as the effective depth auxiliary information.

[0019] Preferably, in step S4, the features of the input tensor are enhanced and optimized in the semantic parsing network through context enhancement and detail structure enhancement, including:

[0020] The RGB-D four-channel input tensor is fed into the encoder, and multi-scale features are extracted through multi-layer convolution.

[0021] The multi-scale features are processed using dilated convolutions with different dilation rates to achieve context enhancement.

[0022] The context-enhanced features are concatenated with the shallow features output from each layer of the encoder via skip connections, and then upsampled by the decoder to obtain a decoded feature map. The decoded feature map is then concatenated with the RGB-D four-channel input tensor and convolved to complete the detail structure enhancement.

[0023] Preferably, the semantic structure parsing results and effective depth-assisted prediction results output in step S4 include:

[0024] The enhanced features are convolutionally mapped to obtain the probability map and category label map of each pixel belonging to each semantic category, which serve as the semantic structure parsing result.

[0025] The decoded feature map is convolved to obtain a binary classification probability map in which each pixel belongs to the depth reliable and depth unreliable categories; the maximum value of the binary classification probability map along the category dimension is taken to obtain an effective depth-assisted prediction label map; the binary classification probability map and the effective depth-assisted prediction label map are used together as the effective depth-assisted prediction result.

[0026] Preferably, step S3 includes:

[0027] If the pixel area of ​​the target fruit region determined by the semantic annotation corresponding to the sample data is less than the preset first area threshold, then the sample data is judged as a small target fruit difficult sample, and the pixels in the target fruit region are judged as difficult pixel regions.

[0028] If the pixel area of ​​the connection structure region determined by the semantic annotation corresponding to the sample data is less than the preset second area threshold, then the sample data is judged as a difficult sample with a small area connection structure, and the pixels in the connection structure region are judged as difficult pixel regions.

[0029] If the effective depth ratio determined based on the effective depth auxiliary information corresponding to the sample data is lower than the preset first ratio threshold, then the sample data is judged as a low effective depth difficult sample; the effective depth ratio refers to the percentage of the number of effective pixels in the depth map to the total number of pixels in the depth map;

[0030] If the sample data is determined to have been collected from a camera device or collection environment different from the main domain of the training sample set based on the sample source identifier, then the sample data is identified as a difficult sample in the new camera domain. The main domain of the training sample set refers to the collection condition distribution that most samples in the training set share, including camera device type, lens parameters, and environmental conditions during collection.

[0031] Preferably, step S4 includes:

[0032] During training iteration sampling, if the sample data used in the current iteration of training is a difficult sample, the sampling probability of the sample data used in the current iteration of training is higher than the sampling probability of the non-difficult sample.

[0033] Based on the sampling results, data augmentation processing is performed on the RGB images and depth maps in the selected sample data used in the current iteration of training;

[0034] Generate an enhanced RGB-D four-channel input tensor based on the data-augmented RGB image and depth map;

[0035] The enhanced RGB-D four-channel input tensor is input into the semantic parsing network. After forward inference, the network outputs the semantic structure parsing result and the effective depth-assisted prediction result. The binary classification probability map in the output effective depth-assisted prediction result is compared with the effective depth-assisted information generated from the current sample data in step S2 to obtain the effective depth-assisted loss. The output semantic structure parsing result is used as the predicted value, and the semantic label corresponding to the current sample data is used as the ground truth value. The predicted value is compared with the ground truth value to obtain the category classification loss. When determining the category classification loss, if the current sample data includes difficult pixel regions, the loss weight of pixels in difficult pixel regions is set higher than that of pixels in non-difficult pixel regions.

[0036] The total loss is determined based on the category classification loss and the effective depth-assisted loss; the semantic parsing network is then jointly trained using the total loss to update the network parameters.

[0037] Preferably, step S4 further includes:

[0038] Based on the set of pixels belonging to the target fruit category and the set of pixels belonging to the connection structure category in the semantic annotation of the current sample data, the boundaries of the target fruit region and the boundaries of the connection structure region are extracted as the true boundaries.

[0039] Based on the set of pixels belonging to the target fruit category and the set of pixels belonging to the connection structure category in the semantic structure parsing results output in step S4, the boundary of the target fruit prediction region and the boundary of the connection structure prediction region are extracted respectively as the prediction boundary.

[0040] Determine the boundary loss based on the actual boundary and the predicted boundary;

[0041] The step of determining the total loss based on the category classification loss and the effective depth auxiliary loss includes: determining the total loss based on the category classification loss, the effective depth auxiliary loss, and the boundary loss.

[0042] Preferably, in step S4, the sampling probability of the sample data is determined in the following way:

[0043] The difficulty sampling weight factor of each sample is determined based on the difficulty level value of each sample in each difficulty dimension in the training sample set;

[0044] Specifically, the sample is determined according to the following formula. Difficult sampling weighting factor:

[0045]

[0046] in, Indicates the number of the current sample; For the sample Difficult sampling weighting factor; Indicates sample The difficulty of achieving the target fruit size, based on the sample The ratio between the pixel area of ​​the target fruit region and the first area threshold is determined; Indicates sample The difficulty of the connection structure size, based on the sample The ratio between the pixel area of ​​the connected structure region and the second area threshold is determined; Indicates sample The depth quality difficulty is set to a value of 1. Subtract samples The effective depth ratio is determined based on the effective depth auxiliary information. Indicates sample The difficulty of domain difference, based on the sample The source of the samples in the middle was identified; These are the first preset weight coefficient, the second preset weight coefficient, the third preset weight coefficient, and the fourth preset weight coefficient, respectively, satisfying... And all weights are greater than 0; This is a preset temperature regulation hyperparameter, and its value is a positive number;

[0047] The hard sampling weight factor of each sample in the training sample set is normalized to obtain the actual sampling probability of each sample; the training samples are then sampled according to the actual sampling probability of each sample.

[0048] Preferably, in step S4, the loss weights of the pixels in the difficult pixel region are determined in the following way:

[0049] When there are difficult pixel regions in the current sample, the pixel difficulty weight coefficient of each pixel is determined based on the local neighborhood contrast of each pixel in the current sample and the determination result of whether it belongs to a difficult pixel region.

[0050] Specifically, the pixel difficulty weight coefficient for each pixel is determined according to the following formula:

[0051]

[0052] in, This represents the number of any pixel in the current sample; For pixels Difficult pixel indicator value, when pixel The value is 1 when the pixel is identified as belonging to a difficult pixel region, and 0 otherwise. For pixels Local neighborhood contrast; Local neighborhood contrast of all hard pixels in the current sample The mean; The preset base weight enhancement coefficient, The preset contrast modulation coefficient. ; This is the steepness adjustment parameter for the sigmoid function. ; For pixels The pixel difficulty weighting coefficient;

[0053] When determining the classification loss of the current sample, the pixel-level classification loss of each pixel is determined. The pixel difficulty weight coefficient of each pixel is multiplied by the corresponding pixel-level classification loss to obtain the weighted loss value of each pixel. Then, the weighted loss values ​​of all pixels are averaged to obtain the classification loss of the current sample.

[0054] On the other hand, embodiments of the present invention also provide a semantic parsing and hard sample training system for images of clustered fruits, comprising:

[0055] The sample acquisition module is used to acquire a training sample set, which includes sample data of multiple clusters of fruits. Each sample data includes at least an RGB image, a depth map aligned with the RGB image, a semantic annotation for identifying the category to which the pixel belongs, and a sample source identifier.

[0056] The data processing module is used to generate an RGB-D four-channel input tensor for each sample data in the training sample set based on the RGB image and the depth map, and to generate effective depth auxiliary information based on the depth map;

[0057] The judgment processing module is used to determine, for each sample data in the training sample set, whether the sample data is a difficult sample or whether there is a difficult pixel region, based on at least one of the semantic annotation, effective depth auxiliary information, and sample source identifier corresponding to the sample data, and to obtain a judgment result.

[0058] The training module, during each training iteration, augments the RGB-D four-channel input tensor of the sample data collected in the current iteration before inputting it into the semantic parsing network. In the semantic parsing network, the features of the input tensor are enhanced and optimized through contextual augmentation and detail augmentation. Based on the enhanced and optimized features, the semantic parsing network performs category parsing and effective depth prediction on each pixel corresponding to the input tensor, outputting the corresponding semantic structure parsing results and effective depth auxiliary prediction results. The semantic parsing network is iteratively trained based on the semantic annotations corresponding to the sample data, the effective depth auxiliary information, the semantic structure parsing results, the effective depth auxiliary prediction results, and the judgment results.

[0059] The beneficial effects of the above-mentioned technical solutions provided in the embodiments of the present invention include at least the following:

[0060] The technical solution provided in this invention improves the segmentation stability of targets with similar colors by fusing RGB images and depth maps into an RGB-D four-channel input tensor. This allows the semantic parsing network to simultaneously utilize color and spatial distance information for feature extraction. By incorporating context enhancement and detail structure enhancement modules into the network, features are enhanced and optimized from two dimensions: global semantic association and local edge details, improving the network's segmentation accuracy for the boundaries of target fruit regions and fine connection structures. A multi-dimensional hard sample judgment mechanism comprehensively identifies hard samples from the perspectives of target fruit size, connection structure size, depth map quality, and sampling domain differences. Based on this, iterative training is performed using differentiated sampling probabilities and pixel-level loss weight strategies, solving the generalization problem in small targets, fine structures, and cross-domain scenarios. Furthermore, by employing a multi-task learning framework that includes semantic classification branches and effective depth assistance branches, the network outputs effective depth assistance prediction results while simultaneously outputting semantic structure parsing results. This provides a reliable depth information assessment basis for downstream 3D point cloud generation and grasping pose estimation modules, which is beneficial for improving the overall success rate of harvesting operations.

[0061] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0062] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0063] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0064] Figure 1 This is a flowchart of the semantic parsing and hard sample training method for fruit cluster images provided in an embodiment of the present invention. Detailed Implementation

[0065] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0066] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," "outer," "far," "near," "front," and "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0067] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0068] Example 1

[0069] This invention provides a method for semantic parsing and hard sample training of images of clustered fruits, such as... Figure 1 As shown, it includes:

[0070] Step S1: Obtain a training sample set, which includes sample data of multiple clusters of fruits. Each sample data includes at least an RGB image, a depth map aligned with the RGB image, a semantic annotation for identifying the category to which the pixel belongs, and a sample source identifier.

[0071] Step S2: For each sample data in the training sample set, generate an RGB-D four-channel input tensor based on the RGB image and the depth map, and generate effective depth auxiliary information based on the depth map;

[0072] The effective depth auxiliary information is auxiliary information that identifies whether the depth value of each pixel in the depth map is reliable;

[0073] Step S3: For each sample data in the training sample set, determine whether the sample data is a difficult sample or whether there is a difficult pixel region based on at least one of the semantic annotation, effective depth auxiliary information, and sample source identifier corresponding to the sample data, and obtain the determination result.

[0074] Step S4: During each iteration of training, for the sample data sampled in the current iteration, the RGB-D four-channel input tensor is augmented and then input into the semantic parsing network. In the semantic parsing network, the features of the input tensor are enhanced and optimized through context enhancement and detail structure enhancement. Based on the enhanced and optimized features, the semantic parsing network performs category parsing and effective depth prediction on each pixel corresponding to the input tensor, and outputs the corresponding semantic structure parsing results and effective depth auxiliary prediction results. Based on the semantic annotations corresponding to the sample data, the effective depth auxiliary information, the semantic structure parsing results, the effective depth auxiliary prediction results, and the judgment results, the semantic parsing network is iteratively trained.

[0075] The method provided in this embodiment is applicable to image processing of clustered fruits. Clustered fruits refer to a type of fruit in which multiple individual fruits grow in a cluster or spike on the same pedicel or inflorescence axis, including but not limited to grapes, tomatoes (including cherry tomatoes), lychees, longans, rambutans, and other fruits that grow in clusters.

[0076] The method first obtains a training sample set in step S1, which contains sample data of multiple clusters of fruits. In each sample, the RGB image provides the fruit's color and texture information, the depth map provides the fruit's spatial distance information, semantic annotations provide pixel-level ground truth for subsequent supervised learning, and the sample source identifier records the image acquisition device information and / or acquisition environment information for each sample. The RGB image and depth map must maintain pixel-level alignment; that is, each pixel position in the RGB image and the corresponding pixel position in the depth map describe the same spatial point.

[0077] In step S2, for each sample in the training sample set, the three-channel color data of the RGB image and the single-channel depth data of the depth map are concatenated along the channel dimension to form a four-channel input tensor. The depth map may contain invalid values ​​(such as areas where the sensor did not detect distance), outliers, or holes. Therefore, effective depth auxiliary information needs to be generated based on the actual situation of the depth map to identify which pixel locations have reliable depth values. This step is performed once for each sample before training begins, and the generated results are used in subsequent training phases.

[0078] In step S3, for each sample in the training sample set, the difficulty level of the sample is determined from multiple dimensions based on the sample's semantic annotation, effective depth auxiliary information, and sample source identifier. Specifically, semantic annotation can determine whether the area of ​​the target fruit region or the connecting structure region is too small; effective depth auxiliary information can determine whether the effective depth ratio of the entire depth map is too low; and sample source identifier can determine whether the sample comes from a collection condition different from the main domain of the training sample set. This step is performed once for each sample before training begins, and the determination results are used to adjust the sampling probability and loss weights in subsequent training phases.

[0079] In step S4, during each training iteration, the sampled data is first augmented, and then the augmented RGB-D four-channel input tensor is fed into the semantic parsing network. This network first extracts multi-scale features through an encoder, and then performs contextual augmentation and detail enhancement. Contextual augmentation expands the network's receptive field, enabling it to capture semantic information over a wider range; detail enhancement restores and preserves details such as edges and textures in the image, improving the accuracy of segmentation boundaries. Based on the augmented and optimized features, the semantic parsing network performs two tasks simultaneously: first, it classifies each pixel into a semantic category and outputs the semantic structure parsing result; second, it assesses the credibility of the depth value of each pixel and outputs the effective depth-assisted prediction result. Then, the output prediction result is compared with the semantic annotations and effective depth-assisted information of the sample to calculate various losses and update the network parameters. This process is repeated in each training iteration until the network converges.

[0080] The beneficial effects of this embodiment are as follows: by fusing RGB images and depth maps into a four-channel input, the network can simultaneously utilize color and spatial information for semantic parsing; through context enhancement and detail structure enhancement, the network's segmentation accuracy for target fruits, other fruits, and connection structures in fruit cluster images is improved; by setting a hard sample judgment mechanism and a differentiated training strategy, the network can specifically learn the features of hard samples, improving the model's generalization ability in complex scenes; and by simultaneously outputting semantic structure parsing results and effective depth-assisted prediction results through a multi-task learning framework, richer perceptual information is provided for downstream tasks.

[0081] Example 2

[0082] In this embodiment, step S1 can be implemented as follows:

[0083] Semantic annotation is performed on the RGB images in the sample data, and each pixel in the region related to picking in the RGB image is labeled as one of the following semantic categories: background category, target fruit category, other fruit category, and connection structure category; the connection structure is at least one of the following structures: calyx, pedicel, and main stem.

[0084] Specifically, during the sample data acquisition process in step S1, pixel-level semantic annotation of the RGB image is required. During annotation, each pixel in the region of the RGB image related to picking is divided into four categories: background category, target fruit category, other fruit category, and connection structure category.

[0085] The harvest-related region refers to the area in the image containing clusters of fruit to be picked, other fruit clusters, or connecting structures. This region can be pre-determined by manually annotated bounding boxes or automatically identified and cropped using object detection algorithms. The harvest-related region is determined during the training sample preprocessing stage and is only used to limit the semantic annotation range during training; this region is completed during the training dataset creation stage and is not dynamically generated during network inference. Image regions other than the harvest-related region (such as pure background regions far from the fruit) do not participate in semantic annotation; these regions are ignored during training and the loss is not calculated.

[0086] Background category refers to all objects or areas in the image that are not part of the picking target within the area related to picking, including leaves, branches (except for connecting structures), sky, ground, and other non-fruit, non-connecting structures, etc.

[0087] The target fruit category refers to the individual cluster of fruits currently to be picked. In a cluster fruit picking scenario, the same image may contain multiple clusters of fruits, of which only one cluster may be the target object of the current picking operation, while the rest of the fruits are distractors.

[0088] Other fruit categories refer to individual clusters of fruits in the image that are not the target fruit. Although these fruits may look similar to the target fruit, they need to be distinguished from the target fruit in the picking decision.

[0089] The category of connecting structures refers to plant structures that connect the fruit to the main body of the plant. Connecting structures include at least one of the following: calyx, pedicel, and main stem. The calyx is the persistent calyx portion at the top of the fruit that connects to the pedicel; the pedicel is a short stalk connecting the fruit to the main stem; the main stem is the main trunk and branch of the plant, from which the pedicel branches off. These structures are key parts that harvesting robots need to identify when performing cutting or grasping operations.

[0090] The beneficial effects of this embodiment are as follows: Semantic categories are divided into four types: background, target fruit, other fruits, and connection structures. The distinction between target fruit and other fruits takes into account the practical needs of multiple fruits in the same frame during fruit picking scenarios. The setting of the connection structure category provides semantic support for subsequent cutting point localization and grasping planning. By limiting the annotation range to areas related to picking, the interference of invalid background pixels on the training process is reduced, while also lowering the annotation workload.

[0091] Example 3

[0092] In this embodiment, step S2 includes performing the following operations for each sample data in the training sample set:

[0093] The RGB image is normalized to obtain normalized RGB three-channel data;

[0094] The depth map is transformed or normalized according to the depth scale to obtain normalized depth channel data.

[0095] Normalized RGB three-channel data and normalized depth channel data are concatenated along the channel dimension to generate an RGB-D four-channel input tensor;

[0096] Based on invalid values, outliers, void regions, or depth boundaries in the depth map, effective depth region indication information or depth confidence information is generated as the effective depth auxiliary information.

[0097] Specifically, in step S2, the RGB image is normalized by mapping the pixel values ​​of each channel from the original integer range (usually 0-255) to the floating-point range (usually 0-1 or between -1 and 1) to eliminate the problem of inconsistent numerical scales between different images due to differences in lighting conditions.

[0098] When processing depth maps, conversion or normalization can be performed based on the model of the depth camera and the physical meaning of the depth values. Depth maps typically store depth values ​​in millimeters or meters. Different cameras may have different depth ranges, so conversion according to the depth scale is necessary to map the depth values ​​to a uniform numerical range.

[0099] After normalization, the three-channel RGB data and the single-channel depth data are concatenated along the channel dimension to form a four-channel tensor, which is then used as the input to the semantic parsing network.

[0100] The generation process of effective depth auxiliary information is as follows: When the depth camera acquires a depth map, due to factors such as object surface material (e.g., transparent, highly reflective surfaces), occlusion, and sensor noise, some pixel locations may not obtain effective depth values. These locations are usually marked as invalid values ​​(e.g., 0 values ​​or specific large values) in the depth map. In addition, the depth map may also contain outliers caused by excessively large depth value jumps, as well as hollow areas caused by sensor measurement failures. Depth boundaries refer to locations in the depth map where the depth value difference between adjacent pixels exceeds a preset threshold. These locations typically correspond to object edges or occlusion boundaries. Pixels near depth boundaries may have depth values, but their reliability is low. Based on the location and range of these invalid values, outliers, hollow areas, and depth boundaries, effective depth region indication information is generated. This information can be a binary mask of the same size as the depth map, where effective depth locations are marked as 1, invalid depth locations are marked as 0, and pixels near depth boundaries can be marked with lower confidence values. Alternatively, depth confidence information can be generated based on the depth value confidence of each pixel, such as calculating the confidence score of each pixel based on the consistency between the depth value and its neighboring depth values ​​and whether it is close to the depth boundary.

[0101] The beneficial effects of this embodiment are as follows: by normalizing the RGB image and the depth map respectively, the problem of inconsistent data scale caused by differences in acquisition conditions between different samples is eliminated; by concatenating the RGB three channels with the depth single channel into a four-channel tensor, the network can process color information and spatial information simultaneously in a unified input space; by incorporating the depth boundary into the basis for generating effective depth auxiliary information, the network can perceive edge regions with drastic changes in depth values ​​and perform weighted processing on the depth reliability of these regions.

[0102] Example 4

[0103] In this embodiment, step S4, where the features of the input tensor are enhanced and optimized in the semantic parsing network through context enhancement and detail structure enhancement, can be implemented as follows:

[0104] The RGB-D four-channel input tensor is fed into the encoder, which extracts multi-scale features through multi-layer convolution, including visual features and depth features.

[0105] Dilated convolutions with different dilation rates are used to process multi-scale features, capture multi-scale contextual information, and complete context enhancement.

[0106] The context-enhanced features are concatenated with the shallow features output from each layer of the encoder through skip connections to preserve shallow details. The features are then upsampled by the decoder to restore spatial resolution, resulting in a decoded feature map. The decoded feature map is then concatenated with the input RGB-D four-channel input tensor, and the concatenated features are transformed by a convolutional layer to restore spatial details, thus completing the detail structure enhancement.

[0107] Specifically, the semantic parsing network can adopt an encoder-decoder architecture.

[0108] The encoder consists of multiple stacked convolutional layers. As the network depth increases, the spatial resolution of the feature maps decreases layer by layer, while the number of channels increases. The features output by each layer of the encoder have different scales: shallow features have high resolution and contain more spatial details (such as edges and textures); deep features have low resolution and contain more semantic information (such as object category and overall structure). The multi-scale features extracted by the encoder include visual features extracted from the RGB channels and depth features extracted from the depth channels.

[0109] The specific implementation of context enhancement involves using dilated convolutions with different dilation rates to process multi-scale features after one or more intermediate layers of the encoder. The intermediate layers of the encoder refer to the outputs of convolutional layers between the shallowest and deepest layers. These feature maps already possess a certain degree of semantic abstraction while retaining sufficient spatial resolution. Performing dilated convolutions on these feature layers allows for the capture of contextual information within a larger receptive field without losing the details needed for precise localization due to low resolution. Dilated convolutions expand the receptive field of the convolutional kernel without increasing the number of parameters or computational cost by inserting holes (i.e., zero-value intervals) between the sampling points of the convolutional kernel. Multiple convolutional operations are performed on the feature maps in parallel using different dilation rates (e.g., dilation rates of 1, 2, 4, and 8). Then, the outputs of each branch are concatenated along the channel dimension or added element-wise, thereby capturing contextual information across different scales. This process enables the network to simultaneously perceive both local fine structures and global contextual relationships.

[0110] The detailed structure enhancement is implemented as follows: the context-enhanced features are concatenated with the shallow features output from each layer of the encoder via skip connections. Skip connections directly transmit the high-resolution detail information preserved in the shallow layers of the encoder to the decoder, compensating for the spatial resolution loss in deep features. The concatenated features are then used to gradually restore spatial resolution through upsampling layers in the decoder, resulting in a decoded feature map. Based on this, the decoded feature map is concatenated with the original RGB-D four-channel input tensor, and then one or more convolutional layers are used to perform feature transformation on the concatenated features, enabling the network to directly learn how to recover lost spatial detail information from the original input.

[0111] The beneficial effects of this embodiment are as follows: multi-scale feature extraction of RGB-D four-channel input is achieved through the encoder-decoder structure; context enhancement is performed through dilated convolutions with different dilation rates, enabling the network to capture multi-scale context information and improving the recognition ability of targets of different sizes; detail enhancement is performed through skip connections and feature concatenation with the original input tensor, enabling the network to retain and restore the spatial detail information of the image and improve the accuracy of segmentation boundaries.

[0112] Example 5

[0113] In this embodiment, step S4, which outputs the corresponding semantic structure parsing result and the effective depth-assisted prediction result, can be specifically implemented as follows:

[0114] Convolutional mapping is performed on the features after context enhancement and detail structure enhancement, respectively, and the probability map and category label map of each pixel belonging to each semantic category corresponding to the RGB-D four-channel input tensor are output as the semantic structure parsing result;

[0115] The decoded feature map is convolved to convert the number of channels in the decoded feature map into the number of binary classification channels, resulting in a binary classification probability map of each pixel corresponding to the RGB-D four-channel input tensor, indicating whether the pixel belongs to the depth reliable category or the depth unreliable category. The category (argmax) corresponding to the maximum probability value along the category dimension of the binary classification probability map is taken to obtain the effective depth-assisted prediction label map of each pixel. The label value of each pixel in the effective depth-assisted prediction label map is used to indicate whether the depth of the pixel is reliable. The binary classification probability map and the effective depth-assisted prediction label map are used together as the effective depth-assisted prediction result.

[0116] Specifically, a semantic parsing network can contain two output branches: a semantic classification branch and an effective deep auxiliary branch.

[0117] The semantic classification branch performs convolutional mapping on the feature maps after context enhancement and detail structure enhancement. Specifically, a convolutional layer converts the number of channels in the feature map into the number of semantic categories (four categories in this example: background, target fruit, other fruits, and connection structures). Then, a softmax function is applied to each pixel location to convert the pixel's score in each category into a probability value, resulting in a probability map. Each pixel location in the probability map contains a probability vector of length equal to the number of categories, with the sum of its components being 1. Based on this, the category with the highest probability for each pixel is taken as the predicted category for that pixel, resulting in a category label map. The probability map and the category label map together constitute the semantic structure parsing result.

[0118] The effective depth-assisted branch performs convolutional mapping on the decoded feature map output by the decoder. Specifically, a convolutional layer converts the number of channels in the decoded feature map to 2, resulting in a binary probability map for each pixel, classifying it as either a reliable depth category or a reliable depth category. A reliable depth category indicates that the depth value at that pixel location is reliable and can be used for subsequent 3D reconstruction or pose estimation; a reliable depth category indicates that the depth value at that pixel location is unreliable and should be discarded or assigned a lower weight. The binary probability map is then processed along its category dimension, taking the category corresponding to the highest probability value (i.e., the argmax operation), and each pixel is assigned to the category with the higher probability, resulting in the effective depth-assisted prediction label map. The label value of each pixel (e.g., 0 for unreliable depth, 1 for reliable depth) indicates whether the pixel's depth is reliable. The binary probability map and the effective depth-assisted prediction label map together constitute the effective depth-assisted prediction result, where the probability map is used for loss calculation during the training phase, and the label map is used for pixel selection during the inference phase.

[0119] The beneficial effects of this embodiment are as follows: by performing semantic classification and effective depth prediction tasks through two output branches respectively, multi-task learning is achieved. The probability map and category label map output by the semantic classification branch provide semantic category information for each pixel, providing structured semantic understanding results for downstream tasks. The binary classification probability map and prediction label map output by the effective depth auxiliary branch can identify unreliable pixel regions in the depth map, providing a basis for depth quality screening for the downstream 3D point cloud generation module. Incorporating the probability map into the prediction results provides a reasonable supervision target for loss calculation during the training phase.

[0120] Example 6

[0121] In this embodiment, step S3 can be implemented as follows:

[0122] If the pixel area of ​​the target fruit region determined by the semantic annotation corresponding to the sample data is less than the preset first area threshold, then the sample data is judged as a small target fruit difficult sample, and the pixels in the target fruit region are judged as difficult pixel regions.

[0123] If the pixel area of ​​the connection structure region determined by the semantic annotation corresponding to the sample data is less than the preset second area threshold, then the sample data is judged as a difficult sample with a small area connection structure, and the pixels in the connection structure region are judged as difficult pixel regions.

[0124] If the effective depth ratio determined based on the effective depth auxiliary information corresponding to the sample data is lower than the preset first ratio threshold, then the sample data is judged as a low effective depth difficult sample; the effective depth ratio refers to the percentage of the number of effective pixels in the depth map to the total number of pixels in the depth map;

[0125] If the sample data is determined to have been collected from a camera device or collection environment different from the main domain of the training sample set based on the sample source identifier, then the sample data is identified as a difficult sample in the new camera domain. The main domain of the training sample set refers to the collection condition distribution that most samples in the training set share, including camera device type, lens parameters, and environmental conditions during collection.

[0126] Specifically, in step S3, the difficulty level of the sample is determined from four dimensions.

[0127] First, the determination is based on the pixel area of ​​the target fruit region. The total number of pixels in the set of pixels belonging to the target fruit category in the semantic annotation is counted and used as the pixel area of ​​the target fruit region. If this area is less than a preset first area threshold (this threshold can be statistically determined based on the pixel area distribution of the target fruit region in the training dataset, for example, taking the lower quartile of the target fruit region area of ​​all samples), it indicates that the target fruit in this sample occupies a small proportion of the image, belonging to a small target. The network has difficulty extracting enough features for recognition, so this sample is classified as a small target fruit difficult sample, and all pixels within this target fruit region are also classified as difficult pixel regions. When there are multiple connected regions of target fruits in the same image (for example, multiple individual fruits on the same bunch of fruits are labeled as multiple connected regions), when calculating the pixel area of ​​the target fruit region, the pixel areas of all pixels belonging to the target fruit category are accumulated and summed, and this total area is used as the determination criterion to avoid the same image being misclassified as a difficult sample due to differences in annotation methods.

[0128] Second, the determination is based on the pixel area of ​​the connected structure region. The total number of pixels in the set of pixels belonging to the connected structure category in the semantic annotation is counted and used as the pixel area of ​​the connected structure region. Connecting structures (calyx, pedicel, main stem) are usually smaller than fruits and occupy fewer pixels in the image. If the pixel area of ​​the connected structure region is less than a preset second area threshold, it indicates that the connected structure in the sample is too small, making accurate segmentation difficult for the network. Therefore, this sample is classified as a difficult sample with a small connected structure, and all pixels within this connected structure region are classified as difficult pixel regions. The second area threshold can be determined statistically based on the pixel area distribution of connected structure regions in the training dataset, for example, by taking the lower quartile or lower decimal of the area of ​​the connected structure regions of all samples.

[0129] Third, a determination is made based on the effective depth ratio. Based on the effective depth auxiliary information generated in step S2, the proportion of effective depth pixels in the depth map to the total number of pixels is calculated. If this proportion is lower than a preset first proportion threshold, it indicates that the depth map quality of the sample is poor, the depth information in most areas is unusable, and the network cannot easily use the depth information for assisted inference. Therefore, the sample is determined as a low-effective-depth hard sample. It should be noted that determining hard samples based on the effective depth ratio is a sample sampling strategy, used only to adjust the sampling probability of the sample; this global statistical value does not directly participate in the supervised calculation of the network loss.

[0130] Fourth, the determination is based on the sample source identifier. The sample source identifier records the camera equipment model and / or environmental information (such as lighting conditions, shooting angle, etc.) used to collect the sample. During training, the camera equipment and environmental conditions to which most samples belong are defined as the main domain. If the source identifier of a sample indicates that it was collected from a camera equipment or environmental conditions different from the main domain, then the sample is a domain-offset sample. The network often performs poorly in new, unseen domains, so this sample is classified as a difficult sample in the new camera domain.

[0131] This embodiment systematically identifies and classifies difficult samples from four dimensions: target fruit size, connection structure size, depth map quality, and acquisition domain differences. It covers the main types of technical difficulties in the picking of clustered fruits and provides a basis for subsequent differentiated training strategies.

[0132] Example 7

[0133] In this embodiment, step S4 can be specifically implemented as follows:

[0134] During training iteration sampling, if the sample data used in the current iteration of training is a difficult sample, the sampling probability of the sample data used in the current iteration of training is higher than the sampling probability of the non-difficult sample.

[0135] Based on the sampling results, data augmentation processing is performed on the RGB images and depth maps in the selected sample data used in the current iteration of training. Specifically, color space transformation and spatial transformation (including but not limited to random rotation, random scaling, random cropping, horizontal flipping, brightness adjustment, contrast adjustment, and saturation adjustment) are performed on the RGB images, and spatial transformation (including but not limited to random rotation, random scaling, random cropping, and horizontal flipping) is performed on the depth maps to match the RGB images, so as to ensure that the physical meaning of the depth values ​​is not destroyed and pixel-level alignment is maintained.

[0136] Generate an enhanced RGB-D four-channel input tensor based on the data-augmented RGB image and depth map;

[0137] The enhanced RGB-D four-channel input tensor is input into the semantic parsing network. After forward inference, the network outputs the semantic structure parsing result and the effective depth-assisted prediction result. The binary classification probability map in the output effective depth-assisted prediction result is compared with the effective depth-assisted information generated from the current sample data in step S2 to obtain the effective depth-assisted loss. The output semantic structure parsing result is used as the predicted value, and the semantic label corresponding to the current sample data is used as the ground truth value. The predicted value is compared with the ground truth value to obtain the category classification loss. When determining the category classification loss, if the current sample data includes difficult pixel regions, the loss weight of pixels in difficult pixel regions is set higher than that of pixels in non-difficult pixel regions.

[0138] The total loss is determined based on the category classification loss and the effective depth-assisted loss; the semantic parsing network is then jointly trained using the total loss to update the network parameters.

[0139] Specifically, during the sampling phase of the training iteration, if the sample used in the current training iteration is determined to be a difficult sample by step S3, the sampling probability of that sample is increased to be higher than that of a non-difficult sample. A possible implementation is to assign a sampling weight to each sample when constructing the training data loader, with the sampling weight of difficult samples being several times that of ordinary samples (e.g., 2 or 3 times), thus increasing the probability of difficult samples being selected in each training iteration.

[0140] Then, data augmentation is performed on the RGB images and depth maps in the sample data used for the current iteration of training. After data augmentation, the augmented RGB-D four-channel input tensor is regenerated according to step S2.

[0141] It should be noted that the effective depth auxiliary information pre-generated in step S2 is a binary mask aligned with the pixels of the original depth map. During data augmentation spatial transformations (rotation, scaling, cropping, horizontal flipping), this mask needs to undergo a spatial transformation identical to that of the RGB image and depth map to obtain the transformed effective depth auxiliary information, which is used for loss calculation. The determination of whether the depth values ​​of each pixel marked in the effective depth auxiliary information are valid does not change with the spatial transformation, but their pixel coordinates need to move synchronously with the spatial transformation. Color-based transformations are not performed.

[0142] The enhanced RGB-D four-channel input tensor is input into the semantic parsing network, and the semantic structure parsing result and the effective depth-assisted prediction result are obtained through forward inference.

[0143] The effective depth-assisted loss can be calculated as follows: the binary probability map in the effective depth-assisted prediction result output by the network is compared with the effective depth-assisted information generated from the depth map in step S2 (as the true value of the supervision signal), and the depth loss is calculated using the binary cross-entropy loss function. This depth loss is the effective depth-assisted loss.

[0144] The category classification loss is calculated as follows: the semantic structure parsing result output by the network (i.e., the probability of each pixel belonging to each semantic category) is compared with the manually annotated semantic labels in step S1 (as the ground truth of the supervision signal), and the category classification loss is calculated using the multi-class cross-entropy loss function. When calculating the category classification loss, if there are difficult pixel regions in the current sample (determined by step S3), the loss value for these pixels is multiplied by a weight coefficient greater than 1 (e.g., 2 or 3), while pixels in non-difficult pixel regions use a normal weight coefficient (e.g., 1). This weighting method makes the network pay more attention to learning difficult pixel regions during backpropagation.

[0145] The total loss is obtained by weighted summation of the category classification loss and the effective depth-assisted loss. This total loss is then used for backpropagation of the semantic parsing network to update its parameters. The training process is iterative until the network converges.

[0146] The beneficial effects of this embodiment are as follows: by increasing the sampling probability of difficult samples, the network has more opportunities to encounter difficult samples during training; by increasing the loss weight of difficult pixel regions, the network pays more attention to regions that are difficult to segment during optimization; by jointly training the semantic classification task and the effective depth prediction task, the two tasks share the feature extraction network and promote each other; data augmentation operations further improve the generalization ability of the model, and the appropriate augmentation strategies for RGB images and depth maps respectively ensure the rationality and effectiveness of the augmentation.

[0147] Example 8

[0148] In this embodiment, step S4 may further include the following steps:

[0149] Based on the set of pixels belonging to the target fruit category and the set of pixels belonging to the connection structure category in the semantic annotation of the current sample data, the boundaries of the target fruit region and the boundaries of the connection structure region are extracted as the true boundaries.

[0150] Based on the set of pixels belonging to the target fruit category and the set of pixels belonging to the connection structure category in the semantic structure parsing results output in step S4, the boundary of the target fruit prediction region and the boundary of the connection structure prediction region are extracted respectively as the prediction boundary.

[0151] The boundary loss is determined based on the true boundary and the predicted boundary. The boundary loss can be a binary cross-entropy loss function, which calculates the difference between the predicted result and the true label at the boundary pixel location; it can also be the Dice loss function, which calculates the degree of overlap between the predicted boundary and the true boundary; or it can be the boundary distance loss, which calculates the average distance from the predicted boundary to the true boundary.

[0152] At this point, determining the total loss based on the category classification loss and the effective depth auxiliary loss can be implemented as follows: determining the total loss based on the category classification loss, the effective depth auxiliary loss, and the boundary loss.

[0153] Specifically, the boundary loss is determined as follows: First, from the semantic annotations of the current sample, all pixels belonging to the target fruit category are extracted to form the target fruit region, and all pixels belonging to the connection structure category are extracted to form the connection structure region. Then, the boundaries of these two regions are calculated respectively. Region boundaries can be extracted using image morphology operations, such as using edge detection algorithms to extract the region contours. The extracted boundaries are taken as the ground truth boundaries.

[0154] Similarly, from the semantic structure parsing results output by the network, pixels predicted to belong to the target fruit category are extracted to form the target fruit prediction region, and pixels predicted to belong to the connection structure category are extracted to form the connection structure prediction region. Then, the boundaries of these two prediction regions are extracted as prediction boundaries.

[0155] The true boundaries are compared with the predicted boundaries to calculate the boundary loss. The purpose of the boundary loss is to encourage the network to improve the accuracy of its identification of the edge locations of target fruits and connected structures, thereby enhancing the boundary precision of the segmentation results.

[0156] Accordingly, the total loss can be obtained by weighted summation of the category classification loss, the effective depth assistance loss, and the boundary loss.

[0157] The beneficial effects of this embodiment are as follows: by introducing boundary loss, the network learns not only pixel-level classification during training but also the prediction of region boundaries. The introduction of boundary loss improves the network's segmentation accuracy for the edges of target fruit regions and connected structure regions, and has an improving effect on the boundary localization of small targets such as connected structures.

[0158] Example 9

[0159] In this embodiment, step S5 can be implemented as follows:

[0160] Obtain an RGB-D image of the cluster of fruits to be processed, the RGB-D image of the cluster of fruits to be processed includes an RGB image to be processed and a depth map to be processed aligned with the RGB image to be processed;

[0161] Generate the corresponding RGB-D four-channel input tensor and effective depth auxiliary information based on the RGB image to be processed and the depth map to be processed;

[0162] The generated RGB-D four-channel input tensor is input into the trained semantic parsing network to obtain the probability map and category label map of each pixel belonging to each semantic category in the RGB-D image of the fruit cluster to be processed, which are used as the semantic structure parsing results; the effective depth-assisted prediction results corresponding to the RGB-D image of the fruit cluster to be processed are output.

[0163] Based on the maximum category probability value corresponding to each pixel in the semantic structure parsing result, a semantic parsing confidence prompt is generated. Based on the effective depth-assisted prediction result, an effective depth region quality prompt is generated. The semantic parsing confidence prompt and the effective depth region quality prompt are used as quality prompts. The effective depth-assisted information generated from the depth map to be processed during the inference stage does not participate in the internal calculation of the semantic parsing network. As an information channel independent of the network output, it is output to the downstream 3D point cloud generation module along with the quality prompts, so that the downstream module can use it as an independent quality reference for depth value selection when generating point clouds.

[0164] Specifically, step S5 is the model inference stage, which applies the trained semantic parsing network to the real-world scenario.

[0165] First, acquire the RGB-D image of the cluster of fruits to be processed. This image is acquired simultaneously by an RGB camera and a depth camera, and the RGB image and the depth map have been registered and aligned, meaning that the pixel positions of the two images correspond one-to-one.

[0166] Following the same procedure as step S2, the RGB image to be processed is normalized, and the depth map to be processed is scaled or normalized. Then, the normalized RGB three-channel data is concatenated with the depth single-channel data along the channel dimension to generate an RGB-D four-channel input tensor. Simultaneously, valid depth auxiliary information is generated based on invalid values, outliers, holes, and depth boundaries in the depth map to be processed. During the inference phase, this valid depth auxiliary information does not participate in the internal computation of the semantic parsing network but is directly output to downstream modules as an independent information channel separate from the network output.

[0167] The generated RGB-D four-channel input tensor is input into the trained semantic parsing network. The network forward inference outputs semantic structure parsing results (including probability maps and category label maps of each pixel belonging to each semantic category) and effective depth-assisted prediction results (including binary classification probability maps and prediction label maps).

[0168] Along with the inference results, a quality indicator message is also generated. The quality indicator message can consist of two parts:

[0169] Semantic parsing confidence score hint: Based on the probability map in the semantic structure parsing results, the maximum class probability value of each pixel is taken as the confidence score of the semantic parsing result for that pixel. The higher the confidence score, the more confident the network is in determining the class of that pixel.

[0170] Effective depth region quality cues: Based on the prediction label map in the effective depth auxiliary prediction results, obtain the depth reliable / unreliable label for each pixel, generate effective depth region quality cues, and identify which pixel locations have reliable depth information.

[0171] In this embodiment, not only semantic structure parsing results and effective deep-assisted prediction results are output during the inference stage, but quality prompt information is also output. This provides downstream tasks with a quantitative assessment of the reliability of the output results, enabling downstream modules to filter and weight the results based on the quality prompts.

[0172] Example 10

[0173] In one embodiment, the method may further include:

[0174] The inference result is output to the downstream 3D point cloud generation module and / or the grasping pose estimation module. The following describes a typical application scenario of the method of the present invention. The 3D point cloud generation module and the grasping pose estimation module are external downstream modules served by the method of the present invention and are not steps that must be included in the method itself. Specifically, this includes:

[0175] The pixel regions corresponding to the target fruit category, the pixel regions corresponding to other fruit categories, and the pixel regions corresponding to the connection structure category in the semantic structure parsing result, along with the quality prompts, are jointly transmitted to the downstream 3D point cloud generation module.

[0176] The downstream 3D point cloud generation module performs pixel filtering on the pixel regions corresponding to the target fruit category, the pixel regions corresponding to other fruit categories, and the pixel regions corresponding to the connection structure category, based on the semantic parsing confidence level and effective depth region quality level in the quality prompts. After removing pixels with semantic parsing confidence levels lower than a preset confidence threshold and / or unreliable depth, it generates target fruit instance point clouds, other fruit instance point clouds, and connection structure point clouds, and then passes the generated point clouds to the downstream grasping pose estimation module.

[0177] This embodiment describes the specific process of applying the reasoning results of a semantic parsing network to downstream tasks.

[0178] The semantic parsing network is a multi-task neural network that includes two output branches: a semantic classification branch and an effective depth auxiliary branch. It can simultaneously perform two tasks: semantic structure parsing (pixel-level semantic classification) and effective depth prediction (pixel-level depth credibility judgment).

[0179] After inference is completed, all pixels belonging to the target fruit category, all pixels belonging to other fruit categories, and all pixels belonging to the connection structure category in the semantic structure parsing result are extracted to form three pixel region sets. These pixel region sets, along with quality cues (including semantic parsing confidence cues and effective depth region quality cues), are then passed to the downstream 3D point cloud generation module.

[0180] Before generating the point cloud, the 3D point cloud generation module first filters pixels based on quality criteria. The filtering criteria include two dimensions: semantic classification confidence (if a pixel's semantic classification confidence is lower than a preset confidence threshold, the pixel is discarded); and depth reliability (if a pixel is determined to have unreliable depth by effective depth-aided prediction results, the pixel is discarded). After these two rounds of filtering, pixels with both reliable semantic classification and reliable depth values ​​are retained.

[0181] Then, the 3D point cloud generation module uses the camera intrinsics of the RGB-D image to transform the selected pixels from the image coordinate system to 3D spatial points in the camera coordinate system, generating point clouds of target fruit instances, other fruit instances, and connection structures, respectively. When generating the point clouds, each point carries not only 3D coordinate information but also RGB color information and semantic category labels.

[0182] The generated point clouds are passed to the downstream grasping pose estimation module. Based on the spatial relationship between the target fruit point cloud, other fruit point clouds, and connecting structure point clouds, the grasping pose estimation module determines the grasping point and cutting point positions of the harvesting robot and plans the grasping pose of the robotic arm.

[0183] This embodiment achieves a complete link from 2D image understanding to 3D spatial perception by jointly transmitting the semantic structure parsing results and effective depth-assisted prediction results to the downstream 3D point cloud generation module. Double-screening of pixels through quality prompts eliminates pixels with unreliable semantic classification and depth values, improving the quality and accuracy of the generated point cloud. By generating instance point clouds of the target fruit, other fruits, and connection structures separately, structured 3D scene information is provided to the downstream grasping pose estimation module. The design of the multi-task neural network allows the two tasks to share the feature extraction network, reducing computational overhead.

[0184] Example 11

[0185] In another embodiment, in step S4, the sampling probability of the sample data is determined in the following way:

[0186] Step S611: Determine the difficulty sampling weight factor of the corresponding sample based on the difficulty level value of each sample in the training sample set in each difficulty dimension; wherein, each difficulty dimension may include four dimensions: target fruit size, connection structure size, depth map quality, and acquisition domain difference; the division of these four dimensions is based on the judgment criteria for difficult samples;

[0187] Specifically, the difficult sampling weighting factor can be determined according to the following formula (1):

[0188] (1)

[0189] in, Indicates the number of the current sample; For the sample Difficult sampling weighting factor;

[0190] Indicates sample The difficulty of determining the target fruit size is obtained as follows: when the sample... When there is only one target fruit region, the reciprocal of the ratio of the pixel area of ​​the target fruit region to the first area threshold is taken, and then normalized to minimum-maximum linear normalization. The interval is obtained; when the sample When there are multiple target fruit regions, the ratio of the pixel area of ​​each target fruit region to the first area threshold is calculated, and the reciprocal is then normalized using a minimum-maximum linear normalization process. The interval is used as the maximum value. The target fruit region refers to the set of pixels belonging to the target fruit category in the semantic annotation; in calculating the... When calculating the pixel area of ​​the target fruit region, the pixel areas of all pixels belonging to the target fruit category are summed, and this total area is used as the criterion for judgment. If the same cluster of fruits is labeled as multiple connected components, it does not affect the calculation of the summation of pixel areas. The min-maximum linear normalization refers to linear normalization performed with the global minimum and global maximum values ​​of the reciprocal values ​​of all samples in the training sample set as the upper and lower limits. Specifically, for each sample in the training sample set, its reciprocal value is calculated, and the global minimum and global maximum values ​​are found. Then, the global minimum value is subtracted from the reciprocal value of each sample, and then divided by the difference between the global maximum value and the global minimum value. The calculation result is used as the normalized value, so that the normalized values ​​of all samples fall within the interval of 0 to 1.

[0191] Indicates sample The difficulty of connecting the structural dimensions is determined as follows: when the sample When there is only one connected structure region, the ratio of the pixel area of ​​this connected structure region to the second area threshold is taken as the reciprocal and then normalized to minimum-maximum linear normalization. The interval is obtained; when the sample When there are multiple connected structural regions, the ratio of the pixel area of ​​each connected structural region to the second area threshold is calculated, and the reciprocal is then normalized using a minimum-maximum linear normalization. The interval is used as the maximum value. The connected structure region refers to the set of pixels belonging to the connected structure category in the semantic annotation; in calculating the region used for... When determining the pixel area of ​​the connected structure region, the pixel areas of all pixels belonging to the connected structure category are accumulated and summed, and this total area is used as the criterion for judgment.

[0192] Indicates sample The depth quality difficulty is set to a value of 1. Subtract samples The effective depth ratio is determined based on the effective depth auxiliary information. The lower the effective depth ratio, the lower the effective depth ratio. The larger;

[0193] Indicates sample The difficulty of domain difference, based on the sample When the sample source is identified, it means that the sample was collected from a camera device or collection environment that is different from the main domain of the training sample set. The value is 1 if it is set to 1, otherwise the value is 0.

[0194] These are the first, second, third, and fourth preset weight coefficients, which correspond to the four difficulty dimensions and satisfy... And all weights are greater than 0. For example, we can take... , , , Those skilled in the art can adjust the weight allocation of each dimension according to the actual application scenario; for example, in an indoor scenario with stable lighting conditions, the weight allocation of each dimension can be appropriately reduced. The weight of (depth quality) can be increased, while in outdoor scenes it can be improved. The weights;

[0195] The preset temperature regulation hyperparameter is a positive number; for example, it can be set to... or To enhance the differentiation of sampling weights between difficult samples and ordinary samples; The value of determines the discriminative power of the sampling probability between difficult samples and ordinary samples. The larger the value, the more significant the increase in the probability of difficult samples being sampled relative to ordinary samples; those skilled in the art can select based on the proportion of difficult samples in the training set. A larger value can be selected when the proportion of difficult samples is low. Value to enhance attention;

[0196] As an alternative to formula (1), the difficult sampling weight factor can also be calculated using the following exponential form (formula (2)):

[0197] (2)

[0198] in This is a weighting factor for difficult sampling using an exponential function.

[0199] Both formula (1) and the exponential form described above can achieve the technical effect of a higher sampling probability for difficult samples than for non-difficult samples, and either can be used. The difference lies in that the sigmoid form has a compressive effect on the weight difference between difficult and ordinary samples, while the exponential form can amplify this difference. When difficult samples are extremely scarce in the training set, the exponential form can provide stronger discrimination. In practical applications, those skilled in the art can choose either form to implement based on the distribution of difficult samples in the training data.

[0200] Step S612: Normalize the difficulty sampling weight factor of each sample in the training sample set to obtain the actual sampling probability of each sample; specifically, divide the difficulty sampling weight factor of each sample by the sum of the difficulty sampling weight factors of all samples to obtain the actual sampling probability of each sample; perform random sampling with replacement on the training samples according to the actual sampling probability, so that the sample with higher difficulty has a greater probability of being sampled, and use the sampled samples to iteratively train the semantic parsing network.

[0201] The beneficial effects of this embodiment are as follows: It transforms the multi-dimensional (target fruit size, connection structure size, depth map quality, and acquisition domain differences) hard sample judgment results into continuously adjustable sampling weight factors. The weighted difficulty level is mapped to a weight value, and then global normalization is performed to obtain the actual sampling probability of each sample. This allows samples with different levels of difficulty to receive differentiated attention during training. The weight coefficients for the four dimensions are... to This feature is independently adjustable, allowing this embodiment to adapt to changes in the importance of each difficulty dimension in different scenarios. The difficulty can be displayed in any of the four dimensions. (If the value is large), it will increase. The value of is determined to increase the probability that the sample will be sampled.

[0202] Example 12

[0203] In another embodiment, the loss weights of the pixels in the difficult pixel region are determined in the following manner:

[0204] Step S621: When there are difficult pixel regions in the current sample, determine the pixel difficulty weight coefficient of each pixel based on the local neighborhood contrast of each pixel and the determination result of the difficult pixel region.

[0205] Specifically, the pixel difficulty weight coefficient for each pixel can be determined according to the following formula (3):

[0206] (3)

[0207] in, This represents the number of any pixel in the current sample; For pixels Difficult pixel indicator value, when pixel The value is 1 when the pixel is identified as belonging to a difficult pixel region, and 0 otherwise. For pixels The local neighborhood contrast is calculated by pixel. and its surroundings The average Euclidean distance of each pixel in the neighborhood in the RGB color space is obtained, where each RGB channel has been normalized to... interval;

[0208] Local neighborhood contrast of all hard pixels in the current sample The mean; as an optional implementation, Alternatively, you can retrieve all pixels in the current sample (including non-difficult pixels). mean Alternatively, a globally fixed contrast threshold can be pre-determined from statistics obtained from the training dataset. This can be achieved by using the contrast threshold of all difficult pixels in the current sample. When averaging, if the overall contrast of difficult pixel areas is high, it may make the edge pixels appear... Below This weakens edge gain; using a globally fixed contrast threshold can avoid this problem, and those skilled in the art can choose one of the above methods according to the actual training data distribution.

[0209] The preset base weight enhancement coefficient, This α value controls the overall weight increase for difficult pixel regions. A larger α value results in a higher base weight for difficult pixel regions in the loss calculation. In practice, α can be adjusted within the range of 0.5 to 2.0. When difficult pixel regions account for a low proportion in the training data, the α value can be appropriately increased to enhance attention to these regions.

[0210] The preset contrast modulation coefficient. β is used to control the degree to which local neighborhood contrast affects pixel weights. The larger the β value, the more significant the modulation effect of contrast on weights; when β=0, contrast modulation is disabled, and all hard pixels receive the same basic weight enhancement. In practical applications, β can be adjusted within the range of 0 to 1.0.

[0211] This is the steepness adjustment parameter for the sigmoid function. γ is used to control the concentration of additional gain for pixels with contrast levels higher than the average. A larger γ value results in more concentrated gain on edge pixels with significantly higher contrast than the average; a smaller γ value results in a more even distribution of gain among pixels with different contrast levels. In practical applications, γ can be adjusted within the range of 3.0 to 10.0.

[0212] For pixels The pixel difficulty weight coefficient, when hour ,when hour ;

[0213] Step S622: Determine the category classification loss based on the semantic annotation and semantic structure parsing results corresponding to the current sample; wherein, when determining the category classification loss, first determine the pixel-level classification loss of each pixel, multiply the pixel difficulty weight coefficient of each pixel by the corresponding pixel-level classification loss to obtain the weighted loss value of each pixel, and then take the average of the weighted loss values ​​of all pixels to obtain the category classification loss of the current sample; thus, the contribution of pixels in difficult pixel regions to the category classification loss is higher than that of pixels in non-difficult pixel regions; iteratively train the semantic parsing network based on the category classification loss of each sample.

[0214] Specifically, in step S621, it is first determined whether the current sample contains a difficult pixel region (the determination result is given in step S3). If it does, for each pixel in the sample, the weight coefficient corresponding to the pixel is calculated by combining its local neighborhood contrast and whether the pixel belongs to a difficult pixel region.

[0215] In step S621, the meaning of formula (3) can be understood in two parts. The first part Indicates: For pixels belonging to the difficult pixel region (i.e. Its weight is at least ), This means obtaining a basic enhancement; for pixels that do not belong to the difficult pixel region (i.e. Its weight is 1 and remains unchanged. Part Two This indicates that within the difficult pixel region, based on the local contrast of the pixel... Secondary modulation is performed. Among them... The physical meaning is: a pixel and its surroundings The average Euclidean distance of other pixels in the neighborhood in the RGB color space. The larger the value, the more likely the pixel is in a region with drastic changes in visual features (such as the edge of an image or a region with rich texture). The physical meaning is: all difficult pixels in the current sample. The average value represents the general contrast level within the difficult pixel area; the following explanation uses this value as an example. The modulation principle remains the same when other alternative methods are used.

[0216] when Greater than When, the Sigmoid function The output is greater than 0.5, making Further increase; when Less than When the Sigmoid output is less than 0.5, it makes The relative increment decreases. This makes pixels near the edge of a region, which are already considered difficult to segment (such as small fruit regions or regions with fine connecting structures), typically have higher local neighborhood contrast and are more critical for determining the segmentation boundary. Therefore, assigning higher weights to these pixels helps improve the network's accuracy in locating the boundary position. Its function is to adjust the steepness of the sigmoid function. The larger the value, the more concentrated the extra weight gain when the contrast exceeds the mean. The smaller the value, the more evenly the extra weight is distributed across all contrast levels.

[0217] In step S622, the relationship between pixel-level classification loss and class classification loss is as follows: The output of the semantic parsing network is a probability map of the same size as the input image. Each pixel location corresponds to a probability vector, representing the probability distribution of the pixel belonging to each class. By comparing the probability vector of each pixel with the manually annotated semantic label (i.e., the true class label of the pixel) in step S1, the classification loss of that pixel can be calculated. This loss is the pixel-level classification loss. After obtaining the pixel-level classification loss for all pixels, the loss value of each pixel is multiplied by its corresponding pixel difficulty weight coefficient. The weighted loss value for each pixel is obtained; then, the arithmetic mean of the weighted loss values ​​for all pixels is taken to obtain a scalar value, which is the classification loss for the current sample. This classification loss is used for backpropagation to update the network parameters.

[0218] The pixel-hard weight coefficients are calculated only during the training phase and not during the network inference phase (i.e., when processing images from actual harvesting scenarios). Furthermore, no weighting is applied to the output. This is because weighting is only used during training to adjust the optimization direction of the network parameters; during inference, the network parameters are fixed and no further weighting is needed.

[0219] The beneficial effects of this embodiment are as follows: It addresses difficult pixel regions at the loss function level. By applying differentiated weights to each pixel when calculating the category classification loss, pixels in difficult pixel regions (small target fruit regions, regions with fine connection structures) occupy a higher proportion in the loss, guiding the network to focus on learning the feature representations of these regions during training. Compared to training methods that apply uniform weights to all pixels, differentiated weights can improve the network's segmentation accuracy for targets with small pixel proportions and insignificant visual features. Furthermore, this embodiment utilizes local neighborhood contrast for secondary differentiation within difficult pixel regions, giving extra weights to pixels at the region edges, which helps improve the network's ability to perceive segmentation boundaries. Contrast modulation uses the Sigmoid function... The (mean contrast value of the difficult pixel region) serves as the dividing point, ensuring a smooth and continuous weight difference between edge pixels and pixels within the region. The hyperparameters in the formula... , , All of them can be adjusted independently, providing flexibility to adapt to training needs in different scenarios.

[0220] Example 13

[0221] In one embodiment, step S4 can also be implemented as follows:

[0222] Step S631: Normalize the difficult sampling weight factor of each sample in the training sample set (determined according to the aforementioned formulas (1) and (2)) to obtain the actual sampling probability of each sample, and sample the training samples according to the actual sampling probability.

[0223] The process involves normalizing the hard sampling weight factors for all samples by dividing each sample's hard sampling weight factor by the sum of all sample hard sampling weight factors to obtain the actual sampling probability for each sample. Training samples are then sampled according to these probabilities, ensuring that samples with higher difficulty levels are selected more frequently during training.

[0224] Step S632: Determine the classification loss and boundary loss for each sample in the training sample set as follows:

[0225] The RGB image and depth map of the current sample are augmented to generate an augmented RGB-D four-channel input tensor. The augmented RGB-D four-channel input tensor is then input into a semantic parsing network, which outputs semantic structure parsing results and effective depth-assisted prediction results via forward inference. The pixel difficulty weight coefficient of each pixel is determined based on the local neighborhood contrast and the determination results of difficult pixel regions in the current sample. The category classification loss is determined based on the semantic annotation and semantic structure parsing results corresponding to the current sample. Specifically, when determining the category classification loss, the pixel-level classification loss of each pixel is first determined, and the pixel difficulty weight coefficient of each pixel is multiplied by the corresponding pixel-level classification loss to obtain the weighted loss value of each pixel. Then, the weighted loss values ​​of all pixels are averaged to obtain the category classification loss of the current sample.

[0226] The pixel difficulty weight coefficient for each pixel can be determined according to the aforementioned formula (3);

[0227] Based on the set of pixels belonging to the target fruit category and the set of pixels belonging to the connection structure category in the semantic annotations corresponding to the current sample data, the boundaries of the target fruit region and the connection structure region are extracted as the ground truth boundaries; based on the set of pixels belonging to the target fruit category and the set of pixels belonging to the connection structure category in the semantic structure parsing results output by the network, the boundaries of the predicted target fruit region and the predicted connection structure region are extracted as the predicted boundaries; the boundary loss is determined based on the ground truth boundaries and the predicted boundaries.

[0228] Step S633: Determine the effective depth auxiliary loss for each sample based on the effective depth auxiliary prediction results and the effective depth auxiliary information generated in step S2.

[0229] Specifically, the effective depth-assisted prediction result is compared with the effective depth-assisted information generated from the depth map in step S2 to determine the effective depth-assisted loss for the current sample. The effective depth-assisted prediction result is the network's reliable prediction of the depth of each pixel (binary classification probability map), while the effective depth-assisted information is a supervision signal (true label) directly generated from the depth map. The difference between the two can be calculated using the binary classification cross-entropy loss function to obtain the effective depth-assisted loss for the current sample.

[0230] Step S634: Determine the total loss for each sample based on the category classification loss, boundary loss, and effective depth-assisted loss. This total loss consists of three parts: category classification loss, boundary loss, and effective depth-assisted loss. During actual training, preset weight coefficients can be assigned to each of these three loss parts. The total loss is obtained by multiplying each loss by its corresponding weight coefficient and summing the results. Those skilled in the art can adjust the values ​​of each weight coefficient according to the convergence speed of each task to balance the contribution of different learning tasks to network parameter optimization. The semantic parsing network is then jointly trained using the total loss for each sample to update the network parameters.

[0231] Alternatively, the above operations can be performed in batches to obtain the total loss for each sample in the current training batch. The total losses of all samples in the current training batch are then aggregated (e.g., by averaging) to obtain a scalar value. This scalar value is then used for backpropagation to update the parameters of the semantic parsing network. A training batch refers to the set of samples used for each parameter update during deep learning training. In practical engineering implementations, due to limitations in computing resources (especially GPU memory capacity), the training samples can be divided into multiple smaller batches, each containing several images (e.g., 8, 16, or 32). The total loss of each sample within a batch is averaged before a parameter update is performed.

[0232] The beneficial effects of this embodiment are as follows: By combining a sample-level hard sampling strategy with a pixel-level hard loss weighting strategy in the same training process, the semantic parsing network is trained from two levels. During the sampling phase, hard samples receive a higher probability of being sampled, ensuring the network has more opportunities to encounter them during training iterations. During the loss calculation phase, pixels in hard pixel regions receive higher loss weights, guiding the network to focus more on difficult-to-segment regions during backpropagation. The synergistic effect of these two strategies allows the network to both see hard samples (through sampling) and learn hard regions (through loss weighting) during training, which helps improve the model's generalization ability and segmentation accuracy in complex scenarios. Furthermore, this embodiment incorporates semantic classification and effective depth prediction tasks into the same training process for joint optimization, allowing the two tasks to share a feature extraction network, which is beneficial for the network to learn feature representations that are advantageous to both tasks. Additionally, batch training is possible, where the total loss of each sample in the current batch is aggregated and the network parameters are updated uniformly, maintaining the stability and convergence efficiency of the training process.

[0233] Example 14

[0234] Corresponding to the methods provided in any of the foregoing embodiments, this embodiment of the invention also provides a semantic parsing and hard sample training system for images of clustered fruits, comprising:

[0235] The sample acquisition module is used to acquire a training sample set, which includes sample data of multiple clusters of fruits. Each sample data includes at least an RGB image, a depth map aligned with the RGB image, a semantic annotation for identifying the category to which the pixel belongs, and a sample source identifier.

[0236] The data processing module is used to generate an RGB-D four-channel input tensor for each sample data in the training sample set based on the RGB image and the depth map, and to generate effective depth auxiliary information based on the depth map. The effective depth auxiliary information is auxiliary information that identifies whether the depth value of each pixel in the depth map is reliable.

[0237] The judgment processing module is used to determine, for each sample data in the training sample set, whether the sample data is a difficult sample or whether there is a difficult pixel region, based on at least one of the semantic annotation, effective depth auxiliary information, and sample source identifier corresponding to the sample data, and to obtain a judgment result.

[0238] The training module, during each training iteration, augments the RGB-D four-channel input tensor of the sample data collected in the current iteration before inputting it into the semantic parsing network. In the semantic parsing network, the features of the input tensor are enhanced and optimized through contextual augmentation and detail augmentation. Based on the enhanced and optimized features, the semantic parsing network performs category parsing and effective depth prediction on each pixel corresponding to the input tensor, outputting the corresponding semantic structure parsing results and effective depth auxiliary prediction results. The semantic parsing network is iteratively trained based on the semantic annotations corresponding to the sample data, the effective depth auxiliary information, the semantic structure parsing results, the effective depth auxiliary prediction results, and the judgment results.

[0239] The technical solutions of the system provided in the embodiments of the present invention correspond one-to-one with the technical solutions of the methods provided in the embodiments of the present invention. The technical principles and beneficial effects are described in the methods described above, and will not be repeated here.

[0240] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. This disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A method for semantic parsing and hard sample training of images of clustered fruits, characterized in that, include: Step S1: Obtain a training sample set, which includes sample data of multiple clusters of fruits. Each sample data includes at least an RGB image, a depth map aligned with the RGB image, a semantic annotation for identifying the category to which the pixel belongs, and a sample source identifier. Step S2: For each sample data in the training sample set, generate an RGB-D four-channel input tensor based on the RGB image and the depth map, and generate effective depth auxiliary information based on the depth map; Step S3: For each sample data in the training sample set, determine whether the sample data is a difficult sample or whether there is a difficult pixel region based on at least one of the corresponding semantic annotation, effective depth auxiliary information, and sample source identifier, and obtain the determination result. Step S4: During each iteration of training, for the sample data sampled in the current iteration, its RGB-D four-channel input tensor is augmented and then input into the semantic parsing network. In the semantic parsing network, the features of the input tensor are enhanced and optimized through context enhancement and detail structure enhancement. Based on the enhanced and optimized features, the semantic parsing network performs category parsing and effective depth prediction on each pixel corresponding to the input tensor, and outputs the corresponding semantic structure parsing results and effective depth auxiliary prediction results. Based on the semantic annotations corresponding to the sample data, the effective depth auxiliary information, the semantic structure parsing results, the effective depth auxiliary prediction results, and the corresponding judgment results, the semantic parsing network is iteratively trained.

2. The method according to claim 1, characterized in that, Step S2 includes performing the following operations for each sample data in the training sample set: The RGB image is normalized to obtain normalized RGB three-channel data; The depth map is transformed or normalized according to the depth scale to obtain normalized depth channel data. Normalized RGB three-channel data and normalized depth channel data are concatenated along the channel dimension to generate an RGB-D four-channel input tensor; Based on invalid values, outliers, void regions, or depth boundaries in the depth map, effective depth region indication information or depth confidence information is generated as the effective depth auxiliary information.

3. The method according to claim 1, characterized in that, In step S4, the features of the input tensor are enhanced and optimized in the semantic parsing network through context enhancement and detail structure enhancement, including: The RGB-D four-channel input tensor is fed into the encoder, and multi-scale features are extracted through multi-layer convolution. The multi-scale features are processed using dilated convolutions with different dilation rates to achieve context enhancement. The context-enhanced features are concatenated with the shallow features output from each layer of the encoder via skip connections, and then upsampled by the decoder to obtain a decoded feature map. The decoded feature map is then concatenated with the RGB-D four-channel input tensor and convolved to complete the detail structure enhancement.

4. The method according to claim 3, characterized in that, The semantic structure parsing results and effective depth-assisted prediction results output in step S4 include: The enhanced features are convolutionally mapped to obtain the probability map and category label map of each pixel belonging to each semantic category, which serve as the semantic structure parsing result. The decoded feature map is convolved to obtain a binary classification probability map in which each pixel belongs to the depth reliable and depth unreliable categories; the maximum value of the binary classification probability map along the category dimension is taken to obtain the effective depth-assisted prediction label map; the binary classification probability map and the effective depth-assisted prediction label map are used together as the effective depth-assisted prediction result.

5. The method according to claim 1, characterized in that, Step S3 includes: If the pixel area of ​​the target fruit region determined by the semantic annotation corresponding to the sample data is less than the preset first area threshold, then the sample data is judged as a small target fruit difficult sample, and the pixels in the target fruit region are judged as difficult pixel regions. If the pixel area of ​​the connection structure region determined by the semantic annotation corresponding to the sample data is less than the preset second area threshold, then the sample data is judged as a difficult sample with a small area connection structure, and the pixels in the connection structure region are judged as difficult pixel regions. If the effective depth ratio determined based on the effective depth auxiliary information corresponding to the sample data is lower than the preset first ratio threshold, then the sample data is judged as a low effective depth difficult sample; the effective depth ratio refers to the percentage of the number of effective pixels in the depth map to the total number of pixels in the depth map; If the sample data is determined to have been collected from a camera device or collection environment different from the main domain of the training sample set based on the sample source identifier, then the sample data is identified as a difficult sample in the new camera domain. The main domain of the training sample set refers to the collection condition distribution that most samples in the training set share, including camera device type, lens parameters, and environmental conditions during collection.

6. The method according to claim 1, characterized in that, Step S4 includes: During training iteration sampling, if the sample data used in the current iteration of training is a difficult sample, the sampling probability of the sample data used in the current iteration of training is higher than the sampling probability of the non-difficult sample. Based on the sampling results, data augmentation processing is performed on the RGB images and depth maps in the selected sample data used in the current iteration of training; Generate an enhanced RGB-D four-channel input tensor based on the data-augmented RGB image and depth map; The enhanced RGB-D four-channel input tensor is input into the semantic parsing network. After forward inference, the network outputs the semantic structure parsing result and the effective depth-assisted prediction result. The binary classification probability map in the output effective depth-assisted prediction result is compared with the effective depth-assisted information generated from the current sample data in step S2 to obtain the effective depth-assisted loss. The output semantic structure parsing result is used as the predicted value, and the semantic label corresponding to the current sample data is used as the ground truth value. The predicted value is compared with the ground truth value to obtain the category classification loss. When determining the category classification loss, if the current sample data includes difficult pixel regions, the loss weight of pixels in difficult pixel regions is set higher than that of pixels in non-difficult pixel regions. The total loss is determined based on the category classification loss and the effective depth-assisted loss; the semantic parsing network is then jointly trained using the total loss to update the network parameters.

7. The method according to claim 6, characterized in that, Step S4 further includes: Based on the set of pixels belonging to the target fruit category and the set of pixels belonging to the connection structure category in the semantic annotation of the current sample data, the boundaries of the target fruit region and the boundaries of the connection structure region are extracted as the true boundaries. Based on the set of pixels belonging to the target fruit category and the set of pixels belonging to the connection structure category in the semantic structure parsing results output in step S4, the boundary of the target fruit prediction region and the boundary of the connection structure prediction region are extracted respectively as the prediction boundary. Determine the boundary loss based on the actual boundary and the predicted boundary; The step of determining the total loss based on the category classification loss and the effective depth auxiliary loss includes: determining the total loss based on the category classification loss, the effective depth auxiliary loss, and the boundary loss.

8. The method according to claim 1, characterized in that, In step S4, the sampling probability of the sample data is determined in the following way: The difficulty sampling weight factor of each sample is determined based on the difficulty level value of each sample in the training sample set in each difficulty dimension; Specifically, the sample is determined according to the following formula. Difficult sampling weighting factor: in, Indicates the number of the current sample; For the sample Difficult sampling weighting factor; Indicates sample The difficulty of achieving the target fruit size, based on the sample The ratio between the pixel area of ​​the target fruit region and the first area threshold is determined; Indicates sample The difficulty of the connection structure size, based on the sample The ratio between the pixel area of ​​the connected structure region and the second area threshold is determined; Indicates sample The depth quality difficulty is set to a value of 1. Subtract samples The effective depth ratio is determined based on the effective depth auxiliary information. Indicates sample The difficulty of domain difference, based on the sample The source of the samples in the middle was identified; These are the first preset weight coefficient, the second preset weight coefficient, the third preset weight coefficient, and the fourth preset weight coefficient, respectively, satisfying... And all weights are greater than 0; This is a preset temperature regulation hyperparameter, and its value is a positive number; The hard sampling weight factor of each sample in the training sample set is normalized to obtain the actual sampling probability of each sample; the training samples are then sampled according to the actual sampling probability of each sample.

9. The method according to claim 1, characterized in that, In step S4, the loss weights of the pixels in the difficult pixel region are determined in the following way: When there are difficult pixel regions in the current sample, the pixel difficulty weight coefficient of each pixel is determined based on the local neighborhood contrast of each pixel in the current sample and the determination result of whether it belongs to a difficult pixel region. Specifically, the pixel difficulty weight coefficient for each pixel is determined according to the following formula: in, This represents the number of any pixel in the current sample; For pixels Difficult pixel indicator value, when pixel The value is 1 when the pixel is identified as belonging to a difficult pixel region, and 0 otherwise. For pixels Local neighborhood contrast; Local neighborhood contrast of all hard pixels in the current sample The mean; The preset base weight enhancement coefficient, The preset contrast modulation coefficient. ; This is the steepness adjustment parameter for the sigmoid function. ; For pixels The pixel difficulty weighting coefficient; When determining the classification loss of the current sample, the pixel-level classification loss of each pixel is determined. The pixel difficulty weight coefficient of each pixel is multiplied by the corresponding pixel-level classification loss to obtain the weighted loss value of each pixel. Then, the weighted loss values ​​of all pixels are averaged to obtain the classification loss of the current sample.

10. A semantic parsing and hard sample training system for images of clustered fruits, characterized in that, include: The sample acquisition module is used to acquire a training sample set, which includes sample data of multiple clusters of fruits. Each sample data includes at least an RGB image, a depth map aligned with the RGB image, a semantic annotation for identifying the category to which the pixel belongs, and a sample source identifier. The data processing module is used to generate an RGB-D four-channel input tensor for each sample data in the training sample set based on the RGB image and the depth map, and to generate effective depth auxiliary information based on the depth map; The judgment processing module is used to determine, for each sample data in the training sample set, whether the sample data is a difficult sample or whether there is a difficult pixel region, based on at least one of the semantic annotation, effective depth auxiliary information, and sample source identifier corresponding to the sample data, and to obtain a judgment result. The training module, during each training iteration, augments the RGB-D four-channel input tensor of the sample data collected in the current iteration before inputting it into the semantic parsing network. In the semantic parsing network, the features of the input tensor are enhanced and optimized through contextual augmentation and detail augmentation. Based on the enhanced and optimized features, the semantic parsing network performs category parsing and effective depth prediction on each pixel corresponding to the input tensor, outputting the corresponding semantic structure parsing results and effective depth auxiliary prediction results. The semantic parsing network is iteratively trained based on the semantic annotations corresponding to the sample data, the effective depth auxiliary information, the semantic structure parsing results, the effective depth auxiliary prediction results, and the judgment results.

Images