Paddy field high-density shielding scene three-dimensional instance segmentation method in combination with multi-modal fusion network

Abstract

The invention relates to a three-dimensional instance segmentation method in a paddy field high-density shielding scene in combination with a multi-modal fusion network. The method comprises the steps of collecting a multi-view RGB image and three-dimensional point cloud data of a paddy field for preprocessing; inputting the multi-view RGB image in the multi-source data set into the image segmentation model to generate an image instance segmentation result; matching the image instance segmentation result with the three-dimensional point cloud data in the multi-source data set; and inputting the point cloud data containing the multi-view RGB image information into the improved three-dimensional instance segmentation network model, and outputting a three-dimensional instance segmentation result. According to the method, accurate segmentation in a high-density shielding scene is realized, and the segmentation accuracy is remarkably improved; effective fusion of multi-modal data is realized, the robustness and precision of instance segmentation are effectively improved, and the method is especially suitable for a fine-grained segmentation task in a dense vegetation scene. The calculation complexity is reduced, the processing efficiency is improved, and the long-distance dependency relationship in the point cloud can be captured while the efficient calculation performance is kept.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, deep learning and agricultural information technology, and in particular to a three-dimensional instance segmentation method in a rice field high-density occlusion scene combined with a multimodal fusion network. Background Art

[0002] In recent years, with the rapid development of agricultural informatization and intelligent technology, the demand for precise monitoring and management in agricultural production has continued to grow. Especially for major food crops such as rice, how to accurately monitor their growth and evaluate their yield has become an important topic in agricultural research. As a core tool in precision agriculture, crop phenotyping technology can help improve the efficiency and quality of crop production through detailed analysis of crop morphology, structure and other characteristics. Traditional crop phenotyping mostly relies on two-dimensional image technology, which performs well in relatively simple crop scenes. However, in complex environments such as rice fields, the high-density planting and overlapping of rice ears and leaves make two-dimensional image processing methods face great challenges. In such scenes with severe occlusion, two-dimensional image technology often finds it difficult to accurately distinguish individual rice ears or leaves, resulting in a decrease in segmentation accuracy, which affects subsequent monitoring, analysis and yield evaluation.

[0003] In order to overcome the limitations of two-dimensional image analysis, three-dimensional point cloud technology has gradually become an important tool in the field of crop phenotyping. Point cloud data can not only provide three-dimensional spatial information of objects, but also more intuitively show the morphological characteristics of crops in complex scenes. However, current point cloud processing methods are mostly used in sparse or unobstructed scenes, such as indoor objects or other simple environments. In high-density planting environments such as rice fields, the occlusion and overlap of rice ears and leaves make the existing point cloud segmentation algorithms perform poorly in processing accuracy and efficiency, and cannot meet the needs of precision agriculture.

[0004] In order to solve the segmentation problem in complex scenes, multi-source data fusion technology has gradually become a research hotspot. Multimodal fusion methods can fully utilize the advantages of each data by combining two-dimensional images with three-dimensional point cloud data, and significantly improve the segmentation accuracy and robustness in complex scenes. In high-density occlusion scenes, by fusing the detailed information of the image with the spatial information of the point cloud, the accuracy problem that cannot be solved by a single data source can be effectively improved, especially in the precise segmentation task of densely planted crops. Although multimodal fusion technology has shown wide application potential in the field of crop phenotyping, there is still a lack of specialized research on high-density occlusion scenes such as rice fields. How to efficiently integrate multi-view images and three-dimensional point cloud data is still a technical problem in current agricultural intelligent research. Effective fusion algorithms not only need to deal with the information alignment problem between different data sources, but also need to ensure computational efficiency and segmentation accuracy when processing large-scale data. Therefore, multimodal segmentation methods for complex rice field scenes urgently need to be further explored and developed. Summary of the invention

[0005] In order to solve the problems of insufficient instance segmentation accuracy and high missed detection rate of rice ears, leaves and stems in complex scenes in the prior art, the purpose of the present invention is to provide a three-dimensional instance segmentation method in high-density occlusion scenes of rice fields combined with a multimodal fusion network, which can achieve accurate segmentation in high-density occlusion scenes, effectively cope with complex rice field environments, and significantly improve the segmentation accuracy.

[0006] To achieve the above object, the present invention adopts the following technical solution: a three-dimensional instance segmentation method in a rice field high-density occlusion scene combined with a multimodal fusion network, comprising:

[0007] (1) Collect multi-view RGB images and three-dimensional point cloud data of rice fields, pre-process the collected data, and form a multi-source data set after the pre-processing;

[0008] (2) The multi-view RGB images in the multi-source dataset are input into the image segmentation model. The image segmentation model processes the multi-view RGB images and generates image instance segmentation results.

[0009] (3) According to the geometric mapping algorithm, the image instance segmentation results are matched with the three-dimensional point cloud data in the multi-source data set, and the pixel coordinates in each image are mapped to the point cloud coordinate system to obtain point cloud data containing multi-view RGB image information;

[0010] (4) The 3D instance segmentation network model based on long sequence point cloud processing is improved to obtain an improved 3D instance segmentation network model, and the point cloud data containing multi-view RGB image information is input into the improved 3D instance segmentation network model, and the improved 3D instance segmentation network model outputs the 3D instance segmentation result.

[0011] The step (1) specifically comprises the following steps:

[0012] (1a) A depth camera is used to obtain multi-view RGB images. After calibrating the camera parameters, the neural radiation field NeRf is used to reconstruct the 3D structure of rice and extract point cloud data.

[0013] (1b) De-noising the point cloud data and performing color correction on the multi-view RGB images;

[0014] (1c) Downsampling the point cloud data.

[0015] The step (2) specifically comprises the following steps:

[0016] (2a) Inputting the multi-view RGB image into the image segmentation model to generate a segmentation result, and generating multiple masks from different viewpoints according to the segmentation result, corresponding to the rice ear, leaf and stem respectively;

[0017] (2b) All masks are merged into a single grayscale image, and the masks are sorted according to the confidence level. Each mask instance is marked with a different weight. According to the mask instance, for the overlapping area of ​​the multi-view projection, the image instance segmentation result G is calculated by selecting the point closest to the multi-view RGB image plane. m :

[0018]

[0019] Among them, M i is the i-th mask, w i is the corresponding weight, and k is the number of multi-view RGB images.

[0020] The step (3) specifically comprises the following steps:

[0021] (3a) First, calculate the projection matrix, using the external parameter matrix E i and the internal parameter matrix I i , the image instance segmentation result G m Project to the 3D point cloud space and generate the corresponding 3D point cloud label P:

[0022]

[0023] Among them, U, v are the image instance segmentation results G m The pixel coordinates of , D is the depth value;

[0024] (3b) During the projection process, the point cloud coordinates closest to the multi-view RGB image plane are preferentially selected according to the depth value D;

[0025] (3c) After projection, in order to ensure that each point cloud instance has a unique ID, the density-based clustering algorithm DBSCAN is used to cluster the point clouds in the 3D instance segmentation results to obtain the point cloud points belonging to the same instance;

[0026] (3d) To distinguish different instances obtained by clustering, a minimum three-dimensional bounding box is generated according to each clustering instance, and the overlap between instances is calculated using the intersection-over-union ratio:

[0027]

[0028] Where A' and B' represent the three-dimensional bounding boxes of the two instances respectively;

[0029] (3e) When the IoU value of two instances exceeds a preset threshold, the two instances are merged into the same instance to obtain point cloud data containing multi-view RGB image information.

[0030] The step (4) specifically includes the following steps: the three-dimensional instance segmentation network model based on long sequence point cloud processing includes a point cloud embedding module, an encoder and a decoder; a fusion processing module is added to the encoder, a long sequence feature extraction module is used to replace the convolution module and the self-attention mechanism module inside the encoder, the long sequence feature extraction module, the self-attention module and the cross attention module form a new decoder, and the decoder is replaced by the new decoder;

[0031] The point cloud data is divided into multiple point cloud blocks of fixed size, and each point cloud block is mapped to a high-dimensional feature vector space. The formula is expressed as:

[0032] X' = ​​MLP(PatchEmbed(X))

[0033] Among them, X is the original point cloud data with labels, X' is the feature representation after embedding; PatchEmbed means converting each point cloud block into a high-dimensional feature vector; MLP is a multi-layer perceptron;

[0034] In the fusion processing module, the neighborhood features of X' are extracted, and the neighborhood features are smoothed using adaptive average pooling to eliminate local random noise; then they are input into the multi-layer perceptron for further processing to enhance the feature expression capability; the features processed by the multi-layer perceptron are fused with X' to obtain the fused feature representation X. fusion :

[0035] X fusion =MLP(AdaptiveAvgPool(KNN(X')))+X'

[0036] Among them, AdaptiveAvgPool represents adaptive average pooling, and KNN represents K-nearest neighbor algorithm;

[0037] The long sequence feature extraction module in the encoder receives X fusion , and extract long sequence dependency information through the state space model, as follows:

[0038] First, X is updated by the state update formula fusion To process:

[0039] h t =Ah t-1 +Bx t

[0040] y t =Ch t +Dx t

[0041] Among them, h t represents the hidden state at the current moment, h t-1 represents the hidden state at the previous moment, x t Indicates the feature representation of the current input, i.e. X fusion ; A, B, C, D are matrix parameters, A and B are responsible for the input x t Perform linear transformation, C and D are used to transform h t Mapping to y t ,y t Represents the output features of the current moment containing global geometric information;

[0042] In the decoder, the long sequence feature extraction module uses the state space model to extract the encoder output y t To process, the long sequence feature extraction module first updates the state of the features output by the encoder:

[0043] h k =Ah k-1 +Bx k

[0044] Among them, h k Represents the hidden state of the current time step, h k-1 represents the hidden state of the previous time step, x k Represents the output feature of the encoder, i.e., y t ;

[0045] The output features are generated by the following formula:

[0046] y k =Ch k +Dx k

[0047] Among them, y k It is the output feature of the long sequence feature extraction module in the decoder;

[0048] In the self-attention module of the decoder, the output feature y from the long sequence feature extraction module is k Input to the self-attention module, calculate the correlation between each feature, extract y k The context information of self-attention is calculated as:

[0049]

[0050] Where Q is the feature query vector; K is the key vector; V is the value vector; d k is a scaling factor; Attention represents self-attention; softmax represents the softmax function;

[0051] Through the self-attention mechanism, we get the output z t ;

[0052] In the decoder’s criss-cross attention module, the criss-cross attention module accepts the output z from the self-attention module t , the formula for cross attention is:

[0053]

[0054] Among them, Q C is a randomly initialized matrix and updated via backpropagation during training;

[0055] Generating new feature representations through criss-cross attention modules Will After Softmax and argmax processing, the final three-dimensional rice ear, leaf and stem instance segmentation results, that is, the three-dimensional instance segmentation results, are obtained. The specific calculation formula is:

[0056]

[0057] Label point = arg max(P class )

[0058] Among them, P class is the probability distribution of the category, Label point is the 3D instance segmentation result.

[0059] It can be seen from the above technical scheme that the beneficial effects of the present invention are: first, the present invention improves the segmentation accuracy in occluded scenes: by fusing multi-perspective RGB images with three-dimensional point cloud data, accurate segmentation in high-density occluded scenes is achieved. Compared with the traditional single-modal segmentation method, the joint processing of images and point clouds can effectively cope with complex rice field environments, especially when rice ears, leaves and stems occlude each other, which significantly improves the accuracy of segmentation; second, efficient multimodal data fusion is performed: by introducing image segmentation algorithms and combining their results with point cloud data, the present invention realizes the effective fusion of multimodal data, and the prior information provided by the image is combined with the spatial features of the point cloud, which effectively improves the robustness and accuracy of instance segmentation, and is particularly suitable for fine-grained segmentation tasks in dense vegetation scenes; third, the computational complexity is reduced, and the image segmentation algorithm is used to perform segmentation. clutter, and improves processing efficiency: the present invention adopts a long sequence feature extraction module, and performs point cloud modeling through an optimized long sequence matrix. Compared with the traditional method, the matrix can capture long-distance dependencies in the point cloud while maintaining efficient computing performance, avoiding the rapid growth of computational complexity, and is suitable for large-scale point cloud data processing; Fourth, it has higher segmentation robustness and accuracy: the present invention ensures the extraction of global features and the suppression of noise through point cloud serialization processing, combined with an optimized matrix modeling method and a multi-layer perceptron. Through this feature extraction and optimization, the present invention can generate high-quality segmentation results under complex backgrounds, reducing the situation of mis-segmentation and missed segmentation; Fifth, it provides automated and intelligent agricultural application support: the present invention can be applied to the refined management of crops, and automatically monitors the growth status and health of crops. The efficient and accurate segmentation results provide strong data support for the phenotypic analysis, pest monitoring and yield evaluation of crops, and have a wide range of application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the method of the present invention;

[0061] Figure 2 It is a schematic diagram of collecting multi-source data sets of the present invention;

[0062] Figure 3 It is a schematic diagram of the three-dimensional instance segmentation result of the present invention. DETAILED DESCRIPTION

[0063] like Figure 1 As shown, a 3D instance segmentation method in a rice field high-density occlusion scene combined with a multimodal fusion network includes:

[0064] (1) Collect multi-view RGB images and three-dimensional point cloud data of rice fields, pre-process the collected data, and form a multi-source data set after the pre-processing;

[0065] (2) The multi-view RGB images in the multi-source dataset are input into the image segmentation model. The image segmentation model processes the multi-view RGB images and generates image instance segmentation results.

[0066] (3) According to the geometric mapping algorithm, the image instance segmentation results are matched with the three-dimensional point cloud data in the multi-source data set, and the pixel coordinates in each image are mapped to the point cloud coordinate system to obtain point cloud data containing multi-view RGB image information;

[0067] (4) The 3D instance segmentation network model based on long sequence point cloud processing is improved to obtain an improved 3D instance segmentation network model. The point cloud data containing multi-view RGB image information is input into the improved 3D instance segmentation network model. The improved 3D instance segmentation network model outputs a 3D instance segmentation result, such as Figure 3 shown.

[0068] like Figure 2 As shown, the step (1) specifically includes the following steps:

[0069] (1a) A depth camera is used to obtain multi-view RGB images. After calibrating the camera parameters, the neural radiation field NeRf is used to reconstruct the 3D structure of rice and extract point cloud data.

[0070] (1b) De-noising the point cloud data and performing color correction on the multi-view RGB images;

[0071] (1c) Downsampling the point cloud data.

[0072] The step (2) specifically comprises the following steps:

[0073] (2a) Inputting the multi-view RGB image into the image segmentation model to generate a segmentation result, and generating multiple masks from different viewpoints according to the segmentation result, corresponding to the rice ear, leaf and stem respectively;

[0074] (2b) All masks are merged into a single grayscale image, and the masks are sorted according to the confidence level. Each mask instance is marked with a different weight. According to the mask instance, for the overlapping area of ​​the multi-view projection, the image instance segmentation result G is calculated by selecting the point closest to the multi-view RGB image plane. m :

[0075]

[0076] Among them, M i is the i-th mask, w i is the corresponding weight, and k is the number of multi-view RGB images.

[0077] The step (3) specifically comprises the following steps:

[0078] (3a) First, calculate the projection matrix, using the external parameter matrix E i and the internal parameter matrix I i , the image instance segmentation result G m Project to the 3D point cloud space and generate the corresponding 3D point cloud label P:

[0079]

[0080] Among them, U, v are the image instance segmentation results G m The pixel coordinates of , D is the depth value;

[0081] (3b) During the projection process, the point cloud coordinates closest to the multi-view RGB image plane are preferentially selected according to the depth value D;

[0082] (3c) After projection, in order to ensure that each point cloud instance has a unique ID, the density-based clustering algorithm DBSCAN is used to cluster the point clouds in the 3D instance segmentation results to obtain the point cloud points belonging to the same instance;

[0083] (3d) To distinguish different instances obtained by clustering, a minimum three-dimensional bounding box is generated according to each clustering instance, and the overlap between instances is calculated using the intersection-over-union ratio:

[0084]

[0085] Where A' and B' represent the three-dimensional bounding boxes of the two instances respectively;

[0086] (3e) When the IoU value of two instances exceeds a preset threshold, the two instances are merged into the same instance to obtain point cloud data containing multi-view RGB image information.

[0087] Through the geometric mapping algorithm, the segmentation information of the image is accurately matched with the three-dimensional point cloud data. Specifically, the segmentation information is used as the prior knowledge of point cloud processing, and combined with the spatial coordinate information of the point cloud, the three-dimensional instance segmentation of rice ears, leaves and stems in the rice field is effectively realized. In order to ensure the accuracy of the projection result, the present invention accurately maps the pixel coordinates in each image to the point cloud coordinate system by using the external parameter matrix and the internal parameter matrix.

[0088] The step (4) specifically includes the following steps: the three-dimensional instance segmentation network model based on long sequence point cloud processing includes a point cloud embedding module, an encoder and a decoder; a fusion processing module is added to the encoder, a long sequence feature extraction module is used to replace the convolution module and the self-attention mechanism module inside the encoder, the long sequence feature extraction module, the self-attention module and the cross attention module form a new decoder, and the decoder is replaced by the new decoder;

[0089] The point cloud data is divided into multiple point cloud blocks of fixed size, and each point cloud block is mapped to a high-dimensional feature vector space. The formula is expressed as:

[0090] X' = ​​MLP(PatchEmbed(X))

[0091] Among them, X is the original point cloud data with labels, X' is the feature representation after embedding; PatchEmbed means converting each point cloud block into a high-dimensional feature vector; MLP is a multi-layer perceptron;

[0092] In the fusion processing module, the neighborhood features of X' are extracted, and the neighborhood features are smoothed using adaptive average pooling to eliminate local random noise; then they are input into the multi-layer perceptron for further processing to enhance the feature expression capability; the features processed by the multi-layer perceptron are fused with X' to obtain the fused feature representation X. fusion :

[0093] X fusion =MLP(AdaptiveAvgPool(KNN(X')))+X'

[0094] Among them, AdaptiveAvgPool represents adaptive average pooling, and KNN represents K-nearest neighbor algorithm;

[0095] The long sequence feature extraction module in the encoder receives X fusion , and extract long sequence dependency information through the state space model, as follows:

[0096] First, X is updated by the state update formula fusion To process:

[0097] h t =Ah t-1 +Bx t

[0098] y t =Ch t +Dx t

[0099] Among them, h t represents the hidden state at the current moment, h t-1 represents the hidden state at the previous moment, x t Indicates the feature representation of the current input, i.e. X fusion ; A, B, C, D are matrix parameters, A and B are responsible for the input x t Perform linear transformation, C and D are used to transform h t Mapping to y t ,y tRepresents the output features of the current moment containing global geometric information;

[0100] In the decoder, the long sequence feature extraction module uses the state space model to extract the encoder output y t To process, the long sequence feature extraction module first updates the state of the features output by the encoder:

[0101] h k =Ah k-1 +Bx k

[0102] Among them, h k Represents the hidden state of the current time step, h k-1 represents the hidden state of the previous time step, x k Represents the output feature of the encoder, i.e., y t ;

[0103] The output features are generated by the following formula:

[0104] y k =Ch k +Dx k

[0105] Among them, y k It is the output feature of the long sequence feature extraction module in the decoder;

[0106] In the self-attention module of the decoder, the output feature y from the long sequence feature extraction module is k Input to the self-attention module, calculate the correlation between each feature, extract y k The context information of self-attention is calculated as:

[0107]

[0108] Where Q is the feature query vector; K is the key vector; V is the value vector; d k is a scaling factor; Attention represents self-attention; softmax represents the softmax function;

[0109] Through the self-attention mechanism, we get the output z t ;

[0110] In the decoder’s criss-cross attention module, the criss-cross attention module accepts the output z from the self-attention module t , the formula for cross attention is:

[0111]

[0112] Among them, Q Cis a randomly initialized matrix and updated via backpropagation during training;

[0113] Generating new feature representations through criss-cross attention modules Will After Softmax and argmax processing, the final three-dimensional rice ear, leaf and stem instance segmentation results, that is, the three-dimensional instance segmentation results, are obtained. The specific calculation formula is:

[0114]

[0115] Label point = arg max(P class )

[0116] Among them, P class is the probability distribution of the category, Label point is the 3D instance segmentation result.

[0117] In summary, the present invention improves the segmentation accuracy in occluded scenes: by fusing multi-perspective RGB images with three-dimensional point cloud data, accurate segmentation in high-density occluded scenes is achieved. Compared with traditional single-modal segmentation methods, the joint processing of images and point clouds can effectively cope with complex rice field environments, especially when rice ears, leaves and stems occlude each other, which significantly improves the accuracy of segmentation; efficient multimodal data fusion is performed: by introducing image segmentation algorithms and combining their results with point cloud data, the present invention achieves effective fusion of multimodal data, and the prior information provided by the image is combined with the spatial features of the point cloud, which effectively improves the robustness and accuracy of instance segmentation, and is particularly suitable for fine-grained segmentation tasks in dense vegetation scenes; the computational complexity is reduced and the processing efficiency is improved: the present invention adopts a long sequence feature extraction module and performs point cloud modeling through an optimized long sequence matrix. Compared with traditional methods, the matrix can capture long-distance dependencies in point clouds while maintaining efficient computing performance.

Claims

1. A 3D instance segmentation method in a rice field high-density occlusion scene combined with a multimodal fusion network, characterized by: The method comprises the following steps in order: (1) Collect multi-view RGB images and three-dimensional point cloud data of rice fields, pre-process the collected data, and form a multi-source data set after the pre-processing; (2) The multi-view RGB images in the multi-source dataset are input into the image segmentation model. The image segmentation model processes the multi-view RGB images and generates image instance segmentation results. (3) According to the geometric mapping algorithm, the image instance segmentation results are matched with the three-dimensional point cloud data in the multi-source data set, and the pixel coordinates in each image are mapped to the point cloud coordinate system to obtain point cloud data containing multi-view RGB image information; (4) The 3D instance segmentation network model based on long sequence point cloud processing is improved to obtain an improved 3D instance segmentation network model, and the point cloud data containing multi-view RGB image information is input into the improved 3D instance segmentation network model, and the improved 3D instance segmentation network model outputs the 3D instance segmentation result.

2. The three-dimensional instance segmentation method in a rice field high-density occlusion scene combined with a multimodal fusion network according to claim 1 is characterized in that: The step (1) specifically comprises the following steps: (1a) A depth camera is used to obtain multi-view RGB images. After calibrating the camera parameters, the neural radiation field NeRf is used to reconstruct the 3D structure of rice and extract point cloud data. (1b) De-noising the point cloud data and performing color correction on the multi-view RGB images; (1c) Downsampling the point cloud data.

3. The three-dimensional instance segmentation method in a rice field high-density occlusion scene combined with a multimodal fusion network according to claim 1, characterized in that: The step (2) specifically comprises the following steps: (2a) Inputting the multi-view RGB image into the image segmentation model to generate a segmentation result, and generating multiple masks from different viewpoints according to the segmentation result, corresponding to the rice ear, leaf and stem respectively; (2b) All masks are merged into a single grayscale image, and the masks are sorted according to the confidence level. Each mask instance is marked with a different weight. According to the mask instance, for the overlapping area of ​​the multi-view projection, the image instance segmentation result G is calculated by selecting the point closest to the multi-view RGB image plane. m : Among them, M i is the i-th mask, w i is the corresponding weight, and k is the number of multi-view RGB images.

4. The 3D instance segmentation method in a rice field high-density occlusion scene combined with a multimodal fusion network according to claim 1, characterized in that: The step (3) specifically comprises the following steps: (3a) First, calculate the projection matrix, using the external parameter matrix E i and the internal parameter matrix I i , project the image instance segmentation result into the three-dimensional point cloud space and generate the corresponding three-dimensional point cloud label P: Among them, U and v are the pixel coordinates of the image instance segmentation result, and D is the depth value; (3b) During the projection process, the point cloud coordinates closest to the multi-view RGB image plane are preferentially selected according to the depth value D; (3c) After projection, in order to ensure that each point cloud instance has a unique ID, the density-based clustering algorithm DBSCAN is used to cluster the point clouds in the 3D instance segmentation results to obtain the point cloud points belonging to the same instance; (3d) To distinguish different instances obtained by clustering, a minimum three-dimensional bounding box is generated according to each clustering instance, and the overlap between instances is calculated using the intersection-over-union ratio: Where A' and B' represent the three-dimensional bounding boxes of the two instances respectively; (3e) When the IoU value of two instances exceeds a preset threshold, the two instances are merged into the same instance to obtain point cloud data containing multi-view RGB image information.

5. The 3D instance segmentation method in a rice field high-density occlusion scene combined with a multimodal fusion network according to claim 1, characterized in that: The step (4) specifically includes the following steps: the three-dimensional instance segmentation network model based on long sequence point cloud processing includes a point cloud embedding module, an encoder and a decoder; a fusion processing module is added to the encoder, a long sequence feature extraction module is used to replace the convolution module and the self-attention mechanism module inside the encoder, the long sequence feature extraction module, the self-attention module and the cross attention module form a new decoder, and the decoder is replaced by the new decoder; The point cloud data is divided into multiple point cloud blocks of fixed size, and each point cloud block is mapped to a high-dimensional feature vector space. The formula is expressed as: X' = ​​MLP(PatchEmbed(X)) Among them, X is the original point cloud data with labels, X' is the feature representation after embedding; PatchEmbed means converting each point cloud block into a high-dimensional feature vector; MLP is a multi-layer perceptron; In the fusion processing module, the neighborhood features of X' are extracted, and the neighborhood features are smoothed using adaptive average pooling to eliminate local random noise; then they are input into the multi-layer perceptron for further processing to enhance the feature expression capability; the features processed by the multi-layer perceptron are fused with X' to obtain the fused feature representation X. fusion : X fusion =MLP(AdaptiveAvgPool(KNN(X')))+X' Among them, AdaptiveAvgPool represents adaptive average pooling, and KNN represents K-nearest neighbor algorithm; The long sequence feature extraction module in the encoder receives X fusion , and extract long sequence dependency information through the state space model, as follows: First, X is updated by the state update formula fusion To process: h t =Ah t-1 +Bx t y t =Ch t +Dx t Among them, h t represents the hidden state at the current moment, h t-1 represents the hidden state at the previous moment, x t Indicates the feature representation of the current input, i.e. X fusion ; A, B, C, D are matrix parameters, A and B are responsible for the input x t Perform linear transformation, C and D are used to transform h t Mapping to y t ,y t Represents the output features of the current moment containing global geometric information; In the decoder, the long sequence feature extraction module uses the state space model to extract the encoder output y t To process, the long sequence feature extraction module first updates the state of the features output by the encoder: h k =Ah k-1 +Bx k Among them, h k Represents the hidden state of the current time step, h k-1 represents the hidden state of the previous time step, x k Represents the output feature of the encoder, i.e., y t ; The output features are generated by the following formula: y k =Ch k +Dx k Among them, y k It is the output feature of the long sequence feature extraction module in the decoder; In the self-attention module of the decoder, the output feature y from the long sequence feature extraction module is k Input to the self-attention module, calculate the correlation between each feature, extract y k The context information of self-attention is calculated as: Where Q is the feature query vector; K is the key vector; V is the value vector; d k is a scaling factor; Attention represents self-attention; softmax represents the softmax function; Through the self-attention mechanism, we get the output z t ; In the decoder’s criss-cross attention module, the criss-cross attention module accepts the output z from the self-attention module t , the formula for cross attention is: Among them, Q C is a randomly initialized matrix and updated via backpropagation during training; Generating new feature representations through criss-cross attention modules Will After Softmax and argmax processing, the final three-dimensional rice ear, leaf and stem instance segmentation results, that is, the three-dimensional instance segmentation results, are obtained. The specific calculation formula is: Label point =argmax(P class ) Among them, P class is the probability distribution of the category, Label point is the 3D instance segmentation result.

Images