Green feed loading information acquisition and identification system and method based on IaPFusion model

Abstract

The invention discloses a green feed loading information acquisition and identification system and method based on an IaPFusion model, and belongs to the technical field of automatic filling and throwing of silage harvesters, and the system comprises a data acquisition module, a communication module, a data processing module and a fusion evaluation module. According to the invention, a deep learning technology is researched and utilized, accompanying vehicle hopper identification research is carried out, on the basis of obtaining two-dimensional and three-dimensional data of a feed box, a double-branch network based on an IaPFusion model is provided, image segmentation and point cloud segmentation technologies are combined, a data fusion method is adopted, automatic identification is carried out on the feed box and the green feed in the green feed loading process, and the identification efficiency of the feed box and the green feed is improved. The scientificity of silage harvesting is further improved, and forage loss is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic filling and throwing of a silage harvester, and in particular to a silage loading information collection and identification system and method based on an IaPFusion model. Background Art

[0002] Silage plants such as silage corn are the main raw materials for green fodder in animal husbandry. Using a self-propelled silage harvester to harvest, chop, throw and collect in the field at one time can reduce labor costs and time costs and improve harvesting efficiency and economic benefits.

[0003] In traditional self-propelled forage harvesters, the direction and landing point of bulk material throwing are generally determined by the driver's experience, or a special observer is assigned to control the rotation of the throwing drum through a hydraulic system.

[0004] Although the traditional bulk material throwing process can realize the collection of bulk materials, the collection of bulk materials can only be realized through the subjective judgment of the driver or observer on the frame of the material box, the falling point of the bulk materials, the loading situation in the material box, and the situation of the bulk materials falling out of the material box, and the real-time detection of the above situations cannot be achieved. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes a green fodder loading information collection and identification system and method based on the IaPFusion model, which combines deep learning technology to real-time identify the position of the feed box, green fodder accumulation and material flow during the green fodder harvesting process.

[0006] The present invention is achieved by adopting the following technical solutions: A green fodder loading information collection and identification system based on the IaPFusion model, comprising: A data acquisition module is used to collect video of the empty feed box, the forage loaded feed box and the full feed box during the operation of the self-propelled silage machine, and transmit the collected information to the communication module; A communication module, used to store the video sent by the data acquisition module and transmit it to the data processing module; The data processing module includes an image processing network and a point cloud processing network, which are used to extract images and point clouds from the collected videos, and to produce image data sets and point cloud data sets respectively after annotation. The above data sets are respectively imported into the image processing network and the point cloud processing network for training, and the weights and target segmentation files output by the model are saved, and the weights of the retrieved training results are transferred to the fusion evaluation module; The fusion evaluation module is used to fuse the image and point cloud recognition effects, set the threshold, and evaluate the detection effects of the two by comparing the intersection ratio with the threshold.

[0007] Furthermore, the data acquisition module includes an information collection box, a fixed bracket and a camera. The information collection box is placed in the cab and connected to the camera through a data cable passing through the fixed bracket. The fixed bracket is detachably placed above the throwing bucket of the self-propelled silage machine. The camera is connected to one end of the fixed bracket and is electrically connected to the communication module for collecting video image information of the empty, working and fully loaded conditions of the material box.

[0008] Furthermore, the preparation of the image dataset includes: using Labelme to perform image segmentation on the upper edge of the feed box, the top of the silage pile in the feed box, and the material flow in the image, and setting the labels as Container, Silage, and Materialflow respectively, establishing a dataset and dividing it into a training set, a validation set, and a test set.

[0009] Furthermore, the image processing network introduces a UIB search module, adds a Mobile MQA attention module, adds a SHViT model that reduces computation and memory consumption, and replaces the YOLOv9 backbone network with a lightweight MobileNetV4 backbone network. Each bottleneck layer in MobileNetV4 includes a depthwise separable convolution and a linear layer, that is, no activation function is performed, and the feature map is directly output. The amount of computation can be expressed as: , The total amount of computation can be obtained by adding up the computation of all layers: , Where L is the number of layers in the network; The number of parameters of the MobileNetV4 backbone network can be obtained by calculating the weight parameters of each layer and summing them up. The number of parameters of the depthwise separable convolution is calculated as: , .

[0010] Furthermore, the production of the point cloud data set includes: preprocessing the point cloud data, manually annotating the preprocessed point cloud data, annotating the upper edge of the material box, the top of the silage pile in the material box and the material flow, and setting the labels to Container, Silage, Materialflow, establishing a data set and dividing it into a training set, a validation set and a test set.

[0011] Furthermore, the preprocessing of the point cloud data includes: extracting single-frame point clouds from the video file multiple times and saving them in pcd format, filtering the point cloud data to remove noise and outliers in the point cloud, flipping the point cloud to ensure that the upper edge plane of the material box is parallel to the xoy plane in the point cloud coordinates, and the rotation transformation can be defined by the rotation matrix around the x, y, and z axes: Rotation around the X axis: , Rotation around the Y axis: , Rotation around the Z axis: , For a point , the new point after rotation P ’ It can be obtained by the following formula: , Where R is the rotation matrix and T is the translation vector, from which the point cloud coordinates corresponding to each point in the point cloud map can be obtained.

[0012] Furthermore, the point cloud processing network uses a residual MLP structure to replace the feature extraction of each local area in the PointNet network. Local features are extracted by residual MLP blocks in each local area of ​​PointNet. Each MLP block contains two layers of MLP, and feature enhancement is achieved through residual connections. Its mathematical expression is that the input feature is x, and the output after two fully connected layers is y. The calculation of the residual block can be expressed as: , Among them, F(x, W) represents the feature extraction function after two layers of MLP, and W is a learnable weight parameter. In the specific implementation, F(x) can be: , where σ is the activation function.

[0013] Furthermore, the point cloud processing network also introduces a geometric radiation module, whose expression formula is: Let the relative position of point p and point q be To obtain, then combine the geometric features with the local features , the final expression is: , F rcs It is the residual MLP structure.

[0014] Furthermore, the method for fusing the image and point cloud recognition effect includes: Retrieve the training results separately, project the point cloud segmentation onto the two-dimensional image plane, obtain the overlapping area with the image segmentation, calculate the overlapping part with the point cloud segmentation area for each segmentation bounding box, use the camera's intrinsic and extrinsic parameters for perspective projection, and calculate the homogeneous coordinates of each three-dimensional point , where (X, Y, Z) are the coordinates of the point in three-dimensional space, and the last line is the expansion of the homogeneous coordinates, which can be expressed as follows when combined with the camera's projection matrix: , Where P is the projection matrix, fx and fy are the focal lengths, (cx, cy) is the optical center position, R is the 3×3 rotation matrix, and t is the 3×1 displacement vector; The final formula for converting these points into pixel coordinates on the image plane can be expressed as , calculate the spatial intersection of the segmented two-dimensional projection area and the YOLO detection box, determine the bounding box of the overlapping area, and for each detected bounding box, calculate the overlapping part with the point cloud segmentation area, and then obtain the IoU value by calculating the ratio of the area of ​​the overlapping area to the area of ​​the two areas. Let A be the image recognition area and B be the area projected to the two-dimensional point cloud recognition. According to the calculation formula of the intersection-over-union ratio IoU: .

[0015] The present invention also provides a method for collecting and identifying green fodder loading information based on the IaPFusion model. The method is implemented based on any one of the systems described above, and the method includes the following steps: Collect video image information of the self-propelled silage machine when the feed box is empty, loaded with forage, and fully loaded; Extract image information and point cloud information from the collected video, and create image datasets and point cloud datasets after annotation; Import the above dataset into the image processing network and point cloud processing network in the IaPFusion model for training, and save the weight and target segmentation files output by the model; Retrieve the training results separately, fuse the image and point cloud recognition effects, set the threshold, compare the intersection over union (IoU) value with the threshold. If the IoU value exceeds the preset threshold, the detection box is considered to match the segmented area. Comprehensively evaluate the performance of the model using the precision, recall, and F1 score indicators, and evaluate the model processing results.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This study uses deep learning technology to carry out research on the identification of accompanying vehicle hoppers, and proposes a dual-branch network of image processing network and point cloud processing network based on the IaPFusion model. On the basis of obtaining two-dimensional and three-dimensional data of the feed box, image segmentation and point cloud segmentation technology are used, and data fusion method is adopted to automatically identify the feed box and green fodder during the green fodder loading process, which further improves the scientific nature of silage harvesting and effectively reduces forage loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a framework diagram of a green fodder loading information collection and identification system based on the IaPFusion model.

[0018] Figure 2 The present invention is a schematic diagram of the structure of a data acquisition module of a green fodder loading information acquisition and identification system based on the IaPFusion model.

[0019] Figure 3 The present invention is a workflow diagram of a method for collecting and identifying green fodder loading information based on the IaPFusion model.

[0020] In the figure: 1. Information collection box; 2. Long tube; 3. Tube clamp fixing plate; 4. Angle adjuster; 5. Short tube; 6. Crab claw type clamping device; 7. Pan head; 8. Camera; 9. Neodymium magnet with hole; 10. Limiting tube clamp; 11. Positioning tube clamp. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the examples of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Although the bulk material can be collected during the throwing process of the traditional self-propelled silage harvester, the collection of the bulk material can only be achieved through the subjective judgment of the driver or observer on the frame of the material box, the falling point of the bulk material, the loading condition in the material box, and the situation of the bulk material falling out of the material box, and the real-time detection of the above situations cannot be achieved; and the automatic following and throwing system of the self-propelled silage harvester during the green fodder harvesting process cannot be detachable, cannot adapt to the automatic following and throwing work of various self-propelled silage harvesters, has a single target recognition principle, and has no target recognition adjustment capability. The target group for the accompanying vehicle material box recognition is single, and it is only applicable to specific accompanying vehicles.

[0023] This study uses deep learning technology to carry out research on accompanying vehicle hopper recognition. On the basis of obtaining two-dimensional and three-dimensional data of the feed box, a dual-branch network based on the IaPFusion model is proposed. Combined with image segmentation and point cloud segmentation technology, a data fusion method is used to automatically identify the feed box and green fodder during green fodder loading, which further improves the scientific nature of silage harvesting and effectively reduces forage loss.

[0024] Example 1: Figure 1 As shown, an embodiment of the present invention provides a green fodder loading information collection and identification system based on the IaPFusion model, including a data collection module, a communication module, a data processing module, and a fusion evaluation module.

[0025] The data acquisition module is used to collect video of the empty feed box, the forage loaded feed box and the full feed box during the operation of the self-propelled silage machine, and transmit the collected information to the communication module.

[0026] like Figure 2 As shown, the data acquisition module includes an information collection box 1, a fixed bracket and a camera 8. The data acquisition module includes an information collection box 1, a fixed bracket and a camera 8. The information collection box 1 is placed in the cab and is connected to the camera 8 through a data cable passing through the fixed bracket. The fixed bracket is detachably placed above the throwing bucket of the self-propelled silage machine. The camera 8 is connected to one end of the fixed bracket and is electrically connected to the communication module for collecting video image information of the empty, working and fully loaded conditions of the material box.

[0027] Among them, the fixed bracket includes a long tube 2, a tube clamp fixing plate 3, an angle adjuster 4, a short tube 5, a crab claw type clamping device 6, a pan head 7, a neodymium magnet with a hole 9, a limiting tube clamp 10 and a positioning tube clamp 11.

[0028] Among them, the camera 8 is the core component of the data acquisition module, which is connected to the pan / tilt 7 and fixed on the crab-claw type clamping device 6. The crab-claw type clamping device 6 is clamped on the top of the short tube 5. The end of the short tube 5 is connected to the long tube 2 through the angle adjuster 4. After the fixed position, it is convenient to adjust the angle to ensure that the end of the throwing bucket will not block the data acquisition device. The short tube 5 is used to support and increase the horizontal height of the camera, so that the camera has a larger field of view. The camera data transmission line passes through the tube and is fixed on the hydraulic oil pipe of the throwing bucket by a rolling belt. The end of the data line is connected to the information collection box 1 to avoid the transmission line from being suspended in the air and hindering the bulk material scattering effect, and at the same time avoid the data transmission line from being entangled with the silage machine throwing barrel.

[0029] Among them, the long tube 2 is clamped and locked by the limiting tube clamp 10 and the positioning tube clamp 11. A rubber pad is installed on the inner side of the limiting tube clamp 10 to reduce the influence of low-frequency vibration on the collection effect, and at the same time increase friction to prevent the long tube 2 from rotating and damaging the overall equipment. In addition to balancing the force points of the clamping position and clamping the long tube 2 with the limiting tube clamp 10, the positioning tube clamp 11 is connected to the tube clamp fixing plate 3, and a rubber pad is installed in the middle to provide buffering. The tube clamp fixing plate 3 is connected to the perforated neodymium magnet 9, which reduces vibration while realizing the flexibility and detachability of the data acquisition module.

[0030] During operation, the data acquisition module is magnetically mounted on the upper end of the throwing barrel to ensure that the end of the guide plate of the throwing barrel does not block the camera's field of view, so that the fixed position of the acquisition device remains unchanged. The angle of the swinging short tube can be adjusted according to the calculation formula: , Where: α is the angle between the short tube and the horizontal, β is the angle between the guide plate and the horizontal, a is the length of the guide plate, and b is the distance between the crab clamping position and the rotation center.

[0031] It should be noted that, under normal circumstances, the angle between the camera normal and the horizontal is 45°-90°. The angle between the camera normal and the horizontal is adjusted according to the following position of the material truck, the height of the material box from the ground, and the height of the throwing bucket from the ground. After fixing, the camera 8 is used to shoot the self-propelled silage machine accompanying the vehicle material box when it is working and save it as an SVO format video. The collected content is to record the entire harvesting process from the time the self-propelled silage machine is ready to harvest. The recording ends when the material truck is fully loaded and leaves the field of view. The complete video is temporarily stored in the information collection box 1, and the file is compressed and transmitted to the terminal via wireless information transmission. Each video is shot first and then saved. During the data transmission process, the collection device starts recording the next operation.

[0032] The communication module is used to store the video sent by the data acquisition module and transmit it to the data processing module.

[0033] Specifically, the communication module includes a server and a mobile terminal. The server is a microcomputer used to receive data information and send it to the terminal for data processing. The terminal is an administrator's computer.

[0034] The data processing module includes an image processing network and a point cloud processing network, which are used to extract images and point clouds from the collected videos, and produce image datasets and point cloud datasets respectively after annotation. The above datasets are respectively imported into the image processing network and the point cloud processing network for training, and the weights and target segmentation files output by the model are saved, and the weights of the training results are transferred to the fusion evaluation module.

[0035] Furthermore, the preparation of the image dataset includes: using Labelme to perform image segmentation on the upper edge of the feed box, the top of the silage pile in the feed box, and the material flow in the picture, and setting the labels as Container, Silage, and Materialflow respectively, using Python to perform data enhancement on the annotated files to improve the generalization performance of the model, and establishing a dataset and dividing it into training set, validation set, and test set.

[0036] Furthermore, the image processing network introduces the UIB search module, adds the Mobile MQA attention module, adds the SHViT model to reduce computation and memory consumption, and replaces the YOLOv9 backbone network with a lightweight MobileNetV4 backbone network. Each bottleneck layer in MobileNetV4 includes a depthwise separable convolution and a linear layer, that is, no activation function is performed, and the feature map is directly output. The amount of computation can be expressed as: , The total amount of computation can be obtained by adding up the computation of all layers: , Where L is the number of layers in the network; The number of parameters of the MobileNetV4 backbone network can be obtained by calculating the weight parameters of each layer and summing them up. The number of parameters of the depthwise separable convolution is calculated as: , .

[0037] It should be noted that the UIB module in the network improves the inverted bottle module and adds two optional deep convolutions. This module integrates several important micro-architectures and introduces a new variant, which provides flexibility in spatial and channel mixing, receptive field expansion, and computational efficiency. The Mobile MQA attention module shares the key values ​​of all heads, which can significantly reduce memory access requirements and thus improve operational intensity. In the early stage of the network model, convolution is used instead of the attention layer, and a single attention head is used instead of a multi-head mechanism, which reduces computational complexity and makes the branch network more lightweight, reduces internal space resource usage, and has low latency for segmentation and recognition of large targets and high accuracy of segmentation and recognition training results.

[0038] Furthermore, the production of the point cloud data set includes: preprocessing the point cloud data, manually annotating the preprocessed point cloud data, annotating the upper edge of the material box, the top of the silage pile in the material box and the material flow, and setting the labels as Container, Silage, Materialflow, establishing a data set and dividing it into a training set, a validation set and a test set.

[0039] Furthermore, the preprocessing of point cloud data includes: extracting single-frame point clouds from video files multiple times and saving them in pcd format, filtering the point cloud data to remove noise and outliers in the point cloud, flipping the point cloud to ensure that the upper edge plane of the material box is parallel to the xoy plane in the point cloud coordinates, and the rotation transformation can be defined by the rotation matrix around the x, y, and z axes: Rotation around the X axis: , Rotation around the Y axis: , Rotation around the Z axis: , For a point , the new point after rotation P ’ It can be obtained by the following formula: , Where R is the rotation matrix and T is the translation vector, from which the point cloud coordinates corresponding to each point in the point cloud map can be obtained.

[0040] Furthermore, the point cloud processing network uses a residual MLP structure to replace the feature extraction of each local area in the PointNet network. In each local area of ​​PointNet, complex local geometric feature extraction is not used, but local features are extracted by residual MLP blocks. Each MLP block contains two layers of MLP, and feature enhancement is achieved through residual connections. Its mathematical expression is that the input feature is x, and the output after two fully connected layers is y. The calculation of the residual block can be expressed as: , Among them, F(x, W) represents the feature extraction function after two layers of MLP, and W is a learnable weight parameter. In the specific implementation, F(x) can be: , where σ is the activation function.

[0041] Furthermore, the point cloud processing branch also introduces a geometric radiation module in the network. The geometric radiation module is used to enhance the geometric feature extraction capability of the point cloud. It is mainly achieved by considering the relative position between points. The feature can be enhanced by introducing the distance or angle between points. The expression formula is: Let the relative position of point p and point q be expressed by To obtain, then combine the geometric features with the local features , the final expression is: , F rcs It is the residual MLP structure.

[0042] At the same time, the residual MLP structure in the point cloud processing network allows the network to learn more complex features by introducing jump connections. This design can effectively alleviate the gradient vanishing problem in deep network training, thereby improving the expressiveness of the model. In the feature extraction process, the local features and geometric features are combined, and the network can capture the details and global information of the point cloud data. This fusion improves the adaptability to different shapes and structures, and has significant efficiency and accuracy for feature extraction of large targets with good light.

[0043] The fusion evaluation module is used to fuse the image and point cloud recognition effects, set the threshold, and evaluate the detection effects of the two by comparing the intersection ratio with the threshold.

[0044] The fusion method of the image and the point cloud recognition effect is: Retrieve the training results separately, project the point cloud segmentation onto the two-dimensional image plane, obtain the overlapping area with the image segmentation, calculate the overlapping part with the point cloud segmentation area for each segmentation bounding box, use the camera's intrinsic and extrinsic parameters for perspective projection, and calculate the homogeneous coordinates of each three-dimensional point , where (X, Y, Z) are the coordinates of the point in three-dimensional space, and the last line is the expansion of the homogeneous coordinates, which can be expressed as follows when combined with the camera's projection matrix: , Where P is the projection matrix, fx and fy are the focal lengths, (cx, cy) is the optical center position, R is the 3×3 rotation matrix, and t is the 3×1 displacement vector; The final formula for converting these points into pixel coordinates on the image plane can be expressed as , the position of the projected point cloud and the segmented area can be accurately located through the camera parameters; the spatial intersection calculation of the segmented two-dimensional projection area and the YOLO detection frame is performed to determine the bounding box of the overlapping area. For each detected bounding box, the overlapping part with the point cloud segmentation area is calculated, and then the IoU value is obtained by calculating the ratio of the area of ​​the overlapping area to the area of ​​the two areas. Let A be the image recognition area and B be the area projected to the two-dimensional point cloud recognition. According to the calculation formula of the intersection-over-union ratio IoU: .

[0045] During the training process, the point cloud segmentation loss and image detection loss are combined to design a composite loss function to optimize the model performance, and data enhancement is performed on the image and point cloud data to improve the generalization ability of the model.

[0046] Example 2: Figure 3 As shown, the embodiment of the present invention also provides a method for collecting and identifying green fodder loading information based on the IaPFusion model, comprising the following steps: S1. Collect video image information of the empty feed box, the feed box loaded with forage, and the full feed box during the operation of the self-propelled silage machine; S2. Extract image information and point cloud information from the collected video data, and create image datasets and point cloud datasets after annotation; S3. Import the above dataset into the image processing network and point cloud processing network in the IaPFusion model for training, and save the weight and target segmentation files output by the model; S4. Retrieve the training results separately, fuse the image and point cloud recognition effects, set the threshold, compare the intersection over union (IoU) value with the threshold. If the IoU value exceeds the preset threshold, it is considered that the detection box matches the segmented area. The performance of the model is evaluated by comprehensive precision, recall, and F1 score indicators, and the model processing results are evaluated.

Claims

1. A green fodder loading information collection and identification system based on the IaPFusion model, characterized in that: include: A data acquisition module is used to collect video of the empty feed box, the forage loaded feed box and the full feed box during the operation of the self-propelled silage machine, and transmit the collected information to the communication module; A communication module, used to store the video sent by the data acquisition module and transmit it to the data processing module; The data processing module includes an image processing network and a point cloud processing network, which are used to extract images and point clouds from the collected videos, and to produce image data sets and point cloud data sets respectively after annotation. The above data sets are respectively imported into the image processing network and the point cloud processing network for training, and the weights and target segmentation files output by the model are saved, and the weights of the retrieved training results are transferred to the fusion evaluation module; The fusion evaluation module is used to fuse the image and point cloud recognition effects, set the threshold, and evaluate the detection effects of the two by comparing the intersection ratio with the threshold.

2. The green fodder loading information collection and identification system based on the IaPFusion model according to claim 1 is characterized in that: The data acquisition module includes an information collection box, a fixed bracket and a camera. The information collection box is placed in the cab and connected to the camera through a data cable passing through the fixed bracket. The fixed bracket is detachably placed above the throwing bucket of the self-propelled silage machine. The camera is connected to one end of the fixed bracket and is electrically connected to the communication module for collecting video image information of the empty, working and fully loaded conditions of the material box.

3. The green fodder loading information collection and identification system based on the IaPFusion model according to claim 1 is characterized in that: The preparation of the image data set includes: using Labelme to perform image segmentation on the upper edge of the feed box, the top of the silage pile in the feed box, and the material flow in the picture, and setting labels as Container, Silage, and Materialflow respectively, establishing a data set and dividing it into a training set, a validation set, and a test set.

4. The green fodder loading information collection and identification system based on the IaPFusion model according to claim 3 is characterized in that: The image processing network introduces the UIB search module, adds the Mobile MQA attention module, adds the SHViT model to reduce computation and memory consumption, and replaces the YOLOv9 backbone network with a lightweight MobileNetV4 backbone network. Each bottleneck layer in MobileNetV4 includes a depth-separable convolution and a linear layer, that is, no activation function is performed, and the feature map is directly output. The computational complexity can be expressed as: ， The total amount of computation can be obtained by adding up the computation of all layers: ， Where L is the number of layers in the network; The number of parameters of the MobileNetV4 backbone network can be obtained by calculating the weight parameters of each layer and summing them up. The number of parameters of the depthwise separable convolution is calculated as: ， 。 5. The green fodder loading information collection and identification system based on the IaPFusion model according to claim 1 is characterized in that: The production of the point cloud data set includes: preprocessing the point cloud data, manually annotating the preprocessed point cloud data, annotating the upper edge of the material box, the top of the silage pile in the material box and the material flow, and setting the labels as Container, Silage, Materialflow, establishing a data set and dividing it into a training set, a validation set and a test set.

6. The green fodder loading information collection and identification system based on the IaPFusion model according to claim 5 is characterized in that: The preprocessing of the point cloud data includes: extracting single-frame point clouds from the video file multiple times and saving them in pcd format, filtering the point cloud data to remove noise and outliers in the point cloud, flipping the point cloud to ensure that the upper edge plane of the material box is parallel to the xoy plane in the point cloud coordinates, and the rotation transformation can be defined by the rotation matrix around the x, y, and z axes: Rotation around the X axis: ， Rotation around the Y axis: ， Rotation around the Z axis: ， For a point , the new point after rotation P ’ It can be obtained by the following formula: ， Where R is the rotation matrix and T is the translation vector, from which the point cloud coordinates corresponding to each point in the point cloud map can be obtained.

7. The green fodder loading information collection and identification system based on the IaPFusion model according to claim 5 is characterized in that: The point cloud processing network uses the residual MLP structure to replace the feature extraction of each local area in the PointNet network. The local features are extracted by the residual MLP block in each local area of ​​PointNet. Each MLP block contains two layers of MLP, and feature enhancement is achieved through residual connection. Its mathematical expression is that the input feature is x, and the output after two fully connected layers is y. The calculation of the residual block can be expressed as: ， Among them, F(x, W) represents the feature extraction function after two layers of MLP, and W is a learnable weight parameter. In the specific implementation, F(x) can be: ， where σ is the activation function.

8. The green fodder loading information collection and identification system based on the IaPFusion model according to claim 7 is characterized in that: The point cloud processing network also introduces a geometric radiation module, whose expression formula is: Let the relative position of point p and point q be To obtain, then combine the geometric features with the local features , the final expression is: ， F rcs It is the residual MLP structure.

9. The green fodder loading information collection and identification system based on the IaPFusion model according to claim 1 is characterized in that: The method for fusing the image and point cloud recognition effects includes: Retrieve the training results separately, project the point cloud segmentation onto the two-dimensional image plane, obtain the overlapping area with the image segmentation, calculate the overlapping part with the point cloud segmentation area for each segmentation bounding box, use the camera's intrinsic and extrinsic parameters for perspective projection, and calculate the homogeneous coordinates of each three-dimensional point , where (X, Y, Z) are the coordinates of the point in three-dimensional space, and the last line is the expansion of the homogeneous coordinates, which can be expressed as follows when combined with the camera's projection matrix: ， Where P is the projection matrix, fx and fy are the focal lengths, (cx, cy) is the optical center position, R is the 3×3 rotation matrix, and t is the 3×1 displacement vector; The final formula for converting these points into pixel coordinates on the image plane can be expressed as , calculate the spatial intersection of the segmented two-dimensional projection area and the YOLO detection box, determine the bounding box of the overlapping area, and for each detected bounding box, calculate the overlapping part with the point cloud segmentation area, and then obtain the IoU value by calculating the ratio of the area of ​​the overlapping area to the area of ​​the two areas. Let A be the image recognition area and B be the area projected to the two-dimensional point cloud recognition. According to the calculation formula of the intersection-over-union ratio IoU: 。 10. A method for collecting and identifying green fodder loading information based on the IaPFusion model, characterized in that: The method is implemented based on the system according to any one of claims 1 to 9, and the method comprises the following steps: Collect video image information of the self-propelled silage machine when the feed box is empty, loaded with forage, and fully loaded; Extract image information and point cloud information from the collected video data, and create image datasets and point cloud datasets after annotation; Import the above dataset into the image processing network and point cloud processing network in the IaPFusion model for training, and save the weight and target segmentation files output by the model; Retrieve the training results separately, fuse the image and point cloud recognition effects, set the threshold, compare the intersection over union (IoU) value with the threshold. If the IoU value exceeds the preset threshold, the detection box is considered to match the segmented area. Comprehensively evaluate the performance of the model using the precision, recall, and F1 score indicators, and evaluate the model processing results.

Images