A pig posture detection method based on fusion of RGB image and depth image

Abstract

The application discloses a pig posture detection method based on RGB image and depth image fusion, relates to the field of intelligent livestock breeding technology, and comprises the following steps: acquiring an RGB image and a depth image of group-raised pigs under the same field of view, determining a cross-modal alignment offset, obtaining a depth information matrix corresponding to the RGB image, respectively extracting features of the RGB image and the depth information matrix, obtaining an RGB feature map and a depth feature map, and fusing the RGB feature map and the depth feature map to obtain a multi-modal fusion feature map. The pig posture detection method based on RGB image and depth image fusion improves the integrity and reliability of the pig posture detection result in a shielding scene, and provides more accurate data basis for subsequent standing, lying, feeding, fighting, abnormal stillness and other behavior recognition, health monitoring, disease early warning and fine breeding management.

Description

Technical Field

[0001] This invention relates to the field of intelligent livestock farming technology, specifically to a method for pig pose detection based on the fusion of RGB images and depth images. Background Technology

[0002] As modern animal husbandry develops towards large-scale, refined, and intelligent operations, precision farming and intelligent management are gradually becoming important technological means to improve farming efficiency, reduce labor costs, and enhance animal welfare. In large-scale pig farming, real-time acquisition of pigs' health status, behavioral patterns, and growth and development through visual monitoring technology can provide data support for disease early warning, feeding strategy optimization, reproductive management, and abnormal behavior identification. Therefore, image- or video-based pig individual recognition, pose estimation, and behavior analysis technologies have significant application value in the field of smart farming.

[0003] In existing technologies, RGB images, depth images, or RGB-D images are typically used for target detection, individual segmentation, and keypoint recognition in pig posture detection. RGB images can reflect the apparent information such as the pig's surface texture, color, and contour edges, while depth images can reflect the spatial distance and three-dimensional structural information between the pig's body parts and the acquisition device. Both of these types of information play a role in pig posture recognition, achieving a certain degree of posture detection in scenarios with a small number of pigs, minimal occlusion between individuals, and a relatively stable acquisition environment. However, in actual group-house pig environments, multiple pigs are often in a state of frequent movement, close contact, and crowding, easily leading to body overlap, contour intersection, and limb occlusion. When relying solely on RGB images for posture detection, it is easily affected by factors such as changes in lighting, similar body colors, fecal contamination, and missing texture in occluded areas, making it difficult to distinguish the body boundaries of adjacent pigs. When relying solely on depth images for posture detection, it is easily affected by factors such as depth noise, local depth loss, and the close spatial position of adjacent pigs, causing body parts of different pigs to be misidentified as the same target area. Furthermore, in existing solutions that combine RGB and depth images, issues such as viewpoint deviation, pixel misalignment, or inconsistent edges between the two types of images can lead to inaccurate correspondence between apparent and spatial information, affecting subsequent feature fusion, overlapping region identification, and individual boundary separation. Particularly in areas occluded by pigs, the visible contours in the RGB image and the depth variations in the depth image are not always perfectly aligned. If the reliability of these two types of information in different regions cannot be effectively assessed, fusion feature distortion can easily occur, causing confusion regarding pig boundaries in overlapping areas. Further, in severe occlusion scenarios, the head, legs, torso edges, or joints of some pigs may be completely occluded by other pigs. For example, in a feeding area, when multiple pigs gather simultaneously, the bodies of pigs in the front row may obscure the heads and forelimbs of pigs in the back row; in a resting area, when multiple pigs lie overlapping, the leg or abdominal contours of some pigs may not be fully observable. For such occluded areas, existing methods typically rely on limited visible contours or local depth information for judgment, making it difficult to reliably infer the spatial location of occluded joints and the complete limb structure. Since the results of pig posture detection depend on accurate individual boundaries, body contours and joint positions, inaccurate identification of overlapping areas, incorrect segmentation of individual boundaries or unreliable positioning of occluded joints will directly affect the identification results of behavioral states such as standing, lying down, feeding, fighting, and abnormal stillness, thereby reducing the accuracy of health monitoring, disease early warning and refined management. Summary of the Invention

[0004] The purpose of this invention is to provide a pig pose detection method based on the fusion of RGB images and depth images, thereby solving the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a pig pose detection method based on the fusion of RGB images and depth images, comprising: S1, acquiring RGB images and depth images of group-housed pigs under the same field of view, determining cross-modal alignment offset, and obtaining a depth information matrix corresponding to the RGB images; S2, extracting features from the RGB images and depth information matrix respectively, obtaining RGB feature maps and depth feature maps, and fusing the RGB feature maps and depth feature maps to obtain a multimodal fusion feature map; S3, calculating the overlap confidence of pixels belonging to the overlapping region of pigs based on the apparent similarity, depth difference, and spatial adjacency relationship between pixels in the multimodal fusion feature map, and if the overlap confidence is greater than a preset overlap threshold, determining that the corresponding pixel belongs to the overlapping region, and obtaining an overlapping region mask containing multiple pigs; S4, for the multiple overlapping regions corresponding to the overlapping region mask... The modal fusion feature map extracts the apparent gradient change features and depth gradient change features within the overlapping area to determine the individual boundaries of each pig within the overlapping area; S5, the visible body contour within the individual boundary is obtained, and the matching degree between the visible body contour and the pre-established complete pig morphology template is calculated. If the matching degree is less than the preset matching threshold, it is determined that there is limb occlusion in the visible body contour, and the missing limb region is obtained; S6, for the missing limb region, the three-dimensional coordinates of the visible joints are determined according to the depth information matrix, and the topological relationship between the visible joints in the missing limb region is extracted. The three-dimensional coordinates of the occluded joints are predicted according to the three-dimensional coordinates of the visible joints and the topological relationship to obtain the spatial position of the occluded joints; S7, according to the spatial position of the occluded joints and the three-dimensional coordinates of the visible joints, the complete skeleton structure of the pig corresponding to the missing limb region is reconstructed to obtain the posture detection results of each pig.

[0006] Preferably, step S1 includes acquiring RGB images and depth images of group-raised pigs under the same field of view; processing the RGB images to obtain a color contour feature set; extracting boundary continuity features based on the color contour feature set and extracting a depth transition feature set from the depth image; calculating a transition consistency distribution map based on the boundary continuity features and the depth transition feature set; if the local extrema of the transition consistency distribution map are greater than a preset threshold, determining a cross-modal alignment offset, and using the cross-modal alignment offset for pixel-level alignment to obtain a depth information matrix corresponding to the RGB images.

[0007] Preferably, step S2 includes acquiring an RGB image of a pig and a depth information matrix, extracting RGB feature maps and depth feature maps through a convolutional neural network; extracting occluded areas of the pig from the RGB feature maps and depth feature maps, determining the degree of texture loss based on the color features of the occluded areas, and obtaining the depth boundary clarity based on the depth features of the occluded areas; determining the color modality confidence and depth modality confidence based on the degree of texture loss and the depth boundary clarity; and performing weighted fusion of the RGB feature maps and depth feature maps based on the color modality confidence and depth modality confidence to obtain a multimodal fusion feature map.

[0008] Preferably, step S3 includes acquiring a multimodal fusion feature map, calculating the apparent similarity and depth difference of pixels based on the multimodal fusion feature map; extracting spatial adjacency relationships based on the apparent similarity and depth difference, calculating the overlap confidence of pixels based on the spatial adjacency relationships; if the overlap confidence is greater than a preset overlap threshold, extracting the corresponding pixels to generate an initial overlapping region; and performing connected component labeling on the initial overlapping region to obtain an overlapping region mask containing multiple pigs.

[0009] Preferably, step S4 includes obtaining a multimodal fusion feature map corresponding to the mask of the overlapping region; processing the multimodal fusion feature map using an edge detection operator to obtain an initial gradient matrix; extracting apparent gradient change features and depth gradient change features based on the initial gradient matrix to obtain a fusion gradient magnitude map; removing non-edge pixels from the fusion gradient magnitude map to obtain a refined edge feature map; generating a candidate boundary map containing strong edge points and weak edge points based on the refined edge feature map; and performing connected component analysis based on the strong edge points and weak edge points in the candidate boundary map to determine the individual boundaries of each pig within the overlapping region.

[0010] Preferably, step S5 includes extracting the visible body contour based on the individual boundary and constructing a target topology map of the visible body contour; performing spatial mapping between the target topology map and the template topology map of a pre-established complete pig morphology template to obtain an aligned contour map; comparing the aligned contour map with the complete pig morphology template to obtain a set of differing pixels and a matching degree; if the matching degree is less than a preset matching threshold, it is determined that there is limb occlusion in the visible body contour, and the missing limb region is extracted based on the set of differing pixels.

[0011] Preferably, step S6 includes extracting depth pixel values ​​corresponding to the missing limb region from the depth information matrix, determining the three-dimensional coordinates of the visible joints based on the depth pixel values, using a graph convolutional network to extract feature vectors from the skeleton node graph constructed based on the three-dimensional coordinates, and generating an adjacency matrix based on the feature vectors to determine the topological relationship between the visible joints.

[0012] Preferably, step S6 further includes constructing a coordinate prediction model based on three-dimensional coordinates and topological relationships, predicting the three-dimensional coordinates of the occluded joint through the coordinate prediction model, and obtaining the spatial position of the occluded joint.

[0013] Preferably, step S7 includes obtaining an initial depth value of the obscured joint, the initial depth value being extracted from the region of the missing limb; and constructing a three-dimensional spatial vector based on the initial depth value and the three-dimensional coordinates of the visible joint to correct the spatial position of the obscured joint.

[0014] Preferably, step S7 further includes reconstructing the complete skeleton structure based on the spatial position of the occluded joints and the three-dimensional coordinates of the visible joints; extracting the skeletal spatial features of the complete skeleton structure; and obtaining the posture detection results of each pig based on the skeletal spatial features.

[0015] As can be seen from the above technical solution, the present invention has the following beneficial effects: This pig pose detection method based on the fusion of RGB and depth images acquires RGB and depth images of group-housed pigs under the same field of view. It utilizes the boundary continuity of the visible contours of the pigs and the consistency of depth transitions for pixel-level alignment, ensuring accurate correspondence between the appearance information in the RGB images and the spatial information in the depth images. A convolutional neural network extracts RGB and depth features separately, and the modal confidence is determined based on the degree of texture loss and depth boundary clarity within the occluded areas of the pigs before fusion. This reduces recognition errors caused by variations in illumination, texture loss, depth noise, or local depth loss in single image information. The method calculates the overlap confidence of pixels belonging to overlapping areas of pigs and combines this with the appearance gradient. Degree variation features and depth gradient variation features determine the individual boundaries of each pig, which can improve the accuracy of individual separation when multiple pigs are crowded together, have overlapping outlines, and have overlapping bodies in group rearing scenarios. At the same time, by using a complete pig morphology template to determine whether there are limb occlusions on the visible body outline, and by using the three-dimensional coordinates of visible joints and their topological relationships to predict the spatial position of occluded joints, the complete skeletal structure of the pig can be reconstructed even when parts of the head, legs, torso edges, or joints are not visible. This improves the completeness and reliability of pig posture detection results in occluded scenarios, and provides a more accurate data foundation for subsequent recognition of behaviors such as standing, lying down, feeding, fighting, and abnormal stillness, as well as health monitoring, disease early warning, and refined breeding management. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention; Figure 2 This is a timing diagram of the present invention; Figure 3 This is a system block diagram of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: As Figure 1 and Figure 3 As shown, the present invention provides a technical solution: a method for pig pose detection based on the fusion of RGB images and depth images, comprising: S1. Obtain RGB and depth images of group-raised pigs under the same field of view. Based on the boundary continuity of the visible outline of the pigs in the RGB image and the consistency of the depth transition in the depth image, determine the cross-modal alignment offset and perform pixel-level alignment on the RGB and depth images to obtain the depth information matrix corresponding to the RGB image. S2. Use a convolutional neural network to extract features from the RGB image and the depth information matrix to obtain RGB feature maps and depth feature maps. Determine the modal credibility based on the degree of texture loss and the clarity of the depth boundary in the pig's occluded area. Then fuse the RGB feature maps and the depth feature maps to obtain a multimodal fusion feature map. S3. Based on the apparent similarity, depth difference and spatial adjacency relationship between pixels in the multimodal fusion feature map, calculate the overlap confidence of the pixel belonging to the overlapping region of pigs. If the overlap confidence is greater than the preset overlap threshold, the corresponding pixel is determined to belong to the overlapping region, and the overlapping region mask containing multiple pigs is obtained. S4. For the multimodal fusion feature map corresponding to the mask of the overlapping region, the edge detection operator is used to extract the apparent gradient change features and depth gradient change features in the overlapping region. Based on the apparent gradient change features and depth gradient change features, the individual boundary of each pig in the overlapping region is determined. S5. Obtain the visible body outline within the individual boundary, calculate the matching degree between the visible body outline and the pre-established complete pig morphology template. If the matching degree is less than the preset matching threshold, it is determined that there is limb occlusion in the visible body outline, and the incomplete limb area is obtained. S6. For the missing limb area, determine the three-dimensional coordinates of the visible joints based on the depth information matrix, and extract the topological relationship between the visible joints in the missing limb area through a graph convolutional network. Based on the three-dimensional coordinates of the visible joints and the topological relationship, predict the three-dimensional coordinates of the occluded joints and obtain the spatial position of the occluded joints. S7. Based on the spatial location of the obscured joints and the three-dimensional coordinates of the visible joints, reconstruct the complete skeletal structure of the pig corresponding to the missing limb area, and obtain the posture detection results of each pig based on the complete skeletal structure.

[0019] In one possible embodiment, the image acquisition device includes an RGB image acquisition module and a depth image acquisition module. The acquisition range of the RGB image acquisition module and the depth image acquisition module covers the same pig pen area, and acquires RGB images and depth images of the group-housed pigs at the same acquisition time.

[0020] Among them, RGB images are used to provide information on the color, texture, outline and apparent edge of the pigs, while depth images are used to provide information on the distance between the pig's body surface and the image acquisition device, as well as spatial hierarchy information between different pigs.

[0021] Furthermore, in S1, cross-modal alignment is performed on the RGB image and depth image acquired under the same field of view; due to differences in installation position, imaging angle, focal length, resolution or optical center between the RGB image acquisition module and the depth image acquisition module, the pixel position of the same pig in the RGB image and depth image is offset.

[0022] To eliminate the offset, the visible outline of the pig is first extracted from the RGB image. The pig's body edge is obtained through grayscale processing, filtering, edge detection, and contour tracking. The boundary continuity is determined based on whether adjacent contour points meet a preset distance threshold. At the same time, depth change regions are extracted from the depth image. The depth transition regions at the pig's edge are obtained by calculating the depth difference or depth gradient between adjacent pixels. The consistency of depth transition is determined based on whether the depth transition direction is consistent, whether the depth transition amplitude reaches a preset depth transition threshold, and whether the depth transition boundary length reaches a preset boundary length threshold.

[0023] Subsequently, the continuous boundaries in the RGB image are matched with the depth transition boundaries in the depth image, and the lateral and vertical offsets between them are calculated. These lateral and vertical offsets are then used as cross-modal alignment offsets. Based on these cross-modal alignment offsets, coordinate mapping and interpolation resampling are performed on the depth image to make the depth values ​​in the depth image correspond to the pixel positions in the RGB image, thereby obtaining a depth information matrix corresponding to the RGB image. The elements in this depth information matrix are used to represent the depth values ​​at the corresponding pixel positions in the RGB image.

[0024] Furthermore, in S2, the RGB image and depth information matrix are respectively input into the convolutional neural network for feature extraction.

[0025] Specifically, after passing through convolutional layers, normalization layers, activation layers, and pooling layers, the RGB image yields an RGB feature map containing color, texture, contour, and local appearance structure; after passing through the depth feature extraction branch, the depth information matrix yields a depth feature map containing depth variation, spatial distance, body surface undulations, and depth boundaries between individuals.

[0026] Specifically, for occluded regions, the system calculates the local variance of texture, edge continuity length, and color gradient magnitude in the RGB image, and compares the calculation results with a preset texture loss judgment threshold to determine the RGB modality reliability. Simultaneously, the system calculates the depth hole area, depth gradient magnitude, and depth boundary connectivity length in the depth image, and compares the calculation results with a preset depth boundary judgment threshold to determine the depth modality reliability. Based on the RGB modality reliability and the depth modality reliability, the RGB feature map and the depth feature map are weighted fusion, channel splicing fusion, or attention fusion to obtain a multimodal fusion feature map.

[0027] Furthermore, in S3, overlapping areas of pigs are determined based on the multimodal fusion feature map; for pixels or local pixel blocks in the multimodal fusion feature map, the system calculates the apparent similarity, depth difference and spatial adjacency relationship between them and their neighboring pixels.

[0028] Among them, apparent similarity is used to reflect the degree of closeness of adjacent regions in terms of color, texture and contour features; depth difference is used to reflect the hierarchical changes in spatial distance between adjacent regions; spatial adjacency is used to reflect the connectivity state between pixels or pixel blocks on the image plane; when a region satisfies the connectivity condition on the image plane and the depth difference reaches the preset depth difference threshold, or the fusion feature difference reaches the preset fusion feature difference threshold, the system calculates the overlap confidence of the pixels in the region belonging to the overlapping region of pigs; when the overlap confidence is greater than the preset overlap threshold, the corresponding pixel is determined to be an overlapping region pixel; after performing connected component analysis, boundary filling and mask generation on all overlapping region pixels, the overlapping region mask containing multiple pigs is obtained.

[0029] In this embodiment, in S4, the individual boundaries of each pig within the overlapping area are determined for the area defined by the overlapping area mask.

[0030] Specifically, edge detection is performed on overlapping regions in the multimodal fusion feature map, and Sobel operator, Canny operator, Laplacian operator or other edge detection operators are used to extract apparent gradient change features and depth gradient change features. Apparent gradient change features are used to represent the changes in color, texture and brightness of pig body surface in local areas. Depth gradient change features are used to represent the distance changes between different pig body surfaces or between pig body and background.

[0031] Specifically, when the apparent gradient magnitude reaches a preset apparent gradient threshold and the depth gradient magnitude reaches a preset depth gradient threshold, and both satisfy a preset spatial consistency condition within the same pixel neighborhood, the location is designated as a candidate location for the individual boundary. When the apparent gradient magnitude does not reach the preset apparent gradient threshold, but the depth gradient magnitude does, the candidate location for the individual boundary is determined based on the depth gradient. When the depth gradient magnitude does not reach the preset depth gradient threshold, but the apparent gradient magnitude reaches the preset apparent gradient threshold, the candidate location for the individual boundary is determined based on the apparent gradient.

[0032] Subsequently, continuity constraints and connectivity corrections were applied to the candidate locations of individual boundaries to obtain the individual boundaries corresponding to each pig within the overlapping region.

[0033] In a preferred embodiment S5, after determining the individual boundary, the visible body outline within the individual boundary is extracted, and the visible body outline is matched with a pre-established complete pig morphology template.

[0034] Specifically, the complete pig morphology template is established based on the complete body outline of a pig in a normal standing, walking, lying down, or side-lying posture. The template includes the trunk outline, head outline, limb positions, back curve, abdominal curve, and the relative positional relationship between major joints. During matching, the system calculates the outline overlap, shape similarity, key part proportion consistency, and template deformation error between the visible body outline and the complete pig morphology template, and obtains the matching degree based on the above calculation results.

[0035] Specifically, when the matching degree is less than the preset matching threshold, it is determined that the currently visible body outline has a missing outline relative to the complete pig form template, the pig is identified as having limb occlusion, and the area corresponding to the missing outline is identified as the incomplete limb area.

[0036] Furthermore, in S6, for the missing limb area, the three-dimensional coordinates of the visible joints are determined by combining the depth information matrix.

[0037] Specifically, the two-dimensional image coordinates of visible joints are first identified within the individual boundary. Then, the depth value of the corresponding pixel position is read from the depth information matrix. Combined with the imaging parameters of the image acquisition device or the preset coordinate transformation relationship, the two-dimensional image coordinates and depth values ​​of the visible joints are converted into three-dimensional spatial coordinates.

[0038] Then, a joint topology graph is constructed using visible joints as graph nodes and the connection relationships between joints in the pig skeleton as graph edges. The topological relationships between visible joints are then extracted using a graph convolutional network. Based on the spatial location of visible joints, the connection relationships between adjacent joints, the distance constraints between joints, and the prior knowledge of the pig skeleton structure, the graph convolutional network predicts the three-dimensional coordinates of the occluded joints, thereby obtaining the spatial location of the occluded joints.

[0039] Furthermore, in S7, the predicted spatial location of the occluded joint is combined with the three-dimensional coordinates of the visible joint, and the trunk, head and limb joints are connected according to the topological relationship of the pig skeleton to reconstruct the complete skeletal structure of the pig corresponding to the missing limb area.

[0040] Specifically, the reconstructed complete skeleton structure is used to represent the pig's posture in three-dimensional space, including the positional relationship of each joint, limb orientation, and body posture. Based on this complete skeleton structure, the posture detection results of each pig are output, including the individual pig boundary, joint coordinates, skeleton connection lines, posture category, or posture visualization image.

[0041] Through the above process, this method utilizes the complementary appearance information of RGB images and the spatial information of depth images to achieve individual separation, occlusion joint prediction, and pose detection of group-raised pigs in occluded and overlapping scenarios.

[0042] To verify this embodiment, a comparative experiment on the posture detection of group-housed pigs was also set up. The experimental scenario was a closed pigpen, and the image acquisition device was installed above the pigpen to acquire RGB images and depth images under the same field of view. The experimental samples included situations such as unobstructed pigs, pigs crowding each other, pigs with overlapping bodies, pigs with limb occlusion, and pigs with partially invisible joints. During the experiment, the RGB images, depth images, individual pig boundaries, overlapping areas, visible joints, occluded joints, and posture categories were manually labeled as evaluation benchmarks.

[0043] The experiment compared three methods: Method 1 was a method for pig pose detection using only RGB images; Method 2 was a method for pose detection by directly stitching RGB images and depth images; Method 3 was the method proposed in this application, which first performs cross-modal pixel-level alignment, then performs credibility fusion based on the degree of texture loss and the clarity of depth boundaries, and combines overlapping region segmentation, morphological template matching, and occlusion joint prediction to output a complete skeleton structure; all three methods used the same training and testing samples, and no manual correction was performed on the manually labeled results during the testing phase.

[0044] The experimental procedure is as follows: 1. Collect RGB and depth images of group-raised pigs under the same field of view, and complete time synchronization and coordinate calibration; after collecting images, remove samples with blurred images, invalid depth images, and pigs that have completely left the field of view.

[0045] 2. Manually label the test samples. The labeling content includes the overlapping area mask of pigs, the individual boundary of each pig, the position of visible joints, the position of occluded joints, and the posture category, which includes standing, lying down, lying on one's side, and walking.

[0046] 3. The same test set is processed using Method 1, Method 2 and the method of this application respectively, and the overlapping area recognition results, individual boundaries, complete skeleton structure and posture category of each pig are output.

[0047] 4. Calculate evaluation indicators based on manual annotation results. The accuracy of overlapping region identification is evaluated by the F1 value of overlapping regions; the individual segmentation effect is evaluated by the average intersection-union ratio of individual boundaries; the occluded joint prediction effect is evaluated by the average positioning error of occluded joints; the skeleton completion effect is evaluated by the skeleton integrity rate; and the posture detection effect is evaluated by the posture category accuracy rate.

[0048] 5. Compare the experimental results of different methods to verify the detection effect of the method of this application under the conditions of overlapping, occlusion and invisible joints of group-raised pigs.

[0049] Table 1 Experimental Sample Setup

[0050] As shown in Table 1, the test samples cover the overlapping areas, occluded joints, and posture categories in the detection of group-raised pigs, which can be used to verify the detection effect of the method in complex farming scenarios.

[0051] Table 2 Comparison of detection results from different methods

[0052] Table 2 shows that the RGB image method alone has errors in detecting overlapping areas and individual boundaries when pigs have similar body surface colors, overlapping contours, and limb occlusion. The RGB and depth direct stitching method introduces spatial information and improves the detection results, but it does not specifically address modal reliability and occluded joints, and is still affected by depth noise and local depth loss. The method in this application, through pixel-level alignment, reliability fusion, overlapping area identification, and occluded joint prediction, achieves an F1 value of 0.902 for overlapping areas, an average intersection-union ratio of 0.872 for individual boundaries, reduces the average error of occluded joints to 5.4 cm, and achieves a pose accuracy of 93.5%. These results indicate that the method in this application can improve the completeness and accuracy of pose detection in group-housed pigs.

[0053] Table 3 Ablation test results of the method in this application

[0054] As shown in Table 3, after removing pixel-level alignment, the correspondence between the contour positions in the RGB image and the spatial boundaries in the depth image weakens, and the average intersection-union ratio of individual boundaries drops to 0.825. After removing modal confidence fusion, color features and depth features cannot be weighted according to the texture state and depth boundary state of the occluded area, and the average error of occluded joints increases to 6.4 cm. After removing occluded joint prediction, although the boundaries of the visible body area can still be obtained, the joints corresponding to the occluded limbs cannot be completed, and the skeleton integrity rate drops to 82.1%. The complete method of this application can simultaneously maintain the accuracy of individual boundary segmentation and the skeleton completion effect, indicating that cross-modal alignment, confidence fusion, and occluded joint prediction all make practical contributions to the pose detection effect of this application.

[0055] In summary, the experimental results show that the method of this application can complement the appearance information of RGB images and the spatial information of depth images, thereby improving the accuracy of individual pig separation, the accuracy of occluded joint prediction, and the completeness of posture detection in group rearing scenarios where pigs are crowded together, their bodies overlap, and their limbs are partially occluded. These experimental results support the application of this application in pig behavior recognition, health monitoring, disease early warning, and refined breeding management.

[0056] Example 2: S1 includes acquiring RGB images and depth images of group-raised pigs under the same field of view. The RGB images are processed to obtain a color contour feature set. Boundary continuity features are extracted from the color contour feature set, and a depth transition feature set is extracted from the depth image. A transition consistency distribution map is calculated based on the boundary continuity features and the depth transition feature set. If the local extrema of the transition consistency distribution map are greater than a preset threshold, a cross-modal alignment offset is determined, and pixel-level alignment is performed using the cross-modal alignment offset to obtain a depth information matrix corresponding to the RGB images.

[0057] In one possible embodiment, the RGB image and the depth image are acquired by an image acquisition device that is fixedly installed at the same acquisition location. Before acquisition, the RGB acquisition module and the depth acquisition module are synchronized in time so that they can acquire image data within the same pigpen field of view at the same acquisition time.

[0058] The RGB image stores the red, green, and blue channel values ​​of each pixel, while the depth image stores the distance value corresponding to each pixel.

[0059] Furthermore, after acquiring the RGB image, a smoothing filter is applied to the RGB image to eliminate the influence of isolated noise points on contour extraction; then the color difference between each pixel and its 8 neighboring pixels is calculated.

[0060] Specifically, the color difference is calculated as follows: the difference between the current pixel and the neighboring pixels in the red, green and blue channels is calculated respectively, and the absolute values ​​of the differences in the three channels are added together to obtain the color difference value of the current pixel relative to the neighboring pixel; among the color difference values ​​obtained in the eight neighborhood directions, the result with the largest value is taken as the color contour response value of the current pixel.

[0061] Specifically, when the color contour response value is greater than the color contour threshold, the current pixel is determined as a candidate pixel for the color contour. The color contour threshold is determined by a calibration image, in which the boundary pixels and non-boundary pixels of the pig's body are pre-labeled. Different candidate thresholds are judged separately, and the number of unrecognized boundary pixels and the number of misrecognized non-boundary pixels are counted. The two are added together to obtain the total number of classification errors. The candidate threshold corresponding to the minimum value of the total number of classification errors is taken as the color contour threshold.

[0062] Furthermore, after obtaining the candidate pixels of the color contour, connected component analysis is performed on the candidate pixels of the color contour.

[0063] Specifically, for adjacent color contour candidate pixels, if the pixel distance between them in the image does not exceed the preset adjacency distance threshold, they are classified into the same connected component. The adjacency distance threshold is determined according to the resolution of the image acquisition device. After installation and calibration are completed, the pixel interval between adjacent effective edge points of the pig contour in the RGB image is selected as the adjacency distance threshold. For each connected component, the number of pixels, endpoint positions, contour direction, and color contour response value are calculated, and connected components with a number of pixels less than the connected component number threshold are deleted.

[0064] The connectivity threshold is determined based on the minimum identifiable width of a pig's limb in the RGB image. Specifically, the number of pixels corresponding to continuous segments of the pig's leg boundary is measured in the calibration image, and half of this number is rounded down to the nearest integer as the connectivity threshold. The retained connected components form a set of color contour features.

[0065] Furthermore, based on the color contour feature set, boundary continuity features are extracted; contour pixels within the same connected region are sorted according to their adjacency relationship, and the distance and direction changes between two adjacent contour pixels are calculated.

[0066] Specifically, when the pixel distance between two adjacent contour pixels does not exceed the adjacency distance threshold and the angle between their contour directions does not exceed the direction change threshold, the two contour pixels are identified as continuous boundary points. The direction change threshold is determined by the calibration image, in which continuous boundary segments of the same pig's body contour are pre-labeled. Boundary connections are performed for different candidate direction thresholds, and the number of manually labeled continuous boundary segments that are incorrectly broken is counted. The candidate direction threshold corresponding to the minimum number of incorrect breaks is taken as the direction change threshold.

[0067] By connecting the contour pixels sequentially according to the above rules, a continuous boundary point sequence is obtained. The continuous boundary point sequence, boundary endpoints, boundary length, and boundary direction together constitute the boundary continuity feature.

[0068] In this embodiment, when processing the depth image, invalid depth pixels are first removed.

[0069] Specifically, invalid depth pixels include pixels with a depth value of 0 and pixels with a depth value exceeding the range of the depth acquisition module; for valid depth pixels, the depth difference between them and their horizontally adjacent pixels, vertically adjacent pixels, and two diagonally adjacent pixels is calculated; the depth difference in each direction is obtained by subtracting the depth value of the current pixel from the depth value of its corresponding neighboring pixels and taking the absolute value; among the depth differences in the four directions, the result with the largest value is taken as the depth transition response value of the current pixel; when the depth transition response value is greater than the depth transition threshold, the current pixel is determined to be a depth transition candidate pixel.

[0070] Specifically, the depth transition threshold is determined by a depth calibration image, in which pig boundary pixel pairs and non-boundary pixel pairs are pre-labeled. Different candidate thresholds are judged separately, and the number of unrecognized pig boundary pixel pairs and the number of misrecognized non-boundary pixel pairs are counted. The two are added together to obtain the total number of classification errors. The candidate threshold corresponding to the minimum value of the total number of classification errors is taken as the depth transition threshold.

[0071] Furthermore, after obtaining the candidate pixels for the depth transition, connected component analysis is performed on them, and the number of pixels, depth transition direction, depth transition response value and endpoint position of each connected component are calculated.

[0072] Specifically, the direction of the depth transition is determined by the direction of the neighborhood that produces the maximum depth difference; if the number of pixels in a certain depth transition connected region is less than the number of transitions threshold, the connected region is deleted; the number of transitions threshold is determined based on the shortest effective continuous segment of the pig's body boundary in the depth image, specifically: measure the number of pixels corresponding to the continuous segment of the manually labeled pig boundary in the depth calibration image, take half of the number of pixels and round it down, and use it as the number of transitions threshold. The retained depth transition connected regions constitute the depth transition feature set.

[0073] Furthermore, after obtaining the boundary continuity features and depth transition feature sets, the transition consistency distribution map is calculated; firstly, the offset search range is determined, which is obtained by converting the maximum mechanical deviation between the RGB acquisition module and the depth acquisition module after installation into the number of pixels.

[0074] Specifically, during processing, horizontal and vertical candidate offsets are selected one by one within the search range, and each continuous boundary point in the boundary continuity feature is translated to the depth image coordinate system according to the current candidate offset. After translation, depth transition points with a distance not exceeding the matching distance threshold are searched in the depth transition feature set. The matching distance threshold is determined by the pixel error after resampling the RGB image and the depth image. In the calibration image, the maximum pixel deviation between the manually aligned RGB boundary points and the depth transition points is used as the matching distance threshold.

[0075] If the distance between the translated continuous boundary point and a certain depth transition point does not exceed the matching distance threshold, and the angle between their directions does not exceed the matching direction threshold, then the continuous boundary point and the depth transition point are recorded as a valid matching point pair.

[0076] Furthermore, the matching direction threshold is determined by the manually aligned calibration image. Specifically, the angle between the direction of the manually corresponding RGB boundary point and the direction of the depth transition point is calculated, and the boundary angle that can retain the manually corresponding point pairs and exclude non-corresponding point pairs is taken as the matching direction threshold.

[0077] For each valid matching point pair, a consistency contribution value is calculated based on the distance between points and the directional angle. The closer the distance between points is to 0, the greater the consistency contribution value; the closer the directional angle is to 0, the greater the consistency contribution value.

[0078] Specifically, the consistency contribution value is obtained through a pre-established table corresponding to distance, direction angle, and contribution value. The table is generated by the calibration samples, and the distance and direction angle are stored discretely at fixed intervals. The consistency contribution values ​​of all valid matching point pairs under the same candidate offset are added together to obtain the transition consistency value corresponding to the candidate offset. The above process is repeated for all candidate offsets within the search range, and the transition consistency values ​​corresponding to each candidate offset are arranged in the order of horizontal and vertical offset positions to obtain the transition consistency distribution map.

[0079] Furthermore, in the transition consistency distribution map, the transition consistency value of each candidate offset position is compared with that of its 8 neighboring candidate offset positions.

[0080] Specifically, when the transition consistency value of a candidate offset position is greater than the transition consistency value of its 8 neighboring candidate offset positions, the candidate offset position is determined as a local extremum position. Then, the transition consistency value of the local extremum position is compared with the consistency threshold. If the transition consistency value is greater than the consistency threshold, the lateral offset and longitudinal offset corresponding to the local extremum position are determined as cross-modal alignment offsets. The consistency threshold is determined through manually aligned calibration samples. Specifically, the transition consistency values ​​of the correct offset position and the incorrect offset position are calculated in the calibration samples, and the boundary value that can retain the correct offset position and exclude the incorrect offset position is selected as the consistency threshold.

[0081] If there are more than two local extrema that meet the conditions, the local extrema with the largest transition consistency value is selected first; if the transition consistency values ​​are the same, the local extrema with the largest number of effective matching point pairs is selected; if the number of effective matching point pairs is the same, the local extrema with the smallest sum of effective matching point pair distances is selected.

[0082] As a preferred embodiment, after determining the cross-modal alignment offset, the depth image is pixel-level aligned using the cross-modal alignment offset.

[0083] Specifically, for each pixel position in the RGB image, the corresponding pixel position in the depth image is determined according to the cross-modal alignment offset. If the corresponding pixel position is within the valid range of the depth image and the depth value at that position does not belong to an invalid depth pixel, then the depth value is written into the depth information matrix at the same row and column position as the RGB pixel position. If the corresponding pixel position falls on a non-integer pixel position, then the depth values ​​of its four neighboring depth pixels are read, and weights are assigned according to the distance from the non-integer pixel position to the four neighboring depth pixels. The closer the neighboring depth pixels are, the greater the weight they receive. The four depth values ​​are multiplied by their corresponding weights and then added together to obtain the interpolated depth value, which is then written into the depth information matrix.

[0084] Specifically, if the corresponding pixel position is outside the valid range of the depth image, or the corresponding depth value is an invalid depth value, then the position in the depth information matrix is ​​marked as an invalid element; when there is a valid depth value in the 8-neighborhood of the invalid element, the arithmetic mean of all valid depth values ​​in the 8-neighborhood is taken and filled into the invalid element; when there is no valid depth value in the 8-neighborhood, the invalid mark of the position is maintained; after completing the mapping of all pixel positions of the RGB image, the depth information matrix corresponding to the RGB image is obtained.

[0085] Example 3: S2 includes acquiring an RGB image of a pig and a depth information matrix, extracting RGB feature maps and depth feature maps through a convolutional neural network; extracting occluded areas of the pig from the RGB feature maps and depth feature maps, determining the degree of texture loss based on the color features of the occluded areas, and obtaining the depth boundary clarity based on the depth features of the occluded areas; determining the color modality confidence and depth modality confidence based on the degree of texture loss and the depth boundary clarity; and performing weighted fusion of the RGB feature maps and depth feature maps based on the color modality confidence and depth modality confidence to obtain a multimodal fusion feature map.

[0086] In this embodiment, during processing, the RGB image of the pig and the depth information matrix are first cropped or scaled to the input size of the convolutional neural network.

[0087] Specifically, the red, green, and blue channels of the RGB image are normalized. During normalization, the pixel value of each channel is subtracted from the mean of that channel in the training samples and then divided by the standard deviation of that channel in the training samples. The effective depth value of the depth information matrix is ​​normalized according to the effective range of the depth acquisition module. Pixels with a depth value of 0 or exceeding the effective range are marked as invalid depth pixels, and the input value of invalid depth pixels is replaced by a preset invalid padding value.

[0088] In this embodiment, the normalized RGB image is input into the color feature extraction branch of the convolutional neural network.

[0089] Specifically, the color feature extraction branch performs convolution, normalization, activation, and downsampling processes in sequence; when each convolution kernel slides on the RGB image, it performs a weighted summation of the pixel values ​​of each channel within the coverage window, and uses the weighted summation result as the convolution response at that position.

[0090] The process involves multiple convolutions to obtain RGB feature maps representing color variations, fur texture, body contours, and local morphology of the pig's body. A depth information matrix is ​​input to the depth feature extraction branch of the convolutional neural network. This branch processes the depth input using the same convolution method as the color feature extraction branch, but its input channels are depth values ​​and invalid depth labels. After convolution, a depth feature map representing distance variations, depth edges, and spatial undulations on the pig's body surface is obtained. If the RGB feature map and the depth feature map have different sizes, bilinear upsampling is used to unify their sizes to a preset fusion size, ensuring that the feature positions of both maintain a correspondence with the RGB image coordinates.

[0091] Furthermore, when extracting the occluded area of ​​the pig, the boundary response is calculated for the RGB feature map and the depth feature map respectively.

[0092] Specifically, for any feature location in the RGB feature map, the feature values ​​of each channel between that location and its 8 neighboring locations are read, the absolute value of the difference between the corresponding channels is calculated, and the absolute values ​​of the differences between all channels are added together to obtain the color feature difference of that location relative to a neighboring location. The above calculation is performed in all 8 neighboring directions, and the color feature difference with the largest value is taken as the apparent boundary response of that location. For the depth feature map, the same neighborhood calculation method is used to obtain the depth boundary response, that is, the depth feature channel values ​​of the current feature location and its 8 neighboring locations are read, the sum of the absolute values ​​of the channel differences is calculated, and the largest result among the 8 directions is taken as the depth boundary response.

[0093] Subsequently, the apparent boundary response and depth boundary response were used to determine occlusion candidate regions.

[0094] Specifically, when the apparent boundary response at a certain feature location is less than the apparent boundary threshold and the depth boundary response is greater than the depth boundary threshold, it indicates that the location does not form a distinguishable body surface boundary in the color feature, but has a spatial boundary in the depth feature. Therefore, the location is marked as an occlusion candidate location. When the apparent boundary response at a certain feature location is greater than the apparent boundary threshold, the depth boundary response is greater than the depth boundary threshold, and there are more than two boundary directions with different orientations in the neighborhood of the location, the location is also marked as an occlusion candidate location. Specifically, both the apparent boundary threshold and the depth boundary threshold are determined by the calibration samples. When determining the apparent boundary threshold, the visible body surface boundaries and non-boundary regions of the pig are marked in the calibration samples. Candidate thresholds are judged one by one, and the number of unrecognized body surface boundaries and the number of non-boundary regions recognized as boundaries are counted. The candidate threshold with the minimum sum of the two types of errors is selected. The depth boundary threshold is determined in the same way as the apparent boundary threshold. Its marking objects are the front and rear layer boundaries and non-layer boundaries of the pig in the depth image. Connected components are extracted from the occlusion candidate positions. Connected components with a pixel count less than the occlusion area threshold are deleted, and the remaining connected components are taken as the occlusion area of ​​the pig. The occlusion area threshold is determined by the minimum effective occlusion area in the calibration samples. Specifically, occlusion areas that can affect the recognition of limb boundaries are marked in the calibration samples, the pixel count of each occlusion area is counted, and the minimum pixel count is taken as the occlusion area threshold.

[0095] Furthermore, after obtaining the occluded area of ​​the pig, the degree of texture loss is determined based on the color features of the occluded area; for each feature position within the occluded area, the color feature values ​​of that position and its 8 neighboring positions in the RGB feature map are read, and the texture response value of that position is obtained according to the aforementioned calculation method of color feature difference.

[0096] The process involves summing all texture response values ​​within the occluded area and dividing by the number of feature locations involved in the calculation to obtain the average texture response value of the occluded area. When the average texture response value reaches the texture preservation threshold, the texture loss level is recorded as 0. When the average texture response value does not reach the texture preservation threshold, the texture preservation threshold is subtracted from the average texture response value, and the difference is taken as the texture loss level. The texture preservation threshold is determined by the calibration samples before training. The calibration samples are labeled with regions where the texture can be used to distinguish individual pigs and regions where the texture cannot be used to distinguish individual pigs. Candidate thresholds are classified one by one, and the number of classification errors for the two types of labeled regions is counted. The candidate threshold with the minimum sum of classification errors is taken as the texture preservation threshold.

[0097] As a preferred embodiment, the depth boundary sharpness is calculated based on the depth features within the occluded area.

[0098] Specifically, the boundary pixels of the occluded area are processed one by one. First, the normal direction of the boundary pixel is determined, and then a preset number of depth feature positions are selected on both sides of the normal direction. The average value of each channel feature value of the depth feature positions on both sides is calculated, and then the sum of the absolute values ​​of the differences between the average feature values ​​on both sides is calculated as the depth boundary response of the boundary pixel. If the depth boundary response is greater than the depth boundary threshold, the boundary pixel is recorded as a valid depth boundary pixel. The number of valid depth boundary pixels is divided by the total number of boundary pixels of the occluded area, and the result is used as the depth boundary sharpness. The depth boundary threshold is determined by the calibration samples. The calibration samples are marked with depth boundaries that can correctly separate occluded pig individuals and depth boundaries that cannot correctly separate occluded pig individuals. Candidate thresholds are judged one by one, and the candidate threshold that has the minimum sum of classification errors is selected.

[0099] If the number of effective depth pixels in the occluded area is less than the effective depth number threshold, the depth boundary sharpness is recorded as 0. The effective depth number threshold is determined by the minimum number of effective depth pixels required to complete the occlusion boundary recognition in the calibration samples.

[0100] Furthermore, after determining the degree of texture loss and depth boundary sharpness, the confidence levels of the color modality and depth modality are calculated.

[0101] Specifically, the confidence level of the color modality is determined by the degree of texture loss: when the degree of texture loss is 0, the confidence level of the color modality is taken as the preset maximum confidence value; when the degree of texture loss reaches the texture loss threshold, the confidence level of the color modality is taken as the preset minimum confidence value; when the degree of texture loss is between 0 and the texture loss threshold, the confidence level of the color modality is read according to the pre-established correspondence table between the degree of texture loss and the confidence level of the color modality; the correspondence table is generated by the calibration samples, and the recognition results of the RGB feature maps of the pig boundary under different degrees of texture loss are recorded during calibration, and the confidence value with the smallest number of recognition errors is written into the correspondence table; the texture loss threshold is determined by the occlusion area in the calibration samples that causes the RGB boundary recognition error, and the minimum degree of texture loss corresponding to this type of occlusion area is taken as the texture loss threshold.

[0102] In this embodiment, the depth modality confidence is determined by the depth boundary sharpness: when the depth boundary sharpness reaches the depth sharpness threshold, the depth modality confidence takes the preset maximum confidence value; when the depth boundary sharpness is 0, the depth modality confidence takes the preset minimum confidence value; when the depth boundary sharpness is between 0 and the depth sharpness threshold, the depth modality confidence is read according to the pre-established correspondence table between depth boundary sharpness and depth modality confidence.

[0103] Specifically, the correspondence table is generated from the calibration samples. During calibration, the separation results of the depth feature map of the occluded pig individual boundary under different depth boundary clarity are recorded, and the confidence value with the smallest number of recognition errors is written into the correspondence table. The depth clarity threshold is determined by the depth boundary that can correctly separate the occluded individual in the calibration samples, and the minimum depth boundary clarity corresponding to this type of depth boundary is taken as the depth clarity threshold.

[0104] Furthermore, after obtaining the color modality confidence and depth modality confidence, the RGB feature map and depth feature map are weighted and fused.

[0105] Specifically, for each fusion location, first read the color feature value of that location in the RGB feature map and the depth feature value of that location in the depth feature map; then add the color modality confidence and the depth modality confidence to obtain the total confidence value; the color feature weight is obtained by dividing the color modality confidence by the total confidence value, and the depth feature weight is obtained by dividing the depth modality confidence by the total confidence value; if the total confidence value is 0, then both the color feature weight and the depth feature weight take the preset default weight; then, multiply the color feature value by the color feature weight, multiply the depth feature value by the depth feature weight, and then add the two weighted results to obtain the multimodal fusion feature value of that fusion location; perform the above processing on all fusion locations to obtain the multimodal fusion feature map.

[0106] Example 4: S3 includes acquiring a multimodal fusion feature map, calculating the apparent similarity and depth difference of pixels based on the multimodal fusion feature map; extracting spatial adjacency relationships based on apparent similarity and depth difference, calculating the overlap confidence of pixels based on spatial adjacency relationships; if the overlap confidence is greater than a preset overlap threshold, extracting the corresponding pixels to generate an initial overlapping region; and performing connected component labeling on the initial overlapping region to obtain an overlapping region mask containing multiple pigs.

[0107] In this embodiment, each pixel position in the multimodal fusion feature map contains color-related features and depth-related features. The color-related features are derived from the color, texture, and contour information in the RGB image, while the depth-related features are derived from the distance variation and spatial boundary information in the depth information matrix. During processing, the multimodal fusion feature map is read point by point according to the pixel coordinates, and the 8-neighborhood of the current pixel is used as the neighborhood comparison range.

[0108] Specifically, when calculating the apparent similarity of pixels, the color-related feature value of the current pixel is read first, and then the color-related feature values ​​of its 8 neighboring pixels are read respectively.

[0109] Specifically, for the current pixel and any neighboring pixel, the absolute value of the difference between their color-related feature values ​​is calculated channel by channel, and the absolute values ​​of the differences in each channel are added together to obtain the apparent difference value in that neighborhood direction. Then, the apparent similarity is calculated based on the apparent difference value: when the apparent difference value is 0, the apparent similarity is taken as a preset upper limit; when the apparent difference value reaches the upper limit, the apparent similarity is taken as a preset lower limit; when the apparent difference value is between 0 and the upper limit, the similarity value is read according to a pre-established correspondence table between apparent difference value and apparent similarity value. This correspondence table is generated by calibration samples, in which continuous areas on the body surface of the same pig and contact areas of different pigs are pre-labeled. The recognition results corresponding to different apparent difference values ​​are statistically analyzed, and the correspondence that minimizes the sum of the number of disconnected continuous areas on the body surface of the same pig and the number of connected contact areas of different pigs is selected as the correspondence table between apparent difference value and apparent similarity value.

[0110] In this process, after calculating the apparent similarity of the current pixel in its eight neighborhood directions, the apparent similarity of each direction is retained.

[0111] Furthermore, when calculating the depth difference of pixels, the depth-related feature value of the current pixel is read first, and then the depth-related feature values ​​of its 8 neighboring pixels are read respectively.

[0112] Specifically, for the current pixel and any neighboring pixel, the absolute value of the difference between their depth-related feature values ​​is calculated for each channel, and the absolute values ​​of the differences in each channel are added together to obtain the depth feature difference value in that neighborhood direction.

[0113] Specifically, when the original depth value corresponding to the depth information matrix is ​​retained in the multimodal fusion feature map, the absolute value of the difference between the original depth values ​​of the current pixel and the neighboring pixels is also calculated. Subsequently, the depth feature difference value and the absolute value of the difference between the original depth values ​​are weighted and summed to obtain the depth difference in that neighborhood direction. The weights of the depth feature difference value and the absolute value of the difference between the original depth values ​​are determined by the calibration samples. Specifically, the overlapping and non-overlapping boundaries of pigs are pre-labeled in the calibration samples, the recognition results are calculated one by one for the candidate weight combinations, the number of overlapping boundaries that are not recognized and the number of non-overlapping boundaries that are misidentified are counted, and the weight combination that minimizes the sum of the above two types of quantities is selected as the weight for calculating the depth difference.

[0114] Furthermore, after obtaining the apparent similarity and depth difference in each neighborhood direction, spatial adjacency relationships are extracted.

[0115] Specifically, for the current pixel and any neighboring pixel, if the apparent similarity in the neighborhood direction is greater than the apparent adjacency threshold and the depth difference in the neighborhood direction is greater than the depth boundary threshold, the adjacency relationship between the current pixel and the neighboring pixel is recorded as an overlapping adjacency relationship.

[0116] Specifically, this determination indicates that two pixels satisfy the continuum surface determination condition in terms of color appearance, and simultaneously satisfy the front-to-back layer boundary condition in terms of depth. The apparent adjacency threshold is determined by the calibration samples, which pre-label continuous pixel pairs on the same pig's body surface and pixel pairs on different pigs' body surfaces. The candidate apparent adjacency thresholds are determined one by one, and the number of continuous pixel pairs on the same pig's body surface that are broken and the number of pixel pairs on different pigs' body surfaces that are connected are counted. The candidate threshold with the minimum sum of the two is taken as the apparent adjacency threshold. The depth boundary threshold is determined by the calibration samples, which pre-label pixel pairs on both sides of the front and back occlusion boundaries of different pigs and adjacent pixel pairs on the same pig's body surface. The candidate depth boundary thresholds are determined one by one, and the number of pixel pairs on both sides of the front and back occlusion boundaries that are not recognized and the number of adjacent pixel pairs on the same pig's body surface that are incorrectly determined as depth boundaries are counted. The candidate threshold with the minimum sum of the two types of numbers is taken as the depth boundary threshold.

[0117] In this embodiment, when calculating the overlap confidence of a pixel, the overlapping adjacency relationship within the 8-neighborhood of the current pixel is used as input.

[0118] Specifically, the number of directions that are recorded as overlapping adjacencies within the 8-neighborhood of the current pixel is first counted, and the direction score is read from the preset direction score table based on the number of directions. The direction score table is established through calibration samples. When establishing the table, the number of directions in which overlapping adjacencies occur in the 8-neighborhood of overlapping region pixels and non-overlapping region pixels are counted respectively, and a score is assigned to each direction, so that the sum of the number of discrimination errors of overlapping region pixels and non-overlapping region pixels reaches the minimum value.

[0119] Subsequently, for each neighborhood direction recorded as overlapping adjacency, its apparent similarity and depth difference are read; based on the apparent similarity, the apparent score is read from the preset apparent score table, and based on the depth difference, the depth score is read from the preset depth score table; both the apparent score table and the depth score table are generated from samples with overlapping region annotations. The generation method is to verify different score assignment schemes and select the assignment scheme that minimizes the sum of the number of missed detections of pixels in the overlapping region and the number of misdetected detections of pixels in the non-overlapping region.

[0120] Finally, the orientation score, appearance score, and depth score are weighted and summed according to preset weights to obtain the overlap confidence of the current pixel. Each weight is determined through calibration samples. When determining the weight, the overlap confidence is calculated for each candidate weight combination, and the weight combination that minimizes the sum of the number of missed detections in the overlapping area and the number of misdetected detections in the non-overlapping area is selected.

[0121] Furthermore, after obtaining the overlap confidence of the current pixel, the overlap confidence is compared with the preset overlap threshold.

[0122] Specifically, if the overlap confidence is greater than the overlap threshold, the current pixel is extracted as an overlapping pixel; if the overlap confidence is less than or equal to the overlap threshold, the current pixel is not extracted as an overlapping pixel. The overlap threshold is determined by manually labeled calibration samples, which include pixels in overlapping areas and pixels in non-overlapping areas. A judgment result is generated for each candidate overlap threshold, and the number of overlapping area pixels that are not extracted and the number of non-overlapping area pixels that are incorrectly extracted are counted. The candidate threshold with the minimum sum of the two types of numbers is taken as the overlap threshold.

[0123] Furthermore, after traversing all pixels in the multimodal fusion feature map, all overlapping pixels are written into a binary map to form an initial overlapping region. In this binary map, overlapping pixels are assigned a first label value, and non-overlapping pixels are assigned a second label value.

[0124] When labeling connected components in the initial overlapping region, the process begins with any unlabeled overlapping pixel in the binary graph and searches according to the 8-neighborhood connectivity rule. All overlapping pixels directly adjacent to this pixel or indirectly adjacent via other overlapping pixels are assigned the same connected component number. When no unlabeled adjacent overlapping pixels exist, the labeling of one connected component is complete. This process is repeated with the next unlabeled overlapping pixel until all overlapping pixels in the initial overlapping region have been labeled as connected components.

[0125] Furthermore, after the connected components are labeled, the number of pixels, the outer boundary, and the area of ​​the internal holes of each connected component are calculated.

[0126] Specifically, if the number of pixels in a connected region is less than the overlapping area threshold, the connected region is deleted. The overlapping area threshold is determined by the smallest manually labeled area in the calibration samples that can represent contact or occlusion between two pigs. If there is a hole inside a connected region and the hole area is less than the hole filling threshold, the hole is filled as an overlapping pixel. The hole filling threshold is determined through the calibration samples. When determining the threshold, candidate hole filling thresholds are processed one by one. The number of real non-overlapping gaps that are filled and the number of broken holes that are not filled are counted. The candidate threshold with the minimum sum of the two types of numbers is taken as the hole filling threshold.

[0127] Furthermore, after connected component filtering and hole filling, the remaining connected components constitute an overlapping region mask; the pixel coordinates of this overlapping region mask are consistent with the pixel coordinates of the multimodal fusion feature map. The marked positions in the mask represent areas containing the physical contact or occlusion relationship of multiple pigs, and the unmarked positions are not used as objects for subsequent overlapping boundary extraction.

[0128] Example 5: S4 includes obtaining the multimodal fusion feature map corresponding to the mask of the overlapping region, processing the multimodal fusion feature map with an edge detection operator to obtain an initial gradient matrix; extracting the apparent gradient change features and depth gradient change features based on the initial gradient matrix to obtain a fused gradient magnitude map; removing non-edge pixels from the fused gradient magnitude map to obtain a refined edge feature map; generating a candidate boundary map containing strong edge points and weak edge points based on the refined edge feature map; and performing connected component analysis based on the strong edge points and weak edge points in the candidate boundary map to determine the individual boundaries of each pig in the overlapping region.

[0129] In this embodiment, the overlapping region mask is first mapped to the multimodal fusion feature map in coordinates. The pixel positions marked as overlapping regions in the mask are used as the edge extraction range, and the pixel positions outside the mask do not participate in gradient calculation. For each pixel located in the overlapping region, its channel feature values ​​in the multimodal fusion feature map are read. The channel feature values ​​include appearance feature values ​​from the RGB feature map and depth feature values ​​from the depth feature map.

[0130] Furthermore, when using edge detection operators to process multimodal fusion feature maps, a neighborhood window of a preset size is selected around each pixel to be processed.

[0131] Specifically, the edge detection operator includes a horizontal operator and a vertical operator. When calculating the horizontal gradient response, the feature value of each pixel in the neighborhood window is multiplied by the coefficient at the corresponding position in the horizontal operator, and then the multiplication results are summed to obtain the horizontal gradient response of the current pixel. When calculating the vertical gradient response, the same processing is performed on the same neighborhood window using the vertical operator to obtain the vertical gradient response of the current pixel. For a multimodal fusion feature map containing two or more feature channels, the horizontal and vertical gradient responses of each feature channel are calculated separately, and then the absolute values ​​of the gradient responses of each channel in the same direction are summed to obtain the total gradient response of the pixel. After all pixels in the overlapping area have undergone the above processing, the initial gradient matrix is ​​obtained.

[0132] The initial gradient matrix records the horizontal gradient response, the vertical gradient response, and the gradient direction determined by the two directional responses for each pixel.

[0133] Furthermore, when extracting apparent gradient change features based on the initial gradient matrix, only the apparent feature channels derived from the RGB feature map are called.

[0134] Specifically, for each pixel, the absolute value of the horizontal gradient response and the absolute value of the vertical gradient response on the apparent feature channel are added together to obtain the apparent gradient change value. When extracting the depth gradient change feature, only the depth feature channel from the depth feature map is called, and the absolute values ​​of the horizontal gradient response and the absolute values ​​of the vertical gradient response on the depth feature channel are added together in the same way to obtain the depth gradient change value.

[0135] Subsequently, the apparent gradient change value and the depth gradient change value are numerically normalized. The normalization benchmark is determined by the minimum and maximum gradient change values ​​of each pixel in the current overlapping area, so that the two fall into the same numerical range. Then, the normalized apparent gradient change value and the depth gradient change value are weighted and summed according to the apparent gradient weight and the depth gradient weight to obtain the fused gradient magnitude map. The apparent gradient weight and the depth gradient weight are determined by the calibration sample. The actual individual boundaries between each pig in the overlapping area are marked in the calibration sample. Boundary results are generated for different candidate weight combinations. The number of pixels that are not retained by the actual individual boundaries and the number of non-boundary pixels that are mistakenly retained are counted. The candidate weight combination that has the minimum sum of the above two types of quantities is selected as the actual weight.

[0136] Furthermore, after obtaining the fused gradient magnitude map, non-edge pixels are removed.

[0137] For each pixel in the fused gradient magnitude map, the gradient direction of the pixel is first determined based on the initial gradient matrix. Then, the fused gradient magnitudes of its forward and backward neighboring pixels are read along this gradient direction. If the fused gradient magnitude of the current pixel is less than or equal to the fused gradient magnitude of its forward neighboring pixel, or less than or equal to the fused gradient magnitude of its backward neighboring pixel, the current pixel is discarded. If the fused gradient magnitude of the current pixel is greater than both the fused gradient magnitudes of its forward and backward neighboring pixels, the pixel is retained. After this process, a refined edge feature map is obtained. The retained pixels in the refined edge feature map are used to represent candidate boundary trajectories within the overlapping region.

[0138] In this embodiment, when generating candidate boundary maps based on the refined edge feature map, a first edge threshold and a second edge threshold are used to classify the retained pixels, wherein the first edge threshold is greater than the second edge threshold.

[0139] Specifically, the fusion gradient magnitude in the refined edge feature map is read pixel by pixel. When the fusion gradient magnitude is greater than the first edge threshold, the pixel is marked as a strong edge point; when the fusion gradient magnitude is less than the second edge threshold, the pixel is deleted; when the fusion gradient magnitude is not less than the second edge threshold and not greater than the first edge threshold, the pixel is marked as a weak edge point. The first edge threshold is determined by the calibration samples. Specifically, the real individual boundary pixels labeled by humans are used as boundary samples, and the non-boundary pixels inside the overlapping area are used as non-boundary samples. The candidate thresholds are judged one by one. The number of real individual boundary pixels that are not marked as strong edge points and the number of non-boundary pixels that are mistakenly marked as strong edge points are counted. The candidate threshold with the minimum sum of the two types of numbers is taken as the first edge threshold.

[0140] In addition, the second edge threshold is also determined by the calibration samples. Specifically, edge extension pixels that can connect adjacent real individual boundaries are used as retained samples, and isolated noise pixels are used as discarded samples. Candidate thresholds are judged one by one, and the number of edge extension pixels deleted and the number of isolated noise pixels retained are counted. The candidate threshold with the minimum sum of the two types of numbers is taken as the second edge threshold. After classification, a candidate boundary map containing strong edge points and weak edge points is generated.

[0141] Furthermore, when determining individual boundaries based on the candidate boundary map, strong edge points are used as the starting points of the boundary connected domains, and the search proceeds outward according to the 8-neighborhood connectivity rule.

[0142] Specifically, if a weak edge point is directly adjacent to a strong edge point, or indirectly adjacent to a strong edge point through other weak edge points, then the weak edge point is incorporated into the same boundary connected component. If a weak edge point cannot be connected to any strong edge point, then the weak edge point is deleted. During connectivity determination, directional consistency and depth continuity checks are also performed on adjacent edge points. The directional consistency check is performed by calculating the angle between the gradient directions of two adjacent edge points; if this angle is not greater than the directional continuity threshold, the direction is considered to meet the continuity condition.

[0143] Specifically, the depth continuity verification is as follows: read the depth gradient change values ​​corresponding to two adjacent edge points, calculate the absolute value of the difference between them, and when the absolute value of the difference is not greater than the depth continuity threshold, the depth change is considered to meet the continuity condition. The direction continuity threshold and the depth continuity threshold are both determined by the calibration samples. Specifically, the direction angle and depth gradient change difference of adjacent edge points are counted in the manually annotated real individual boundaries, and the corresponding values ​​are counted in the erroneous connection samples. Candidate thresholds are judged one by one, and the candidate threshold with the minimum sum of the number of real individual boundary breaks and the number of erroneous connections is selected respectively.

[0144] Furthermore, after completing the connected component analysis, the boundary length, endpoint positions, and depth boundaries on both sides of each boundary connected component are calculated.

[0145] Specifically, the boundary length is obtained by counting the number of edge points within the connected component; the endpoint position is obtained by finding the position in the connected component that has only one adjacent edge point; the depth boundary result on both sides of the boundary is obtained by reading the depth feature values ​​of the pixels on both sides of the boundary normal direction and calculating the average of the absolute values ​​of the difference between the depth feature values ​​on both sides; when the boundary length is less than the boundary length threshold, the boundary connected component is deleted; the boundary length threshold is determined by the number of pixels of the shortest effective individual boundary line in the calibration sample; when the boundary length is not less than the boundary length threshold and the depth boundary result on both sides of the boundary is greater than the depth boundary threshold, the boundary connected component is retained as an effective boundary line; the depth boundary threshold is determined by the difference in depth features on both sides of the real individual boundary in the calibration sample and the difference in depth features inside the body surface of the same pig. Candidate thresholds are judged one by one, and the candidate threshold with the minimum sum of the number of real individual boundaries that are not retained and the number of mis-segmented parts inside the body surface of the same pig is selected.

[0146] As a preferred embodiment, the retained valid boundary lines are used together with the outer contour of the overlapping region mask for region division.

[0147] Specifically, if the two endpoints of an effective boundary line are connected to the outer contour of the overlapping area mask, the effective boundary line is directly used as the individual boundary between adjacent pigs; if two or more effective boundary lines form a closed separation relationship in the overlapping area, the effective boundary lines forming the closed relationship are merged into an individual boundary; if there are discontinuities at the endpoints of an effective boundary line, and the distance between the endpoints is not greater than the endpoint connection threshold, and the endpoint connection direction is consistent with the extension direction of the two effective boundary lines, the endpoints are connected to form a continuous individual boundary; the endpoint connection threshold is determined by the pixel distance between the broken endpoints of the real individual boundary in the calibration sample, and a candidate threshold that can connect the real broken boundary and exclude the connection of endpoints that are not the same boundary is selected.

[0148] Example 6: S5 includes extracting the visible body contour based on the individual boundary and constructing a target topology map of the visible body contour; performing spatial mapping between the target topology map and the template topology map of the pre-established complete pig morphology template to obtain an aligned contour map; comparing the aligned contour map with the complete pig morphology template to obtain a set of differing pixels and a matching degree; if the matching degree is less than a preset matching threshold, it is determined that there is limb occlusion in the visible body contour, and the missing limb region is extracted based on the set of differing pixels.

[0149] In this embodiment, for each individual boundary, the image region enclosed by the individual boundary is first extracted from the RGB image, depth information matrix, and multimodal fusion feature map, and the individual boundary is used as the outer contour constraint of the foreground region. An 8-neighborhood search is then performed point-by-point along the individual boundary, filling the pixels connected to the inner side of the individual boundary with the pig's body region, obtaining a visible body region mask. The outer boundary pixel sequence is then extracted from the visible body region mask as the visible body contour. The boundary pixel sequence is arranged according to the connection order of adjacent pixels, and the coordinates, adjacency relationships, and boundary directions of each boundary pixel are recorded.

[0150] In addition, when there are more than two closed boundaries within the same body boundary, the number of pixels enclosed by each closed boundary is calculated, the closed boundary with the largest number of pixels is retained as the visible body outline, and the remaining closed boundaries are deleted as interference boundaries.

[0151] Furthermore, after obtaining the visible body outline, a target topology map of the visible body outline is constructed.

[0152] Specifically, contour points are selected along the visible body outline at fixed sampling intervals, determined by calibration samples: the pixel perimeter of the complete contour is counted in unobstructed pig samples, and the interval that ensures the contour turning points of the head, torso, forelimbs, and hindlimbs are all captured is taken as the sampling interval; each selected contour point is used as a node in the target topology map, and connecting edges are established between adjacent sampling points; for each node, the coordinate distance between the node and the previous node, the coordinate distance between the node and the next node, and the angle between the two connecting edges are calculated, and the above calculation results are used as the geometric attributes of the node; the coordinate distance is obtained by adding the absolute values ​​of the differences between the horizontal and vertical coordinates of the two nodes; the angle is obtained by the direction difference between the directions of the previous and next connecting edges; the target topology map is constructed from the node coordinates, node connection relationships, and node geometric attributes.

[0153] In this embodiment, the complete pig morphology template is pre-created from an unobstructed pig image.

[0154] In the specific process, the complete pig body outline is first extracted from the unobstructed samples, and a template topology map is constructed according to the same sampling interval as the visible body outline. Then, based on the pig's body structure, the template topology map is divided into head, trunk, forelimb, and hindlimb segments. Each template node in the template topology map records the template coordinates, the connection relationship between adjacent nodes, and the body segment to which it belongs. In order to make the template correspond to the current pig scale, the unobstructed samples are scaled before the template is built. The normalization reference is the length and width of the bounding rectangle of the pig body outline. The normalized outline coordinates are used to form the complete pig morphology template.

[0155] In this embodiment, when performing spatial mapping based on the target topology map and the template topology map, the center of the outer rectangle of the target topology map and the center of the outer rectangle of the template topology map are first calculated, and the target topology map is translated to a position where the center of its outer rectangle coincides with the center of the outer rectangle of the template topology map.

[0156] Then, the length and width of the bounding rectangle of the target topology diagram and the template topology diagram are calculated, and the target topology diagram is scaled according to the length ratio and width ratio respectively, so that the target topology diagram and the template topology diagram are on the same size reference. After translation and scaling, the principal axis direction of the target topology diagram is compared with the principal axis direction of the template topology diagram. The principal axis direction is determined by the coordinate distribution range of the contour points in the horizontal and vertical directions, and the direction in which the coordinate distribution range reaches the maximum value is taken as the principal axis direction. The target topology diagram is rotated around its center until the principal axis direction of the target topology diagram is consistent with the principal axis direction of the template topology diagram.

[0157] Furthermore, after directional alignment, node mapping is performed between the target topology map and the template topology map. Node mapping is completed in the order of contour arc length, that is, the head endpoints of the target topology map and the template topology map are used as the starting nodes, and then nodes are matched sequentially along the same contour direction.

[0158] Specifically, the head endpoint is determined by the contour position corresponding to the head segment of the template, and the node in the target topology map that has the minimum distance from the head endpoint of the template is used as the target starting node. When the number of nodes in the target topology map is inconsistent with the number of nodes in the template topology map, the number of nodes is adjusted according to the contour arc length ratio: when the number of nodes is insufficient, nodes are inserted between adjacent nodes according to the arc length position; when the number of nodes exceeds the limit, nodes whose distance from adjacent nodes does not reach the sampling interval are deleted; after the number of nodes is consistent, the node coordinates in the target topology map are mapped to the template coordinate system to form an aligned contour map.

[0159] As a preferred embodiment, when comparing the aligned outline map with the complete pig morphology template, the aligned outline map is first filled with the target area mask, and the complete pig morphology template is filled with the template area mask.

[0160] Then, the two masks are compared pixel by pixel. If a pixel belongs to the pig's body region in the template region mask but not in the target region mask, the pixel is recorded as a missing difference pixel. If a pixel belongs to the pig's body region in the target region mask but not in the template region mask, the pixel is recorded as an expanded difference pixel. The missing difference pixels and the expanded difference pixels together form the difference pixel set.

[0161] In addition, to eliminate discrete differences caused by contour extraction errors, an 8-neighborhood connected component analysis is performed on the set of differing pixels, and connected regions whose number of pixels does not reach the difference area threshold are deleted. The difference area threshold is determined by the unoccluded calibration samples. Specifically, the number of pixels in the contour difference region generated after template alignment is calculated in the unoccluded samples, and the maximum number of pixels is taken as the difference area threshold.

[0162] In this embodiment, the matching degree is calculated based on the overlap between the target region mask and the template region mask.

[0163] Specifically, the number of overlapping pixels belonging to both the target region mask and the template region mask is first counted. Then, the number of pixels covered by the merged target region mask and template region mask is counted. The matching degree is obtained by dividing the number of overlapping pixels by the number of merged covered pixels. The matching threshold is determined by calibration samples, which include unoccluded pig samples and pig samples with limb occlusion. For each candidate matching threshold, the number of unoccluded pig samples that are judged as occluded and the number of pig samples with limb occlusion that are not judged as occluded are counted. The two types of error counts are added together, and the candidate matching threshold with the minimum sum of error counts is taken as the matching threshold.

[0164] Furthermore, when the matching degree is less than the matching threshold, it is determined that there is limb occlusion in the visible body outline, and the missing limb region is extracted based on the set of difference pixels.

[0165] In the specific extraction process, missing difference pixels are selected first, because these correspond to areas that exist in the complete pig morphology template but are missing from the currently visible body outline. The missing difference pixels are then labeled with 8-neighbor connected components to obtain difference connected regions. The number of pixels, centroid coordinates, and the number of overlapping pixels with each body segment of the template are then calculated for each difference connected region. The centroid coordinates are obtained by averaging the x-coordinates and y-coordinates of all pixels within the connected region.

[0166] Specifically, for any differentially connected region, when the number of overlapping pixels between it and the forelimb or hindlimb segment reaches the site attribution threshold, the differentially connected region is identified as a missing limb region.

[0167] Furthermore, the body part attribution threshold was determined by samples with limb occlusion annotations.

[0168] Specifically, the following steps are taken: Mark the actual missing limb regions in the calibration samples, and calculate the number of overlapping pixels between the actual missing limb regions and the template forelimb and hindlimb segments respectively; simultaneously calculate the number of overlapping pixels between the non-missing limb regions and the forelimb and hindlimb segments; determine the candidate site assignment threshold one by one, count the number of actual missing limb regions that are not identified and the number of non-missing limb regions that are misclassified as missing limb regions, and take the candidate threshold with the minimum sum of the two types of errors as the site assignment threshold.

[0169] Furthermore, when two or more differentially connected regions meet the criteria for determining a missing limb region, the corresponding missing limb regions are output respectively; when there are two or more adjacent differentially connected regions within the same limb segment, and the distance between the adjacent differentially connected regions is within the connection distance threshold, they are merged into the same missing limb region.

[0170] Specifically, the connection distance threshold is determined by the distance between adjacent difference regions formed by the contour breakage of the same missing limb in the calibration sample. The candidate distance that can connect the same missing limb region but not different limb regions is taken as the connection distance threshold. After the above processing, the visible body contour can be obtained from the individual boundary, and the spatial alignment and pixel comparison between the visible body contour and the complete pig morphology template can be completed. The missing limb region is extracted when the matching degree does not reach the matching threshold.

[0171] Example 7: S6 includes extracting the depth pixel values ​​corresponding to the missing limb region from the depth information matrix, determining the three-dimensional coordinates of the visible joints based on the depth pixel values; using a graph convolutional network to extract feature vectors from the skeleton node graph constructed based on the three-dimensional coordinates, generating an adjacency matrix based on the feature vectors to determine the topological relationship between the visible joints; constructing a coordinate prediction model based on the three-dimensional coordinates and the topological relationship, predicting the three-dimensional coordinates of the occluded joints through the coordinate prediction model, and obtaining the spatial position of the occluded joints.

[0172] In this embodiment, the missing limb region is first mapped to the same pixel coordinate position in the depth information matrix, and the depth pixel value in that region is read. Pixels with a depth value of 0, exceeding the effective range of the depth acquisition module, or marked as invalid in S1 are not directly involved in the coordinate conversion.

[0173] Specifically, for invalid depth pixels located in the joint neighborhood, if there are valid depth pixels in its 8-neighborhood, the arithmetic mean of all valid depth pixels in the 8-neighborhood is taken as the compensation depth value at that position; if there are no valid depth pixels in its 8-neighborhood, the pixel remains marked as invalid and is not used as a joint depth sampling point.

[0174] Furthermore, the location of the visible joints is determined by the area of ​​the missing limb and its adjacent visible body outline.

[0175] Specifically, the skeleton centerline is extracted near the boundary of the missing limb area, and joint candidate points are set at the positions where the skeleton centerline intersects or is close to the turning point of the visible limb contour. The skeleton centerline can be obtained by refining the mask of the visible body area. During the refining process, edge pixels that do not change connectivity are deleted layer by layer until the centerline with a single pixel width is retained. The endpoints, bifurcation points, and positions corresponding to the contour turning points of the centerline are filtered to obtain joint candidate points. The contour turning point is determined by the direction difference between adjacent contour segments. Contour points are read sequentially along the visible body contour, and the directions before and after the current contour point are calculated respectively. When the angle between the two directions is greater than the joint turning threshold, the contour point is taken as the contour turning point.

[0176] The joint transition threshold is determined by the calibration samples. The calibration samples are pre-labeled with shoulder joints, hip joints, knee joints, tarsal joints and hoof joints. The candidate transition thresholds are judged one by one. The number of real joints that are not recognized and the number of non-joint points that are recognized as joints are counted. The candidate threshold with the minimum sum of the two types of errors is taken as the joint transition threshold.

[0177] Furthermore, for each joint candidate point, the depth pixel values ​​in its neighborhood are read, and the joint depth value is calculated.

[0178] The specific processing is as follows: a neighborhood window of a preset size is established with the candidate joint point as the center, and the number of effective depth pixels within the window is counted; when the number of effective depth pixels reaches the effective depth number threshold, the arithmetic mean of all effective depth values ​​within the window is taken as the depth value of the candidate joint point; when the number of effective depth pixels does not reach the effective depth number threshold, the candidate joint point is deleted; the effective depth number threshold is determined by the calibration samples, specifically by counting the number of effective depth pixels in the neighborhood of the joint in the samples that can stably annotate the 3D coordinates of the joint, and taking the minimum number that can make the joint coordinate conversion error meet the calibration requirements as the effective depth number threshold.

[0179] Furthermore, when determining the three-dimensional coordinates of the visible joint based on the depth pixel value, the horizontal and vertical pixel coordinates of the visible joint in the RGB image coordinate system are first obtained, and then the previously determined joint depth value is read.

[0180] Subsequently, the depth acquisition module and the RGB acquisition module are called to complete the calibration of the camera intrinsic parameters. The camera intrinsic parameters include the lateral focal length parameter, the longitudinal focal length parameter, the principal point x-coordinate, and the principal point y-coordinate. When calculating the lateral spatial coordinates, the lateral pixel coordinates of the visible joints are subtracted from the principal point x-coordinates, the difference is multiplied by the joint depth value, and then divided by the lateral focal length parameter.

[0181] In calculating the longitudinal spatial coordinates, the longitudinal pixel coordinates of the visible joints are subtracted from the longitudinal coordinates of the principal point, the difference is multiplied by the joint depth value, and then divided by the longitudinal focal length parameter; the depth spatial coordinates are directly taken from the joint depth value; after the above conversion, each retained joint candidate point is determined as a visible joint and obtains three-dimensional coordinates in the same camera coordinate system.

[0182] Furthermore, after obtaining the three-dimensional coordinates of the visible joints, a skeleton node graph is constructed.

[0183] Specifically, each visible joint is treated as a skeleton node, and the node attributes include joint type, lateral spatial coordinates, longitudinal spatial coordinates, depth direction spatial coordinates, and visibility markers. The joint type is determined based on the position of the visible joint in the pig body template. For example, nodes located in the forelimb template segment are marked as forelimb joints, and nodes located in the hindlimb template segment are marked as hindlimb joints. The visibility markers are used to indicate that the joint is directly determined by the image and depth information.

[0184] The initial connection relationships between the skeleton nodes are set according to the pre-established pig skeleton structure, which includes the connection relationships between trunk nodes and forelimb nodes, trunk nodes and hindlimb nodes, and adjacent joints of the same limb.

[0185] Furthermore, when using a graph convolutional network to extract feature vectors, the skeleton node graph is input into the graph convolutional network.

[0186] For any skeleton node, the graph convolutional network first reads the node's 3D coordinates, joint type, and visibility marker, and then reads the attributes of neighboring nodes that have an initial connection relationship with the node. When performing feature aggregation on neighboring nodes, it calculates the absolute value of the difference between the current node and its neighboring nodes in the horizontal, vertical, and depth spatial coordinates, and inputs the coordinate difference, joint type combination relationship, and visibility marker into the graph convolutional layer.

[0187] Specifically, the graph convolutional layer performs a weighted summation of the attributes of the current node and those of its neighboring nodes, followed by activation processing to obtain the node features of the current node. After continuous graph convolution processing, each skeleton node obtains a corresponding feature vector, which contains the spatial position of the joint itself, the positional relationship of adjacent joints, and the connection information of the joint type. The weight parameters of the graph convolutional network are obtained from training samples, which include unoccluded pig skeleton samples and occluded pig skeleton samples. During training, the 3D coordinates of the complete joints labeled manually are used as supervision values ​​to minimize the spatial distance error between the predicted joint coordinates and the manually labeled joint coordinates.

[0188] As a preferred embodiment, when generating the adjacency matrix based on the feature vector, a connection determination is made for any two visible joints.

[0189] Specifically, connection determination includes joint type determination, spatial distance determination, and limb direction determination.

[0190] Among them, the joint type determination is used to determine whether two joints belong to the joint combination that is allowed to be directly connected in the preset pig skeleton structure; the spatial distance determination is used to determine whether the three-dimensional distance between two joints falls within the corresponding limb length range; and the limb direction determination is used to determine whether the spatial direction formed by two joints conforms to the extension direction range of the corresponding limb segment.

[0191] Specifically, the calculation process for the three-dimensional distance is as follows: The absolute values ​​of the differences between the two joints in the horizontal, vertical, and depth spatial coordinates are calculated, and these three absolute values ​​are summed to obtain the spatial distance between the two joints. The corresponding limb length range is determined by unobstructed calibration samples. Specifically, the three-dimensional distances between adjacent joints in pigs at the same growth stage are statistically analyzed, and the minimum and maximum length values ​​that retain true adjacent joint connections and exclude non-adjacent joint connections are taken as the length range of the limb segment. The limb direction range is determined by the direction of the lines connecting adjacent joints in the unobstructed calibration samples. Candidate direction ranges are judged one by one, and the direction range with the minimum sum of the number of true skeleton disconnections and the number of erroneous skeleton connections is taken as the limb direction range. When two joints simultaneously meet the above judgment conditions, the corresponding positions are marked as connected in the adjacency matrix; otherwise, they are marked as disconnected. The resulting adjacency matrix represents the topological relationship between visible joints.

[0192] Furthermore, after obtaining the three-dimensional coordinates and topological relationships of the visible joints, a coordinate prediction model is constructed.

[0193] Specifically, the coordinate prediction model takes the visible joint 3D coordinates, adjacency matrix, joint type label, and missing limb region label as input.

[0194] The model first identifies visible joints with skeletal connections to the joint to be predicted based on the adjacency matrix, and then reads the 3D coordinates and feature vectors of these visible joints. Subsequently, it determines whether the joint to be predicted belongs to the forelimb joint, hindlimb joint, or hoof joint based on the joint type label, and calls the corresponding length and angle constraints. The length constraint is determined by the 3D distance between adjacent joints in the unoccluded calibration sample; the angle constraint is determined by the angle between adjacent limb segments in the unoccluded calibration sample. When validating the candidate length range and candidate angle range, the number of real skeletons that are excluded and the number of erroneous skeletons that are retained are counted, and the candidate range with the minimum sum of the two types of errors is taken as the corresponding constraint range.

[0195] Furthermore, when the coordinate prediction model predicts the 3D coordinates of the occluded joint, it first generates candidate coordinates based on the adjacent visible joints of the joint to be predicted.

[0196] Specifically, the process of generating candidate coordinates is as follows: based on the three-dimensional coordinates of adjacent visible joints, and according to the length and direction constraints of the limb segment to which the joint belongs, spatial candidate points that satisfy the length and direction ranges are generated in the camera coordinate system.

[0197] Subsequently, the coordinate prediction model scores each candidate coordinate based on the feature vector output by the graph convolutional network. The scoring criteria include the length deviation between the candidate coordinate and the adjacent visible joint, the joint angle deviation formed by the candidate coordinate, and the consistency between the skeleton topology corresponding to the candidate coordinate and the adjacency matrix. The length deviation is obtained by subtracting the corresponding standard limb length from the 3D distance between the candidate coordinate and the adjacent visible joint and taking the absolute value. The angle deviation is obtained by subtracting the corresponding standard joint angle from the angle between the candidate limb segment and the adjacent limb segment and taking the absolute value. The topology consistency is determined based on whether the candidate joint maintains the connection relationship in the adjacency matrix. The length deviation, angle deviation, and topology consistency scores are combined into a candidate coordinate score according to the weights obtained from training, and the candidate coordinates whose scores meet the preset selection rules are selected as the 3D coordinates of the occluded joint.

[0198] The preset selection rule is that the candidate coordinates score the best result, and the candidate coordinates satisfy the corresponding limb length range and angle range.

[0199] Furthermore, when there are more than two candidate coordinates for the same occluded joint that can be inferred from different adjacent visible joints, the total length deviation and total angle deviation between each candidate coordinate and all related visible joints are calculated respectively, and the candidate coordinate with the smallest weighted result of the total length deviation and total angle deviation is selected as the three-dimensional coordinate of the occluded joint.

[0200] Specifically, the weights used for weighting are determined by the calibration samples. During the determination process, each candidate weight combination is verified, the spatial distance error between the predicted coordinates and the manually labeled coordinates is statistically analyzed, and the candidate weight combination with the minimum spatial distance error is taken as the actual weight.

[0201] Furthermore, after obtaining the three-dimensional coordinates of the occluded joint, the coordinates are written into the corresponding pig skeleton node, and the node is marked as a prediction node. The prediction node establishes skeleton connection edges with the adjacent visible joints according to the adjacency matrix, thereby obtaining the spatial position of the occluded joint in the camera coordinate system.

[0202] Example 8: S7 includes obtaining the initial depth value of the occluded joint, which is extracted from the missing limb region; constructing a three-dimensional spatial vector based on the initial depth value and the three-dimensional coordinates of the visible joint to correct the spatial position of the occluded joint; reconstructing the complete skeleton structure based on the spatial position of the occluded joint and the three-dimensional coordinates of the visible joint; extracting the skeletal spatial features of the complete skeleton structure, and obtaining the posture detection results of each pig based on the skeletal spatial features.

[0203] In this embodiment, the initial depth value of the occluded joint is first obtained.

[0204] The specific processing is as follows: the missing limb region obtained by S5 is mapped to the same pixel coordinate position in the depth information matrix, and the depth value of each pixel in the missing limb region is read; pixels with a depth value of 0, exceeding the effective range of the depth acquisition module, or marked as invalid in the pixel-level alignment stage are not included in the calculation.

[0205] Then, the centroid coordinates of the missing limb region are calculated. The centroid coordinates are obtained by the average horizontal and vertical coordinates of all valid pixels in the region. A depth sampling region is selected with the centroid coordinates as the center, and the valid depth values ​​in the depth sampling region are counted. All valid depth values ​​are added together and divided by the number of valid depth values. The result is used as the initial depth value of the occluded joint.

[0206] In this embodiment, the radius of the depth sampling region is determined by the calibration sample.

[0207] Specifically, the calibrated samples pre-label the missing limb region and the projection position of the occluded joint. The pixel distance from the projection position of the occluded joint to the centroid of the missing limb region is calculated, and the candidate sampling radius is verified one by one. The number of actual occluded joint projection positions that do not fall into the sampling region and the number of depth pixels of adjacent pigs that are included in the sampling region are counted. The candidate radius that is the minimum of the sum of the above two types of quantities is taken as the depth sampling region radius. If the number of effective depth values ​​in the depth sampling region is less than the depth sampling quantity threshold, the initial depth value is not used to correct the occluded joint.

[0208] The depth sampling number threshold is determined by the calibration samples. Specifically, it is the minimum number of effective depth pixels required to meet the calibration requirements when the depth error of the occluded joint is statistically analyzed, and this number is used as the depth sampling number threshold.

[0209] Furthermore, after obtaining the initial depth value, a three-dimensional spatial vector is constructed based on the initial depth value and the three-dimensional coordinates of the visible joints.

[0210] Specifically, first, identify the visible joints that have a direct skeletal connection with the obscured joint; this connection is determined by the pre-established pig skeletal structure, for example, when the forelimb is missing, select the shoulder joint, elbow joint or wrist joint adjacent to the obscured forelimb joint, and when the hindlimb is missing, select the hip joint, knee joint or tarsal joint adjacent to the obscured hindlimb joint; then convert the centroid coordinates of the missing limb area and the initial depth value into the three-dimensional coordinates of the spatial reference point.

[0211] During the transformation, the horizontal pixel coordinates of the centroid are subtracted from the horizontal coordinates of the camera principal point, the result is multiplied by the initial depth value, and then divided by the camera's horizontal focal length parameter to obtain the horizontal spatial coordinates of the spatial reference point. The vertical pixel coordinates of the centroid are subtracted from the vertical coordinates of the camera principal point, the result is multiplied by the initial depth value, and then divided by the camera's vertical focal length parameter to obtain the vertical spatial coordinates of the spatial reference point. The depth coordinates of the spatial reference point are taken from the initial depth value. Using the 3D coordinates of adjacent visible joints as the vector starting point and the 3D coordinates of the spatial reference point as the vector ending point, the differences between the ending point and the starting point in the horizontal, vertical, and depth coordinates are calculated respectively to obtain the 3D spatial vector.

[0212] As a preferred embodiment, when correcting the spatial position of the obscured joint, the three-dimensional coordinates of the obscured joint predicted by S6 are first read, and the spatial distance between the predicted three-dimensional coordinates and the three-dimensional coordinates of the adjacent visible joints is calculated.

[0213] Specifically, the spatial distance is obtained by adding the absolute values ​​of the differences in the horizontal spatial coordinates, the vertical spatial coordinates, and the depth coordinates; then it is determined whether the spatial distance falls within the range of the corresponding limb length, and whether the absolute value of the difference between the predicted depth coordinates and the initial depth value is not greater than the depth correction threshold; when the above two conditions are met simultaneously, the three-dimensional coordinates predicted by S6 are retained as the spatial position of the occluded joint.

[0214] Specifically, the corresponding limb length range is determined by unobstructed calibration samples.

[0215] The calibration samples were divided according to the growth stages of the pigs and the three-dimensional coordinates of adjacent joints were pre-labeled. The spatial distance between adjacent joints in the same limb segment was statistically analyzed, and the lower and upper length limits that could retain the true connection of adjacent joints and exclude the incorrect connection of joints were selected as the corresponding limb length range. The depth correction threshold was determined by the calibration samples with occluded joint labels. Specifically, the absolute value of the difference between the true depth value of the occluded joint and the initial depth value of the missing limb area was calculated. Candidate thresholds were judged one by one, and the number of true joint depths that were incorrectly excluded and the number of incorrect depth positions that were incorrectly retained were counted. The candidate threshold with the minimum sum of the two types of numbers was taken as the depth correction threshold.

[0216] Furthermore, when the predicted three-dimensional coordinates do not meet the aforementioned correction conditions, candidate spatial positions are generated along the three-dimensional spatial vector.

[0217] The specific generation process is as follows: Starting from the three-dimensional coordinates of adjacent visible joints, candidate points are generated sequentially within the corresponding limb length range according to the direction of the three-dimensional spatial vector; each candidate point has horizontal spatial coordinates, vertical spatial coordinates, and depth direction coordinates; for each candidate point, the length deviation, depth deviation, and direction deviation are calculated; the length deviation is the absolute value of the difference between the spatial distance between the candidate point and the adjacent visible joint and the standard length of the corresponding limb; the standard length of the corresponding limb is obtained by the arithmetic mean of the spatial distances between adjacent joints in the limb segment in the unobstructed calibration sample; the depth deviation is the absolute value of the difference between the depth direction coordinate of the candidate point and the initial depth value; the direction deviation is the angle between the spatial direction formed by the candidate point relative to the adjacent visible joint and the direction of the three-dimensional spatial vector.

[0218] The length deviation, depth deviation, and direction deviation are then multiplied by their respective weights and summed to obtain the candidate review value. The candidate point with the smallest review value that also meets the corresponding limb length range is selected as the corrected spatial position of the occluded joint. The weights are determined by the calibration samples. Each candidate weight combination is verified, and the spatial distance error between the corrected joint coordinates and the manually labeled joint coordinates is calculated. The candidate weight combination with the smallest spatial distance error is taken as the actual weight.

[0219] Furthermore, after obtaining the corrected spatial position of the occluded joint, the complete skeleton structure is reconstructed based on the spatial position of the occluded joint and the three-dimensional coordinates of the visible joint.

[0220] Specifically, during reconstruction, both visible and occluded joints are treated as skeleton nodes. Each skeleton node records the joint type, 3D coordinates, and visibility state. The visibility state of visible joints is marked as direct observation, while the visibility state of occluded joints is marked as prediction correction.

[0221] The connection edges between skeletal nodes are generated according to the pre-established anatomical connection relationships of pigs, including connections between trunk nodes, connections between trunk nodes and forelimb nodes, connections between trunk nodes and hindlimb nodes, and connections between adjacent joints within the same limb. In the case where there are more than two obscured joints in the same limb, they are connected sequentially from the trunk connection end to the limb end, and the spatial distance between adjacent skeletal nodes is checked segment by segment. When the spatial distance of a connection segment exceeds the length range of the corresponding limb, the spatial position of the end joint of the connection segment is corrected again according to the candidate spatial position generation method until the connection segment meets the length range of the corresponding limb. After all nodes are connected, the complete skeletal structure of the corresponding pig is obtained.

[0222] Furthermore, after the complete skeleton structure is generated, the spatial features of the skeleton are extracted.

[0223] Specifically, the skeletal spatial features include joint three-dimensional coordinates, spatial distance between adjacent joints, angle between adjacent limb segments, trunk axis direction, positional relationship of limb ends relative to the trunk, and joint visibility status. The spatial distance between adjacent joints is calculated as the sum of the absolute values ​​of the differences between the two joints in the lateral, longitudinal, and depth coordinates. The angle between adjacent limb segments is determined by the difference in the spatial directions of the two limb segments, and the spatial direction of each limb segment is determined by the difference in the three-dimensional coordinates of the joints at both ends of the limb segment. The trunk axis direction is determined by the coordinate distribution of the head node, shoulder node, hip node, and tail node. Specifically, the three-dimensional coordinates of the above nodes are read, and the direction in which the maximum value of the coordinate distribution range is taken as the trunk axis direction. The positional relationship of the limb ends relative to the trunk is determined by the spatial distance from the hoof end node to the trunk axis and the orientation of the hoof end node on the trunk axis.

[0224] As a preferred embodiment, when obtaining the posture detection results of each pig based on the skeletal spatial features, the skeletal spatial features are input into a pre-established posture determination rule or posture classification model.

[0225] Specifically, when using the posture determination rule, the distance from the torso node to the ground is first calculated based on the ground calibration plane, and then the support state is determined based on the position of the limb end node relative to the main axis of the torso. The ground calibration plane is determined by the empty pen depth data collected when the acquisition device is installed, and is obtained by fitting the three-dimensional coordinates of the calibration point on the pen floor.

[0226] The rules for determining the standing posture are as follows: the distance from the trunk node to the ground is greater than the standing threshold, and both the forelimb and hindlimb distal nodes fall into the support area below the trunk; the rules for determining the prone posture are as follows: the distance from the trunk node to the ground is less than the prone threshold, and the distal nodes fall into the lateral area of ​​the trunk; the rules for determining the walking posture are as follows: the distance from the trunk node to the ground is greater than the standing threshold, and at least one distal node of the forelimb or hindlimb is displaced relative to the support area on the same side; the rules for determining the lateral posture are as follows: the angle between the trunk axis and the ground calibration plane falls into the lateral angle range, and the distal nodes of both limbs are concentrated in the same area of ​​the trunk.

[0227] In this embodiment, the standing determination threshold, the prone determination threshold, the support area, the lateral area, the displacement threshold, and the lateral angle range are all determined by the posture calibration samples.

[0228] In this process, the posture calibration samples are pre-labeled with categories such as standing, walking, lying down, and side-lying. Each candidate threshold or candidate range is judged one by one, and the number of misclassifications for each posture category is counted. The candidate value with the minimum sum of misclassifications is taken as the corresponding threshold or range. When using the posture classification model, the skeleton space features are input into the classification model, the classification model outputs the score of each posture category, and the posture category with the highest score is taken as the posture detection result of the corresponding pig.

[0229] The final output of the posture detection results includes the individual pig number, complete skeletal structure, 3D coordinates of visible joints, 3D coordinates of occluded joints, occluded joint markers, and posture category.

[0230] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for pig pose detection based on the fusion of RGB images and depth images, characterized in that, include: S1. Obtain RGB and depth images of group-raised pigs under the same field of view, determine the cross-modal alignment offset, and obtain the depth information matrix corresponding to the RGB images; S2. Extract features from the RGB image and the depth information matrix respectively to obtain RGB feature maps and depth feature maps. Then fuse the RGB feature maps and depth feature maps to obtain a multimodal fusion feature map. S3. Based on the apparent similarity, depth difference and spatial adjacency relationship between pixels in the multimodal fusion feature map, calculate the overlap confidence of the pixel belonging to the overlapping region of pigs. If the overlap confidence is greater than the preset overlap threshold, the corresponding pixel is determined to belong to the overlapping region, and the overlapping region mask containing multiple pigs is obtained. S4. For the multimodal fusion feature map corresponding to the overlapping region mask, extract the apparent gradient change features and depth gradient change features in the overlapping region to determine the individual boundaries of each pig in the overlapping region. S5. Obtain the visible body outline within the individual boundary, calculate the matching degree between the visible body outline and the pre-established complete pig morphology template. If the matching degree is less than the preset matching threshold, it is determined that there is limb occlusion in the visible body outline, and the incomplete limb area is obtained. S6. For the missing limb area, determine the three-dimensional coordinates of the visible joints based on the depth information matrix, extract the topological relationship between the visible joints in the missing limb area, predict the three-dimensional coordinates of the occluded joints based on the three-dimensional coordinates of the visible joints and the topological relationship, and obtain the spatial position of the occluded joints. S7. Based on the spatial location of the obscured joints and the three-dimensional coordinates of the visible joints, reconstruct the complete skeletal structure of the pig corresponding to the missing limb area, and obtain the posture detection results of each pig.

2. The pig pose detection method based on RGB image and depth image fusion according to claim 1, characterized in that: S1 includes: RGB and depth images of group-raised pigs under the same field of view are acquired, and the RGB images are processed to obtain a set of color contour features. Boundary continuity features are extracted from the color contour feature set, and depth transition feature set is extracted from the depth image; Calculate the transition consistency distribution map based on the boundary continuity characteristics and the depth transition characteristics set; If the local extrema of the transition consistency distribution map are greater than the preset threshold, the cross-modal alignment offset is determined, and pixel-level alignment is performed using the cross-modal alignment offset to obtain the depth information matrix corresponding to the RGB image.

3. The pig pose detection method based on RGB image and depth image fusion according to claim 1, characterized in that: S2 includes: RGB images and depth information matrices of pigs are acquired, and RGB feature maps and depth feature maps are extracted using a convolutional neural network. The occluded areas of pigs are extracted from the RGB feature map and the depth feature map. The degree of texture loss is determined based on the color features of the occluded areas, and the depth boundary clarity is obtained based on the depth features of the occluded areas. The confidence levels of the color modality and depth modality are determined based on the degree of texture loss and the clarity of the depth boundary. The RGB feature map and the depth feature map are weighted and fused based on the confidence levels of the color modality and the depth modality to obtain a multimodal fused feature map.

4. The pig pose detection method based on RGB image and depth image fusion according to claim 1, characterized in that: S3 includes: Obtain a multimodal fusion feature map, and calculate the appearance similarity and depth difference of pixels based on the multimodal fusion feature map; Spatial adjacency relationships are extracted based on apparent similarity and depth difference, and pixel overlap confidence is calculated based on spatial adjacency relationships. If the overlap confidence is greater than the preset overlap threshold, the corresponding pixel is extracted to generate the initial overlap region. Connectivity labeling is performed on the initial overlapping region to obtain the overlapping region mask containing multiple pigs.

5. The pig pose detection method based on RGB image and depth image fusion according to claim 1, characterized in that: S4 includes: Obtain the multimodal fusion feature map corresponding to the overlapping region mask, and process the multimodal fusion feature map using an edge detection operator to obtain the initial gradient matrix; Based on the initial gradient matrix, the apparent gradient change features and the deep gradient change features are extracted to obtain the fused gradient magnitude map; Non-edge pixels are removed from the fused gradient magnitude map to obtain a refined edge feature map. A candidate boundary map containing strong and weak edge points is generated based on the refined edge feature map. Connectivity analysis is performed on strong and weak edge points in the candidate boundary map to determine the individual boundaries of each pig within the overlapping region.

6. The pig pose detection method based on RGB image and depth image fusion according to claim 1, characterized in that: S5 includes: Extract visible body contours based on individual boundaries and construct a target topology map of the visible body contours; Spatial mapping is performed between the target topology map and the template topology map of the pre-established complete pig morphology template to obtain the aligned contour map; The aligned outline image is compared with the complete pig morphology template to obtain the set of differences in pixels and the matching degree. If the matching degree is less than the preset matching threshold, it is determined that there is limb occlusion in the visible body outline, and the missing limb region is extracted based on the set of difference pixels.

7. The pig pose detection method based on RGB image and depth image fusion according to claim 1, characterized in that: S6 includes: The depth pixel values ​​corresponding to the missing limb regions are extracted from the depth information matrix, and the three-dimensional coordinates of the visible joints are determined based on the depth pixel values. A graph convolutional network is used to extract feature vectors from the skeleton node graph constructed based on 3D coordinates, and an adjacency matrix is ​​generated based on the feature vectors to determine the topological relationship between visible joints.

8. The pig pose detection method based on RGB image and depth image fusion according to claim 7, characterized in that: S6 further includes: A coordinate prediction model is constructed based on three-dimensional coordinates and topological relationships. The three-dimensional coordinates of the occluded joint are predicted by the coordinate prediction model, and the spatial position of the occluded joint is obtained.

9. The pig pose detection method based on RGB image and depth image fusion according to claim 1, characterized in that: S7 includes: Obtain the initial depth value of the obscured joint, which is extracted from the region of the missing limb; A three-dimensional spatial vector is constructed based on the initial depth value and the three-dimensional coordinates of the visible joints to correct the spatial position of the occluded joints.

10. The pig pose detection method based on RGB image and depth image fusion according to claim 9, characterized in that: The S7 also includes: The complete skeleton structure is reconstructed based on the spatial position of the obscured joints and the three-dimensional coordinates of the visible joints; Extract the skeletal spatial features of the complete skeleton structure, and obtain the posture detection results of each pig based on the skeletal spatial features.

Images