A multi-modal information fusion-based cattle identity recognition method

Abstract

Provided is a cattle identity recognition method based on multi-modal information fusion, comprising: acquiring an image of a face of a to-be-recognized cattle; generating a 2D feature vector according to the image of the face of the to-be-recognized cattle; generating a 3D point cloud from the cattle face image; performing cropping on the 3D point cloud to remove a 3D point cloud area corresponding to a cattle cheek area and to retain a 3D point cloud area above the cheek area; generating a 3D feature vector according to the cropped 3D point cloud; acquiring, according to the 2D feature vector, a first plurality of candidate results most similar to the 2D feature vector from a cattle feature database in order of similarity; acquiring, according to the 3D feature vector, a second plurality of candidate results most similar to the 3D feature vector from the cattle feature database in order of similarity; if the first plurality of candidate results and the second plurality of candidate results have an intersection, and the confidence of the candidate results in the intersection is greater than a threshold value, taking the candidate results in the intersection with the confidence greater than the threshold value as an identity recognition result of the to-be-recognized cattle.

Description

Technical Field

[0001] This invention relates to livestock identification technology, and more specifically, to a method for automatically identifying individual cattle based on biometrics. Background Technology

[0002] Identification applications aim to determine the uniqueness of an individual target. This differs from category identification, which focuses on the type of target, by identifying which of multiple objects of the same type the target belongs to. Currently, physical identification methods such as ear tags and radio frequency identification (RFID) are commonly used in livestock farms. However, these methods suffer from problems such as fragility, loss, the need for close contact, and the risk of animal stress. Therefore, the industry is dedicated to researching non-contact, stress-free biometric identification technologies to achieve more intelligent and traceable livestock management. These technologies have significant application value in areas such as dairy cow performance testing, precision feeding of beef cattle, disease prevention and control, breed registration, and agricultural insurance.

[0003] In the field of biometric-based visual recognition, existing research is mainly divided into two categories: identity recognition based on two-dimensional images and identity recognition based on three-dimensional point clouds.

[0004] 1. A two-dimensional image-based cow face recognition scheme for identifying cattle. This scheme draws on mature facial recognition technology, directly extracting and comparing features from acquired 2D images of cow faces. The implementation process typically involves the following: First, preprocessing the input cow face image, including grayscale conversion, normalization, keypoint detection, and alignment; then, using a deep convolutional neural network (such as FaceNet or VGG) as a feature extractor to map the input image into a high-dimensional feature vector; finally, by calculating the cosine or Euclidean distance between the feature vector of the image to be identified and the pre-stored feature vectors in the database, identity recognition is achieved based on similarity ranking.

[0005] Such methods are widely used in academic research. For example, in patent literature (such as Chinese patent application CN120148068A, "A method for recognizing cow faces based on multi-scale feature interaction and its recognition system"), convolutional neural networks are used to identify the identity of cows.

[0006] 2. A cattle identification scheme based on 3D point cloud for identifying cattle heads. This scheme aims to overcome the shortcomings of two-dimensional images, which are easily affected by environmental interference, by utilizing the three-dimensional geometric structure information of the cow's head for identification. Its typical implementation process is as follows: a three-dimensional point cloud model of the cow's head is reconstructed using a 3D scanning device or from multiple 2D images; subsequently, a deep learning network specifically designed for processing point cloud data is used to learn features and classify the point cloud; the network output is either the corresponding cow's identification ID or a point cloud feature vector used for comparison.

[0007] This type of method falls under the application of 3D target recognition technology in computer vision to the agricultural field. Patent literature (such as Chinese patent application number CN119992588A, "A method for recognizing cow faces based on dense point cloud 3D reconstruction") discloses a technical solution that achieves individual recognition by acquiring dense 3D point clouds of cow faces and using deep learning models for feature matching.

[0008] Although the aforementioned technical solutions have made some progress, they all rely on information from a single mode and have inherent, insurmountable performance bottlenecks. Specific shortcomings are analyzed below: 1. Deficiencies of recognition schemes based on two-dimensional images The core flaw of this scheme lies in its feature stability, which is heavily dependent on the imaging environment. A two-dimensional image is essentially a projection of a three-dimensional scene under specific lighting conditions; therefore: Susceptible to changes in lighting: The facial texture and color of the same cow vary greatly under different times, weather conditions, or indoor and outdoor light sources, causing significant shifts in the extracted feature vectors and drastic fluctuations in recognition accuracy.

[0009] Sensitive to pose and occlusion: When a cow's head turns, tilts, or its face is partially obscured by feed or dirt, the texture information captured in the image is incomplete or deformed, leading to feature matching failure. This deficiency stems from the fact that 2D methods lack an understanding of the target's 3D structure and cannot compensate at the geometric level. Moreover, cows do not actively cooperate with the 2D image acquisition process, and the environment in which they are located is usually not conducive to image acquisition, which further exacerbates the problems caused by pose and occlusion.

[0010] Limited ability to distinguish similar individuals: For cattle of the same breed, with similar coat color and pattern, it is difficult to make a high-confidence distinction based solely on the appearance and texture features, which can easily lead to misidentification.

[0011] 2. Deficiencies of 3D point cloud-based recognition schemes While this scheme avoids some of the environmental sensitivity of two-dimensional images, its main drawbacks lie in the lack of information dimension and insufficient computational characteristics. The method loses crucial texture details: 3D point clouds primarily represent the geometric contours and surface shapes of objects, but they cannot effectively capture highly discriminative texture information for individual cattle, such as nose markings, spots, and subtle hair direction. Therefore, for cattle with highly similar 3D geometric shapes, the discriminative power of this method decreases significantly.

[0012] Feature extraction networks have limitations: Existing point cloud processing networks, such as PointNet++, are primarily designed for object classification or part segmentation. When directly used for identifying similar individuals of the same species that require extremely high discriminative power, the learned features are not sensitive enough for fine-grained discrimination and cannot directly output highly discriminative features for similarity ranking.

[0013] High data acquisition and processing costs: Obtaining high-quality dense point clouds usually requires specialized 3D scanning equipment, or strict shooting conditions and a large number of images during reconstruction, which limits its feasibility for field deployment in ranch environments.

[0014] In summary, existing single-modal recognition schemes suffer from inherent limitations in their information sources, resulting in irreconcilable contradictions in accuracy, robustness, and universality. While two-dimensional image recognition schemes are information-rich, they are fragile; conversely, three-dimensional point cloud recognition schemes are stable but lack comprehensive information. This is precisely the core technical problem that this invention aims to solve: how to design a recognition framework that integrates the advantages of both two-dimensional and three-dimensional modalities while mitigating their respective disadvantages, to achieve highly accurate and stable cattle identification in complex real-world scenarios. Summary of the Invention

[0015] The purpose of this invention is to overcome the shortcomings of existing cattle identification technologies and provide a cattle identification method and system based on multimodal information fusion.

[0016] The shortcomings and problems to be overcome: Existing technologies mostly rely on single-modal image recognition or three-dimensional point cloud recognition. Two-dimensional methods are easily affected by environmental factors such as lighting, pose, and hair occlusion, resulting in poor feature stability; while pure three-dimensional methods are insufficient in texture detail resolution. In complex real-world scenarios, both of these single-modal approaches encounter bottlenecks in recognition accuracy and robustness, making it difficult to meet the requirements of high-reliability identity authentication.

[0017] To address this, this invention proposes an innovative fusion decision-making scheme: a "multimodal collaborative authentication strategy." This strategy aims to construct a collaborative recognition framework by fusing the stereoscopic geometric similarity of 3D point cloud data with the pattern and hair texture features of 2D image data, leveraging the complementarity of their information representation. This framework significantly improves the anti-interference capability and discrimination accuracy of cattle identification, ultimately achieving an accuracy rate exceeding 90%, providing a stable and reliable identification solution for modern livestock management.

[0018] According to a first aspect of this application, a method for cattle identification based on multimodal information fusion is provided, comprising: acquiring a frontal cattle face image, a left 45-degree cattle face image, and a right 45-degree cattle face image of the cattle to be identified; generating a 2D feature vector based on the frontal cattle face image, the left 45-degree cattle face image, and / or the right 45-degree cattle face image of the cattle to be identified; feeding the frontal cattle face image, the left 45-degree cattle face image, and the right 45-degree cattle face image of the cattle to be identified into a Dust3R model to generate a 3D mesh model of the cattle face to be identified; converting the 3D mesh model of the cattle face to be identified into a 3D point cloud; cropping the 3D point cloud, removing the 3D point cloud region corresponding to the cheek area of ​​the cattle face, while retaining the 3D point cloud region above the cheek area; The removal of the 3D point cloud region corresponding to the cow face cheek includes: fitting a cow face overall geometric reference plane from the 3D point cloud, represented by the plane equation n·x + d = 0, where n represents the unit normal vector of the plane equation and d is a plane constant parameter; calculating the position of each point in the 3D point cloud relative to the cow face overall geometric reference plane represented by the plane equation; and deleting points from the 3D point cloud that are located below the cow face overall geometric reference plane relative to it. A 3D feature vector is generated based on the cropped 3D point cloud; a first plurality of candidate results most similar to the 2D feature vector are obtained from the cattle feature database according to similarity sorting; wherein, the cattle feature database records the cattle identity in association with its 2D feature vector and 3D feature vector; a second plurality of candidate results most similar to the 3D feature vector are obtained from the cattle feature database according to similarity sorting; if the first plurality of candidate results and the second plurality of candidate results have an intersection, and the confidence of the candidate results in the intersection is greater than a threshold, the candidate results in the intersection with a confidence greater than the threshold are used as the identification results of the cattle to be identified.

[0019] The cattle identification method according to the first aspect of this application further includes: solving the plane equation to obtain the solution of parameters n and d; if the normal vector of the plane equation represented by n points in a specified direction, flipping the plane equation by n = -n and d = -d; wherein, the specified direction is the direction in which its Z-axis component is negative, and the Z-axis is the coordinate axis perpendicular to the ground and pointing upward in the Cartesian coordinate system.

[0020] According to the cattle identification method of the first aspect of this application, the signed distance from a point x in the 3D point cloud to the overall geometric reference plane of the cattle face is calculated according to s = n·x + d, where x is the coordinate vector of the point in the 3D point cloud in the Cartesian coordinate system; the step of deleting points located below the overall geometric reference plane of the cattle face from the 3D point cloud includes: deleting the corresponding point x from the 3D point cloud when s < 0; and retaining the corresponding point x in the 3D point cloud when s > 0.

[0021] According to the cattle identification method of the first aspect of this application, the step of acquiring the frontal cattle face image, the left 45-degree cattle face image, and the right 45-degree cattle face image of the cattle to be identified includes: when acquiring the cattle face image, identifying the cattle face posture in real time through a cattle face recognition model based on YOLOv8; when the acquired cattle face posture is in the frontal view, the left 45-degree view, or the right view, triggering the capture to acquire the cattle face image. The cow face region is extracted from the acquired cow face image, and the background outside the cow face region is removed to obtain the segmented cow face image. The segmented cow face image is matched with the optimal viewpoint template of the cow face image to output a qualified cow face image with the same posture as the optimal viewpoint template. The optimal viewpoint template includes a preset frontal cow face image, a left 45° cow face image, and a right 45° cow face image. If the obtained qualified cow face images include at least one qualified cow face image from each of the three views, the frontal cow face image, the left 45° cow face image, and the right 45° cow face image, then 3D feature vectors are generated using these qualified cow face images, and 2D feature vectors are generated using the qualified cow face images from the frontal view, the left 45° view, and / or the right 45° view.

[0022] The cattle identification method according to the first aspect of this application further includes: obtaining the confidence level generated when the Dust 3R model generates the 3D mesh model, and identifying the availability of the 3D mesh model; detecting and removing black edges of the 3D mesh model for the 3D mesh model with availability greater than a specified threshold; and converting the 3D mesh model with the black edges removed into the 3D point cloud.

[0023] According to the cattle identification method of the first aspect of this application, the confidence features of the 3D mesh model are obtained, including the best edge confidence score and the second-best edge confidence score output by the Dust 3R model, as well as the proportion of low-confidence points cleaned up, the average confidence score, and the proportion of high-confidence noise points when the Dust 3R model generates the 3D mesh model; the usability of the 3D mesh model is identified using a usability verification model based on the random forest algorithm, according to the indicators used to predict the usability of the 3D mesh model; for each mesh vertex on the edge of the 3D mesh model, the sum of the color component values ​​of the R, G, and B color channels of each mesh vertex is calculated; when the sum of the color component values ​​of the R, G, and B channels of a vertex is detected to be lower than the judgment threshold, the vertex is determined to be a black edge and is removed from the 3D mesh model.

[0024] The cattle identification method according to the first aspect of this application further includes: transforming the pose of the 3D point cloud to be consistent with that of a reference point cloud before cropping the 3D point cloud, wherein the reference point cloud represents a 3D point cloud of a cow's face with a standard pose; and scaling the 3D point cloud to a uniform unit scale.

[0025] According to the cattle identification method of the first aspect of this application, the step of generating a 3D feature vector from the cropped 3D point cloud includes: receiving the cropped 3D point cloud using an H-PointNet++ model to generate a 3D feature vector, wherein the H-PointNet++ model is based on the PointNet++ model, and a new SA layer is added before the three SA layers of the PointNet++ model. The new SA layer uniformly samples 1024 center points from the 3D point cloud according to the farthest point, wherein the local neighborhood sampling radius of each center point is smaller than the local neighborhood of the first SA layer of the PointNet++ model. The sampling radius is such that the number of local neighborhood sampling points at each center point is greater than the number of local neighborhood sampling points in the first SA layer of the PointNet++ model. The H-PointNet++ model replaces the output head of the PointNet++ model with a feature vector output head, wherein the feature vector output head receives the output of the last SA layer of the PointNet++ model and generates a 3D feature vector representing the 3D point cloud. The feature vector output head includes a feature compression layer, a feature distribution normalization layer, a nonlinear mapping layer, a normalization layer, and a vector normalization layer connected in sequence. The vector normalization layer outputs the 3D feature vector.

[0026] According to the cattle identification method of the first aspect of this application, the training samples for training the H-PointNet++ model are triplets including anchor samples, positive samples, and negative samples; wherein, the anchor samples serve as a reference benchmark for identification; the positive samples and anchor samples are 3D point clouds obtained from the face images of the same cattle; the negative samples and anchor samples are 3D point clouds from cattle with different identities; the H-PointNet++ model is trained using the loss function L = max(d(a,p) - d(a,n) + margin,0), where a, p, and n represent the feature vectors of the anchor samples, positive samples, and negative samples, respectively; d() represents the Euclidean distance calculation function, and margin represents the margin threshold. The optimization objective of the training process is: For sample triples, the network parameters are adjusted so that the feature distance between the anchor sample and the positive sample is less than the feature distance between the anchor sample and the negative sample, and the difference between the two must meet the margin requirement. In the early stages of training the H-PointNet++ model, a semi-hard sample triplet is constructed, consisting of an anchor sample, a positive sample, and a negative sample whose feature vector is far from the anchor sample but not completely separated; the H-PointNet++ model is then trained using the semi-hard sample triplet. During the training of the H-PointNet++ model, a mixed sample consisting of semi-hard sample triples and hard sample triples is constructed, and the H-PointNet++ model is trained using the mixed sample; wherein the negative sample in the hard sample triple comes from the heterogeneous sample whose feature vector is closest to that of the anchor sample.

[0027] According to the cattle identification method of the first aspect of this application, in order to construct semi-hard sample triples, the anchor sample and multiple negative samples with different identities from the anchor sample are mapped to feature vectors using the currently trained H-PointNet++ model. The negative samples whose feature vectors are greater than the distance between the feature vectors of the positive samples and the feature vectors of the anchor sample, and whose distance difference is less than the margin, are selected to construct semi-hard sample triples. In order to construct hard sample triples, the anchor sample and multiple negative samples with different identities from the anchor sample are mapped to feature vectors using the currently trained H-PointNet++ model. The negative sample whose feature vectors are less than the distance between the feature vectors of the anchor sample and the feature vectors of the anchor sample are selected to construct hard sample triples.

[0028] According to the cattle identification method of the first aspect of this application, in order to construct semi-hard sample triples, the anchor sample and multiple negative samples with different identities from the anchor sample are mapped to feature vectors using the currently trained H-PointNet++ model. Negative samples whose feature vectors are more distant from the feature vectors of the anchor sample than the distance between the feature vectors of the positive sample and the feature vectors of the anchor sample, and whose distance difference is less than the margin, are selected to construct semi-hard sample triples. In order to construct hard sample triples, the anchor sample and multiple negative samples with different identities from the anchor sample are mapped to feature vectors using the currently trained H-PointNet++ model. Negative samples whose feature vectors are more distant from the feature vectors of the anchor sample than the distance between the feature vectors of the positive sample and the feature vectors of the anchor sample are selected to construct hard sample triples.

[0029] According to the cattle identification method of the first aspect of this application, the currently trained H-PointNet++ model used to construct semi-difficult sample triples is an H-PointNet++ model that has not yet been trained using semi-difficult and difficult samples; the currently trained H-PointNet++ model used to construct difficult sample triples is an H-PointNet++ model trained with semi-difficult samples.

[0030] According to the cattle identification method of the first aspect of this application, during the training process, semi-difficult samples and difficult samples are selected from the training samples; wherein the selection criteria for semi-difficult samples are: the similarity between the feature vector of the negative sample and the feature vector of the anchor point is greater than the distance between the feature vector of the positive sample and the feature vector of the anchor point sample, and this distance difference is less than the margin; the selection criteria for difficult samples are: the distance between the feature vector of the negative sample and the feature vector of the anchor point sample is less than the distance between the feature vector of the positive sample and the feature vector of the anchor point sample.

[0031] According to a second aspect of this application, an information processing apparatus according to the second aspect of this application is provided, including a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the method according to the first aspect of this application. Attached Figure Description

[0032] When read in conjunction with the accompanying drawings, and by referring to the following detailed description of illustrative embodiments, this application, its preferred modes of use, and further objects and advantages will be best understood, wherein the drawings include: Figure 1 The working principle and processing flow of the cattle identification method according to the embodiments of this application are demonstrated.

[0033] Figure 2A schematic diagram illustrating the process of acquiring and preprocessing a cow's face image according to an embodiment of this application is shown.

[0034] Figure 3 A schematic diagram illustrating the process of generating and processing a cow face 3D point cloud according to an embodiment of this application is shown.

[0035] Figure 4A This shows a 3D model view of the cow's face before the black edges were removed.

[0036] Figure 4B A 3D model view of a cow's face after removing black edges using an embodiment of this application is shown.

[0037] Figure 5 The flowchart illustrates the size standardization stage in the 3D point cloud alignment process of the data processing flow of a 3D model according to an embodiment of this application.

[0038] Figure 6 The flowchart illustrates the multi-angle ICP pose alignment stage in the 3D point cloud alignment process of the 3D model data processing flow according to an embodiment of this application.

[0039] Figure 7 The before-and-after effects of 3D point cloud alignment according to an embodiment of this application are shown.

[0040] Figure 8 A flowchart illustrating the key region trimming stage of the data processing flow for a 3D model according to an embodiment of this application is shown.

[0041] Figure 9 The images show a comparison view before and after cropping the 3D point cloud of a cow's face according to an embodiment of this application.

[0042] Figure 10 A schematic diagram illustrating the structure and principle of the H-PointNet++ model according to an embodiment of this application is shown.

[0043] Figure 11 It displays 3D point clouds of different categories of objects.

[0044] Figure 12 This is a schematic diagram of the sampling points and sampling radius of SA1 in the H-Pointnet++ model according to an embodiment of this application.

[0045] Figure 13 A schematic diagram of the feature vector output head according to an embodiment of this application is shown.

[0046] Figure 14 This is a block diagram of an information processing device that implements the embodiments of this application. Detailed Implementation

[0047] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0048] This application addresses the inherent bottlenecks in accuracy and robustness of existing single-modal 2D image recognition or 3D point cloud recognition methods by innovatively proposing a "multimodal collaborative authentication strategy" dual-branch recognition architecture. Its core design logic is as follows: fully utilizing the insensitivity of 3D point cloud features to lighting, pose changes, and dirt interference to accurately extract the 3D geometric features of a cow's face; simultaneously, relying on the high discriminative power of 2D images in details such as patterns and hair texture, to efficiently extract planar texture features. Through a collaborative decision-making mechanism of 3D and 2D features, accurate identification of the cow's identity is achieved.

[0049] Figure 1 The working principle and processing flow of the cattle identification method according to the embodiments of this application are demonstrated.

[0050] The cattle identification system of this application includes a data acquisition device, a computer, and a cattle feature database.

[0051] See Figure 1 Images of cattle heads are acquired using image acquisition devices, such as cameras or mobile phones. To generate 3D point clouds, specific pose images of the cattle faces are acquired: a frontal view, a 45-degree left-side view, and a 45-degree right-side view. For each cow, at least one image of each of these specific poses is acquired. To generate two-dimensional features from the cattle face images, one or more of the frontal, 45-degree left-side, and 45-degree right-side view images can be used.

[0052] Optionally, a YOLOv8-based cow face recognition model is run in the image acquisition device to identify the pose of the cow face in the captured images in real time. When the cow face image acquired by the data acquisition device is a frontal, left 45-degree, or right 45-degree angle, automatic capture is triggered to obtain a cow face image with a specific pose required for subsequent processing. It is understood that running the YOLOv8-based cow face recognition model to achieve automatic capture is not mandatory, and users can capture cow face images with specific poses. Alternatively, video data of cow faces including various poses can be acquired through the image acquisition device, and cow face images with specific poses required for generating 3D point clouds and 2D features can be extracted from the video data in real time, or not required.

[0053] Images of a cow's face captured by an image acquisition device are provided to a computer. The computer-implemented processing flow includes 2D processing branches and 3D processing branches.

[0054] The 2D processing branch receives a cow face image and generates a 2D feature vector from it. Generating 2D feature vectors from images is a pre-existing technique. For example, a pre-trained FaceNet model can be used to process a frontal cow face image to generate a 2D feature vector reflecting the cow's identity. Another example is using a deep convolutional neural network to extract a 256-dimensional feature vector reflecting the cow's texture and patterns.

[0055] The 3D processing branch receives three cow face images for each cow: a frontal view, a 45-degree left-side view, and a 45-degree right-side view, and generates 3D feature vectors. Specifically, it generates 3D point clouds of the cow faces using the Dust 3R deep learning model (Dense and Unconstrained Stereo 3D Reconstruction), and then uses a 3D recognition model for classification or object detection to generate 3D feature vectors of the cow faces based on these point clouds. 3D recognition models include, for example, PointNet, PointNet++, Point Transformer, and Uni3D. The generated 3D feature vectors of the cow faces are, for example, 256-dimensional stereo geometric feature vectors.

[0056] Multimodal fusion decision-making is performed on the extracted 2D and 3D feature vectors of the cow's face to identify the cow's identity. For example, it identifies the identity ID or identifier of the cow captured by the image acquisition device in the cow feature database. The similarity between the 2D and 3D feature vectors of the cow's face and the 2D and 3D feature vectors of the cow's face in the cow database is calculated to obtain the top N cow identification results based on the 2D feature vectors and the top N cow identification results based on the 3D feature vectors of the cow's face. Intersection analysis and confidence judgment are performed on the top-N results of the two modalities, and the final identification result is output according to the preset rules (successfully identified cow, identified as a known cow not belonging to the cow feature database, or prompts to re-acquire cow face images).

[0057] The cattle feature database pre-stores 2D and 3D facial feature vectors of cattle. Before cattle identification, a cattle feature entry process is performed. This entry only needs to be done once. The cattle feature entry process is the same as... Figure 1 The demonstrated process is similar. Ideal images of cow faces are acquired using data acquisition equipment, including a frontal view, a 45-degree left-side view, and a 45-degree right-side view. 2D feature vectors are generated from the frontal, left-side, and / or right-side cow face images. A 3D feature vector is then generated by jointly analyzing these three images. The 2D and 3D feature vectors, along with the cow's identification information, are recorded in a cow feature database for subsequent feature vector similarity calculations.

[0058] Figure 2 A schematic diagram illustrating the process of acquiring and preprocessing a cow's face image according to an embodiment of this application is shown.

[0059] The image acquisition and preprocessing workflow is responsible for transforming the raw images acquired by the image acquisition device into high-quality, standardized input images and determining their subsequent processing path. The workflow is as follows: Figure 2 As shown, the specific steps are as follows: Step 1: Cow Face Data Acquisition. The image acquisition device is supported by a cow face recognition model based on YOLOv8, possessing automated data acquisition and classification capabilities. When acquiring cow face images, the cow face recognition model identifies the cow face posture in real time. When the acquired cow face posture is in a frontal view, a 45-degree left view, or a right view, automatic image capture is triggered, and the cow face image is simultaneously classified by viewpoint to label the posture information corresponding to the cow face image (frontal view, 45-degree left view, or right view).

[0060] Step 2: Cow face region segmentation. A cow face recognition model based on YOLOv8 is used to accurately segment the clean cow face region from the acquired original image, eliminating interference from complex backgrounds.

[0061] Optionally, steps 1 and 2 are performed by video capture, and multiple images of cow faces can be continuously accumulated before performing subsequent steps.

[0062] Step 3: Image quality screening and viewing angle determination.

[0063] After obtaining the segmented cow face image output from step 2, the images are first filtered, and those meeting quality requirements are used for subsequent processing. For example, the image is identified to determine if it contains a complete cow face; images containing a complete cow face are selected as meeting quality requirements. Next, the pose of the cow face in the image is identified. The poses of the cow face from a frontal view, a 45-degree left-side view, or a right-side view are used as the target poses. If the cow face image does not have a target pose, the current image is discarded, and the process returns to the step of obtaining the processed cow face image to obtain the next cow face image, repeating the above processing steps in step 3.

[0064] To identify whether a cow face image represents the target pose, optionally, the cow face image is aligned with preset optimal viewpoint templates: a frontal cow face image, a cow face image at a 45° left angle, and a cow face image at a 45° right angle. The confidence score of the cow face image to be processed matching each optimal viewpoint template is then output. When the confidence score of a certain viewpoint is the highest and exceeds a threshold (e.g., 0.8), the image to be processed is determined to have the target pose. By filtering the quality and viewpoint of the cow face images, high-quality cow face images are selected for subsequent 3D point cloud modeling.

[0065] Step 4: Intelligent scheduling of image processing branches.

[0066] Intelligent scheduling of 2D and 3D processing branches, which sends the selected cow face images to the appropriate processing branches, is one of the important steps in this application, used to provide each processing branch with cow face images that meet its needs.

[0067] For example, for selected cow face images, if there are at least one qualified image each from three perspectives—frontal view, left 45° view, and right 45° view—then these images are sent to the 3D processing branch, and simultaneously sent to the 2D recognition branch in parallel. If the above multi-view condition is not met, but there are qualified frontal or near-frontal view images, then all images are sent to the 2D processing branch.

[0068] Specifically, see Figure 2 Each time a preprocessed cow face image is acquired, if the current cow face image has a target pose (frontal view, left 45° view, or right 45° view), in addition to sending the cow face image to the 2D processing branch, it is further determined whether cow face images of all three views exist. If there is at least one cow face image of each of the three views, then the cow face images of all three views are sent to the 3D processing branch; otherwise, the process returns to the step of acquiring the preprocessed cow face image.

[0069] Optionally, when determining whether cow face images from all three perspectives exist, the number of cow face images already acquired for the current cow is also counted. If the number of cow face images from any perspective exceeds a threshold (e.g., 1), it is further determined whether the total number of cow face images already acquired for the current cow exceeds a threshold (e.g., 5). If the total number of cow face images exceeds the threshold (e.g., 5), all these cow face images are sent to the 2D processing branch; otherwise, the process returns to the step of acquiring preprocessed cow face images. Optionally, if the total number of acquired cow face images exceeds the threshold (e.g., 5), the cow face image from the frontal view or the closest to the frontal view is selected from these images and sent to the 2D processing branch.

[0070] When determining whether the current cow face image has the target pose (frontal view, left 45° view, or right 45° view), if the current cow face image does not have the target pose, the process proceeds to the step of determining whether the total number of cow face images acquired for the current cow exceeds a threshold (e.g., 5). If the total number of cow face images exceeds the threshold, all of these cow face images are sent to the 2D processing branch; otherwise, the process returns to the step of acquiring the preprocessed cow face image.

[0071] See also Figure 1 In the 3D processing branch, the Dust 3R model generates a 3D point cloud of the cow's face based on receiving three cow face images for each cow: a frontal cow face image, a cow face image at a 45-degree angle to the left, and a cow face image at a 45-degree angle to the right.

[0072] Figure 3 A schematic diagram illustrating the process of generating and processing a cow face 3D point cloud according to an embodiment of this application is shown.

[0073] Generating 3D point clouds of cow faces is a fundamental operation in the 3D recognition branch, with the goal of generating high-quality, standardized cow face point cloud data from multi-view images of cattle.

[0074] Step 1: 3D mesh reconstruction.

[0075] Using the Dust3R model (https: / / arxiv.org / abs / 2312.14132, https: / / github.com / naver / dust3r), input three selected images of cow faces (each with a frontal view, a left 45° view, or a right 45° view) to obtain a 3D mesh model of the cow's face.

[0076] Step 2: 3D mesh model usability verification.

[0077] Due to factors such as occlusion at the 2D image acquisition angle, fluctuations in equipment accuracy, and uneven lighting, the 3D models obtained from 3D reconstruction may have problems such as incomplete edges, dense noise, and poor geometric consistency. If such low-quality 3D models are input into subsequent processing, it will lead to distortion of point cloud data, with effective features being masked by noise, ultimately resulting in a decrease in the accuracy of feature extraction, and the accuracy of cow face recognition will be difficult to meet the needs of practical applications.

[0078] In this embodiment, a quantitative quality assessment mechanism is established to screen out high-quality cow face 3D models. This provides "clean and reliable" input data for subsequent point cloud noise filtering, standardization processing, and feature extraction, fundamentally avoiding interference from low-quality models in feature extraction, ensuring the authenticity and integrity of the cow face geometric features, and ultimately improving the robustness and accuracy of the overall recognition system. Low-quality 3D models are not used for subsequent processing.

[0079] Step 3: Black edge detection and filtering.

[0080] During 3D reconstruction, the 3D mesh model of a cow face is prone to black edge noise due to factors such as uneven lighting during image acquisition, gaps in multi-view image stitching, and vignetting in device imaging. This type of noise is essentially the mesh vertices corresponding to black pixels that have no effective geometric information. These vertices are spatially adjacent to the effective features of the cow face (such as horns, bridge of the nose, and periorbital contours), making them difficult to distinguish using traditional simple threshold filtering.

[0081] In this embodiment, for each mesh vertex on the edge of a 3D mesh model that has been screened for usability through availability verification, black edge vertices are detected and removed. Black edges are identified by color-based thresholding and distinguished from valid cow face data, achieving targeted noise removal, ensuring the purity of subsequent point cloud data, and providing a high-quality data source free from noise interference for the feature extraction stage.

[0082] Step 4: 3D mesh model to 3D point cloud conversion.

[0083] The 3D mesh model with black edges removed is converted into an initial point cloud. Optionally, a random downsampling method is used to unify the number of points in the cloud to a fixed size to improve the efficiency of subsequent processing.

[0084] Step 5: 3D Point Cloud Alignment When reconstructing a 3D model of a cow's face from multi-view 2D images, the point cloud of the cow's face in different scenes suffers from two main categories of problems due to the influence of the acquisition angle, acquisition distance (such as close-up shots of adult cows and long-distance shots of calves), and the cow's own posture (such as head turning): Attitude deviation: The spatial attitude of the cow face cloud is inconsistent (such as the head turning left and right, or tilting up and down), which leads to significant differences in the position of key features (such as horns and bridge of the nose) of the same cow in the coordinate system; Scale differences: There are significant differences in the facial size between adult cattle and calves. Even for the same animal, the overall scale of the point cloud will differ at different sampling distances.

[0085] These two types of problems directly lead to poor consistency in cow face point cloud features, causing point clouds of the same cow to be misclassified as different individuals due to differences in pose and scale, and making it difficult to distinguish point clouds of different cows due to scale inconsistencies. Ultimately, this reduces the accuracy of feature extraction and the robustness of cow face recognition. Therefore, in this embodiment, a 3D point cloud alignment process is used to eliminate pose shifts and scale differences in the 3D point clouds of cow faces, ensuring that point clouds of different scenes and different cows are consistent in spatial location and scale, providing a highly consistent data source for subsequent feature extraction.

[0086] 3D point cloud alignment processing calculates the centroid of the 3D point cloud and translates it to the origin of the coordinate system, completing initial position calibration and serving as the target point cloud for subsequent processing. Then, the Iterative Closest Point (ICP) algorithm is used to transform the pose of the target point cloud to match that of the reference point cloud. The reference point cloud represents a 3D point cloud of a cow's face with a standard pose. It is obtained, for example, by selecting typical samples from multiple cow face 3D point cloud samples.

[0087] By calculating the centroid of the 3D point cloud, the centroid is translated to the origin of the coordinate system to complete the initial position calibration, serving as the target point cloud for subsequent processing. Then, the Iterative Closest Point (ICP) algorithm is used to transform the pose of the target point cloud to match that of the reference point cloud. The reference point cloud represents a 3D point cloud of a cow's face with a standard pose. It is obtained, for example, by selecting typical samples from multiple cow face 3D point cloud samples.

[0088] Step 6, Crop key areas Because cattle are ruminants, they frequently engage in rumination, characterized by continuous chewing and resulting in cheek swaying. This introduces noise into the collected images of cattle faces and the resulting 3D point clouds of the cheeks. Regions such as the cheeks, jawline, and neck in the cattle face point clouds exhibit significant individual variations, dynamic noise due to muscle swaying, and low discriminative power. Point cloud data from these areas can obscure the geometric information of core feature regions such as horns, bridge of the nose, periorbital area, and nose, interfering with effective signals during subsequent feature extraction and reducing feature recognition accuracy. Furthermore, retaining these regions increases data redundancy and impacts processing efficiency. These regions are referred to as non-critical regions.

[0089] In the data processing of this application embodiment, non-critical areas are precisely cropped and eliminated, focusing on core features with high discriminative power, reducing interference from invalid information, and improving the purity of point cloud data and the targeting of feature extraction.

[0090] Specifically, the 3D point cloud of the cattle is cropped, removing the 3D point cloud area below the cheeks, while retaining the 3D point cloud area above the cheeks that remains relatively stable even during rumination.

[0091] Through the above data processing flow, a high-quality 3D point cloud model reflecting the 3D features of a cow's face is obtained, which is then fed into a 3D recognition model to generate a 3D feature vector.

[0092] In an optional implementation, the 3D mesh model availability verification step of the data processing flow of the 3D model in this application embodiment also combines the reliability index output by the Dust3R model itself with the newly added evaluation index to comprehensively verify the reliability of the 3D mesh model.

[0093] ① Metrics used to verify the reliability of 3D models Combining the built-in reliability metrics of the 3D reconstruction model (Dust3R) with the geometric quality metrics of the bullface model, this embodiment selects five metrics to verify the reliability of the 3D model. The selected metrics, their categories, meanings, detailed descriptions, and technical significance are listed in Table 1.

[0094] Table 1

[0095] Referring to Table 1, the selected metrics include two categories: custom metrics and Dust3R built-in metrics.

[0096] Dust3R's built-in metrics include edge1_score and edge2_score.

[0097] Dust3R is a mainstream multi-view 3D reconstruction algorithm. Its core advantage lies in achieving high-precision geometric reconstruction through image pair matching. The "best edge confidence score" (edge1_score) and the "second-best edge confidence score" (edge2_score) are the core outputs of Dust3R that quantify the quality of image pair matching. The matching accuracy of the core image pair (best edge, edge1) directly determines the accuracy of the model's geometric structure, while the second-best edge (edge2) reflects the redundancy and anti-interference capability of the reconstruction link. Choosing these two metrics allows for the direct reuse of Dust3R's reconstruction quality evaluation results, ensuring the basic quality of the model from the dimension of "reconstruction link reliability" and avoiding geometric distortion caused by image pair matching errors.

[0098] When reconstructing a 3D model, the Dust3R model outputs `edge1_score` and `edge2_score`, and simultaneously outputs `edge1_i` (best edge source image index), `edge1_j` (best edge target image index), `edge2_i` (second best edge source image index), and `edge2_j` (second best edge target image index). The core function of these parameters is to locate the image pairs corresponding to the positional confidence scores, ensuring the physical meaning of `edge1_score` is traceable. Index verification filters out samples with abnormal image pair matching (such as inconsistent `edge1_i` and `edge2_i`, or duplicate `edge1_j` and `edge2_j`). These samples are removed during the index preprocessing stage to prevent invalid indices from affecting the training accuracy of the verification model.

[0099] The built-in metrics of the Dust3R model only focus on "image pair matching quality" and do not cover the geometric quality characteristics after the model is generated. Therefore, problems such as "high noise in 3D models, poor overall reliability, and insufficient effective feature coverage" in existing technologies cannot be solved by the built-in metrics. Therefore, the embodiments of this application make up for this shortcoming by using three custom metrics to form a full-dimensional verification of "reconstruction link + geometric quality".

[0100] Custom metrics include the proportion of low-confidence points cleared, the average confidence level, and the proportion of high-confidence points.

[0101] Low-confidence point removal ratio: Addressing the issue of excessive noise in 3D models, this quantifies the removal ratio of low-confidence noise points, directly reflecting the initial purity of the model. The confidence level of each noise point is derived from the confidence level of each vertex in the Dust3D model output. During the data processing phase (before usability verification), low-confidence noise points are further removed from the 3D model based on the vertex's confidence level. Accordingly, the low-confidence point removal ratio is calculated based on the proportion of removed low-confidence noise points to the total number of vertices. For example, a criterion for selecting low-confidence points is a vertex with a confidence level < 0.3.

[0102] Average Confidence Score: Addressing the deficiency of "no evaluation standard for overall model reliability," this metric quantifies the confidence level of all vertices from a global perspective. It is obtained by calculating the average confidence score of all vertices in the 3D model reconstructed from the Dust3R model, or the average confidence score of all remaining vertices after removing low-confidence points.

[0103] High-confidence point ratio: Addressing the deficiency of "insufficient effective feature coverage," this metric focuses on the proportion of high-discriminative feature regions, ensuring sufficient high-value data support for subsequent feature extraction. The high-confidence point ratio is the percentage of high-confidence vertices out of the total number of vertices in the model. The selection criteria for high-confidence points are, for example, vertices with a confidence level ≥ 0.8.

[0104] ②Availability Fitting In this embodiment, the usability assessment objective is "whether the 3D model reconstructed by Dust3R meets the requirements for subsequent processing". Three mainstream binary classification methods (random forest, logistic regression, and XGBoost) are used for fitting and training to select the optimal method. The specific process is as follows: (2.1) Sample data preparation: Sample size: 476 sets of 3D model samples of cow faces (covering three breeds: dairy cows, beef cattle, and yellow cattle, including reconstruction results under different collection distances and lighting conditions); the above 5 indicators were obtained from each 3D model sample to form sample data for training.

[0105] Labeling: Manually label the sample data with usability labels (binary classification) to distinguish between qualified and unqualified 3D models. For example, "1" represents qualified models (meeting subsequent processing requirements: complete edges, low noise, and strong geometric consistency), totaling 380 groups; "0" represents unqualified models (having problems such as incomplete edges, dense noise, and geometric distortion), totaling 96 groups.

[0106] Feature preprocessing: The five indicators of the sample data are normalized (mapped to the [0,1] interval) to eliminate the difference in units; outliers (samples with |Z|>3) are removed by Z-score method to avoid outliers affecting the training accuracy of the model.

[0107] (2.2) Model training and validation: Random Forest, Logistic Regression, and XGBoost models were trained using sample data.

[0108] Three-fold cross-validation (80% training set, 20% validation set) is used to ensure the model's generalization ability; validation metrics include: accuracy, precision, recall, and F1 score.

[0109] ③ Model performance comparison and optimal selection Table 2 shows the cross-validation results of the models corresponding to the three binary classification methods.

[0110] Table 2

[0111] Of the three methods, logistic regression performed the worst: it assumes that the features are linearly related to the target variable, but there is a non-linear interaction between the five indicators in this module, which makes the model unable to capture complex feature relationships and has insufficient generalization ability.

[0112] XGBoost's performance is close to that of Random Forest, but its training time is 2.67 times that of Random Forest. It has a large number of parameters and poor real-time performance, which cannot meet the needs of large-scale real-time processing in the ranch.

[0113] Random forests offer the best overall performance: leading in accuracy and F1 score, with fast training speed, simple parameter tuning, and strong resistance to overfitting. They can achieve a balance between "high recognition accuracy" and "high real-time performance", making them perfectly suited for ranch application scenarios.

[0114] Therefore, in the preferred embodiment of this application, a random forest model is used to verify the usability of the cow face 3D model.

[0115] ④ Availability verification In this embodiment, five metrics are obtained from the 3D cow face model reconstructed from the Dust3R model (low-confidence points removed). These five metrics are then input into a trained random forest model, which outputs a usability score in the range of 0-1. A threshold of, for example, 0.75 (the optimal threshold based on ROC curve analysis, balancing precision and recall) is set. A usability score ≥ 0.75 indicates a qualified 3D cow face model, which proceeds to subsequent processing; a score < 0.75 indicates an unqualified model, which is directly discarded (not used for subsequent processing).

[0116] In the process of establishing a database of cattle features and identifying cattle, the reconstructed 3D models of cattle faces undergo the same usability verification and other data processing operations.

[0117] (3) Results and Analysis To verify the effectiveness of the five indicators selected in this application's embodiments, an effectiveness verification was also conducted. Through ablation experiments (removing one indicator at a time), it was found that the model accuracy reached 94.2% when all five indicators worked together. If any one indicator was removed, the accuracy decreased by 4%-13%. This indicates that the two-dimensional indicator system, which removed five specific indicators in this application's embodiments, can comprehensively cover the core evaluation dimensions of model quality, without redundant or invalid indicators.

[0118] The model's generalization ability was also verified. In an additional 107 sets of untrained 3D cow face model samples, the random forest model achieved a validation accuracy of 91.5% and an F1 score of 90.8%. This demonstrates that the random forest model possesses strong generalization ability in the usability verification of the cow face 3D model, and can adapt to different scenarios.

[0119] In an optional implementation, the black edge detection and filtering step in the data processing flow of the 3D model in this application embodiment also employs "divide-view processing". The processing method of "pixel color feature extraction → adaptive threshold determination → multi-view integrated filtering" achieves accurate removal of black edge noise.

[0120] Split-view processing: Since the cow face 3D data is obtained by joint reconstruction of multi-view images using DUSt3R, black edges are detected and removed from the 3D models of each viewpoint one by one to ensure that noise can be completely removed across the entire viewpoint range. The split-view 3D models with black edges removed are used to fuse the overall 3D mesh model of the cow face, rather than removing black edges after obtaining the overall 3D mesh model.

[0121] Pixel color feature extraction: For each cow face 3D model with different viewpoints, traverse all vertices within it, extract the RGB pixel color information of each vertex, and calculate the sum of the values ​​of the three color channel components (R+G+B).

[0122] Adaptive threshold determination: The screening threshold is set based on the color feature distribution pattern of the cow face model. Through color prior analysis of 100 sets of cow face model data containing black edges, the threshold normalized to the [0,1] interval is determined to be 0.03. This threshold can accurately distinguish between black edges and effective features.

[0123] Multi-view integration and denoising: Colors and 3D points below a set threshold in each view are identified as black edge noise and removed. The purified 3D models from all views are then globally integrated to finally output a clean cow face 3D mesh model with black edge interference removed.

[0124] Figure 4AThis shows a 3D model view of the cow's face before the black edges were removed. Figure 4B A 3D model view of a cow's face after removing black edges using an embodiment of this application is shown. A comparison of the two shows that the black edges of the cow's face 3D model have been effectively removed, while retaining important features of the cow's face.

[0125] In an optional implementation, this application embodiment further improves the 3D point cloud alignment processing step in the data processing flow of 3D models. Specifically, it includes a size standardization stage and a multi-angle ICP pose alignment stage.

[0126] Figure 5 The flowchart illustrates the size standardization stage in the 3D point cloud alignment process of the data processing flow of a 3D model according to an embodiment of this application.

[0127] See Figure 5 Obtain the generated 3D point cloud of the cow's face. Calculate the 3D centroid coordinates (Cx, Cy, Cz) of the cow's face 3D point cloud using the formula: Cx = (x1 + x2 + ... + x...). n ) / n、Cy=(y1+y2+...+y n ) / n、Cz=(z1+z2+...+z n ) / n, where n is the total number of points in the point cloud.

[0128] Subtract the centroid coordinates from the coordinates of each point in the 3D point cloud (points = points - centroid) to translate the centroid of the cow face 3D point cloud to the origin (0,0,0), eliminating the translational offset of the point cloud and laying the foundation for subsequent pose alignment. Here, centroid is the centroid coordinate calculated in the previous step.

[0129] Calculate the bounding box dimensions of the 3D point cloud: Determine the maximum and minimum values ​​of the bull face 3D point cloud along the X, Y, and Z axes to obtain the dimensions in three directions: extent_x = Xmax - Xmin, extent_y = Ymax - Ymin, extent_z = Zmax - Zmin. Here, Xmax and Xmin represent the maximum and minimum values ​​of the 3D point cloud along the X-axis, Ymax and Ymin represent the maximum and minimum values ​​of the 3D point cloud along the Y-axis, and Zmax and Zmin represent the maximum and minimum values ​​of the 3D point cloud along the Z-axis.

[0130] Extract the maximum size: Take the maximum value of the size in the three directions (scale = max (extent_x, extent_y, extent_z)), which represents the overall scale of the bullface point cloud.

[0131] When size normalization is required, unit scale scaling is performed. The coordinates of each point in the cow face 3D point cloud are divided by the maximum size (points = points / scale) to unify the maximum scale of different cow face point clouds to 1 unit, eliminating scale differences caused by adult cows and calves, as well as different acquisition distances.

[0132] Normal estimation: Based on an adaptive radius search algorithm, the nearest points of each point in the bull face 3D point cloud are found, and the normal vector (nx, ny, nz) perpendicular to the surface of the bull face is calculated, providing the necessary geometric information for the subsequent point-to-plane ICP pose calibration. This yields a size-normalized bull face 3D point cloud.

[0133] Figure 6 The flowchart illustrates the multi-angle ICP pose alignment stage in the 3D point cloud alignment process of the 3D model data processing flow according to an embodiment of this application.

[0134] See Figure 6 , obtain Figure 5 The processing flow outputs a size-standardized 3D point cloud of a cow's face, which serves as the source point cloud to be aligned.

[0135] Obtain a reference point cloud with a standard pose. The reference point cloud represents a 3D point cloud of a cow's face with a standard pose. The standard pose is, for example, a frontal pose with no pose shift, obtained by, for example, selecting representative samples from multiple cow face 3D point cloud samples. The standard pose has, for example, a rotation angle of 0 relative to the X, Y, and Z axes. The reference point cloud is then... Figure 5 The displayed dimensions are standardized, with a scale consistent with the source point cloud. Its centroid is located at the origin of the coordinate axis, and the positions of key features (horns, bridge of the nose, and around the eyes) are fixed.

[0136] Generate a set of pose rotation combinations. For the possible pose deviations (left-right rotation, up-down pitch, roll, etc.) of the cow's face during 2D image acquisition, generate various pose rotation combinations. Examples of pose rotation combinations include {Rx, Ry, Rx}, where Rx represents the rotation angle around the X-axis, Ry represents the rotation angle around the Y-axis, and Rz represents the rotation angle around the Z-axis. The rotation angles around each coordinate axis have multiple values, thus all values ​​of the pose rotation combination {Rx, Ry, Rx} form the pose rotation combination set, whose elements are pose rotation combinations {Rx, Ry, Rx} with specific values.

[0137] For example, if the selected rotation angles around each coordinate axis are (0°, 90°, 180°, 270°) (4 possible values), then the set of pose rotation combinations will have 4x4x4=64 combinations of the elements around each of the three coordinate axes. The selected rotation angles cover the entire circumferential value space to adapt to the pose distribution of the cow face 3D point cloud. In practice, it was found that although the target pose (frontal, 45 degrees left and right) was filtered when acquiring the cow face image, the pose distribution in the obtained cow point 3D point cloud exists in a larger pose space. Therefore, selecting rotation angles of (0°, 90°, 180°, 270°) is beneficial for providing a suitable pose search space and better identifying the cow face pose in subsequent ICP registration. Optionally, more rotation angles can be selected for each coordinate axis to cover various pose change scenarios of the acquired cow face. Still, optionally, the rotation angle values ​​can have smaller granularity, such as 45°, 30°, 15°.

[0138] Next, the pose rotation combination set is traversed, and the pose rotation combination in the pose rotation combination set is used as a coarse-grained initialization enumeration for the cow face pose. The various poses of the source point cloud after rotation are used as the search starting point to search for the pose matching between the source point cloud and the target point cloud, thereby obtaining the pose of the source point cloud.

[0139] Specifically, see also Figure 6 Obtain a current attitude rotation combination from the attitude rotation combination set, representing a rotation angle of (Rx, Ry, Rz), generate a 3×3 rotation matrix R, and apply it to the normalized preprocessed source point cloud (after normalization). Figure 5 The processing flow outputs the source point cloud (denoted as pcd), and the rotated point cloud (denoted as pcd_rotated = R) is obtained. pcd).

[0140] Perform ICP (Iterative ClosestPoint) iterative registration on the rotated source point cloud (pcd_rotated) and the reference point cloud.

[0141] The ICP iterative registration process includes finding the nearest point pair between the rotated source point cloud (pcd_rotated) and the reference point cloud, using a point-to-plane registration mode, and utilizing the previously calculated normal vectors of each point ( Figure 5Calculate the perpendicular distance between each nearest point pair. Solve for the optimal transformation matrix T (including rotation and translation) to minimize the sum of distances between all point pairs. Update the source point cloud with the optimal transformation matrix T, denoted as p_new = R·p_old + t, where p_old is the source point cloud in the current iteration (initially pcd_rotated), p_new is the updated source point cloud for the next iteration, and R and t are the rotation and translation components of the optimal transformation matrix T, respectively. Calculate the best registration quality score (current_fitness) for the current iteration, which represents, for example, the perpendicular distance between corresponding points in the updated source point cloud and the reference point cloud. Perform the next iteration of registration with the updated source point cloud p_new and the reference point cloud. After multiple iterations of registration, obtain the optimal transformation matrix T and the best registration quality score.

[0142] Traverse the set of attitude rotation combinations. For each rotation attitude combination, perform the above ICP iterative registration process to obtain the optimal transformation matrix T and the best registration quality score. Select the historically best optimal transformation matrix T and best registration quality score from all optimal transformation matrices T and best registration quality scores in current_fitness, and also obtain the rotation matrix R of the attitude rotation combination corresponding to the historically best optimal transformation matrix T.

[0143] The historical best transformation matrix T is applied to the normalized preprocessed source point cloud (pcd) (unnormalized raw data, preserving accuracy) to obtain the pose-aligned bull's face point cloud (pcd_aligned = best_transform). pcd), where best_transform=T R, best_transform represents the bull face point cloud after pose alignment.

[0144] After the pose alignment, the bull face point cloud has a uniform pose (standard frontal pose) and scale (unit scale), and can then proceed to the subsequent key region clipping stage.

[0145] Figure 7 The before-and-after effects of 3D point cloud alignment according to an embodiment of this application are shown.

[0146] Figure 7 middle, Figure 7 (a) represents the unprocessed raw bull face point cloud, showing that its spatial orientation remains in its original state; Figure 7 (b) represents the bull face point cloud after normalization preprocessing and pose alignment; Figure 7 (c) represents the comparison results of the point cloud effect before and after processing, which is used to intuitively show the changes brought about by attitude correction.

[0147] Depend on Figure 7It can be seen that the orientation of the unprocessed raw bull face point cloud is uncertain, manifested as a certain deviation in the bull face orientation relative to the front. Moreover, the deviation is random for different bull face point clouds, resulting in uncertainty in the orientation of multiple bull face point clouds. After normalization and pose alignment processing, the overall orientation of the point cloud is more uniform (facing forward), and the correspondence between the main direction and the reference coordinate system is more stable. Figure 7 (c) Further explanation: the embodiments of this application can effectively reduce the spatial deviation caused by the difference in the original posture between different samples.

[0148] By standardizing the size and aligning the pose of 3D point clouds in stages, the position, scale and density of the point clouds are first unified to reduce the registration complexity. Then, the traversal of the pose rotation combination set is used to avoid the ICP iterative registration from getting stuck in the optimal solution. At the same time, the final historical best transformation matrix is ​​directly applied to the normalized preprocessed source point cloud (pcd) to preserve the geometric detail accuracy. The registration quality is quantified by the fitness score to achieve traceability of the effect. The problem of pose offset, scale difference and density unevenness of bullface point clouds is comprehensively solved from five dimensions: reducing complexity, improving robustness, maintaining accuracy, strong adaptability and quantifiability. Finally, a highly consistent and high-precision aligned point cloud is output.

[0149] Figure 8 A flowchart illustrating the key region trimming stage of the data processing flow for a 3D model according to an embodiment of this application is shown.

[0150] See Figure 8 , obtain Figure 6 The processing flow outputs a pose-aligned 3D point cloud of a bull's face. A global geometric reference plane for the bull's face is then fitted from this 3D point cloud. The equation of this global geometric reference plane is represented by n·x + d = 0, where n (n = (nx, ny, nz)) represents the unit normal vector of the plane equation, and d is a plane constant parameter. Fitting the global geometric reference plane for the bull's face is essentially the process of solving for the parameters n and d from the 3D point cloud.

[0151] To obtain the overall geometric reference plane for the cow's face, the centroid of the 3D point cloud was calculated and then decentered (see also...). Figure 5 , and according to Figure 5 In the embodiments, the calculation of the centroid coordinates and the translation of the 3D point cloud centroid to the origin are the same operations to obtain a standardized 3D point cloud of the bull's face. Then, principal component analysis (PCA) least squares plane fitting is used to fit the overall geometric reference plane of the bull's face.

[0152] Specifically, a 3×3 covariance matrix is ​​constructed based on the decentralized point cloud data. The covariance matrix represents the dispersion of the 3D point cloud along the X, Y, and Z coordinate axes. Its diagonal elements represent the variance of the points in the 3D point cloud in the directions represented by each of the three coordinate axes, and the off-diagonal elements represent the covariance of the points in the 3D point cloud between two coordinate axes.

[0153] The covariance matrix is ​​decomposed into eigenvalues, with its three eigenvalues ​​corresponding to the distribution variance of the point cloud in three orthogonal directions. The eigenvector corresponding to the smallest eigenvalue is taken as the normal vector in the "flattest" direction of the 3D point cloud distribution. This vector is normalized to obtain the unit normal vector n(nx, ny, nz) of the plane. The plane constant parameter d is then calculated, for example, by using the constraint condition of the overall geometric reference plane of the bull's face passing through the centroid of the 3D point cloud, and solving for the parameter d. Finally, the plane equation n·x + d = 0 is obtained, and this plane accurately represents the overall surface geometry of the bull's face.

[0154] The eigenvector corresponding to the smallest eigenvalue represents the normal vector in the "flattest" direction of the 3D point cloud distribution. Based on the inherent shape characteristics of a cow's face, the plane corresponding to the normal vector in the "flattest" direction of the cow's face 3D point cloud distribution is roughly defined by the tip of the nose and the base of the ears. Multiple facial features, such as the eyes, nose, and horns, are located on the same side of this plane, while parts prone to noise, such as the cheeks and mouth, are located on the opposite side. Therefore, this plane allows for the differentiation between noisy portions and features that represent the cow's facial characteristics within the 3D point cloud of the cow's face.

[0155] Optionally, since normalization and pose alignment have already been performed during the data processing for generating the 3D point cloud, the operation of calculating the centroid of the 3D point cloud and performing decentralization is no longer required when constructing the overall geometric reference plane of the cow's face.

[0156] Optionally, the normal vector *n* of the plane equation is also standardized so that for any bullface 3D point cloud, the normal vector *n* of the fitted plane equation uniformly points to the same side of the overall geometric reference plane of the bullface. Specifically, it is determined whether the normal vector *n* of the plane equation points in a specified direction. The specified direction is, for example, above the overall geometric reference plane of the bullface (e.g., the z-axis component is positive). When the normal vector *n* points above the plane, no adjustment to the plane equation is required. When the normal vector *n* points below the plane, the plane equation is flipped, for example, by flipping the plane equation by *n = -n* and *d = -d*. For example, whether it points in a specified direction is determined by the Z-axis component value of the normal vector *n*. A positive Z-axis component value of *n* indicates that *n* points above the plane, while a negative Z-axis component value of *n* indicates that *n* points below the plane.

[0157] Next, for each point x in the 3D point cloud of the cow face, calculate its distance to the overall geometric reference plane of the cow face represented by the plane equation n·x+d=0 (denoted as s=n·x+d). When s>0, it means that point x is above the plane, and the point is retained in the 3D point cloud of the cow face. When s<0, it means that point x is below the plane, and the point is deleted from the 3D point cloud of the cow face. This is to achieve cropping of the 3D point cloud of the cow face, remove the influence of cheek movement on 3D feature extraction, and enhance the stability of the extracted 3D feature vectors. It can be understood that for a point s=0, it is located on the face plane, and it can be deleted from the 3D point cloud or retained in the 3D point cloud.

[0158] Optionally, a mask representing the front and side spaces of the overall geometric reference plane of the cow's face is created (when s>0, mask is True(1); when s≤0, mask is False(0)), where points s>0 correspond to key feature regions on the front side of the cow's face (horns, bridge of the nose, periorbital area, and nose), and points s≤0 correspond to non-key noise regions on the negative side of the cow's face. This mask is applied to filter the 3D point cloud, retaining points that meet the mask conditions and removing non-key regions. For example, multiplying the points in the 3D point cloud by the corresponding mask yields the result, where the point values ​​in the front and side spaces of the overall geometric reference plane of the cow's face are retained, while the points in the negative space of the overall geometric reference plane of the cow's face are set to 0.

[0159] It also outputs a cropped 3D point cloud of the target, focusing on core features. By selectively removing noisy regions, the stability of the output bull-face 3D feature vector is significantly improved.

[0160] In another implementation, yet another method is provided for fitting a global geometric reference plane of a cow face from a 3D point cloud of a cow face.

[0161] Specifically, the front of a cow's head is roughly an inverted trapezoid, with the forehead located on the wide base, the muzzle on the narrow base, and the sides of the face on the two sides. The side of the cow's head is triangular, with one side running from the forehead to the tip of the nose, another side running from the lower edge of the nose to the neck, and the third side running from the horns to the neck. Thus, the frontal region of the cow's head (roughly including the eyes, upper nose, and forehead) forms an approximate plane, which is used as the overall geometric reference plane for the cow's face in this embodiment. In one example, based on the fact that this overall geometric reference plane for the cow's face is located at the front edge region of the 3D point cloud of the cow's face, multiple vertices belonging to the face plane can be obtained by detecting the front edge of the pose-normalized and scale-normalized 3D point cloud. Therefore, for 3D point clouds from different cows, an approximate plane representing the face plane can be fitted. For another example, the plane defined by the tip of the nose and the base of the ears can be used as an approximate plane representing the overall geometric reference plane of the cow's face. Alternatively, in the triangle presented on the side of the bull's head, the cross section formed from the vertex representing the tip of the nose to a designated point on the opposite side (such as the midpoint or the upper third point) can be used as an approximate plane representing the overall geometric reference plane of the bull's face.

[0162] Multiple facial features of a cow, such as the eyes, nose, and horns, are located on the same side of the face plane, while parts prone to wobbling noise, such as the cheeks and mouth, are located on the opposite side. Therefore, based on the overall geometric reference plane of the cow's face, it is possible to distinguish between noisy parts and features that represent the cow's facial characteristics in the 3D point cloud of the cow's face.

[0163] Therefore, in this embodiment of the application, by obtaining the coordinates of the vertices belonging to the overall geometric reference plane of the cow's face, an approximate plane is fitted, which is represented by the plane equation n·x + d = 0.

[0164] Specifically, the plane equation n·x + d = 0 is set, and multiple points in the 3D point cloud of the bull's face located on the overall geometric reference plane of the bull's face are obtained. The values ​​of parameters n and d are obtained by least squares fitting, for example, so as to obtain the plane equation representing the overall geometric reference plane of the bull's face.

[0165] Figure 9 The images show a comparison view before and after cropping the 3D point cloud of a cow's face according to an embodiment of this application.

[0166] Figure 9 In the image, (a) is a view of the 3D point cloud of the cow's face before cropping, and (b) is a view of the 3D point cloud of the cow's face after cropping. After cropping, the cow's cheek area in the middle right region of (a) has been removed in (b).

[0167] In this embodiment, the cropped 3D point cloud of the cow's face is fed into a 3D recognition model to extract 3D feature vectors. Optionally, for the 3D recognition model that outputs classification results, the output of the last hidden layer before its classification output head is used as the extracted 3D feature vector. Common 3D recognition models include, for example, PointNet, PointNet++, PointMLP (https: / / arxiv.org / abs / 2202.07123), and TetraSphere (https: / / arxiv.org / abs / 2211.14456).

[0168] In an alternative embodiment of this application, the PointNet++ network is also specifically improved (referred to as H-PointNet++) to better capture the 3D features of a cow's face.

[0169] PointNet++ (https: / / arxiv.org / pdf / 1706.02413) is a landmark achievement in the field of 3D vision. Through its innovative design of hierarchical sampling and local feature aggregation, it can accurately capture the local details and global structure of 3D point clouds. Existing PointNet++ models are mainly applied to point cloud classification and segmentation tasks, and are not well-suited for cattle face recognition scenarios—its training data consists mostly of point clouds with large class differences, uniform and regular distribution, and no noise. However, the point clouds in cattle identification tasks are characterized by disorder and high similarity between samples, showing a significant difference from the distribution of PointNet++ training data. The original design of the PointNet++ model is primarily for object classification or part segmentation. When directly applied to the identification of similar individuals of the same species requiring extremely high discriminative power, the learned features are not sensitive enough to fine-grained discrimination (such as nose prints, spots, subtle hair direction, and other texture information that has high discriminative power for individual cattle).

[0170] To extract highly discriminative features from the 3D point cloud of a cow's face, the H-PointNet++ model provided in this application is an improvement on the PointNet++ model.

[0171] (I) Overall Technical Concept of H-PointNet++ Model To address the objective challenges in 3D point cloud identity recognition tasks, such as high structural similarity between individuals, open sets in actual application scenarios, complex data sources, and uncontrollable noise, the H-PointNet++ model has made systematic adjustments to the PointNet++ model based on its design goals.

[0172] This invention argues that the core of identity recognition based on 3D point clouds lies not in coarse-grained differentiation of the overall shape of the point cloud, but rather in: 1) At the local level, fully capture fine-grained geometric structures that reflect individual differences; 2) At the global level, the aforementioned local differences are stably mapped to a unified identity feature representation space; 3) At the output level, the network output is no longer dependent on a fixed set of categories, but forms a measurable and scalable identity feature vector.

[0173] Based on the above understanding, this invention proposes an improved point cloud feature extraction network, H-PointNet++. This network is based on the hierarchical feature learning idea of ​​PointNet++, but it is specifically reconstructed around the application goal of "open set identity recognition", and the grouping strategy, feature representation form and training mechanism are redesigned to enable the network to work stably in real production environment.

[0174] PointNet provides mapping functions .

[0175] Its input is an unordered set of points {x1, x2, ..., x}. n The output is a feature vector representing the set of points. The unordered set of points consists of n points, each with C-dimensional coordinates and d-dimensional original features. h and γ are both multilayer perceptron mapping networks (MLPs), and max() represents global max pooling. In PointNet++, this mapping function is also called a PointNet layer.

[0176] PointNet++ proposes a hierarchical grouping network architecture based on PointNet. The backbone network consists of several stacked SA (Set Abstraction) layers. Each SA layer receives a set of points, performs feature abstraction, and outputs a new set of points with fewer points. A single SA layer consists of three layers: a sampling layer, a grouping layer, and a PointNet layer.

[0177] Sampling layer: Selects a subset of points from all input points as the center points of a local region; Grouping layer: Sample neighboring points around each center point to construct several local region point subsets; PointNet layer: Encodes the spatial features of each local neighborhood into a feature vector.

[0178] The input to a single SA layer is a matrix of size N x (d + C): corresponding to N input points, each containing d-dimensional spatial coordinates and C-dimensional original features; the output is a matrix of size N' x (d + C'): N' represents the N' center points obtained by downsampling the input points, carrying d-dimensional spatial coordinates and C'-dimensional new features fused with local context. Therefore, the operation of the SA layer is also simply referred to as grouping and downsampling.

[0179] By concatenating multiple SA layers, the number of points is continuously reduced, and the receptive field is continuously increased, resulting in the output of the backbone network. In classification applications, the backbone network output outputs the classification result through the output header of the fully connected layer.

[0180] (II) Overall Structure of H-PointNet++ H-PointNet++ still employs a hierarchical point cloud feature extraction framework, including multiple cascaded SA layers and a feature vector output head. Unlike PointNet++, H-PointNet++ does not aim for class discrimination as its final goal. Instead, it treats the entire network as a 3D identity feature mapping function, used to map the input 3D point cloud into identity feature vectors of a uniform scale.

[0181] The H-PointNet++ network takes normalized 3D point cloud data of cow faces as input and outputs a fixed-dimensional feature vector representing the identity of a cow. This feature vector is directly used for subsequent similarity calculation and identity determination.

[0182] Figure 10 The H-pointnet++ network structure according to an embodiment of this application is shown.

[0183] (III) First Floor SA (SA1) The PointNet++ network employs a uniform sampling density and neighborhood scale configuration in its first SA layer, primarily designed to achieve general modeling capabilities for different categories of 3D objects. (See also...) Figure 11 When used to distinguish object categories, such as Figure 11 In the examples (a) airplane, (b) water cup, and (c) cow, the PointNet++ model performs well in tasks where there are significant differences in object categories and structures, but it shows obvious shortcomings in tasks involving the identification of highly similar individuals, such as cow faces.

[0184] Specifically, the 3D point clouds of different cattle faces exhibit a high degree of consistency in their overall structure. However, the discriminative information that significantly distinguishes individual cattle is not evenly distributed throughout the point cloud space, but rather concentrated in a few local areas (such as the eyes and horns). If a large-scale neighborhood or sparse sampling strategy is adopted in the first SA layer (SA1), fine-grained differences are easily averaged or weakened in shallow layers, making it difficult to recover such differential information even if the network depth is increased in subsequent layers.

[0185] Therefore, the structural design of the first-layer SA directly determines whether the network has the ability to effectively model individual identity differences. This problem cannot be solved by simply adjusting parameters, but belongs to the technical problem of network structure adaptation.

[0186] To address the aforementioned issues, the H-PointNet++ network in this application redesigns the first SA layer from the perspective of group modeling objectives. That is, the core task of the first SA layer is no longer to quickly abstract the overall structure, but to retain the local geometric information that is discriminative for individual differentiation as completely and precisely as possible.

[0187] Based on this technical concept, the design of the first SA layer must simultaneously meet the following principles: (1) At the spatial coverage level, improve the probability of sampling points covering key local areas; (2) At the local modeling level, avoid introducing large-scale neighborhood information too early; (3) At the feature expression level, enhance the ability to nonlinearly characterize subtle geometric changes.

[0188] Specifically, (1) In order to retain more high-discrimination regions of the cow face, the number of input point cloud points of the first SA layer of H-Pointnet++ is 4096.

[0189] (2) High-density farthest point sampling strategy The sampling density of the first (SA) layer exceeds that of PointNet++'s SA1 layer, making the sampling points more evenly and densely distributed in the 3D point cloud space. This increases the probability that key local regions of the cow face will be selected as center points, avoiding the omission of discrimination regions due to sparse sampling. The first SA layer of H-Pointnet++ has 1024 sampling points.

[0190] (3) Small-scale local neighborhood modeling strategy The first SA layer uses a smaller neighborhood search radius than PointNet++'s SA1, allowing each local region to focus on detailed structural changes and reducing interference from irrelevant regions on local features. This ensures that fine-grained differences are not smoothed out in shallow layers from a structural perspective. The sampling radius of H-Pointnet++'s first SA layer is 0.1.

[0191] (4) Enhanced local feature mapping strategy Within a local neighborhood, a multilayer perceptron structure is used to perform nonlinear mapping on the point set, thereby improving the network's ability to express small changes in local geometry while maintaining computational controllability.

[0192] Through the synergistic effect achieved by adjusting the respective strategies of sampling, grouping, and local feature mapping in the SA1 layer, the features output by the SA1 layer have high identity discrimination potential at the shallow stage, providing a reliable foundation for subsequent layer feature abstraction.

[0193] Figure 12 This is a schematic diagram of the sampling points and sampling radius of SA1 in the H-PointNet++ model according to an embodiment of this application.

[0194] Figure 12 The diagram illustrates 10 local regions obtained by grouping and sampling the 3D point cloud of a cow's face using the SA1 layer. These local regions correspond to sampling points of the SA1 layer and their local neighborhoods. As can be seen from the diagram, the size of the local regions effectively covers important, discriminative structures in the 3D point cloud of the cow's face (e.g., eyes, horns), avoiding the situation where a single local region is too large and simultaneously covers multiple important structures on the cow's face, or too small and only covers a portion of an important structure. It's also important to understand that the sampling points are selected randomly or uniformly, without needing to know the location and features of important structures on the cow's face. By using a high-density sampling point setting (1024 points and a sampling radius r=0.1), any important structure on the cow's face is likely to be effectively covered by several sampling points and their local neighborhoods. Even if some sampling points fail to effectively cover important structures on the cow's face, it does not affect the overall extraction of detailed features from the 3D point cloud of the cow's face.

[0195] (iv) Subsequent SA layer and global identity feature modeling After completing the adaptive grouping and fine-grained feature modeling of the first SA1 layer, the subsequent SA layers of the H-PointNet++ network follow the PointNet++ model. Therefore, in terms of the overall backbone network structure, the first SA layer of this embodiment can be regarded as another SA layer added before the first SA layer of the PointNet++ network.

[0196] Specifically, subsequent SA layers reduce the number of points involved in the computation step by step and correspondingly expand the spatial scale of the local neighborhood, allowing the local discriminative features extracted in the shallow stage to be aggregated over a larger spatial range. This aggregation process is not a simple spatial downsampling, but rather a gradual introduction of larger-scale structural context information while preserving local identity differences.

[0197] Since the first layer SA1 has already performed high-resolution modeling of key local regions of the cow face, the subsequent SA-level feature abstraction process builds upon this foundation, linking multiple local structures with identity-discriminating significance to form an identity-level representation of the overall geometric morphology of the cow face. The feature representation constructed in this way possesses the following characteristics: 1) At the local level, retain detailed information that is sensitive to individual differences; 2) At the global level, it has strong robustness to overall morphological changes under different collection conditions.

[0198] In the H-PointNet++ model, the hierarchical modeling path of "gradually converging local identity features into global identity representation" effectively avoids the problem of unstable identity representation caused by focusing only on local features, and provides a reliable foundation for subsequent feature vector output.

[0199] In the H-PointNet++ implementation, to preserve detailed features, the first layer SA (Self-Sampling) uses farthest point sampling (FPS) to obtain 1024 sampling points as center points from the input 4096-point cow face 3D point cloud. For each center point, sampling is performed within its local neighborhood using a small local neighborhood sampling radius (r=0.1) and a large number of sampling points within the local neighborhood (e.g., 64). This approach not only preserves rich fine-grained features but also allows each local neighborhood to integrate surrounding feature information, achieving an organic combination of fine-grained features and the global structure of the cow face point cloud. This is a key step in balancing local details and overall distribution.

[0200] The subsequent three SA layers follow the core logic of PointNet++, gradually reducing the number of sampling points, increasing the local neighborhood radius and the number of sampling points in the neighborhood, and gradually abstracting and refining fine-grained key features with strong discriminative power, providing stable and effective feature support for the feature vector output head.

[0201] The output of the fourth layer SA is provided to the feature vector output head after passing through the feature extraction network.

[0202] (v) Feature Vector Output Head Design In open-set identity recognition applications for cow faces, the system will inevitably encounter a large number of individuals that did not appear during the training phase. Therefore, the network output must break free from its dependence on a fixed set of categories and use only the similarity of identity features as the criterion for judgment.

[0203] Traditional point cloud classification networks typically use fully connected layers and a softmax function to output class probabilities. This type of output is essentially based on the "class closure assumption," making it unsuitable for calculating similarity for unknown identities or for applications requiring identity registration and dynamic expansion.

[0204] To address the aforementioned issues, this application's embodiments specifically design the output header of the H-PointNet++ network, transforming it from a category-based discriminative output into an identity feature vector output header. This output header is used to map the high-dimensional point cloud features extracted by the backbone network into fixed-dimensional identity feature vectors.

[0205] The H-PointNet++ network feature vector output head includes: (1) a feature compression layer, which reduces the feature dimension and reduces redundant information while maintaining the discriminative ability; (2) a feature distribution normalization layer, which constrains the statistical distribution of features during training and improves the comparability between samples from different batches; (3) a nonlinear mapping layer, which enhances the feature expression ability and makes the identity feature space more discriminative; (4) a regularization layer, which suppresses overfitting and improves the robustness of features to acquisition noise and pose changes; and (5) a vector normalization layer, which maps the output features to a unified scale space.

[0206] The feature vectors obtained after processing the feature vector output head described above can be directly used for similarity calculation based on Euclidean distance or cosine distance, thereby realizing cattle identification under open set conditions.

[0207] Figure 13 A schematic diagram of the network structure of the H-PointNet++ network feature vector output header according to an embodiment of this application is shown.

[0208] See Figure 13 First, a feature compression layer (shown as a fully connected layer in blue) compresses the 1024-dimensional features at each point of the fourth layer (SA) of the backbone network to 512 dimensions at each point. The output features then enter a combination of a feature distribution normalization layer (BN), a nonlinear mapping layer (ReLU), and a regularization layer (Dropout) to complete a full round of normalization and regularization. Subsequently, it passes through a combination of neural network layers including a feature distribution normalization layer (BN), a nonlinear mapping layer (ReLU), and a regularization layer (Dropout), and then through a feature compression layer (fully connected layer) to compress the 512-dimensional features at each point of the previous neural network layer combination to 256 dimensions at each point.

[0209] Finally, the L2 normalization process is performed on the 256-dimensional feature vector of each point through a normalization layer and a linear layer, so that the output feature vector conforms to the requirements of Euclidean distance measurement and ensures the accuracy of similarity calculation.

[0210] This feature vector output head structure can progressively distill the complex geometric information of the captured cow face into 256 ready-to-use, highly discriminative features. This design not only ensures the efficiency and stability of model training, but also makes the output features suitable for downstream tasks requiring high-precision similarity comparisons, such as cow identity verification.

[0211] (vi) Identity Feature Learning Method Based on Triple Constraints In the cow face open set identity recognition task, the system ultimately determines identity based on the similarity between identity features. Therefore, the core objective of network training is not to learn the discrimination boundary between fixed categories, but to construct an identity feature space with clear geometric constraints, so that samples of the same identity are highly clustered in this space, and samples of different identities maintain a stable and separable distance interval.

[0212] Traditional classification-based loss functions aim to maximize class probability, and their constraint object is the class label itself. They are difficult to directly apply to the geometric structure of the feature space and have no constraint ability on samples with unknown identities. Therefore, they are not suitable for open set identity recognition scenarios.

[0213] Based on the above understanding, this invention introduces a constraint method based on triplet relationships in the training phase of the H-PointNet++ model to directly model the geometric structure of the feature space.

[0214] Return to view Figure 10 In this embodiment, triplet loss is used instead of traditional cross-entropy loss to complete end-to-end training of the model. As the core loss function of the metric learning model, the training objective of triplet loss is to construct a highly discriminative feature embedding space—in this space, the feature vectors of similar samples are close to each other, while the feature vectors of dissimilar samples are far apart. This characteristic makes it more suitable for cow face authentication tasks.

[0215] The training mechanism of triplet loss uses "sample triplet" as the core unit. Each triplet contains three types of samples: anchor sample (a), positive sample (p), and negative sample (n). Anchor sample (a) serves as the reference benchmark for identity; positive sample (p) has the same identity as anchor sample (a), and is point cloud data obtained from cow face images corresponding to the same cow; negative sample (n) has a different identity from anchor sample (a), and comes from point cloud data of cows with different identities (different identities).

[0216] During training, for each sample triplet, the H-PointNet++ network extracts the identity feature vectors of the three sample classes respectively, and imposes constraints on their distance relationships in the feature space, ensuring that the feature distance between the anchor sample and the positive sample is less than the feature distance between the anchor sample and the negative sample, and that there is at least a preset margin between them. This constraint relationship can be formally expressed as: L = max(d(a,p) - d(a,n) + margin, 0) In this formula, a, p, and n represent the feature vectors of the anchor point, sample, positive sample, and negative sample in the feature space, respectively; d() represents the Euclidean distance calculation function, and margin represents the margin threshold. This loss function enhances the class discrimination of the feature vectors by directly optimizing the distance distribution of samples in the feature space, and is more in line with the needs of identity verification tasks for the discriminative power of sample features compared to traditional loss functions.

[0217] By imposing the aforementioned constraints on a large number of triplet samples, the network is guided to continuously compress the distribution range of samples with the same identity in the feature space during training, while simultaneously widening the distribution interval between samples with different identities, thereby gradually forming an identity feature space with a clear geometric structure. This approach enables the H-PointNet++ network to explicitly learn the core discrimination criterion of "identity similarity" during the training phase, which is beneficial for improving the generalization recognition ability under unknown identity conditions.

[0218] (vii) Gradual Sample Difficulty Training Strategy Considering that the cow face point cloud data comes from the actual production environment, the samples are inevitably affected by factors such as pose changes, occlusion, lighting differences, noise points, and annotation errors during the collection process. If a large number of high-difficulty triplet samples are directly introduced in the early stage of training, the feature space may be forcibly stretched before it is formed, resulting in training oscillations or even model collapse; on the other hand, if only simple samples are used for training, it is difficult to effectively learn the subtle identity differences between individuals.

[0219] To optimize the feature extraction capabilities of the H-PointNet++ model, this application proposes a progressive adaptive hard sample mining strategy. This strategy simulates the human cognitive pattern of "from easy to difficult," dynamically dividing the model training process into three stages. By adaptively adjusting the difficulty of sample learning, it addresses the problems of poor stability and susceptibility to local optima in traditional hard sample mining during the initial training phase, ensuring the model always learns features under an appropriately challenging intensity.

[0220] The training triplet samples include simple samples, semi-hard samples, and hard samples. Each sample consists of a triple {anchor sample, positive sample, negative sample}, where a, p, and n represent the feature vectors of the anchor sample, positive sample, and negative sample in the feature space, respectively.

[0221] In the initial training phase, semi-difficult triplet samples are constructed first, consisting of anchor samples, positive samples, and negative samples that are far from the anchor samples but not yet completely separated. The training objective of this stage is to guide the network to form a preliminary stable identity feature space structure, avoiding disordered feature distribution due to excessive constraints.

[0222] For example, the semi-hard sample screening criterion is: the distance between the feature vector of the negative sample and the anchor sample is greater than the distance between the feature vector of the positive sample and the anchor sample, but this distance difference is smaller than the margin threshold margin, which can be expressed as 0 < d(a, n) - d(a, p) < margin. In other words, for a semi-hard sample, its positive sample and negative sample can be distinguished, but the degree of distinction is not high.

[0223] As a comparison, the screening criterion for easy samples is that the distance between the negative sample and the anchor sample is greater than the distance between the positive sample and the anchor sample, and this distance difference is greater than the margin threshold margin. The screening criterion for hard samples is that the distance between the negative sample and the anchor point is smaller than the distance between the positive sample and the anchor sample, which can be expressed as d(a, n) < d(a, p). The ideal situation is that the distance between the positive sample and the anchor sample is small, and the distance between the negative sample and the anchor sample is large.

[0224] Optionally, in hard samples, the distance between the feature vector of the negative sample and the anchor sample is the distance of the negative sample that has the minimum distance (or the distance is smaller than a specified threshold) among all negative samples different from the anchor sample.

[0225] Easy samples are not used for training. In the early stage of training, semi-hard samples are used to calculate the loss function and update model parameters, so as to help the H-PointNet++ model establish the basic structure of the feature space with mild learning intensity, and ensure the stability of initial training.

[0226] In the middle stage of training, after the feature space is initially formed, a triplet sample set composed of a mixture of random semi-hard samples and hard samples is gradually introduced, so that the network starts to focus on the boundary regions between different identities while maintaining overall stability, thereby further compressing intra-class distance and expanding inter-class distance.

[0227] In the later stage of training, hard sample triplets composed of anchor samples and different-identity samples with the closest distance in the feature space are mainly used to finely stretch the identity feature space, so that the network has the ability to distinguish subtle differences between highly similar individuals.

[0228] Through the above progressive training strategy of "semi-hard samples → mixed samples → hard samples", the network always takes into account both stability and discriminability throughout the training process, which not only effectively avoids the instability problem in the early stage of training, but also significantly improves the distinguishing ability and convergence efficiency of identity features in complex actual scenarios.

[0229] The trained H-PointNet++ model is used to receive the cow face 3D point cloud obtained from the collected cow face images and output the corresponding 3D feature vector.

[0230] In an optional acquisition method of sample triplets, sample triplets are randomly generated, and then sample triplets meeting the specified conditions are screened from the randomly generated sample triplets according to the screening conditions of semi-hard samples and hard samples, so as to obtain a semi-hard sample set and a hard sample set.

[0231] In another optional acquisition method of sample triplets, sample triplets are constructed according to the required sample type. A sample with a known identity is used as an anchor sample, a sample with the same identity as the anchor sample is acquired as a positive sample, and samples with different identities from the anchor sample are acquired as candidate negative samples. The H-PointNet++ model is used to map the anchor sample and a plurality of candidate negative samples respectively to obtain 3D feature vectors. The distance between the feature vector of the anchor sample and the feature vector of each candidate negative sample is compared to select a negative sample for constructing the sample triplet from the plurality of candidate negative samples. For example, if |d(a, n) - d(a, p)|<margin, the negative sample corresponding to the feature vector n is a negative sample suitable for constructing a semi-hard sample; and the negative sample that minimizes |d(a, n)| among the plurality of negative samples is a negative sample suitable for constructing a hard sample. Optionally, there are multiple candidate positive samples with the same identity as the anchor sample. To construct a sample triplet, any candidate positive sample can be selected to construct the sample triplet, or the H-PointNet++ model can generate feature vectors for the candidate positive samples, and positive samples are selected according to the distance between the candidate positive samples and the anchor sample. To improve the training effect, among the selected positive samples, candidate positive samples that have a larger distance from the anchor sample are preferably selected. A triplet is constructed based on the selected anchor sample, positive sample and negative sample to calculate the loss, and back propagation is performed according to the loss, so as to achieve continuous optimization of the parameters of the H-PointNet++ network, making the features of samples of the same class closer in the feature space and the features of samples of different classes more separated.

[0232] In another alternative method for obtaining sample triples, a batch-based online mining approach using intra-batch feature vectors is employed during training to dynamically generate sample triples. Specifically, 3D point clouds of cow faces from a batch are first input into the H-PointNet++ network to obtain feature vectors corresponding to each cow face 3D point cloud. Then, anchor samples, samples with the same identity label as the anchor sample, and samples with different identity labels are obtained from the batch based on the sample label (cow identity). For each anchor sample, the sample with the largest distance from its own sample is selected as the positive sample; the sample with a semi-hard condition from the out-of-class sample is selected as the semi-hard negative sample, i.e., the distance between the negative sample and the anchor sample is greater than the distance to the positive sample and falls within a preset margin range; alternatively, the sample with the smallest distance from the anchor sample can be selected from the out-of-class sample as the hard negative sample. Finally, the anchor samples and their corresponding positive and negative samples are combined to obtain sample triples. The loss is then calculated based on the constructed sample triples, and backpropagation is performed based on this loss to optimize the parameters of the H-PointNet++ network. In the next phase of training, the 3D point cloud of cow faces from another batch is input into the optimized H-PointNet++ network, and sample triples are constructed using the same method described above. The loss is then calculated, and the H-PointNet++ network parameters are continuously optimized based on the loss.

[0233] In another alternative implementation, the construction of sample triples continues throughout the training process of the H-PointNet++ model. Furthermore, as the H-PointNet++ model is trained progressively, even if the criteria for identifying semi-difficult / difficult samples remain unchanged, the selected negative samples will change. For example, before or in the early stages of training, the distance is calculated using feature vectors generated by the unoptimized H-PointNet++ model and used as the basis for selecting negative samples. In the middle or later stages of training, the H-PointNet++ model is still used to select negative samples, but since it has been optimized, the feature vectors generated for the same anchor sample and candidate negative samples may differ, leading to changes in the negative sample selection results. This achieves dynamic adjustment of sample difficulty based on the training progress.

[0234] According to an embodiment of this application, cattle identification is performed based on the calculated 3D feature vector representing the cow's face. For example, the similarity (e.g., Euclidean distance) between the calculated 3D feature vector and the 3D feature vectors of known cattle in a cattle feature database is calculated. The highest-ranked or top N candidate results in the cattle feature database are returned as the cattle identification result.

[0235] Optionally, a confidence threshold is also used, based on the similarity (distance between feature vectors) between the candidate result and the 3D feature vector of the cattle to be identified. Only one or more candidate results with a similarity exceeding the confidence threshold are recognized as cattle identification results. If the similarity of any candidate result does not exceed the confidence threshold, then a result is returned indicating, for example, rejection, identification failure, or explicit indication that it does not belong to any known cattle.

[0236] Furthermore, the output of the trained H-PointNet++ model directly reflects the spatial distribution of feature vectors without relying on a known set of categories, thus possessing natural adaptability for identifying unknown cattle. It also exhibits good adaptability for newly registered cattle, allowing for dynamic expansion of the cattle feature database during use. For example, for a new cattle, the processing method of this application embodiment uses the same generation method as other feature vectors in the cattle feature database to obtain its facial 3D point cloud, generate a 3D feature vector, and input it along with its identity information into the cattle feature database, thereby achieving identification of the new cattle without retraining the H-PointNet++ model. When the new cattle needs to be identified, a 3D feature vector is generated for it using the method of this application embodiment, and its similarity is calculated with the 3D feature vectors of known cattle in the cattle feature database to obtain the identification result. This meets the practical needs of continuously adding individuals in cattle farming scenarios.

[0237] Return to view Figure 1 After obtaining the 2D and 3D feature vectors of the cow's face to be identified, a multimodal fusion decision is made on the extracted 2D and 3D feature vectors to identify the cow's identity.

[0238] The similarity (e.g., Euclidean distance) between the 2D facial feature vector of the cow to be identified and the 2D feature vectors in the cow feature database is calculated, and the top N (Top-N) candidate list List_2D is returned. Similarly, the similarity (e.g., Euclidean distance) between the 3D facial feature vector of the cow to be identified and the 3D feature vectors in the cow feature database is calculated, and the top N (Top-N) candidate list List_3D is returned. N can take values ​​such as 1, 3, or 5.

[0239] Calculate the intersection S_intersection of the IDs of List_2D and List_3D. Set a confidence threshold T_confidence.

[0240] ● If S_intersection is not empty, and there is a candidate whose highest similarity (confidence score of the candidate fused with two modalities or confidence score of the 2D feature vector) exceeds T_confidence, then the identification is successful, and the candidate ID is used as the identification result of the cattle to be identified.

[0241] ● If S_intersection is not empty, but the confidence scores of all candidates are lower than T_confidence, then it is judged as "insufficient confidence, it is recommended to re-collect".

[0242] ● If S_intersection is empty, it is determined as "target not identified" (not a cow in the database or identification failed).

[0243] The multimodal fusion decision-making method in this application combines set logic (intersection) and confidence measurement (threshold). It utilizes the consensus of bimodal approaches and controls output risk through thresholds, thereby significantly improving the recognition accuracy (over 90%).

[0244] According to embodiments of this application, in order to identify the identity of cattle, a cattle feature database is also established to store the 3D feature vectors of the 3D point cloud of the cattle faces of cattle with known identities. For example, in an insurance application scenario, when insuring cattle, qualified images of the cattle's face from three perspectives—front, left 45°, and right 45°—are taken (at least one image for each perspective).

[0245] Images from three perspectives, based on embodiments of this application (see also...). Figures 1-8 The processing flow is as follows: image preprocessing, Dust3R model to build 3D mesh model, black edge detection and filtering, point cloud standardization and key region cropping, and then fed into the trained H-Point Net++ model to obtain 3D feature vectors.

[0246] All qualified images of the insured cattle from three perspectives are fed into the FaceNet model to obtain 2D feature vectors.

[0247] The obtained 3D feature vectors and 2D feature vectors are combined with the identity ID of the insured cattle to obtain the cattle feature database.

[0248] When using the embodiments of this application to identify whether a target cattle is insured, qualified images of the target cattle's face are collected from three perspectives: frontal, left 45°, and right 45°, based on the embodiments of this application (see also the embodiments of this application). Figures 1-8 The processing flow of the identification results is as follows: output the ID of the identified insured cattle, indicate insufficient confidence, or identify failure (non-insured cattle)

[0249] Optionally, in some cases, due to factors such as quality defects in the acquired cow face images, the 3D processing branch is not executed, and only 2D feature vectors are generated. In this case, only the generated 2D feature vectors are used to query the cow feature database to obtain a list of the top N candidates similar to the queried 2D feature vectors. The results with confidence scores greater than a specified threshold are selected as the cow identity query results. If the confidence scores of all candidates are lower than the threshold, it is determined that "confidence is insufficient, and re-collection is recommended".

[0250] The core of this application lies in constructing a collaborative and complementary multimodal biometric recognition framework, and through a series of targeted improvements, overcoming the inherent limitations of single-modal technologies in livestock identification. Its key technical points include: 1. A multimodal collaborative decision-making architecture that integrates 3D and 2D features This is the top-level architectural technical improvement of this invention. Specifically: 3D stereo geometry channel: The H-PointNet++ network is used to extract the three-dimensional stereo geometry features of the cow face. Based on its high robustness to changes in lighting and pose, the candidate individuals with the highest stereo geometry similarity are retrieved from a large-scale database to form a high-recall geometric candidate set.

[0251] 2D Texture Channel: Simultaneously utilize the FaceNet network to extract two-dimensional texture features of the cow's face, making full use of the rich fine-grained discriminative information (such as patterns) in the image to generate a high-discrimination texture candidate set.

[0252] Cross-validation and consensus formation: The candidate sets of the two channels are intersected to form a consensus candidate set. If the intersection is empty, the system can directly determine that the queried sample is not in the database without further calculation, greatly improving the efficiency of rejection.

[0253] Confidence fusion and final decision: When the consensus candidate set is not empty, the candidate individual with the highest aggregate similarity is selected as the final matching object. The 3D-2D aggregate similarity of this candidate individual is calculated. The aggregate similarity is compared with a preset threshold to make a three-level decision: Successful identification: If the aggregated similarity is greater than or equal to the threshold, output confirmation of identity.

[0254] Recognition failed: Aggregated similarity < threshold, indicating insufficient recognition confidence. It is recommended to re-collect data.

[0255] If the consensus candidate set is empty, it is determined that no target has been identified.

[0256] 2. Precise preprocessing and feature learning techniques for cow face point cloud data Cow Face 3D Point Cloud Cropping: A cow face point cloud cropping method based on least squares fitting is proposed. This method can accurately segment key facial regions for cow identification from a complete cow face point cloud, removing interfering parts such as the cheeks and neck, significantly improving the purity and efficiency of subsequent feature learning.

[0257] Improved Point Cloud Feature Extraction Network: The PointNet++ network was specifically modified for similarity learning, resulting in H-PointNet++, a feature extraction network dedicated to extracting 3D features of cow faces. Loss function design: A triplet loss function is introduced for training, which drives the feature space learned by the network to satisfy the constraint that "the feature distance of the 3D point cloud of the same cow's face is small, and the feature distance of the 3D point cloud of different cows is large", which directly optimizes the metric required for the recognition task.

[0258] Triple Sample Sampling Strategy Optimization: To enhance the optimization efficiency of the triple loss function and improve the feature extraction capability of H-PointNet++, a progressive adaptive hard sample mining strategy that simulates the human cognitive pattern of "from easy to difficult" is proposed. By dynamically dividing the model training into three stages—semi-hard sample mining (initial foundation building), random mixed mining (mid-term signal balancing), and fully hard sample mining (late-term optimal boundary)—the learning difficulty is adaptively adjusted. This solves the problems of instability and easy getting trapped in local optima in the early stage of traditional hard sample mining training, enabling the model to learn efficiently under the appropriate challenge intensity and strengthening the ability to distinguish fine-grained differences.

[0259] 3. Dynamic threshold decision-making mechanism for practical applications To facilitate the transition from laboratory algorithms to stable products, this application designs an intelligent threshold decision-making scheme: Point cloud quality pre-screening threshold: A quality assessment threshold based on point cloud density, integrity, and noise level is established. Before the recognition process begins, the input 3D data is screened, and samples that do not meet the quality standards will be required to be re-acquired or transferred to a pure 2D process, ensuring the reliability of the 3D recognition branch from the source.

[0260] Fusion result rejection threshold: In the final fusion decision stage, a confidence rejection threshold is set. When the consensus between the recognition results of the 3D and 2D modules is insufficient, or the highest score is lower than this threshold, the system will determine it as "recognition failure" or "unknown individual", instead of forcibly outputting a low-confidence result, thereby greatly reducing the false recognition rate and improving the system's practicality in open environments.

[0261] Compared with the prior art, the cattle identification method and system based on multimodal information fusion provided in this application have achieved the following significant beneficial effects: 1. A significant leap in recognition accuracy has been achieved. The core effect of this application's embodiments lies in fundamentally improving recognition accuracy. Through deep fusion of 3D point cloud and 2D image features, the system can simultaneously utilize the stability of three-dimensional morphology and the high resolution of two-dimensional texture, achieving information complementarity.

[0262] Experimental Verification: Table 3 compares the recognition accuracy of various existing methods with the method of the present application embodiment on a test set covering 500 cattle and over 10,000 real-world scene images. In Table 3, Top1 accuracy refers to the result of comparing the input sample with the feature vectors of all cattle in the cattle feature database and selecting the most similar one. Similarly, Top5 accuracy is the result of selecting the top 5 similarity results when comparing with the feature vectors of the cattle feature database. In Table 3, rows labeled "FaceNet" represent the results of recognition based on 2D feature vectors using only the FaceNet model, rows labeled "H-PointNet++" represent the results of recognition based on 3D feature vectors using only the H-PointNet++ model of the present application embodiment, and rows labeled "FaceNet+H-PointNet++" represent the recognition results obtained by combining the 2D and 3D processing branches of the present application embodiment and performing multimodal fusion decision.

[0263] The Top-1 recognition accuracy of this application embodiment consistently reaches 94.5%. This result is significantly better than the single-modality scheme used as a control, demonstrating that the fusion strategy plays a decisive role in overcoming the performance limitations of single-modality schemes. Although the FaceNet model has a high Top-5 accuracy, its applicability is relatively low in scenarios such as identifying whether a target cattle is insured, because it outputs five recognition results and requires manual further screening to identify insured cattle. This not only causes operational inconvenience but also presents difficulties for manual screening of cattle.

[0264] Table 3: Comparison of Recognition Accuracy

[0265] 2. It has good environmental robustness. The embodiments of this application effectively overcome the pain point of traditional methods experiencing a sharp drop in performance under complex environments.

[0266] Combating lighting interference: The 3D point cloud module is completely invariant to changes in lighting, ensuring that the system can still provide a stable candidate set under adverse lighting conditions such as dawn, dusk, and backlighting, thus compensating for the feature distortion of pure 2D models in such scenarios.

[0267] Tolerance for pose variations and partial occlusion: When the cattle's head is not in a frontal pose or is partially occluded, the 3D global contour features can serve as a reliable backup; while when the 3D structures of different individuals are highly similar, the subtle texture features such as patterns and hair textures in the 2D modules become key to accurate identification. This cross-modal fault-tolerant and collaborative mechanism significantly improves the system's recognition success rate under non-ideal acquisition conditions.

[0268] 3. It pioneered a widely applicable technological paradigm. This application successfully applies the concept of multimodal fusion to the specific field of livestock identification and verifies the effectiveness of the efficient architecture of the "multimodal collaborative authentication strategy." It also exhibits good scalability. Its technical framework and methodology can be directly transferred to the identification of other livestock with biological characteristics, such as pigs and sheep, providing an innovative solution to the identification challenges in the entire smart agriculture field.

[0269] In summary, this invention not only achieves a leap forward in technical indicators, but also demonstrates great value in terms of practicality, economy, and industry influence, providing core technical support for the intelligent upgrading of livestock management.

[0270] According to embodiments of this application, it is executed on an information processing device such as a computer or server. Figure 14 This is a block diagram of an information processing device that implements an embodiment of the present invention. Figure 14 The information processing device shown is a computer. The computer is merely one example of a suitable computing environment and is not intended to imply any limitation on the use or scope of the invention. Figure 14 The information processing equipment shown should not be construed as having any dependency or requirement on any of the components or combinations of components shown.

[0271] Information processing devices include memory, one or more processors, one or more displays, I / O components, and a power supply, directly or indirectly coupled to a bus. A bus can represent one or more types of buses (such as an address bus, a data bus, or a combination thereof). However, in practice, the definition of each component is not necessarily as straightforward. Figure 14 In this way, for example, a display device can be considered as an I / O component. Furthermore, the processor may have memory. The inventors of this invention recognize that this is precisely the nature of the art and reiterate... Figure 14 The illustrations are merely for illustrating exemplary computer systems that can be used in conjunction with one or more embodiments of the present invention.

[0272] Information processing devices typically include various types of memory. By way of example and not limitation, memory may include: random access memory (RAM), read-only memory (ROM), electronically erasable programmable read-only memory (EEPROM), flash memory, CD-ROM, DVD or other optical or holographic media, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices. Computer storage media may be non-volatile.

[0273] Information processing devices include one or more processors that read data from various entities such as buses, memory, or I / O groups. One or more displays present data indications to a user or other device. Exemplary displays include display devices, speakers, printing components, vibrating components, flat panel displays, projectors, head-mounted displays, etc. A display can also be an I / O port for coupling display devices, speakers, printing components, vibrating components, flat panel displays, projectors, head-mounted displays, etc. Illustrative I / O components include cameras, microphones, joysticks, game controllers, dish satellite transceiver antennas, scanners, printers, wireless devices, etc.

[0274] Although the examples referenced in this application are described for illustrative purposes only and not for limiting the scope of this application, changes, additions and / or deletions to the implementation may be made without departing from the scope of this application.

[0275] Those skilled in the art who benefit from the teachings presented in the above description and associated drawings will recognize many modifications and other embodiments of the present application described herein. Therefore, it should be understood that the present application is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terminology is used herein, it is used only in its general and descriptive sense and not for limiting purposes.

Claims

1. A method for identifying cattle based on multimodal information fusion, the method comprising: Acquire frontal image, left 45-degree angle image, and right 45-degree angle image of the cow's face to be identified; Generate 2D feature vectors based on the frontal cow face image, the cow face image at a 45-degree angle to the left, and / or the cow face image at a 45-degree angle to the right of the cow to be identified; The frontal, left 45-degree, and right 45-degree images of the cow's face to be identified are fed into the Dust3R model to generate a 3D mesh model of the cow's face to be identified. Convert the 3D mesh model of the cow's face to be identified into a 3D point cloud; The 3D point cloud is cropped to remove the 3D point cloud area corresponding to the cow's cheek, while retaining the 3D point cloud area above the cheek. The removal of the 3D point cloud region corresponding to the cow's cheek includes: The overall geometric reference plane for fitting the bull's face from the 3D point cloud is represented by the plane equation n·x + d = 0, where n represents the unit normal vector of the plane equation and d is the plane constant parameter; Calculate the position of each point in the 3D point cloud relative to the overall geometric reference plane of the bull's face represented by the plane equation; and Delete points in the 3D point cloud that are located below the overall geometric reference plane of the bull's face relative to the overall geometric reference plane of the bull's face; Generate 3D feature vectors from the cropped 3D point cloud; Based on the 2D feature vector, the first plurality of candidate results most similar to the 2D feature vector are obtained from the cattle feature database by sorting them by similarity; wherein, the cattle feature database records the cattle identity and its 2D feature vector and 3D feature vector in association; Based on the 3D feature vector, obtain the second or more candidate results that are most similar to the 3D feature vector from the cattle feature database by sorting them by similarity; If the first plurality of candidate results intersect with the second plurality of candidate results, and the confidence level of the candidate results in the intersection is greater than a threshold, the candidate results in the intersection with a confidence level greater than the threshold shall be used as the identification results of the cattle to be identified.

2. The cattle identification method according to claim 1 further includes: Solve the plane equation to obtain the solution for parameters n and d. If the normal vector of the plane equation represented by n points in a specified direction, flip the plane equation by n = -n and d = -d. The specified direction is the direction in which its Z-axis component is negative, and the Z-axis is the coordinate axis perpendicular to the ground and pointing upwards.

3. The cattle identification method according to claim 2, wherein, The signed distance from point x in the 3D point cloud to the overall geometric reference plane of the bull's face is calculated according to s = n·x + d, where x is the coordinate vector of the point in the 3D point cloud; Deleting points from the 3D point cloud that are located below the face plane relative to the face plane includes: When s < 0, the corresponding point x is deleted from the 3D point cloud; When s>0, the corresponding point x is retained in the 3D point cloud.

4. The cattle identification method according to claim 3, wherein, The acquisition of the frontal image, the left 45-degree angle image, and the right 45-degree angle image of the cow's face to be identified includes: When collecting cow face images, the cow face posture is identified in real time by a cow face recognition model based on YOLOv8. When the cow face posture is in a frontal view, a left 45-degree view, or a right view, the capture is triggered to obtain the cow face image. Extract the cow face region from the acquired cow face image and remove the background outside the cow face region to obtain the segmented cow face image; The segmented cow face image is matched with the optimal viewpoint template of the cow face image to output a qualified cow face image with the same posture as the optimal viewpoint template; wherein the optimal viewpoint template includes a preset frontal cow face image, a left 45° cow face image and a right 45° cow face image. If the obtained qualified cow face images include at least one qualified cow face image from each of the three perspectives: frontal cow face image, left 45° cow face image, and right 45° cow face image, then 3D feature vectors are generated using these qualified cow face images, and 2D feature vectors are generated using the qualified cow face images from the frontal perspective, left 45° perspective, and / or right 45° perspective.

5. The cattle identification method according to claim 4 further includes: Obtain the confidence level generated when the Dust 3R model generates the 3D mesh model, and identify the usability of the 3D mesh model; For the 3D mesh model with availability greater than a specified threshold, detect and remove black edges from the 3D mesh model; and The 3D mesh model with black edges removed is converted into the 3D point cloud.

6. The cattle identification method according to claim 5, wherein, Obtain the confidence features of the 3D mesh model, including the best edge confidence score and the second best edge confidence score output by the Dust 3R model, as well as the proportion of low confidence points cleaned up, the average confidence score, and the proportion of high confidence noise points when the Dust 3R model generates the 3D mesh model; The availability of the 3D mesh model is identified by using a availability verification model based on the random forest algorithm and the indicators used to predict the availability of the 3D mesh model. For each mesh vertex on the edge of the 3D mesh model, calculate the sum of the color component values ​​of the R, G, and B color channels for each mesh vertex; When the color component values ​​of the R, G, and B channels of a vertex are detected to be below the determination threshold, the vertex is determined to be a black edge and is removed from the 3D mesh model.

7. The cattle identification method according to claim 6 further includes: Before cropping the 3D point cloud, the pose of the 3D point cloud is transformed to be consistent with that of the reference point cloud, wherein the reference point cloud represents a 3D point cloud of a cow's face with a standard pose. The 3D point cloud is scaled to a uniform unit scale.

8. The cattle identification method according to claim 7, wherein, The step of generating 3D feature vectors from the cropped 3D point cloud includes: The H-PointNet++ model is used to receive the cropped 3D point cloud and generate 3D feature vectors, where... The H-PointNet++ model is based on the PointNet++ model, with a new SA layer added before the three SA layers of the PointNet++ model. The new SA layer samples 1024 center points uniformly from the 3D point cloud according to the farthest point. The local neighborhood sampling radius of each center point is smaller than the local neighborhood sampling radius of the first SA layer of the PointNet++ model, and the number of local neighborhood sampling points of each center point is greater than the number of local neighborhood sampling points of the first SA layer of the PointNet++ model. The H-PointNet++ model replaces the output header of the PointNet++ model with a feature vector output header, wherein... The feature vector output head receives the output of the last SA layer of the PointNet++ model and generates a 3D feature vector representing the 3D point cloud; wherein, the feature vector output head includes a feature compression layer, a feature distribution normalization layer, a nonlinear mapping layer, a normalization layer and a vector normalization layer connected in sequence; the vector normalization layer outputs the 3D feature vector.

9. The cattle identification method according to claim 8, wherein, The training samples for training the H-PointNet++ model are triplets consisting of anchor samples, positive samples, and negative samples; where anchor samples serve as a reference benchmark for identity; positive samples and anchor samples are 3D point clouds obtained from cow face images corresponding to the same cow; and negative samples and anchor samples are 3D point clouds from cows with different identities. The H-PointNet++ model is trained using the loss function L = max(d(a,p) - d(a,n) + margin,0), where a, p, and n represent the feature vectors of the anchor sample, positive sample, and negative sample, respectively; d() represents the Euclidean distance calculation function; and margin represents the margin threshold. The optimization objective of the training process is to adjust the network parameters to make the feature distance between the anchor sample and the positive sample less than the feature distance between the anchor sample and the negative sample, and the difference between the two must meet the margin threshold requirement. In the early stages of training the H-PointNet++ model, a semi-hard sample triplet is constructed, consisting of an anchor sample, a positive sample, and a negative sample whose feature vector is far from the anchor sample but not completely separated; the H-PointNet++ model is then trained using the semi-hard sample triplet. During the training of the H-PointNet++ model, a mixed sample consisting of semi-hard sample triples and hard sample triples is constructed, and the H-PointNet++ model is trained using the mixed sample; wherein the negative sample in the hard sample triple comes from the heterogeneous sample that is closest to the feature vector of the anchor sample. In the later stages of training the H-PointNet++ model, the H-PointNet++ model is trained using hard sample triples.

10. An information processing apparatus comprising a storage, a processor, and a program stored on the storage and capable of running on the processor, characterized by When the processor executes the program, it implements the method according to any one of claims 1-9.

Images