Livestock identity verification method and system based on multi-modal image analysis

Abstract

The invention discloses a livestock identity verification method and system based on multi-modal image analysis, and belongs to the technical field of image processing. And non-contact identity recognition is realized through collaborative analysis of macroscopic dynamic contours and microcosmic local biological characteristics of the livestock. Comprising the following steps: synchronously acquiring a side body video stream and a head and neck near-field image; key attitude frames are extracted from a video stream, a target monomer mask sequence subjected to form correction is generated, and dynamic contour feature vectors are extracted through a three-dimensional convolutional neural network; the head and neck images are processed through a double-branch feature extraction network, and the network extracts deep semantic features and micro texture features enhanced by directional gradient filtering in parallel and fuses the deep semantic features and the micro texture features to generate local biological feature vectors; and finally, respectively evaluating confidence coefficients of the two types of features, performing adaptive weighted fusion according to the confidence coefficients, and generating target features to be matched with a database. According to the method, the limitation of single feature recognition in a complex scene is effectively overcome, and the accuracy and practicability of identity verification are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and system for livestock identification based on multimodal image analysis. Background Technology

[0002] In modern intensive livestock farming, accurate and efficient identification and management of individual livestock is fundamental to achieving key aspects such as refined feeding, disease prevention and control, pedigree tracing, and production performance recording. Traditional livestock identification methods mainly rely on physical tags, such as ear tags, neck bands, and radio frequency identification (RFID) tags. However, these methods have significant drawbacks: physical tags are easily worn, detached, tampered with, or forged; implanted tags may cause stress, infection, or animal welfare problems. Furthermore, in large-scale ranches, individually capturing livestock to read or replace tags is cumbersome, costly, and easily disturbs the animals.

[0003] With the development of computer vision technology, contactless identification methods based on biometrics have become a research hotspot. Existing solutions mostly focus on identifying livestock using single visual features, primarily including identification based on body patterns or coat color. However, this method is extremely sensitive to changes in lighting and is largely ineffective for breeds with uniform coat color or no obvious markings. Identification based on three-dimensional features is another approach; however, the discriminative power of three-dimensional features is limited, and in real-world scenarios, it's difficult to ensure livestock cooperate with the camera in a fixed posture, resulting in unstable image quality. Identification based on high-precision local features, while possessing uniqueness and stability, requires extremely close acquisition distances and a high degree of animal cooperation, making it difficult to achieve large-scale, rapid, contactless, and automated inspections. Summary of the Invention

[0004] This application provides a livestock identification method and system based on multimodal image analysis to improve the above-mentioned problems.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, this application proposes a livestock identification method based on multimodal image analysis, including:

[0007] Acquire lateral body video streams and head and neck near-field images of the target livestock. The lateral body video streams are used to determine dynamic contour features, and the head and neck near-field images are used to determine local biological features.

[0008] M key pose frames are determined based on the side body video stream, and M target unit masks are determined based on the M key pose frames;

[0009] M target single-unit mask sequences are input into a pre-trained 3D convolutional neural network. Based on the output of the 3D convolutional neural network, the dynamic contour feature vector that changes with motion is determined.

[0010] The near-field image of the head and neck is input into a pre-trained dual-branch feature extraction network. The first branch of the dual-branch feature extraction network is an encoder network for extracting deep semantic features. The second branch of the dual-branch feature extraction network adopts a directional gradient filter group and is used to obtain the micro-spatial texture information composed of skin wrinkles, hair direction or spot texture, and output the corresponding texture features.

[0011] By fusing deep semantic features with texture feature maps, local biological feature vectors are generated.

[0012] Obtain the first confidence level corresponding to the dynamic contour feature vector and the second confidence level corresponding to the local biological feature vector;

[0013] Based on the first and second confidence levels, the local biological feature vector and the dynamic contour feature vector are fused to determine the target features;

[0014] The target features are matched with a livestock feature database for similarity, and the identity is verified based on the matching results.

[0015] In conjunction with the first aspect, optionally, a lateral body video stream and a head and neck near-field image of the target livestock are acquired, wherein the lateral body video stream is used to determine dynamic contour features, and the head and neck near-field image is used to determine local biological features, including:

[0016] Obtain temporal information of motion containing gait periodicity based on lateral body video stream;

[0017] Based on motion temporal information, at least one frame is selected from the head and neck near-field image sequence as the head and neck near-field image.

[0018] In conjunction with the first aspect, optionally, based on motion temporal information, at least one frame is selected from the head and neck near-field image sequence as the head and neck near-field image, including:

[0019] Obtain the sum of texture gradient magnitudes for each image frame in the head and neck near-field image sequence;

[0020] Based on motion timing information, the phase interval in which the target animal is in a stable posture and has the smallest head sway is determined;

[0021] From all image frames corresponding to the phase interval, the image frame with the largest sum of texture gradient magnitudes is determined as the near-field image of the head and neck.

[0022] In conjunction with the first aspect, optionally, temporal information of motion containing gait periodicity is obtained based on the lateral body video stream, including:

[0023] Joint key point detection is performed on the lateral body video stream to obtain the head movement trajectory sequence and at least one leg joint movement trajectory sequence of the target animal;

[0024] Determine the basic gait frequency of the target livestock based on the sequence of leg joint motion trajectories;

[0025] Based on the basic gait frequency, peaks and troughs that conform to periodic patterns in the motion trajectory sequence are located in the time domain, and the time interval between adjacent peaks is defined as a complete gait cycle.

[0026] The average displacement amplitude of the head motion trajectory sequence within each gait cycle is obtained as a quantitative indicator to characterize the degree of head swaying.

[0027] Integrating basic gait frequency, quantitative indicators, and the corresponding phase of the gait cycle constitutes motion timing information.

[0028] In conjunction with the first aspect, optionally, M key pose frames are determined based on the side body video stream, and M target unit masks are determined based on the M key pose frames, including:

[0029] The movement trajectories of the animal's limbs are extracted based on the side body video stream.

[0030] Based on the movement trajectory of the limbs, M video frames in which all four limbs of the livestock are in contact with the ground are identified as key pose frames.

[0031] The key pose frames are input into the instance segmentation network for segmentation.

[0032] The initial mask for the target livestock in the key pose frame is generated using the segmentation results output by the instance segmentation network.

[0033] Obtain the preset torso aspect ratio constraint, and based on the preset torso aspect ratio constraint, correct the initial mask to obtain the target single-unit mask.

[0034] In conjunction with the first aspect, optionally, the sequence of M target individual masks is input into a pre-trained three-dimensional convolutional neural network, and based on the output of the three-dimensional convolutional neural network, the dynamic contour feature vector that changes with motion is determined, including:

[0035] Stack M target unit masks in chronological order to obtain three-dimensional mask sequence volume data;

[0036] The volume data of the three-dimensional mask sequence is input into a pre-trained three-dimensional convolutional neural network for spatiotemporal feature extraction, resulting in a three-dimensional feature tensor.

[0037] Obtain the feature vector of the three-dimensional feature tensor at each time step in the time dimension;

[0038] The feature vector at each time step is input into a fully connected network, and a scalar weight representing the importance of the time step is output.

[0039] Normalize the scalar weights corresponding to all time steps to obtain a normalized weight sequence;

[0040] The three-dimensional feature tensor is weighted and summed in the time dimension using a normalized weight sequence, and the two-dimensional spatial feature map is obtained based on the weighted summation result.

[0041] Global average pooling and global max pooling are performed on the two-dimensional spatial feature map, and the pooling results are concatenated to form a dynamic contour feature vector.

[0042] In conjunction with the first aspect, optionally, the first confidence level corresponding to the dynamic contour feature vector and the second confidence level corresponding to the local biometric feature vector are obtained, including:

[0043] Obtain the cosine similarity sequence of dynamic contour feature vectors within multiple consecutive time windows;

[0044] Based on the standard deviation between each cosine similarity in the cosine similarity sequence, the first confidence level representing the temporal stability of the feature is determined, where the smaller the standard deviation, the higher the first confidence level.

[0045] K registration feature vectors that are most similar to the local biological feature vectors are obtained from the livestock feature database, and N non-matching registration feature vectors are randomly obtained.

[0046] The average distance from the local biometric feature vector to the K registered feature vectors is taken as the intra-class distance, and the average distance from the local biometric feature vector to the N non-matching registered feature vectors is taken as the inter-class distance.

[0047] A second confidence level is determined based on the ratio of intra-class distance to inter-class distance, where a smaller ratio indicates a higher second confidence level.

[0048] Secondly, this application proposes a livestock identification system based on multimodal image analysis, the system being configured as follows:

[0049] Acquire lateral body video streams and head and neck near-field images of the target livestock. The lateral body video streams are used to determine dynamic contour features, and the head and neck near-field images are used to determine local biological features.

[0050] M key pose frames are determined based on the side body video stream, and M target unit masks are determined based on the M key pose frames;

[0051] M target single-unit mask sequences are input into a pre-trained 3D convolutional neural network. Based on the output of the 3D convolutional neural network, the dynamic contour feature vector that changes with motion is determined.

[0052] The near-field image of the head and neck is input into a pre-trained dual-branch feature extraction network. The first branch of the dual-branch feature extraction network is an encoder network for extracting deep semantic features. The second branch of the dual-branch feature extraction network adopts a directional gradient filter group and is used to obtain the micro-spatial texture information composed of skin wrinkles, hair direction or spot texture, and output the corresponding texture features.

[0053] By fusing deep semantic features with texture feature maps, local biological feature vectors are generated.

[0054] Obtain the first confidence level corresponding to the dynamic contour feature vector and the second confidence level corresponding to the local biological feature vector;

[0055] Based on the first and second confidence levels, the local biological feature vector and the dynamic contour feature vector are fused to determine the target features;

[0056] The target features are matched with a livestock feature database for similarity, and the identity is verified based on the matching results.

[0057] In conjunction with the second aspect, optionally, the system is configured as follows:

[0058] Acquire lateral body video streams and near-field images of the head and neck of the target livestock. The lateral body video stream is used to determine dynamic contour features, and the near-field images of the head and neck are used to determine local biological features, including:

[0059] Obtain temporal information of motion containing gait periodicity based on lateral body video stream;

[0060] Based on motion temporal information, at least one frame is selected from the head and neck near-field image sequence as the head and neck near-field image.

[0061] In conjunction with the second aspect, optionally, the system is configured as follows:

[0062] Based on motion temporal information, at least one frame is selected from the head and neck near-field image sequence as the head and neck near-field image, including:

[0063] Obtain the sum of texture gradient magnitudes for each image frame in the head and neck near-field image sequence;

[0064] Based on motion timing information, the phase interval in which the target animal is in a stable posture and has the smallest head sway is determined;

[0065] From all image frames corresponding to the phase interval, the image frame with the largest sum of texture gradient magnitudes is determined as the near-field image of the head and neck.

[0066] In conjunction with the second aspect, optionally, the system is configured as follows:

[0067] Based on the lateral body video stream, temporal information of motion containing gait periodicity is obtained, including:

[0068] Joint key point detection is performed on the lateral body video stream to obtain the head movement trajectory sequence and at least one leg joint movement trajectory sequence of the target animal;

[0069] Determine the basic gait frequency of the target livestock based on the sequence of leg joint motion trajectories;

[0070] Based on the basic gait frequency, peaks and troughs that conform to periodic patterns in the motion trajectory sequence are located in the time domain, and the time interval between adjacent peaks is defined as a complete gait cycle.

[0071] The average displacement amplitude of the head motion trajectory sequence within each gait cycle is obtained as a quantitative indicator to characterize the degree of head swaying.

[0072] Integrating basic gait frequency, quantitative indicators, and the corresponding phase of the gait cycle constitutes motion timing information.

[0073] In conjunction with the second aspect, optionally, the system is configured as follows:

[0074] Based on the side view video stream, M key pose frames are determined, and based on the M key pose frames, M target unit masks are determined, including:

[0075] The movement trajectories of the animal's limbs are extracted based on the side body video stream.

[0076] Based on the movement trajectory of the limbs, M video frames in which all four limbs of the livestock are in contact with the ground are identified as key pose frames.

[0077] The key pose frames are input into the instance segmentation network for segmentation.

[0078] The initial mask for the target livestock in the key pose frame is generated using the segmentation results output by the instance segmentation network.

[0079] Obtain the preset torso aspect ratio constraint, and based on the preset torso aspect ratio constraint, correct the initial mask to obtain the target single-unit mask.

[0080] In conjunction with the second aspect, optionally, the system is configured as follows:

[0081] M target unit mask sequences are input into a pre-trained 3D convolutional neural network. Based on the output of the 3D convolutional neural network, the dynamic contour feature vector that changes with motion is determined, including:

[0082] Stack M target unit masks in chronological order to obtain three-dimensional mask sequence volume data;

[0083] The volume data of the three-dimensional mask sequence is input into a pre-trained three-dimensional convolutional neural network for spatiotemporal feature extraction, resulting in a three-dimensional feature tensor.

[0084] Obtain the feature vector of the three-dimensional feature tensor at each time step in the time dimension;

[0085] The feature vector at each time step is input into a fully connected network, and a scalar weight representing the importance of the time step is output.

[0086] Normalize the scalar weights corresponding to all time steps to obtain a normalized weight sequence;

[0087] The three-dimensional feature tensor is weighted and summed in the time dimension using a normalized weight sequence, and the two-dimensional spatial feature map is obtained based on the weighted summation result.

[0088] Global average pooling and global max pooling are performed on the two-dimensional spatial feature map, and the pooling results are concatenated to form a dynamic contour feature vector.

[0089] In conjunction with the second aspect, optionally, the system is configured as follows:

[0090] Obtain the first confidence score corresponding to the dynamic contour feature vector and the second confidence score corresponding to the local biometric feature vector, including:

[0091] Obtain the cosine similarity sequence of dynamic contour feature vectors within multiple consecutive time windows;

[0092] Based on the standard deviation between each cosine similarity in the cosine similarity sequence, the first confidence level representing the temporal stability of the feature is determined, where the smaller the standard deviation, the higher the first confidence level.

[0093] K registration feature vectors that are most similar to the local biological feature vectors are obtained from the livestock feature database, and N non-matching registration feature vectors are randomly obtained.

[0094] The average distance from the local biometric feature vector to the K registered feature vectors is taken as the intra-class distance, and the average distance from the local biometric feature vector to the N non-matching registered feature vectors is taken as the inter-class distance.

[0095] A second confidence level is determined based on the ratio of intra-class distance to inter-class distance, where a smaller ratio indicates a higher second confidence level.

[0096] A third aspect of the present invention provides an electronic device comprising:

[0097] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method proposed in the first aspect of the present invention.

[0098] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the embodiments of the present invention.

[0099] In summary, the above method has the following technical effects:

[0100] This invention discloses a method and system for livestock identification based on multimodal image analysis. The method achieves contactless identification by collaboratively analyzing the macroscopic dynamic contours and microscopic local biometric features of livestock. Specifically, it includes: simultaneously acquiring lateral body video streams and near-field images of the head and neck; extracting key pose frames from the video stream and generating a morphology-corrected target individual mask sequence, and extracting dynamic contour feature vectors using a three-dimensional convolutional neural network; simultaneously processing the head and neck images through a dual-branch feature extraction network, which extracts deep semantic features and microscopic texture features enhanced by directional gradient filtering in parallel, and fuses them to generate local biometric feature vectors; finally, evaluating the confidence levels of the two types of features respectively, and performing adaptive weighted fusion to generate target features for matching with a database. This invention effectively overcomes the limitations of single-feature recognition in complex scenarios, significantly improving the accuracy, robustness, and practicality of identification verification. Attached Figure Description

[0101] Figure 1 This is a flowchart illustrating a livestock identification method based on multimodal image analysis proposed in an embodiment of this application. Detailed Implementation

[0102] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0103] This application proposes a livestock identification method based on multimodal image analysis. Please refer to [link to relevant documentation]. Figure 1 This includes the following steps:

[0104] S101: Acquire the lateral body video stream and head and neck near-field images of the target livestock, wherein the lateral body video stream is used to determine dynamic contour features, and the head and neck near-field images are used to determine local biological features.

[0105] Understandably, a side-view video stream is a continuous video stream shot from the side of a livestock, fully recording the animal's body movements while walking or standing naturally. The temporal information contained in the video can reflect the unique biological behavioral characteristics of an individual over time, such as body proportions, gait rhythm, and movement habits. In other words, it is used to capture and analyze the macroscopic dynamic contour features of livestock.

[0106] Close-field images of the head and neck are single or multiple images of the head and neck region of livestock, taken at relatively close range, and are characterized by clearer details. This area typically contains stable and discriminative details such as unique patterns of skin wrinkles, microscopic textures of hair, localized distribution of spots or coat color, and the shape of the muzzle.

[0107] Specifically, in this embodiment, for head and neck near-field images, motion timing information containing gait periodicity can be obtained based on the lateral body video stream. Then, based on the motion timing information, at least one frame image is selected from the head and neck near-field image sequence as the head and neck near-field image.

[0108] Understandably, analyzing the lateral body video stream to extract temporal information that characterizes the regular walking rhythm of livestock is crucial, with the most important aspect being the identification of its gait cycle. The gait cycle reveals the cyclical patterns of when a animal lifts its leg, when it lands, and when its body is most relaxed and stable.

[0109] After mastering the movement timing pattern, the best timing indicated by the movement timing information can be selected from the head and neck image sequence acquired at the same time, such as the phase in the gait cycle when the limbs are stable, the head is relatively still, and the body sway is minimal. The frame or several frames with the most upright posture and the least swaying blur can be selected as the final head and neck near-field images for analysis.

[0110] For example, motion timing information can be obtained through steps S201-S205:

[0111] S201: Perform joint key point detection on the lateral body video stream to obtain the head movement trajectory sequence and at least one leg joint movement trajectory sequence of the target animal.

[0112] Specifically, a deep learning-based pose estimation model can be used to analyze each frame of the video stream. The model is trained to identify and locate specific points on the animal's body surface that have stable anatomical significance. For the head, feature points such as the tip of the nose, the corner of the eye, or the base of the ear are typically detected; for the legs, joints such as the shoulder joint, elbow joint, wrist joint, or the upper edge of the hoof are detected. The detection output is the two-dimensional pixel coordinates of each defined keypoint in each frame.

[0113] After obtaining the two-dimensional pixel coordinates of each defined keypoint in each frame, the coordinate positions of the same keypoint in consecutive frames can be continuously recorded. The coordinates of the head keypoints in each frame are arranged in chronological order, thus forming a sequence of head motion trajectories. Similarly, the coordinates of one or more leg joints in each frame are arranged in chronological order, thus forming a sequence of one or more leg joint motion trajectories. Each trajectory sequence is a set of time-indexed two-dimensional coordinate arrays that fully describe the spatial movement path of that body part during video recording.

[0114] S202: Determine the basic gait frequency of the target livestock based on the movement trajectory sequence of leg joints.

[0115] Understandably, inputting a two-dimensional motion trajectory sequence of one or more leg joints—a discrete signal showing the joint positions changing over time, typically containing displacement components in both horizontal and vertical dimensions—results in a scalar value, the base gait frequency. Physically, this represents the number of complete gait cycles completed per unit time. A complete gait cycle is generally defined as the process from the start of a support phase of a leg, through the swing phase, to the start of the next support phase of the same leg.

[0116] For example, the displacement component that best reflects the periodic fluctuations can be selected from the trajectory sequence. Then, spectral analysis, such as Fast Fourier Transform (FFT), is performed on the one-dimensional displacement signal. In the resulting spectrum, the frequency component with the strongest energy is found. This dominant frequency corresponds to the most significant and stable periodic rhythm in the gait motion and is determined as the basic gait frequency.

[0117] S203: Based on the basic gait frequency, locate the peaks and troughs that conform to the periodic pattern in the motion trajectory sequence in the time domain, and define the time period between adjacent peaks as a complete gait cycle.

[0118] Understandably, the extreme points of each periodic oscillation can be marked on the displacement-time signal curve.

[0119] For example, using the approximate period length corresponding to the basic gait frequency as a reference window, the displacement signal is scanned and analyzed in the time domain. Mathematical algorithms are used, such as finding points where the first derivative is zero and the second derivative sign changes, or using peak detection algorithms to identify a series of local maximum points in the signal, i.e., peaks, which usually correspond to the moment when the hoof is raised to its highest point, and local minimum points, i.e. troughs, which usually correspond to a low point when the hoof contacts the ground or is about to leave the ground.

[0120] Of course, the recognition process is constrained by the basic gait frequency to avoid misjudging signal noise or small fluctuations as valid extreme values, and to ensure that the peaks and troughs located roughly conform to the expected periodicity in terms of time interval.

[0121] Furthermore, based on the identified extreme points, the continuous time is segmented to define each independent behavioral cycle.

[0122] For example, the time interval between two adjacent gait peaks can be defined as a complete gait cycle. Peaks are chosen as dividing markers because they typically correspond to an easily identifiable and repeatable posture event kinematically.

[0123] Through this division, the entire continuous displacement time series is transformed into a series of consecutive time periods [T1, T2, ..., Tn], where each time period Ti represents a complete motion cycle that starts from a certain characteristic posture, undergoes a series of changes, and returns to a similar characteristic posture.

[0124] S204: Obtain the average displacement amplitude of the head motion trajectory sequence within each gait cycle, as a quantitative indicator representing the degree of head swaying.

[0125] For example, after obtaining the following head movement trajectory sequence, for each defined gait cycle, all coordinate points belonging to the current cycle can be extracted from the complete head movement trajectory sequence to form a subsequence. Then, the arithmetic mean of the x and y coordinates of all points in the subsequence is calculated to obtain the average center point of the head movement within that cycle. This point represents the equilibrium position around which the head swings within that cycle.

[0126] For each coordinate point in the subsequence, its Euclidean distance to the average center point can be calculated. The Euclidean distance represents the instantaneous displacement of the head from its average position within its period at a given frame.

[0127] The arithmetic mean of all instantaneous displacements within the cycle is calculated. This arithmetic mean is the average head displacement amplitude of the current gait cycle. In other words, it represents the typical deviation of the head's position at each moment within the cycle from its average position. Understandably, a larger value indicates a larger spatial range of head movement within the cycle, i.e., more vigorous shaking; a smaller value indicates a more stable head.

[0128] S205: Integrates basic gait frequency, quantitative indicators, and the phase of the corresponding gait cycle to form motion timing information.

[0129] Understandably, the discrete parameters and indicators generated by the aforementioned analysis steps are fused and encapsulated according to their inherent time logic to construct a motion state description data package.

[0130] Specifically, the three types of information are combined into a composite data structure with a unified time coordinate. A typical integration method is to generate an information list or array, where each element represents an analyzed gait cycle unit and contains the following fields:

[0131] Period identifier / phase: The period number and its start and end boundaries on the time axis.

[0132] The quantitative indicator value for this cycle: that is, the average displacement of the head within this cycle.

[0133] Global baseline gait frequency: As a common parameter, it is associated with each cyclic unit and indicates the rhythmic benchmark on which these cycles are generated.

[0134] Understandably, the output of this step is essentially a machine-readable motion state report. This motion state description data packet uses a common gait cycle phase as a time anchor, enabling optimal timing conclusions derived from dynamic video analysis to be unambiguously mapped to corresponding frames in the head and neck image sequence, achieving cross-modal alignment.

[0135] Furthermore, in this embodiment, near-field images of the head and neck can be obtained through steps S301-S303:

[0136] S301: Obtain the sum of texture gradient magnitudes for each image frame in the head and neck near-field image sequence.

[0137] Understandably, a head and neck near-field image sequence refers to multiple frames of digital images of the head and neck region acquired sequentially over time. The sum of texture gradient magnitudes is a value calculated using standard image processing operations.

[0138] Specifically, gradient operators, such as the Sobel and Prewitt operators, are applied to the input image to calculate the rate of change of brightness for each pixel in the horizontal and vertical directions. The result of this calculation assigns a gradient vector to each pixel, the magnitude of which characterizes the edge strength or the intensity of texture change at that pixel's location. A larger magnitude indicates sharper image details near that point, such as sharper edges of skin wrinkles and hair boundaries, resulting in more pronounced contrast. After calculating the gradient magnitudes of all pixels in the entire image, these magnitudes are summed to obtain a total value. This total value is the sum of the texture gradient magnitudes of that frame of the image.

[0139] Understandably, this sum value is a global image quality metric. A higher value indicates that the frame contains more sharp edges and significant texture details, resulting in a sharper image with richer information content, which is more beneficial for subsequent extraction of local biometric features. A lower value indicates that the image is generally blurry, smooth, or lacks contrast, with less obvious detail features.

[0140] S302: Based on motion timing information, determine the phase interval in which the target animal is in a stable posture and has the smallest head sway.

[0141] Understandably, in this step, motion timing information is used to execute a decision-making process to locate and output an optimal image acquisition time window. That is, based on quantitative motion stability analysis, it is determined which stage of the livestock's movement is most conducive to acquiring high-quality head and neck feature images.

[0142] Specifically, the system can read the sequence of average head displacement amplitude values ​​for each gait cycle recorded in the motion timing information. Then, it compares all the values ​​in the sequence to identify the one or more consecutive gait cycles with the smallest values. The smaller this quantification value, the less head movement and the more stable the posture within that cycle.

[0143] The phases corresponding to the identified gait cycles with the minimum quantification index value, i.e., the start and end times, are determined as the target interval. For example, if the analysis finds that the average head displacement amplitude is the smallest in the third gait cycle, then the time interval [t3, t4] of that cycle is determined as the phase interval where the posture is stable and the head sway amplitude is minimal.

[0144] S303: From all image frames corresponding to the phase interval, determine the image frame with the largest sum of texture gradient magnitudes as the head and neck near-field image.

[0145] Specifically, the sum of texture gradient magnitudes for each frame within that phase interval can be read. By comparison, the frame with the largest value is identified. This frame is then selected as the near-field head and neck image for subsequent local feature extraction.

[0146] This is understandable, because it originates from the phase interval with the smallest head movement amplitude. Therefore, at the moment of acquisition, the spatial displacement of the livestock's head is minimal, minimizing motion blur caused by target movement at the source. Simultaneously, because it possesses the largest sum of texture gradient amplitudes within the aforementioned interval, its visual details are sharpest, texture edges are most distinct, and it contains the richest discernible information.

[0147] S102: Determine M key pose frames based on the side body video stream, and determine M target unit masks based on the M key pose frames.

[0148] Understandably, from a continuous side-body video stream, M static image frames with the highest analytical value are selected. These selected key pose frames typically need to comprehensively and clearly show the typical body shape and posture changes of livestock during movement, thus forming a keyframe sequence that can characterize its dynamic behavior.

[0149] After obtaining each key pose frame, instance segmentation is performed on that frame image. Using computer vision techniques, the complete body pixels of the target livestock in the image are accurately separated from the background environment and other irrelevant foreground objects. The result of the segmentation is to generate a corresponding target unit mask for each frame, i.e., a binary image, in which all pixels belonging to the target livestock are labeled as foreground, and all other parts of the image are labeled as background.

[0150] For example, step S102 may include the following steps:

[0151] S1021: Extract the movement trajectory of the limbs of livestock based on the side body video stream.

[0152] Understandably, it is possible to obtain at least one representative and traceable joint on each leg, such as the hoof or knee joint, and the continuous positional changes of the joint on the video timeline to form a two-dimensional motion trajectory sequence.

[0153] Specifically, for each frame in the video sequence, a trained deep learning pose estimation model is used to detect and locate predefined, anatomically significant joints on the animal's limbs. The model continuously tracks the position of the same joint across all frames to ensure the continuity of the trajectory. Specific models have already been disclosed in existing technical documents and are not limited to in this application.

[0154] The two-dimensional pixel coordinates of each tracked joint point in each frame of the video are recorded in chronological order. For a specific joint point on each leg, its coordinates in all frames are arranged in chronological order, forming a motion trajectory sequence for that point. This sequence is essentially a temporally indexed array of positions. There are four trajectory sequences, each corresponding to the motion path of a joint point on a leg. Each trajectory sequence precisely describes the spatial movement of the corresponding limb part during filming, including its position, velocity, and direction information.

[0155] S1022: Based on the movement trajectory of the limbs, determine the M video frames in which all four limbs of the livestock are in contact with the ground as key attitude frames.

[0156] Understandably, by utilizing quantified limb movement trajectory data, a screening based on well-defined kinematic state criteria can be performed to automatically locate and select the most suitable static image for contour analysis from the lateral body video stream.

[0157] Specifically, by analyzing trajectory data, it's possible to automatically determine whether all four limbs are in contact with the ground in each frame of video. For example, it can be checked whether the vertical coordinates of all four leg tracking joints in the current frame are within their respective minimum ranges throughout the entire trajectory sequence. In a side view, the minimum Y-coordinate of a joint typically corresponds to the moment it is closest to the ground, i.e., in contact with or about to in contact with the ground.

[0158] To further ensure the stability of the posture, the vertical velocity of these joints may be checked at the same time, that is, whether the change in Y value between adjacent frames is close to zero, indicating that it is in a static support state rather than being rapidly lifted or dropped.

[0159] By scanning the entire video stream, all video frames that meet the quantitative criterion of all limbs being in contact with the ground are identified. Furthermore, from all frames that meet the criteria, frames can be selected at equal intervals to cover different motion cycles, or the first M frames that meet the criteria can be selected directly, or the M frames whose spatial distribution of joints best conforms to the standard standing posture model can be selected.

[0160] S1023: Input the key pose frames into the instance segmentation network for segmentation.

[0161] S1024: Using the segmentation results output by the instance segmentation network, generate an initial mask for the target livestock in the key pose frame.

[0162] Understandably, given the selected M key pose frames, a specific individual animal can be completely isolated from the image. Essentially, this involves using a trained deep learning model, specifically an instance segmentation network, to automatically identify and delineate the precise pixel boundaries of the target animal in the image.

[0163] For example, the network first performs depthwise convolution on the input image to extract multiple layers of feature maps. Based on these features, the network identifies all individual animals belonging to the livestock category in the image and determines the target animal.

[0164] After identifying the target instance, the network further assigns a foreground label to each pixel belonging to that instance and a background label to pixels not belonging to that instance, including the background, other livestock, and irrelevant objects. Generally, this is implemented through an encoder-decoder structure, where the encoder compresses and understands the image semantics, and the decoder progressively recovers spatial details to generate an accurate pixel-level mask.

[0165] For each input key pose frame, a binary segmentation mask with the same spatial resolution as the input image is output. In this mask, pixels within the outline of the target livestock are activated, while all pixels outside the outline are suppressed.

[0166] Understandably, the processing results of a single frame image by an instance segmentation network can be converted into binary image data that can be directly used for subsequent calculations and corrections.

[0167] Of course, in some implementations, the input to this step is the result of the instance segmentation network processing a single key pose frame. This result is usually not a direct binary image, but a probability heatmap, i.e., a single-channel matrix with the same resolution as the input image, where the value of each pixel represents the probability that the pixel belongs to the target livestock category.

[0168] For a probabilistic heatmap, all pixels are binarized by applying a predetermined confidence threshold. Pixels with probability values ​​greater than or equal to the threshold are identified as foreground and assigned a fixed high value; pixels with probability values ​​less than the threshold are identified as background and assigned a fixed low value. This operation generates a binary image.

[0169] S1025: Obtain the preset torso aspect ratio constraint, and based on the preset torso aspect ratio constraint, correct the initial mask to obtain the target single-unit mask.

[0170] Understandably, the preset torso length-to-width ratio constraint is an empirical range derived from statistical analysis of a large amount of body measurement data of livestock of the same species and age. It defines a reasonable range for the length-to-width ratio of the torso of this type of livestock, for example, 2.0:1 to 2.5:1.

[0171] Understandably, the aforementioned biostatistical constraints are applied to the geometric correction of the initial mask. Specifically, the current aspect ratio of the contour represented by the initial mask is first calculated. This can be achieved by calculating the ratio of the longer side to the shorter side of the minimum bounding rectangle of the mask. The calculated current aspect ratio is then compared with a reasonable range specified by the preset torso aspect ratio constraint.

[0172] If the current aspect ratio deviates significantly from a reasonable range—for example, if the mask is too short and wide or too long and thin due to segmentation errors—image processing operations are initiated for correction. For instance, if the mask is too wide, resulting in a low aspect ratio, a moderate erosion operation might be performed on both sides of the mask to narrow its width; if the mask is too thin, resulting in a high aspect ratio, a moderate dilation operation might be performed to increase its width. The goal of the correction is to bring the overall proportions of the mask into a preset reasonable range.

[0173] Understandably, the corrected outline proportions conform to the general biological morphological characteristics of this type of livestock, correcting the severe shape distortion caused by occlusion, lighting, or limitations of the segmentation algorithm.

[0174] S103: Input the sequence of M target single-unit masks into a pre-trained three-dimensional convolutional neural network, and determine the dynamic contour feature vector that changes with motion based on the output of the three-dimensional convolutional neural network.

[0175] Understandably, by jointly learning the spatiotemporal features of M temporally consecutive target individual mask sequences, dynamic contour features that can characterize the unique movement patterns of individual livestock can be extracted. In this application, the architectural advantages of a three-dimensional convolutional neural network (3DCNN) are utilized to automatically learn and abstract identity-related dynamic information from the contour sequences.

[0176] For example, step S103 may include the following steps:

[0177] S1031: Stack M target unit masks in chronological order to obtain three-dimensional mask sequence volume data.

[0178] Understandably, these M two-dimensional mask images are stacked along a new dimension—the time dimension, or the depth dimension. This forms a data block with three dimensions. For example,

[0179] Dimension 1: Image height (H, spatial vertical axis)

[0180] Dimension 2: Image width (W, horizontal axis of space)

[0181] Dimension 3: Time series length (M, time axis or frame sequence axis)

[0182] This 3D data block is the 3D mask sequence volume data. It can be intuitively understood as a binary contour animation composed of M consecutive animal silhouettes.

[0183] S1032: Input the volume data of the three-dimensional mask sequence into a pre-trained three-dimensional convolutional neural network to extract spatiotemporal features and obtain a three-dimensional feature tensor.

[0184] To extract high-order spatiotemporal features that characterize individual motion patterns, a pre-trained 3D CNN can be used as a feature extractor, for example. Specifically, the network uses 3D convolutional kernels that slide across the input volume data. Each convolutional kernel is computed simultaneously within a local spatial region and a local temporal window. This allows the network to directly perceive the changing patterns of specific local regions of the contour over short periods of time, such as how the leg contour transitions from a support phase to a swing phase.

[0185] The initial convolutional layers output a set of basic feature maps that respond to simple spatiotemporal edges or changes. As the network deepens, subsequent convolutional layers further combine and abstract these feature maps to form higher-level features that respond to more complex spatiotemporal patterns, such as complete gait cycles and the coordinated relationship between trunk undulations and limb swings.

[0186] By using a three-dimensional pooling layer to perform moderate downsampling in both spatial and temporal dimensions, the data size is gradually compressed while preserving salient features.

[0187] S1033: Obtain the feature vector corresponding to each time step of the three-dimensional feature tensor in the time dimension.

[0188] Understandably, the spatiotemporal features extracted by the three-dimensional convolutional neural network are structurally transformed from a multidimensional tensor form with implicit spatiotemporal relationships into a series of vector sequences aligned on the time axis, which are convenient for subsequent time series analysis.

[0189] For example, the input is a three-dimensional feature tensor. This tensor typically has a shape of (C, T, H, W), where:

[0190] C represents the number of feature channels, with each channel corresponding to a learned spatiotemporal pattern filter.

[0191] T represents the number of time steps.

[0192] H and W represent the spatial height and width of each feature map.

[0193] The operation proceeds along the time dimension T, with the goal of generating a fixed-length feature vector for each time step t (t=1,2,…,T). For the t-th time step, a corresponding 3D slice of shape (C,H,W) is extracted from the 3D feature tensor. This can be viewed as a stack of all feature maps corresponding to that time step.

[0194] To transform the C HxW feature maps at this time step into a single vector, the spatial information of each feature map needs to be aggregated. The most common method is global average pooling, which calculates the average value of all pixels in each HxW feature map. In this way, each channel (C in total) obtains a scalar value.

[0195] The C scalar values ​​calculated for each of the C channels are concatenated in channel order to form a C-dimensional column vector or row vector. This C-dimensional vector is the feature vector corresponding to the t-th time step. It summarizes the spatial activation responses of all feature channels of the network at that specific time.

[0196] Repeat the above process T times, and finally output a sequence [V1, V2, ..., VT] containing T feature vectors, where each Vt is a C-dimensional vector. This sequence strictly maintains the temporal order of the original input.

[0197] S1034: Input the feature vector of each time step into a fully connected network and output a scalar weight representing the importance of the time step.

[0198] Understandably, it is necessary to assess the importance of time-series feature vector sequences.

[0199] For example, consider the sequence [V1,V2,...,VT], where each Vt is a C-dimensional vector representing all the spatiotemporal features contained at the t-th time step.

[0200] A fully connected network is used as the scoring function. It typically has one or more hidden layers, each ending with a neuron of output dimension 1. Its function is to map the high-dimensional input feature vector Vt to a single scalar value wt through a series of nonlinear transformations.

[0201] Understandably, during training, this fully connected network is optimized along with the backbone 3DCNN. The optimization goal drives the network to learn which time-step feature patterns are more discriminative for the task of correctly identifying livestock. Therefore, the weight parameters learned by the network essentially encode the relative importance of different poses or motion phases for identification.

[0202] For each of the T input feature vectors, the fully connected network processes them sequentially and outputs T scalar values, forming a scalar weight sequence [w1, w2, ..., wT]. Here, wt represents the scalar weight indicating the importance of the t-th time step.

[0203] S1035: Normalize the scalar weights corresponding to all time steps to obtain a normalized weight sequence.

[0204] S1036: Use a normalized weight sequence to perform a weighted summation of the three-dimensional feature tensor in the time dimension, and obtain the two-dimensional spatial feature map based on the result of the weighted summation.

[0205] Understandably, the slice of the 3D feature tensor at time step t is multiplied by the corresponding normalized weight. This is equivalent to performing a global scaling on the feature map at each time step, with the scaling factor being its importance weight.

[0206] All T weighted time-step slices are summed along the time dimension (T). This summation eliminates the time dimension T, outputting a three-dimensional tensor of shape (C, H, W). Since C is the feature channel dimension and (H, W) is the spatial dimension, this output can be structurally viewed as a stack of C two-dimensional spatial feature maps. Understandably, the output two-dimensional spatial feature maps are static feature representations refined by the temporal attention mechanism. Each feature map is no longer a snapshot of a single moment, but rather a weighted average summary of the entire temporal segment across its corresponding feature channels. Through weighting, the contributions of time steps with weak discriminative power or poor quality are reduced, while the information of key motion phases with strong discriminative power is enhanced. This is equivalent to feature purification along the time dimension.

[0207] S1037: Perform global average pooling and global max pooling on the two-dimensional spatial feature map, and concatenate the pooling results to form a dynamic contour feature vector.

[0208] Specifically, the input is a stack of two-dimensional spatial feature maps, with a shape of (C, H, W). Here, C is the number of feature channels, and each HxW feature map encodes the temporally weighted spatial pattern learned by the network on the corresponding channel.

[0209] For each feature map, calculate the arithmetic mean of all H x W pixel values. This operation yields C scalar values, forming a C-dimensional vector. This vector represents the overall average intensity of the response on each feature channel, reflecting the global distribution and prevalence of the feature, and exhibits a certain degree of robustness to noise.

[0210] For each feature map, the maximum value among all H x W pixel values ​​is extracted. This operation also yields C scalar values, forming another C-dimensional vector. This vector captures the strongest local activations in each feature channel, preserving the most salient and discriminative feature patterns, and is sensitive to unique local features.

[0211] The C-dimensional vector obtained by global average pooling and the C-dimensional vector obtained by global max pooling are concatenated in terms of dimension to form a 2C-dimensional composite feature vector. The resulting 2C-dimensional vector is the dynamic contour feature vector.

[0212] Understandably, global average pooling and global max pooling complement each other. Global average pooling focuses on the overall picture, while global max pooling highlights the most unique features. The concatenated vector simultaneously encodes both the general statistics and the unique peaks of the features, forming a more comprehensive description of the features.

[0213] S104: Input the near-field image of the head and neck into a pre-trained dual-branch feature extraction network. The first branch of the dual-branch feature extraction network is an encoder network for extracting deep semantic features. The second branch of the dual-branch feature extraction network adopts a directional gradient filter group and is used to obtain the micro-spatial texture information composed of skin wrinkles, hair direction or spot texture, and output the corresponding texture features.

[0214] Understandably, in order to perform refined feature analysis on the selected head and neck near-field images, a specially designed dual-branch neural network can be used to collaboratively extract local biological features that constitute the key identifiers of livestock individuals from different levels of abstraction and physical characteristics.

[0215] Specifically, its two branches process the same input image in parallel using different technical principles:

[0216] The first branch is typically a deep convolutional neural network, such as a variant of ResNet or VGG, whose structure is called the encoder. Through multiple layers of convolution, non-linear activation, and pooling operations, it performs a progressively deeper abstract understanding of the input image.

[0217] Understandably, this involves learning and extracting high-level, identity-related semantic features from pixel-level information, such as the overall facial structure and the relative positions and shapes of key organs (eyes, nose, mouth), and other macroscopic layout information. The output is a set of highly abstract feature maps or feature vectors, known as deep semantic features.

[0218] The second branch employs a set of gradient filters with different directions, such as multi-directional Gabor filter banks or directional Sobel filters. These filters are designed to be sensitive to edge and texture variations in specific directions.

[0219] Understandably, filter banks perform convolution operations on the input image, specifically to enhance, separate, and encode microscopic spatial texture information such as the direction of skin folds, the pattern of hair arrangement, and the fine boundaries of spots or discolored regions. Their output is a set of texture feature maps that directly reflect the intensity and directional distribution of the microstructure in the local area, offering high resolution and preserving rich spatial details.

[0220] S105: Fusion of deep semantic features with texture feature maps to generate local biological feature vectors.

[0221] Understandably, deep semantic features are typically a low spatial resolution, high channel count 3D tensor or a high-dimensional vector. They encode high-level abstract information about the image, such as its overall structural layout. Texture feature maps, on the other hand, are usually a set of feature maps with high spatial resolution and relatively low channel count. They encode the local microscopic details and directional texture patterns of the image. A well-designed fusion mechanism can be used to fuse and refine features representing different levels and characteristics into a unified and more discriminative feature representation.

[0222] Because the two features often differ in spatial size and / or data format, alignment operations are required. For example, deep semantic features can be upsampled to the same spatial size as the texture feature map, such as through transposed convolution or bilinear interpolation.

[0223] The two aligned features are concatenated along the channel dimension to form a composite feature tensor that integrates macroscopic semantics and microscopic texture. This concatenated composite feature tensor is then input into a lightweight feature refining module.

[0224] Understandably, the output local biometric vector is a comprehensive descriptor that integrates multi-scale information, containing both high-level identity semantic cues and subtle, unique biometric texture patterns. Furthermore, through a learnable fusion and refinement process, the information in the vector is optimized to maximize its ability to distinguish individual identities.

[0225] S106: Obtain the first confidence level corresponding to the dynamic contour feature vector and the second confidence level corresponding to the local biological feature vector.

[0226] Understandably, by calculating two specially designed quantitative indicators, the temporal stability of dynamic features and the inter-class discriminative power of static features can be evaluated respectively.

[0227] For example, step S106 may include the following steps:

[0228] S1061: Obtain the cosine similarity sequence of dynamic contour feature vectors within multiple consecutive time windows.

[0229] Understandably, the original process corresponding to the complete dynamic contour feature vector is divided into multiple temporally continuous or overlapping segments. For each segmented time window, its corresponding feature sub-vector is calculated, and then the cosine similarity between the feature sub-vectors corresponding to adjacent or nearby windows is calculated sequentially. This generates a sequence of cosine similarity values, which intuitively reflects whether the evolution of the feature over time is stable and coherent or fluctuates drastically.

[0230] S1062: Based on the standard deviation between each cosine similarity in the cosine similarity sequence, determine the first confidence level representing the temporal stability of the feature, where the smaller the standard deviation, the higher the first confidence level.

[0231] Specifically, statistical analysis is performed on the aforementioned cosine similarity sequence to generate a scalarized stability index. The standard deviation of all values ​​in the sequence is calculated to measure the dispersion of these similarity values ​​around their mean. A smaller standard deviation indicates that the similarity values ​​in the sequence are closer to each other, meaning that the feature vectors across different time windows maintain high similarity, and the dynamic features exhibit excellent temporal consistency and repeatability, thus earning a higher first confidence level. Conversely, a larger standard deviation implies greater feature fluctuation and instability, resulting in a correspondingly lower confidence level.

[0232] S1063: Obtain K registration feature vectors that are most similar to the local biological feature vectors from the livestock feature database, and randomly obtain N non-matching registration feature vectors.

[0233] A search is performed in a pre-generated livestock feature database to identify the K feature vectors with the highest cosine similarity to the local biometric feature vector to be tested. These vectors are considered to be most likely from the same individual and form a similarity set. Simultaneously, N registered feature vectors that clearly do not belong to the current target livestock are randomly selected from the database, forming a difference set.

[0234] S1064: Obtain the average distance from the local biometric feature vector to the K registered feature vectors as the intra-class distance, and the average distance from the local biometric feature vector to the N non-matching registered feature vectors as the inter-class distance.

[0235] Understandably, this is done to quantify the clustering and separation properties of local features. Specifically, we can calculate the average distance between the feature vector to be tested and K vectors in the similarity set, which is the intra-class distance, reflecting how closely the feature is related to samples of the same class. Next, we calculate the average distance between it and N vectors in the "difference set" to obtain the inter-class distance, reflecting how well the feature distinguishes itself from samples of different classes.

[0236] S1065: Based on the ratio of intra-class distance to inter-class distance, determine the second confidence level used to characterize the discriminative power of the feature, wherein the smaller the ratio, the higher the second confidence level.

[0237] Specifically, the calculation formula is: Distance Ratio = Intra-class Distance / Inter-class Distance. Understandably, the smaller the ratio, the closer the feature vector to the center of its class in the feature space, and the farther away it is from other classes. This indicates that the feature has a very strong ability to distinguish individuals, and therefore is assigned a higher second confidence level. Conversely, the larger the ratio, the weaker the discriminative power and the lower the confidence level.

[0238] S107: Based on the first confidence level and the second confidence level, the local biometric feature vector and the dynamic contour feature vector are fused to determine the target feature.

[0239] Specifically, two confidence scores are typically input into a normalization function (such as Softmax) to generate two fusion weights that sum to 1. These two weights are then used to perform weighted summation or weighted concatenation on the dynamic contour feature vector and the local biometric feature vector, respectively, ultimately generating a comprehensive feature vector of the target feature. When the data quality of a certain modality deteriorates, its low confidence score automatically reduces its weight in the final decision, thus enabling reliance on more reliable modalities acquired in this instance. This improves the accuracy of the entire recognition system in complex real-world scenarios.

[0240] S108: Perform similarity matching between the target features and the livestock feature database, and complete the identity verification based on the matching results.

[0241] Understandably, the similarity between the target feature and each registered livestock feature template in the database, such as cosine similarity, is calculated, and all candidate identities are ranked according to the score. If the highest similarity score exceeds a preset acceptance threshold, the identity is considered a successful match; if the score is below a preset rejection threshold, the verification is considered a failure.

[0242] This invention discloses a livestock identification method based on multimodal image analysis. This method achieves contactless identification by collaboratively analyzing the macroscopic dynamic contours and microscopic local biometric features of livestock. Specifically, it includes: simultaneously acquiring a side body video stream and near-field images of the head and neck; extracting key pose frames from the video stream and generating a morphology-corrected target individual mask sequence, then extracting dynamic contour feature vectors using a 3D convolutional neural network; simultaneously processing the head and neck images through a dual-branch feature extraction network, which extracts deep semantic features and microscopic texture features enhanced by directional gradient filtering in parallel, and fuses them to generate a local biometric feature vector; finally, evaluating the confidence levels of the two types of features separately, and performing adaptive weighted fusion accordingly to generate target features for matching with a database. This invention effectively overcomes the limitations of single-feature recognition in complex scenarios, significantly improving the accuracy, robustness, and practicality of identification verification.

[0243] Based on the same inventive concept, this application proposes a livestock identification system based on multimodal image analysis, the system being configured as follows:

[0244] Acquire lateral body video streams and head and neck near-field images of the target livestock. The lateral body video streams are used to determine dynamic contour features, and the head and neck near-field images are used to determine local biological features.

[0245] M key pose frames are determined based on the side body video stream, and M target unit masks are determined based on the M key pose frames;

[0246] M target single-unit mask sequences are input into a pre-trained 3D convolutional neural network. Based on the output of the 3D convolutional neural network, the dynamic contour feature vector that changes with motion is determined.

[0247] The near-field image of the head and neck is input into a pre-trained dual-branch feature extraction network. The first branch of the dual-branch feature extraction network is an encoder network for extracting deep semantic features. The second branch of the dual-branch feature extraction network adopts a directional gradient filter group and is used to obtain the micro-spatial texture information composed of skin wrinkles, hair direction or spot texture, and output the corresponding texture features.

[0248] By fusing deep semantic features with texture feature maps, local biological feature vectors are generated.

[0249] Obtain the first confidence level corresponding to the dynamic contour feature vector and the second confidence level corresponding to the local biological feature vector;

[0250] Based on the first and second confidence levels, the local biological feature vector and the dynamic contour feature vector are fused to determine the target features;

[0251] The target features are matched with a livestock feature database for similarity, and the identity is verified based on the matching results.

[0252] Optionally, the system is configured as follows:

[0253] Acquire lateral body video streams and near-field images of the head and neck of the target livestock. The lateral body video stream is used to determine dynamic contour features, and the near-field images of the head and neck are used to determine local biological features, including:

[0254] Obtain temporal information of motion containing gait periodicity based on lateral body video stream;

[0255] Based on motion temporal information, at least one frame is selected from the head and neck near-field image sequence as the head and neck near-field image.

[0256] Optionally, the system is configured as follows:

[0257] Based on motion temporal information, at least one frame is selected from the head and neck near-field image sequence as the head and neck near-field image, including:

[0258] Obtain the sum of texture gradient magnitudes for each image frame in the head and neck near-field image sequence;

[0259] Based on motion timing information, the phase interval in which the target animal is in a stable posture and has the smallest head sway is determined;

[0260] From all image frames corresponding to the phase interval, the image frame with the largest sum of texture gradient magnitudes is determined as the near-field image of the head and neck.

[0261] Optionally, the system is configured as follows:

[0262] Based on the lateral body video stream, temporal information of motion containing gait periodicity is obtained, including:

[0263] Joint key point detection is performed on the lateral body video stream to obtain the head movement trajectory sequence and at least one leg joint movement trajectory sequence of the target animal;

[0264] Determine the basic gait frequency of the target livestock based on the sequence of leg joint motion trajectories;

[0265] Based on the basic gait frequency, peaks and troughs that conform to periodic patterns in the motion trajectory sequence are located in the time domain, and the time interval between adjacent peaks is defined as a complete gait cycle.

[0266] The average displacement amplitude of the head motion trajectory sequence within each gait cycle is obtained as a quantitative indicator to characterize the degree of head swaying.

[0267] Integrating basic gait frequency, quantitative indicators, and the corresponding phase of the gait cycle constitutes motion timing information.

[0268] Optionally, the system is configured as follows:

[0269] Based on the side view video stream, M key pose frames are determined, and based on the M key pose frames, M target unit masks are determined, including:

[0270] The movement trajectories of the animal's limbs are extracted based on the side body video stream.

[0271] Based on the movement trajectory of the limbs, M video frames in which all four limbs of the livestock are in contact with the ground are identified as key pose frames.

[0272] The key pose frames are input into the instance segmentation network for segmentation.

[0273] The initial mask for the target livestock in the key pose frame is generated using the segmentation results output by the instance segmentation network.

[0274] Obtain the preset torso aspect ratio constraint, and based on the preset torso aspect ratio constraint, correct the initial mask to obtain the target single-unit mask.

[0275] Optionally, the system is configured as follows:

[0276] M target unit mask sequences are input into a pre-trained 3D convolutional neural network. Based on the output of the 3D convolutional neural network, the dynamic contour feature vector that changes with motion is determined, including:

[0277] Stack M target unit masks in chronological order to obtain three-dimensional mask sequence volume data;

[0278] The volume data of the three-dimensional mask sequence is input into a pre-trained three-dimensional convolutional neural network for spatiotemporal feature extraction, resulting in a three-dimensional feature tensor.

[0279] Obtain the feature vector of the three-dimensional feature tensor at each time step in the time dimension;

[0280] The feature vector at each time step is input into a fully connected network, and a scalar weight representing the importance of the time step is output.

[0281] Normalize the scalar weights corresponding to all time steps to obtain a normalized weight sequence;

[0282] The three-dimensional feature tensor is weighted and summed in the time dimension using a normalized weight sequence, and the two-dimensional spatial feature map is obtained based on the weighted summation result.

[0283] Global average pooling and global max pooling are performed on the two-dimensional spatial feature map, and the pooling results are concatenated to form a dynamic contour feature vector.

[0284] Optionally, the system is configured as follows:

[0285] Obtain the first confidence score corresponding to the dynamic contour feature vector and the second confidence score corresponding to the local biometric feature vector, including:

[0286] Obtain the cosine similarity sequence of dynamic contour feature vectors within multiple consecutive time windows;

[0287] Based on the standard deviation between each cosine similarity in the cosine similarity sequence, the first confidence level representing the temporal stability of the feature is determined, where the smaller the standard deviation, the higher the first confidence level.

[0288] K registration feature vectors that are most similar to the local biological feature vectors are obtained from the livestock feature database, and N non-matching registration feature vectors are randomly obtained.

[0289] The average distance from the local biometric feature vector to the K registered feature vectors is taken as the intra-class distance, and the average distance from the local biometric feature vector to the N non-matching registered feature vectors is taken as the inter-class distance.

[0290] A second confidence level is determined based on the ratio of intra-class distance to inter-class distance, where a smaller ratio indicates a higher second confidence level.

[0291] Based on the same inventive concept, embodiments of this application also propose an electronic device, which includes:

[0292] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the livestock identification method based on multimodal image analysis according to the embodiments of this application.

[0293] Furthermore, to achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the livestock identification method based on multimodal image analysis according to embodiments of this application.

[0294] The following is a detailed introduction to the various components of the electronic device:

[0295] In this context, the processor is the control center of the electronic device. It can be a single processor or a collective term for multiple processing elements. For example, a processor can be one or more central processing units (CPUs), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0296] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0297] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can be referred to the above method embodiment, which will not be repeated here.

[0298] Optionally, the memory can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory can be integrated with the processor or exist independently and coupled to the processor through an interface circuit of an electronic device; the embodiments of the present invention do not specifically limit this.

[0299] A transceiver is used to communicate with network devices or with terminal devices.

[0300] Optionally, the transceiver may include a receiver and a transmitter. The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0301] Optionally, the transceiver can be integrated with the processor or exist independently and coupled to the processor through the router's interface circuit. This embodiment of the invention does not specifically limit this.

[0302] Furthermore, the technical effects of the electronic device can be referred to the technical effects of the data transmission method in the above method embodiments, and will not be repeated here.

[0303] It should be understood that the processor in the embodiments of the present invention can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0304] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0305] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0306] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0307] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0308] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0309] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

Claims

1. A livestock identification method based on multimodal image analysis, characterized in that, include: Acquire a side body video stream and a head and neck near-field image of the target livestock, wherein the side body video stream is used to determine dynamic contour features, and the head and neck near-field image is used to determine local biological features; Based on the side body video stream, M key pose frames are determined, and based on the M key pose frames, M target unit masks are determined. The M target single-unit mask sequences are input into a pre-trained three-dimensional convolutional neural network, and the dynamic contour feature vector that changes with motion is determined based on the output of the three-dimensional convolutional neural network. The near-field image of the head and neck is input into a pre-trained dual-branch feature extraction network. The first branch of the dual-branch feature extraction network is an encoder network for extracting deep semantic features. The second branch of the dual-branch feature extraction network adopts a directional gradient filter group and is used to obtain micro-spatial texture information composed of skin wrinkles, hair direction or spot texture, and output the corresponding texture features. The deep semantic features are fused with the texture feature map to generate a local biological feature vector; Obtain the first confidence level corresponding to the dynamic contour feature vector and the second confidence level corresponding to the local biological feature vector; Based on the first confidence level and the second confidence level, the local biometric feature vector and the dynamic contour feature vector are fused to determine the target feature; The target features are matched with a livestock feature database for similarity, and identity verification is performed based on the matching results.

2. The livestock identification method based on multimodal image analysis according to claim 1, characterized in that, Acquire a lateral body video stream and a head and neck near-field image of the target livestock, wherein the lateral body video stream is used to determine dynamic contour features, and the head and neck near-field image is used to determine local biological features, including: Based on the side body video stream, obtain motion timing information including gait periodicity; Based on the motion timing information, at least one frame is selected from the head and neck near-field image sequence as the head and neck near-field image.

3. The livestock identification method based on multimodal image analysis according to claim 2, characterized in that, Based on the motion timing information, at least one frame is selected from the head and neck near-field image sequence as the head and neck near-field image, including: Obtain the sum of texture gradient magnitudes for each image frame in the head and neck near-field image sequence; Based on the motion timing information, it is determined that the target animal is in a phase interval where its posture is stable and its head sway is minimal. From all the image frames corresponding to the phase interval, the image frame with the largest sum of texture gradient magnitudes is determined as the head and neck near-field image.

4. The livestock identification method based on multimodal image analysis according to claim 2, characterized in that, Based on the lateral body video stream, obtain motion timing information containing gait periodicity, including: Joint key point detection is performed on the side body video stream to obtain the head movement trajectory sequence and at least one leg joint movement trajectory sequence of the target animal; The basic gait frequency of the target livestock is determined based on the sequence of movement trajectories of the leg joints; Based on the basic gait frequency, the peaks and troughs that conform to the periodic pattern in the motion trajectory sequence are located in the time domain, and the time period between adjacent peaks is defined as a complete gait cycle. The average displacement amplitude of the head movement trajectory sequence within each gait cycle is obtained as a quantitative indicator characterizing the degree of head swaying. The motion timing information is formed by integrating the basic gait frequency, the quantization index, and the corresponding phase of the gait cycle.

5. The livestock identification method based on multimodal image analysis according to claim 3, characterized in that, Based on the side-body video stream, M key pose frames are determined, and based on the M key pose frames, M target unit masks are determined, including: Based on the side body video stream, the movement trajectories of the animal's limbs are extracted; Based on the limb movement trajectory, M video frames in which all four limbs of the livestock are in contact with the ground are determined as the key posture frames. The key pose frames are input into an instance segmentation network for segmentation. Using the segmentation results output by the instance segmentation network, an initial mask is generated for the target livestock in the key pose frame; Obtain a preset torso aspect ratio constraint, and based on the preset torso aspect ratio constraint, correct the initial mask to obtain the target single-unit mask.

6. The livestock identification method based on multimodal image analysis according to claim 3, characterized in that, The M target single-unit mask sequences are input into a pre-trained 3D convolutional neural network. Based on the output of the 3D convolutional neural network, a dynamic contour feature vector that changes with motion is determined, including: The M target unit masks are stacked in chronological order to obtain three-dimensional mask sequence volume data; The three-dimensional mask sequence volume data is input into the pre-trained three-dimensional convolutional neural network for spatiotemporal feature extraction to obtain a three-dimensional feature tensor; Obtain the feature vector of the three-dimensional feature tensor at each time step in the time dimension; The feature vector of each time step is input into a fully connected network, and a scalar weight representing the importance of the time step is output. The scalar weights corresponding to all the time steps are normalized to obtain a normalized weight sequence; The normalized weight sequence is used to perform a weighted summation on the three-dimensional feature tensor in the time dimension, and a two-dimensional spatial feature map is obtained based on the weighted summation result. The two-dimensional spatial feature map is subjected to global average pooling and global max pooling, and the pooling results are concatenated to form the dynamic contour feature vector.

7. The livestock identification method based on multimodal image analysis according to claim 1, characterized in that, Obtaining the first confidence level corresponding to the dynamic contour feature vector and the second confidence level corresponding to the local biometric feature vector includes: Obtain the cosine similarity sequence of the dynamic contour feature vector within multiple consecutive time windows; Based on the standard deviation between each cosine similarity in the cosine similarity sequence, a first confidence level representing the temporal stability of the feature is determined, wherein the smaller the standard deviation, the higher the first confidence level. K registration feature vectors that are most similar to the local biological feature vectors are obtained from the livestock feature database, and N non-matching registration feature vectors are randomly obtained. The average distance from the local biometric vector to the K registered feature vectors is obtained as the intra-class distance, and the average distance from the local biometric vector to the N non-matching registered feature vectors is obtained as the inter-class distance; A second confidence level is determined based on the ratio of the intra-class distance to the inter-class distance, wherein the smaller the ratio, the higher the second confidence level.

8. A livestock identification system based on multimodal image analysis, characterized in that, The system is configured as follows: Acquire a side body video stream and a head and neck near-field image of the target livestock, wherein the side body video stream is used to determine dynamic contour features, and the head and neck near-field image is used to determine local biological features; Based on the side body video stream, M key pose frames are determined, and based on the M key pose frames, M target unit masks are determined. The M target single-unit mask sequences are input into a pre-trained three-dimensional convolutional neural network, and the dynamic contour feature vector that changes with motion is determined based on the output of the three-dimensional convolutional neural network. The near-field image of the head and neck is input into a pre-trained dual-branch feature extraction network. The first branch of the dual-branch feature extraction network is an encoder network for extracting deep semantic features. The second branch of the dual-branch feature extraction network adopts a directional gradient filter group and is used to obtain micro-spatial texture information composed of skin wrinkles, hair direction or spot texture, and output the corresponding texture features. The deep semantic features are fused with the texture feature map to generate a local biological feature vector; Obtain the first confidence level corresponding to the dynamic contour feature vector and the second confidence level corresponding to the local biological feature vector; Based on the first confidence level and the second confidence level, the local biometric feature vector and the dynamic contour feature vector are fused to determine the target feature; The target features are matched with a livestock feature database for similarity, and identity verification is performed based on the matching results.

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by at least one of the processors, which are executed by at least one of the processors to enable the at least one of the processors to perform a livestock identification method based on multimodal image analysis as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements a livestock identification method based on multimodal image analysis as described in any one of claims 1-7.

Images