Multi-object tracking methods, devices, and equipment based on improved TrackR-CNN

Abstract

The application discloses a multi-target tracking method and device based on an improved Track R-CNN, and relates to the technical field of deep learning and computer vision. The method comprises the following steps: acquiring image data to be detected, wherein the image data contains target pigs; processing the image data through an improved Track R-CNN model to obtain the identity of the target pigs, so as to identify and track the target pigs; the improved Track R-CNN model comprises a backbone network, an RPN branch, an RoI Align module, a mask prediction branch and an appearance vector calculation branch; wherein the output of the backbone network is connected with the input of the RPN branch, the output of the RPN branch is connected with the input of the RoI Align module, and the output of the RoI Align module is connected with the input of the mask prediction branch and the appearance vector calculation branch respectively; the mask prediction branch is composed of multiple layers of deconvolution layers and multiple layers of hollow convolution layers which are connected in an alternating manner; and the SE channel attention module is arranged between the RoI Align module and the full connection layer in the appearance vector calculation branch.

Description

Technical Field

[0001] This application relates to the fields of deep learning and computer vision technology, and more specifically, to a multi-object tracking method, apparatus, and device based on an improved Track R-CNN. Background Technology

[0002] As livestock and poultry farming develops towards large-scale and intensive operations, accurate identification and continuous tracking of individual livestock have become key requirements for intelligent livestock management. Traditional manual inspection methods are inefficient, while electronic ear tags (RFID) suffer from problems such as stress response, high cost, and discontinuous data collection.

[0003] Deep learning-based computer vision technology can achieve identification of unlabeled animals. Track R-CNN is a general-purpose model that integrates detection, instance segmentation, and multi-object tracking. However, its direct application in pigpen environments has many shortcomings: insufficient mask segmentation accuracy, making it difficult to meet the needs of precise body size measurement in farming scenarios; poor individual tracking stability, when multiple pigs have similar appearances, their appearance vector distances are close, making it easy for identity swapping and tracking errors to occur; and insufficient adaptability to farming scenarios, with poor robustness to complex environments such as pen occlusion, uneven lighting, and pig stacking, making it difficult to meet the actual needs of precision farming management. Summary of the Invention

[0004] The main objective of this application is to provide a multi-target tracking method, apparatus, and device based on an improved Track R-CNN to address the problems of low mask segmentation accuracy, poor individual tracking stability, and insufficient adaptability to complex environments in existing Track R-CNN systems for aquaculture scenarios.

[0005] To achieve the above objectives, the first aspect of this application proposes a multi-object tracking method based on an improved Track R-CNN, comprising: Acquire image data to be detected, wherein the image data contains the target pig; The image data is processed by an improved Track R-CNN model to obtain the identification of the target pig, so as to identify and track the target pig. The improved Track R-CNN model includes a backbone network, an RPN branch, a RoI Align module, a mask prediction branch, and an appearance vector calculation branch. The backbone network output is connected to the input of the RPN branch, the RPN branch output is connected to the input of the RoI Align module, and the RoI Align module output is connected to the inputs of the mask prediction branch and the appearance vector calculation branch, respectively. The mask prediction branch is composed of multiple deconvolutional layers and multiple dilated convolutional layers connected in series. The appearance vector calculation branch sets an SE channel attention module between the RoI Align module and the fully connected layer.

[0006] Secondly, a multi-object tracking device based on an improved Track R-CNN is proposed, including: The acquisition module is used to acquire image data to be detected, wherein the image data contains the target pig; The processing module is used to process the image data through an improved Track R-CNN model to obtain the identity identifier of the target pig, so as to identify and track the target pig. The improved Track R-CNN model includes a backbone network, an RPN branch, a RoI Align module, a mask prediction branch, and an appearance vector calculation branch. The backbone network output is connected to the input of the RPN branch, the RPN branch output is connected to the input of the RoI Align module, and the RoI Align module output is connected to the inputs of the mask prediction branch and the appearance vector calculation branch, respectively. The mask prediction branch is composed of multiple deconvolutional layers and multiple dilated convolutional layers connected in series. The appearance vector calculation branch sets an SE channel attention module between the RoI Align module and the fully connected layer.

[0007] Thirdly, an electronic device is proposed, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0008] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the embodiments of this disclosure.

[0009] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of the present disclosure.

[0010] The technical solutions provided by the embodiments of this application may include the following beneficial effects: In this application, a mask prediction branch consisting of alternating deconvolutional layers and dilated convolutional layers is constructed. This restores the spatial resolution of the feature map while expanding the receptive field, effectively improving the edge detail and overall accuracy of the pig segmentation mask, thus meeting the needs for precise measurement of pig size in farming scenarios. Simultaneously, by adding an SE channel attention module between RoIAlign and the fully connected layer, different feature channels are recalibrated, strengthening individual-discriminating feature channels and suppressing interference from irrelevant channels. This effectively increases the feature vector distance between individuals with similar appearances, significantly reducing the probability of identity swapping and tracking errors, and improving the stability and robustness of multi-target tracking. These two aspects work together to enable the improved Track R-CNN model to achieve accurate identification and stable tracking of individual pigs in complex farming environments. Attached Figure Description

[0011] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 A flowchart illustrating a multi-object tracking method based on an improved Track R-CNN provided for this application; Figure 2 A schematic diagram of the structure of the improved Track R-CNN model provided in this application; Figure 3 This is a schematic diagram of the SE channel attention module provided in this application; Figure 4 A schematic diagram of a multi-target tracking device based on an improved Track R-CNN provided in this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0013] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0015] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0016] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linked," and "socketing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0017] Figure 1 This application provides a flowchart illustrating a multi-object tracking method based on an improved Track R-CNN, which may include: S110. Obtain the image data to be detected, wherein the image data contains the target pig.

[0018] In this embodiment of the application, image data is obtained by taking pictures of the breeding scene through an image acquisition device. The image data may contain one or more individual pigs that need to be identified, segmented and tracked.

[0019] Image acquisition equipment can be RGB-D depth cameras deployed above the pigsty to simultaneously acquire color and depth images of the pigs. The image acquisition equipment can continuously acquire images of the breeding scene at a preset acquisition frequency, such as acquiring several frames per second, to achieve continuous tracking and long-term monitoring of individual pigs.

[0020] In practical applications, after acquiring the image data to be detected, the image data can be preprocessed, such as image denoising, size normalization, and contrast enhancement, to improve the accuracy and robustness of the model in processing image data.

[0021] S120. The image data is processed by the improved Track R-CNN model to obtain the identity of the target pig, so as to identify and track the target pig. The improved Track R-CNN model includes a backbone network, RPN branch, RoI Align module, mask prediction branch and appearance vector calculation branch. The backbone network output is connected to the input of the RPN branch, the RPN branch output is connected to the input of the RoI Align module, and the RoI Align module output is connected to the inputs of the mask prediction branch and the appearance vector calculation branch, respectively. The mask prediction branch is composed of multiple deconvolutional layers and multiple dilated convolutional layers connected in series. The appearance vector calculation branch sets an SE channel attention module between the RoI Align module and the fully connected layer.

[0022] In this embodiment, an improved Track R-CNN model is used to perform end-to-end feature extraction, target detection, instance segmentation, and identity association calculations on the acquired pig images. It simultaneously outputs a unique identifier for each target pig in the image and a high-precision instance segmentation mask, enabling contactless, ear-tag-free pig identification and continuous tracking. The improved Track R-CNN model may include a backbone network, an RPN branch, a RoI Align module, a mask prediction branch, and an appearance vector calculation branch.

[0023] The backbone network serves as the feature extraction base for the model. Its output is connected to the input of the RPN branch, and it is used to perform multi-layer convolution operations on the input pig image to extract multi-scale feature maps containing texture, contour, and semantic information.

[0024] The RPN branch receives a multi-scale feature map output from the backbone network at its input and connects to the input of the RoI Align module at its output. This allows it to slide across the feature map output by the backbone network to generate candidate regions (RoIs), which are candidate bounding boxes that may contain the target pigs. This branch performs foreground / background binary classification and preliminary bounding box regression using preset multi-scale anchor boxes. It locates all suspected pig target regions from the entire feature map, eliminating a large number of invalid background regions and significantly reducing the computational scope of subsequent fine-tuning. This ensures detection coverage while improving model inference efficiency.

[0025] The RoI Align module receives all candidate regions from the RPN branch at its input, and its output is connected to the inputs of the mask prediction branch and the appearance vector calculation branch, respectively, thus unifying the feature size. This layer uses bilinear interpolation to map the features of candidate regions of different sizes and locations to a standard feature map of a fixed size (e.g., 7×7×256 dimensions), avoiding the loss of feature details caused by quantization errors. At the same time, it provides consistent input features to the subsequent two task branches through split output, ensuring the feature homology between the segmentation and tracking tasks.

[0026] The internal network structure of the mask prediction branch is improved by employing a structure of alternating deconvolutional layers and dilated convolutional layers. The deconvolutional layers restore the spatial resolution of the feature maps through progressive upsampling, gradually amplifying the low-resolution features output by RoI Align to ultimately output a higher-resolution pig segmentation mask. The dilated convolutional layers, interspersed between the deconvolutional layers, expand the receptive field of the features without increasing the number of parameters or computational cost by setting a dilation factor, capturing large-scale detailed features such as the pig's back contour and body edges. Compared to the original Track R-CNN's direct interpolation upsampling mask branch, this alternating concatenation structure significantly improves the edge precision of the mask, effectively alleviating mask adhesion and contour blurring issues in scenarios involving pig stacking and pen occlusion, providing a high-precision segmentation foundation for subsequent pig body size measurement and vital sign parameter extraction.

[0027] An SE channel attention module is added within the appearance vector calculation branch. This module is embedded between the RoIAlign output and the final fully connected layer. The SE channel attention module automatically strengthens feature channels related to individual pig differences (such as body shape and local texture features) and suppresses irrelevant interference channels such as background stains, lighting changes, and pen shadows through global average pooling to compress spatial information, learning channel weights in the fully connected network, and weighting and calibrating each channel. This makes the extracted appearance features more focused on individual distinguishing information. The feature map weighted by the SE module is fed into the subsequent fully connected layer and mapped to a fixed-dimensional pig appearance feature vector. The fusion similarity between the appearance feature vectors of each pig in the current frame and historical features in the database is calculated. A matching cost matrix is ​​constructed based on the similarity, and the optimal inter-frame matching is achieved using the Hungarian algorithm. A similarity threshold is set to distinguish the matching results: if a match is found, the original unique identifier is retained and historical features are updated; if no match is found, a new identifier is assigned. Features of pigs that have not matched for multiple consecutive frames are temporarily cached, and the original identifier is restored when the match recurs in a short period of time. Ultimately, this achieves continuous and stable output of pig identifiers across frames. Assigning a unique and persistent identity to the same pig effectively increases the feature distance between similar pigs of the same breed, reduces the probability of identity switching in stacked or occluded scenarios, and improves tracking stability.

[0028] According to the embodiments of this application, by constructing a mask prediction branch composed of alternating series of multiple deconvolutional layers and multiple dilated convolutional layers, the receptive field is expanded while restoring the spatial resolution of the feature map, effectively improving the edge fineness and overall accuracy of the pig segmentation mask, thereby meeting the needs for precise measurement of pig body size in breeding scenarios. At the same time, by adding an SE channel attention module between RoI Align and the fully connected layer, the weights of different feature channels are recalibrated, strengthening the feature channels with individual discrimination and suppressing interference from irrelevant channels, effectively increasing the feature vector distance between individuals with similar appearances, significantly reducing the probability of identity swapping and tracking errors, and improving the stability and robustness of multi-target tracking. The two work together to enable the improved Track R-CNN model to achieve accurate identification and stable tracking of individual pigs in complex breeding environments.

[0029] In one implementation, the mask prediction branch may sequentially include: The first deconvolutional layer has a 4×4 kernel, a stride of 2, and 128 output channels. It is used to upsample the feature map from 7×7 to 14×14×128. The first layer is a dilated convolutional layer with a 3×3 kernel, a dilation factor of 2, a padding factor of 2, and 128 output channels. The second deconvolutional layer has a 4×4 kernel, a stride of 2, and 64 output channels. It is used to upsample the feature map from 14×14 to 28×28×64. The second dilated convolutional layer has a 3×3 kernel, a dilation factor of 4, a padding factor of 4, and 64 output channels. The third deconvolutional layer has a 4×4 kernel, a stride of 2, and 32 output channels. It is used to upsample the feature map from 28×28 to 56×56×32. The third dilated convolutional layer has a 3×3 kernel, a dilation factor of 6, a padding factor of 6, and 32 output channels. The final convolutional layer has a 1×1 kernel and 1 output channel. It is used to map the feature map output by the three dilated convolutional layers and output a binary segmentation mask after Sigmoid activation.

[0030] In this embodiment, the mask prediction branch consists of three deconvolutional layers, three dilated convolutional layers, and a final convolutional layer, receiving a 7×7×256 feature map output by RoI Align. The specific processing flow is as follows: The first deconvolutional layer uses a 4×4 kernel with a stride of 2 and 128 output channels to perform the first upsampling operation on the input 7×7×256 feature map, outputting a 14×14×128 feature map, thus initially restoring the spatial resolution of the features.

[0031] The first dilated convolutional layer uses a 3×3 kernel with a dilation factor of 2, a padding value of 2, and 128 output channels to extract features from a 14×14×128 feature map. Without changing the feature map size, the receptive field is expanded to 7×7 to capture the contextual information of the local contour of the pig's body.

[0032] The second deconvolutional layer uses a 4×4 kernel with a stride of 2 and 64 output channels to perform a second upsampling on the 14×14×128 feature map output from the previous layer, outputting a 28×28×64 feature map to further restore spatial details.

[0033] The second dilated convolutional layer uses a 3×3 kernel with a dilation factor of 4, a padding value of 4, and 64 output channels to extract features from the 28×28×64 feature map; the receptive field is expanded to 13×13 to cover a larger area of ​​contour features such as the back and abdomen of the pig.

[0034] The third deconvolutional layer uses a 4×4 kernel with a stride of 2 and 32 output channels to perform a third upsampling on the 28×28×64 feature map output from the previous layer, outputting a 56×56×32 feature map, thus completing the construction of high-resolution features.

[0035] The third dilated convolutional layer uses a 3×3 kernel, a dilation factor of 6, a padding value of 6, and 32 output channels to extract features from a 56×56×32 feature map; the receptive field is expanded to 19×19 to fully capture global details such as the overall edge and body contours of the pig.

[0036] Final convolutional layer: Using a 1×1 convolutional kernel with 1 output channel, the 56×56×32 feature map output by the third dilated convolution layer is mapped to the channel dimension. After processing by the Sigmoid activation function, a 56×56 single-channel probability map is output, which finally generates a binary segmentation mask corresponding to a single pig.

[0037] According to the embodiments of this application, the mask output resolution is increased from the original 28×28 to 56×56 by progressively upsampling through three layers of deconvolution. At the same time, three dilated convolutional layers with increasing dilation factors are interspersed to gradually expand the feature receptive field without increasing the number of parameters. The alternating and cascaded structure not only ensures the spatial resolution and edge detail accuracy of the mask, but also retains sufficient global semantic information. This can effectively improve the precision of pig contour segmentation, alleviate the mask adhesion and edge blurring problems in stacked and occluded scenes, and provide a reliable segmentation basis for the accurate calculation of subsequent vital signs such as backfat thickness and weight.

[0038] Taking a 1280×720 resolution pigsty monitoring image as an example, this image contains three pregnant sows. After processing by the backbone network, RPN branch, and RoI Align, each sow outputs a 7×7×256 dimensional candidate region feature map. This feature map is then input into the mask prediction branch, and sequentially undergoes three deconvolutions to progressively enlarge the feature map size. This is combined with three layers of dilated convolutions with increasing dilation factors to gradually expand the receptive field. Finally, each sow outputs a 56×56 resolution binary segmentation mask. Compared to the original Track R-CNN's 28×28 resolution mask output, this embodiment increases the total number of mask pixels by four times, resulting in smoother and more accurate segmentation contours for the pig's ears, abdomen, and limb edges. Even if two pigs are only partially adjacent, their body boundaries can be clearly distinguished, and there is no mask adhesion.

[0039] In one implementation, the SE channel attention module may include: The Squeeze unit is used to perform global average pooling on the feature map output by the RoI Align module to obtain the feature channel description vector. The Excitation unit is used to sequentially pass the feature channel description vector through a first fully connected layer to compress the dimension, activate ReLU, restore the original channel dimension through a second fully connected layer, and activate Sigmoid to generate a feature channel weight vector. The Reweight unit is used to multiply the feature channel weight vector by the feature map channel by channel to obtain a weighted feature map.

[0040] In this embodiment, the SE channel attention module is connected between the RoI Align output and the fully connected layer of the appearance vector branch. It takes the fixed-size feature map output by RoI Align as input, performs channel weighting on the uniformly sized feature map, and then feeds it into the fully connected layer to generate the target appearance feature vector. This SE channel attention module may include a Squeeze unit, an Excitation unit, and a Reweight unit. The specific execution process of each unit is as follows: The Squeeze unit receives the candidate region feature map output by RoI Align, performs global average pooling on the feature map, averages the pixel values ​​of all spatial locations within each channel, and compresses the H×W×C spatial features into a 1×1×C feature channel description vector. This step achieves spatial dimensional information aggregation, making each channel correspond to a global semantic representation, providing a basis for subsequent weight learning.

[0041] The Excitation unit takes the feature channel description vector as input and sequentially passes it through two fully connected network layers to learn the weights: the first fully connected layer compresses and reduces the channel dimension by introducing a non-linear transformation through the ReLU activation function; the second fully connected layer restores the dimension to the original number of channels and maps the values ​​to the 0-1 range through the Sigmoid activation function, finally outputting a feature channel weight vector with the same number of input channels. This step automatically determines the importance of different feature channels by autonomously learning the dependencies between channels through the fully connected network.

[0042] The Reweight unit multiplies the feature channel weight vectors output from the Excitation unit, channel by channel, with the original input feature map. Specifically, all spatial pixels in each channel are multiplied by the corresponding weight value, resulting in a weighted and calibrated feature map. This step achieves adaptive recalibration of feature channels, enhancing those useful for the task and suppressing interference from irrelevant channels such as background and noise.

[0043] Taking a 1280×720 image of a pigpen containing three pregnant sows of the same breed and similar body size as an example, after RoIAlign processing, each pig corresponds to a 7×7×256-dimensional candidate region feature map. Without the SE module, the appearance vector branch directly calculates the appearance vector based on the original feature map. The feature vectors of different pigs are relatively close in space, which can easily lead to identity matching errors when pigs are stacked and occluded. After adding the SE module, the model aggregates global information from 256 channels through the Squeeze unit. After autonomous learning by the Excitation unit, the feature channel weights corresponding to the pig's back contour, ear shape, and local coat color texture are increased to above 0.8, while the interference channel weights corresponding to pigpen floor stains, pen shadows, and light spots are reduced to below 0.2. The final output weighted feature map focuses more on the individual differences of the pigs, and the distinguishability of the appearance vectors of the three pigs is significantly improved. Even if the pigs are briefly stacked or partially occluded by the fence, cross-frame identity matching can be accurately completed, effectively reducing identity swapping errors.

[0044] According to the embodiments of this application, through the three-level processing structure of "Squeeze global information aggregation - Excitation adaptive weight learning - Reweight feature channel-by-channel calibration", the feature channels related to individual pig differentiation can be automatically strengthened, and the background and lighting interference channels can be suppressed. This effectively improves the individual differentiation of appearance feature vectors, widens the feature distance between similar pigs of the same breed, reduces the probability of identity swapping in complex breeding scenarios such as stacking, occlusion, and uneven lighting, and improves the stability and accuracy of cross-frame multi-target tracking.

[0045] In one implementation, before processing the image data using the improved Track R-CNN model, the method may further include: The mask prediction branch is trained using boundary-aware binary cross-entropy loss, with higher computational weights assigned to pixels in the mask edge region than to those in the non-edge region. The appearance vector calculation branch is trained using Triplet Loss. The interval α of Triplet Loss is set to an adaptive value. The expression of the interval α is α=0.5+0.1·sim(a,p), where a is the anchor sample, p is the positive sample of the same target, and sim(a,p) is the feature similarity between the anchor sample and the positive sample.

[0046] In this embodiment of the application, the model needs to be trained before processing the image data using the improved Track R-CNN model.

[0047] Using pig images labeled with real segmentation masks as training samples, the model's predicted mask and the real mask are substituted into the boundary-aware binary cross-entropy loss function to complete the loss calculation. For each pixel position (i,j) in the mask, the loss function is defined as: L_mask= Σ(i,j)w(i,j)·[y(i,j)·log(p(i,j))+(1 y(i,j))·log(1 p(i,j))], where p(i,j) is the probability that the model predicts the pixel belongs to the foreground of a pig, y(i,j) is the ground truth label (value is 0 or 1), and w(i,j) is the adaptive weight corresponding to the pixel position. The weight w(i,j) is determined by the distance d(i,j) from the pixel to the nearest edge, and its expression is: w(i,j)=1+α·exp( d²(i,j) / 2σ²), where d(i,j) represents the Euclidean distance between the pixel position (i,j) and its nearest target real edge pixel, α is a hyperparameter that controls the magnitude of weight increase (e.g., a value of 2.0), and σ is a hyperparameter that controls the rate of weight decay with distance (e.g., a value of 3.0).

[0048] According to the above formula, when the pixel is located in the edge region (closer to the edge, i.e., d(i,j) is smaller), exp( As d² / 2σ² approaches 1, the weight w(i,j) approaches 1+α (when α=2.0, the weight approaches 3), meaning the weight of pixels in edge regions is increased to about 3 times that of pixels in non-edge regions; when a pixel is located in a region far from the edge (i.e., d(i,j) is large), exp( The weights d² / 2σ² approach 0, and the weights w(i,j) approach 1, which is basically consistent with the weights of non-edge regions. Through this loss function, the model focuses more on the pixel classification accuracy of edge regions during training, thereby guiding the network to learn more accurate pig edge features.

[0049] Using consecutive pig frame images labeled with real individual IDs as training samples, triplet sample pairs are constructed to complete loss calculation and parameter update. For anchor sample a (the appearance vector of a pig in a certain frame), positive sample p (the appearance vector of the same pig in other frames), and negative sample n (the appearance vectors of different pigs), the mathematical expression of Triplet Loss is: L_triplet=max(0,∥f(a)). f(p)∥2² ∥f(a) f(n)∥2²+α), where f(·) represents the appearance vector output by the appearance vector calculation branch, ∥·∥2² represents the square of the Euclidean distance, and α is the margin. The margin α is set as an adaptive value, and its expression is: α=0.5+0.1·sim(a,p), where sim(a,p) is the feature similarity between the anchor sample a and the positive sample p. The value of sim(a,p) is usually between 0 and 1. When the anchor point and the positive sample are highly similar, the value of sim(a,p) is higher, corresponding to a larger value of α, which forces the model to further increase the inter-class distance while maintaining intra-class compactness; when the anchor point and the positive sample have low similarity, the value of α is reduced accordingly to reduce the training difficulty and avoid the model from failing to converge.

[0050] During training, an online hard example mining strategy can also be employed. Specifically, for each batch of samples, when calculating the loss, the model not only uses randomly selected negative samples but also dynamically selects the negative samples that are closest to the anchor sample (i.e., the most difficult to distinguish) from all available negative samples in that batch to participate in the loss calculation. This strategy enables the model to utilize valuable training samples more efficiently, accelerates model convergence, and improves tracking accuracy.

[0051] According to the embodiments of this application, the learning accuracy of the model for the mask edge region is enhanced by boundary-aware binary cross-entropy loss. At the same time, the adaptive interval Triplet Loss combined with the online hard example mining strategy is used to specifically improve the distinguishability of appearance features of similar individuals. The two optimize the model performance from the two core dimensions of segmentation accuracy and tracking stability, which can effectively solve the pain points of rough mask edges of pigs and easy confusion of similar pigs of the same breed in the breeding scenario, and improve the overall robustness and deployment usability of the model in complex environments such as stacking, occlusion and uneven lighting.

[0052] In one embodiment, the backbone network may consist of a ResNet-101 network and a Feature Pyramid Network (FPN), with the output of the ResNet-101 network connected to the input of the FPN, and the output of the FPN connected to the input of the RPN branch. The FPN is used to output feature maps at multiple scales.

[0053] In this embodiment, the backbone network adopts a combined architecture of "ResNet-101+FPN". Starting with the input pig image, it sequentially completes multi-level feature extraction and multi-scale feature fusion, and finally outputs a multi-scale feature map for use by the RPN branch. The specific processing flow is as follows: The input image first enters the ResNet-101 network. The ResNet-101 network typically contains five convolutional stages: the first convolutional stage (Conv1): performs initial convolution and pooling on the input image to initially extract low-level features (such as edges, corners, etc.); the second to fifth convolutional stages (Conv2_x to Conv5_x): through the stacking of multiple bottleneck structures, gradually extracts mid-level (such as texture, local shape) and high-level semantic features (such as object parts, overall structure), and gradually reduces the spatial resolution of the feature maps and increases the number of channels.

[0054] The ResNet-101 network extracts features through progressive downsampling using multiple convolutional and residual modules, outputting four levels of basic feature maps (C2, C3, C4, and C5) from shallow to deep. The resolutions of these four feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original input image, respectively. The shallow C2 feature map preserves fine spatial details such as pig hair texture and body edges, while the deep C5 feature map carries global high-level information such as the overall outline of the pig and category semantics. The outputs of each level of the ResNet-101 feature map are directly connected to the corresponding inputs of the FPN, providing the foundational features for subsequent feature fusion.

[0055] The FPN receives four-level basic feature maps from ResNet-101 and performs multi-scale feature fusion through a three-stage process: Bottom-up path: the forward feature extraction process of ResNet-101 generates four-level basic features with progressively decreasing resolution; Top-down path: starting with the lowest resolution deep feature C5, the feature map size is gradually increased through upsampling, and then matched with the shallow features C4, C3, and C2 in sequence; Lateral connection fusion: the upsampled high-level features are element-wise added to the shallow basic features of the same size, preserving deep semantic information while supplementing shallow detail information. After fusion, the FPN outputs four multi-scale fused feature maps: P2, P3, P4, and P5, with resolutions corresponding to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. All outputs of the FPN are connected to the inputs of subsequent RPN branches, feeding the multi-scale fused features into the candidate region generation stage.

[0056] According to the embodiments of this application, multi-level basic features are extracted through ResNet-101 deep residual network, and multi-scale fusion of deep and shallow features is completed by combining FPN feature pyramid. The output multi-layer feature map has both high-level semantic information and low-level detail information, which can simultaneously adapt to the detection needs of pigs of different sizes and distances, effectively reduce the missed detection of small targets and false detection of background, improve the comprehensiveness and accuracy of target detection in complex breeding scenarios, and provide a high-quality multi-scale feature foundation for subsequent fine mask segmentation and accurate extraction of appearance features.

[0057] In one implementation, the improved Track R-CNN model may further include: The detection branch, whose input is connected to the output of RoI Align, is used to output the bounding box and category information of the target pig.

[0058] In this embodiment, the detection branch is one of the parallel branches of the improved Track R-CNN model. Its input is directly connected to the output of the RoI Align module, and it shares candidate region features with the mask prediction branch and the appearance vector calculation branch, ensuring feature consistency across multiple tasks. The specific processing flow is as follows: Feature reception: Receives fixed-size candidate region feature maps (e.g., 7×7×256 dimensions) output by RoI Align. These features are completely homologous to the inputs of the mask branch and appearance vector branch, ensuring that localization, segmentation, and tracking tasks are carried out based on the same feature foundation.

[0059] Classification sub-branch operation: Convolution and fully connected mapping operations are performed on the input features to output the category confidence score corresponding to each candidate region. For the pig farming scenario, the category output includes two categories: "target pig" and "background". The higher the confidence score, the greater the probability that the candidate region is a pig, thus distinguishing pigs from background elements such as pens, ground, and stains.

[0060] Bounding box regression sub-branch operation: Simultaneously perform bounding box offset regression calculation on the input features, and output the horizontal and vertical coordinates, width and height correction values ​​of the candidate box relative to the real target position; fine-tune the position and size of the coarse candidate box generated by RPN so that the final output bounding box fits the actual outline of the pig's body better.

[0061] Post-processing filtering: Non-maximum suppression is performed on the classification and regression results of all candidate regions to remove redundant detection boxes with duplicates and high overlaps, and retain the effective detection results with the highest confidence. Finally, the precise bounding box coordinates and category information corresponding to each target pig are output.

[0062] According to the embodiments of this application, by setting a detection branch that runs parallel to the mask and appearance vector branches, and sharing the same source features output by RoIAlign, multi-task joint inference can be achieved. The accurate bounding box and category information of the target pig can be output simultaneously. This can effectively filter background false detections, clarify the spatial location of the target, and provide a positioning benchmark for subsequent instance segmentation and cross-frame identity tracking. This ensures the accuracy and execution efficiency of segmentation and tracking tasks, enabling the model to fully cover the detection requirements of the entire process of "localization-classification-segmentation-tracking", and adapt to the intelligent monitoring scenarios of large-scale pig farms.

[0063] Figure 2A schematic diagram of the structure of the improved Track R-CNN model provided in this application is shown below. Figure 2 As shown, the improved Track R-CNN model includes a backbone network 201, an RPN branch 202, a RoI Align module 203, a mask prediction branch 204, an appearance vector calculation branch 205, and a detection branch 206.

[0064] In this embodiment, the image to be detected, containing the target pig, is first fed into the backbone network 201. A ResNet-101 basic convolutional network, combined with an FPN feature pyramid network, performs multi-scale shared feature extraction, outputting a multi-scale feature map that integrates shallow texture and deep semantics. The feature map output from the backbone network 201 is input into the RPN branch 202. The RPN branch 202 generates candidate region boxes for the pig target within the image through an anchor box mechanism and performs preliminary foreground / background classification and coarse coordinate regression on the candidate boxes. The candidate region features output from the RPN branch 202 are fed into the RoI Align module 203. This module eliminates quantization errors through bilinear interpolation, uniformly mapping candidate regions of varying sizes to a standardized feature tensor of fixed dimensions. The RoIAlign module 203 outputs features and simultaneously branches them to three parallel branches: a mask prediction branch 204, an appearance vector calculation branch 205, and a detection branch 206. These three branches share all features extracted from the front end, achieving integrated calculation of detection, segmentation, and individual tracking feature encoding.

[0065] Mask prediction branch 204: The internal network structure of mask prediction branch 204 is improved by adopting a cascaded structure of alternating deconvolutional layers and dilated convolutional layers. The first deconvolutional layer, the first dilated convolutional layer, the second deconvolutional layer, the second dilated convolutional layer, the third deconvolutional layer, and the third dilated convolutional layer are arranged sequentially, and the final convolutional layer is connected at the end to output a binary pig segmentation mask. The deconvolutional layer is used to progressively upsample the feature map and improve the resolution of the output mask; the dilated convolutional layer is used to expand the feature receptive field without increasing the number of parameters and computational cost, and to capture the subtle concave and convex features of the pig's body edges and contours. The alternating stacking of the two can significantly improve the edge fineness of the segmentation mask and avoid the blurry contours and jagged edges of the original model mask.

[0066] Appearance vector calculation branch 205: An SE channel attention module is embedded within appearance vector calculation branch 205. The output features of RoIAlign module 203 first enter the SE channel attention module, and then are fed down to the fully connected layer. The features optimized by the SE channel attention module are sent to the fully connected layer and compressed into a fixed-dimensional appearance feature vector, which is used for cross-frame matching to generate unique identifiers for pigs, greatly reducing ID swapping and tracking errors in scenarios where pigs cluster together or have similar appearances.

[0067] Detection branch 206: Receives standardized features from RoI Align module 203, outputs target classification results and precise bounding box coordinates in parallel, completes target localization and category determination for pigs, and provides basic target range constraints for branch input.

[0068] Figure 3 This is a schematic diagram of the SE channel attention module provided in this application, as shown below. Figure 3 As shown, the SE channel attention module consists of a Squeeze unit 301, an Excitation unit 302, and a Reweight unit 303 connected in series. These three units work together to adaptively adjust the feature channel weights. Specifically, the Squeeze unit 301 uses global average pooling to aggregate all spatial pixel information within a single channel, compressing the two-dimensional spatial features into a one-dimensional channel statistical vector, thus extracting global feature information. The Excitation unit 302 takes the channel statistical vector as input and autonomously learns the importance of each feature channel through a two-level fully connected network, outputting a channel weight vector with values ​​ranging from 0 to 1. The Reweight unit 303 multiplies the channel weight vector with the original input features of the module channel by channel, completing adaptive weighted correction of the features.

[0069] Based on the joint operation of the three-level unit, the SE channel attention module can adaptively increase the weight of channels carrying individual differentiated features such as pig body shape and ear texture, while reducing the weight of background-irrelevant feature channels such as pen shadows, ground stains, and ambient light interference, thereby achieving effective feature enhancement of the target individual and suppression of background redundant noise.

[0070] Figure 4 A schematic diagram of a multi-target tracking device based on an improved Track R-CNN provided in this application, the device may include: Acquisition module 410 is used to acquire image data to be detected, wherein the image data includes the target pig; The processing module 420 is used to process the image data through an improved Track R-CNN model to obtain the identity identifier of the target pig, so as to identify and track the target pig. The improved Track R-CNN model includes a backbone network, an RPN branch, a RoI Align module, a mask prediction branch, and an appearance vector calculation branch. The backbone network output is connected to the input of the RPN branch, the RPN branch output is connected to the input of the RoI Align module, and the RoI Align module output is connected to the inputs of the mask prediction branch and the appearance vector calculation branch, respectively. The mask prediction branch is composed of multiple deconvolutional layers and multiple dilated convolutional layers connected in series. The appearance vector calculation branch sets an SE channel attention module between the RoI Align module and the fully connected layer.

[0071] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown in the figure, an embodiment of this application provides a multi-target tracking device 500 based on an improved Track R-CNN, including a memory 530, a processor 510, and a computer program 520 stored in the memory. The processor 510 executes the computer program 520 to implement the multi-target tracking method based on an improved Track R-CNN described in any of the above embodiments.

[0072] This application provides a multi-target tracking device based on an improved Track R-CNN, which may include a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the multi-target tracking method based on the improved Track R-CNN described in any of the above embodiments.

[0073] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the multi-target tracking method based on the improved Track R-CNN described in any of the above embodiments.

[0074] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0075] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0076] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-target tracking method based on an improved Track R-CNN, characterized in that, include: Acquire image data to be detected, wherein the image data contains the target pig; The image data is processed by an improved Track R-CNN model to obtain the identification of the target pig, so as to identify and track the target pig. The improved Track R-CNN model includes a backbone network, an RPN branch, a RoI Align module, a mask prediction branch, and an appearance vector calculation branch. The backbone network output is connected to the input of the RPN branch, the RPN branch output is connected to the input of the RoIAlign module, and the RoI Align module output is connected to the inputs of the mask prediction branch and the appearance vector calculation branch, respectively. The mask prediction branch is composed of multiple deconvolutional layers and multiple dilated convolutional layers connected in series. The appearance vector calculation branch sets an SE channel attention module between the RoI Align module and the fully connected layer.

2. The method according to claim 1, characterized in that, The mask prediction branch includes, in sequence: The first deconvolutional layer has a 4×4 kernel, a stride of 2, and 128 output channels. It is used to upsample the feature map from 7×7 to 14×14×128. The first layer is a dilated convolutional layer with a 3×3 kernel, a dilation factor of 2, a padding factor of 2, and 128 output channels. The second deconvolutional layer has a 4×4 kernel, a stride of 2, and 64 output channels. It is used to upsample the feature map from 14×14 to 28×28×64. The second dilated convolutional layer has a 3×3 kernel, a dilation factor of 4, a padding factor of 4, and 64 output channels. The third deconvolutional layer has a 4×4 kernel, a stride of 2, and 32 output channels. It is used to upsample the feature map from 28×28 to 56×56×32. The third dilated convolutional layer has a 3×3 kernel, a dilation factor of 6, a padding factor of 6, and 32 output channels. The final convolutional layer has a 1×1 kernel and 1 output channel. It is used to map the feature map output by the three dilated convolutional layers and output a binary segmentation mask after Sigmoid activation.

3. The method according to claim 1, characterized in that, The SE channel attention module includes: The Squeeze unit is used to perform global average pooling on the feature map output by the RoI Align module to obtain the feature channel description vector. The Excitation unit is used to sequentially pass the feature channel description vector through a first fully connected layer to compress the dimension, activate ReLU, restore the original channel dimension through a second fully connected layer, and activate Sigmoid to generate a feature channel weight vector. The Reweight unit is used to multiply the feature channel weight vector by the feature map channel by channel to obtain a weighted feature map.

4. The method according to claim 1, characterized in that, Prior to processing the image data using the improved Track R-CNN model, the method further includes: The mask prediction branch is trained using boundary-aware binary cross-entropy loss, with higher computational weights assigned to pixels in the mask edge region than to those in the non-edge region. The appearance vector calculation branch is trained using Triplet Loss. The interval α of Triplet Loss is set to an adaptive value. The expression of the interval α is α=0.5+0.1·sim(a,p), where a is the anchor sample, p is the positive sample of the same target, and sim(a,p) is the feature similarity between the anchor sample and the positive sample.

5. The method according to claim 1, characterized in that, The backbone network consists of a ResNet-101 network and a Feature Pyramid Network (FPN). The output of the ResNet-101 network is connected to the input of the FPN, and the output of the FPN is connected to the input of the RPN branch. The FPN is used to output feature maps at multiple scales.

6. The method according to claim 1, characterized in that, The improved Track R-CNN model also includes: The detection branch, whose input is connected to the output of RoI Align, is used to output the bounding box and category information of the target pig.

7. A multi-target tracking device based on an improved Track R-CNN, characterized in that, include: The acquisition module is used to acquire image data to be detected, wherein the image data contains the target pig; The processing module is used to process the image data through an improved Track R-CNN model to obtain the identity identifier of the target pig, so as to identify and track the target pig. The improved Track R-CNN model includes a backbone network, an RPN branch, a RoI Align module, a mask prediction branch, and an appearance vector calculation branch. The backbone network output is connected to the input of the RPN branch, the RPN branch output is connected to the input of the RoIAlign module, and the RoI Align module output is connected to the inputs of the mask prediction branch and the appearance vector calculation branch, respectively. The mask prediction branch is composed of multiple deconvolutional layers and multiple dilated convolutional layers connected in series. The appearance vector calculation branch sets an SE channel attention module between the RoI Align module and the fully connected layer.

8. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images