Animal behavior detection method and system based on multi-view semi-supervised training

Abstract

The application provides an animal behavior detection method and system based on multi-view semi-supervised training, based on a semi-supervised learning framework, adopting a multi-view video matching mechanism and a teacher-student semi-supervised learning framework, better fusing information from different views by changing a training strategy, efficiently fusing multi-view data through phased optimization; the confidence screening is introduced through a dynamic pseudo-label filtering mechanism to filter low-quality or noise pseudo-labels, so that the student model only uses reliable pseudo-label signals in the training process, thereby improving the learning effect and performance of the model; the teacher model of the oblique view is used to generate pseudo-label data, which is mapped to the overhead view through a homography matrix, helping the student model of the overhead view to learn more target feature information, through more accurate matching matrix calculation, using the complementary information between multi-views, effectively reducing the identity error switching under the target shielding condition, and realizing the function of improving the accuracy of animal behavior detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an animal behavior detection method and system based on multi-view semi-supervised training. Background Art

[0002] Existing technologies improve the accuracy and robustness of animal behavior monitoring by combining video information from different perspectives. In multi-view monitoring, traditional methods use the geometric relationship between perspectives to match target identities to reduce target identity switching (IDSW). When occlusion is serious, target detection consistency remains challenging. However, the disadvantage of existing methods is their insufficient ability to fuse multi-view data. Although existing technologies can fuse video data from different perspectives, they are still limited in the presence of occlusion or perspective deviation, affecting the robustness of target detection. Pseudo-label noise often leads to the degradation of model training, especially when the confidence level of unlabeled data is low. The quality of pseudo-labels directly affects the training effect of the model. Noisy pseudo-labels may introduce erroneous information, thereby affecting the detection accuracy and the generalization ability of the model.

[0003] When data labeling is difficult, existing technologies use a semi-supervised learning framework to improve the model's detection capabilities using a small amount of labeled data and a large amount of unlabeled data. Pseudo-label generation techniques use unlabeled data for training, but improving the quality of pseudo-labels and model generalization remains a challenge. While existing technologies mitigate the issue of target identity switching through multi-view fusion and target identity matching, target identity switching (IDSW) still frequently occurs in situations with severe occlusion or insufficient view coverage. This is due to insufficient matching information between different viewpoints, or an inaccurate matching mechanism, which prevents the model from properly maintaining target consistency.

[0004] Current multi-view detection methods mostly rely on large amounts of labeled data for training. However, when labeled data is scarce, the model's generalization ability decreases significantly. Even with the use of transfer learning or data augmentation techniques, it is still difficult to achieve ideal detection results in environments with insufficient annotations. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an animal behavior detection method and system based on multi-view semi-supervised training, so as to improve the accuracy of animal behavior detection.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: an animal behavior detection method based on multi-view semi-supervised training, comprising the following steps:

[0007] S1: Taking the animal behavior to be tested as the target, obtain the target's strabismus data and process it to obtain strabismus supervision data and strabismus pseudo-labels, and use the strabismus supervision data to train and optimize the teacher model;

[0008] S2: Based on a semi-supervised learning framework, the teacher model uses multi-view matching to infer the squint data and generate the top-down supervision data and pseudo-labels.

[0009] S3: Combining the knowledge transfer mechanism of the teacher-student model, a transfer learning method is used to train the student model to integrate the features of the bird's-eye view and the oblique view. Knowledge transfer and model optimization are achieved through pseudo-labels and supervised data.

[0010] S4: Detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

[0011] According to the above scheme, in step S1, the specific steps are:

[0012] S11: Acquire squint angle data from multiple video input devices at different angles;

[0013] S12: Manually label the strabismus view data to generate strabismus view supervision data;

[0014] S13: Train the teacher model using the squint-view supervision data;

[0015] S14: Use the trained teacher model to infer the squint view data and generate squint view pseudo labels;

[0016] S15: Manually check and correct the squint pseudo-labels, add the checked and corrected squint pseudo-labels to the squint supervision dataset, and optimize the performance of the teacher model through multiple rounds of iterations.

[0017] Furthermore, in step S15, the specific steps of manually checking and correcting the squint angle pseudo-label are as follows:

[0018] Perform geometric verification on the squint pseudo-label to check whether the position and size of the target box meet the expectations. If not, perform manual correction.

[0019] The squint pseudo-labels are filtered by confidence to ensure that the quality standards of squint supervision data are met.

[0020] According to the above scheme, in step S2, the specific steps of generating the bird's-eye view supervision data are as follows:

[0021] Map the squint view data to the top view through the homography matrix to generate preliminary pseudo labels;

[0022] Manually check the preliminary pseudo-labels to verify the accuracy and completeness of the target boxes and category labels, and perform manual corrections if they are not satisfied;

[0023] The checked and corrected pseudo-labels are added to the overhead supervision dataset.

[0024] According to the above solution, in step S2, the specific steps of generating a top-view pseudo label are as follows:

[0025] Use the trained teacher model to infer the squint view data and generate squint view pseudo labels;

[0026] The squint perspective pseudo label is mapped to the top view perspective through the homography matrix to generate the top view pseudo label.

[0027] Furthermore, in step S2, the robustness of the mapping frame is improved by reducing the mapping frame before mapping and expanding the mapping frame after mapping.

[0028] According to the above scheme, in step S3, the specific steps are:

[0029] S31: The student model is initialized and trained using the top-down perspective supervision data;

[0030] S32: Freeze the low-level features of the student model, train only the high-level features, and use the top-down pseudo-labels for optimization;

[0031] S33: Unfreeze some low-level features of the student model and continue fine-tuning the student model using the top-down supervised data. Switch the optimizer from AdamW to NAdam, reduce the learning rate to ensure training stability, and return the weight of the loss function to normal, ensuring that training focuses on feature extraction of the supervised data.

[0032] S34: Freeze the first 18 layers of the student model and use the weighted loss function to perform knowledge transfer fine-tuning training on the student model using the oblique view supervision data;

[0033] S35: Unfreeze 5 of the first 18 layers of the student model, restore the weights of the loss function, and switch the optimizer to NAdam; fine-tune the student model using the bird's-eye view supervised data to achieve final convergence on the bird's-eye view data.

[0034] According to the above solution, in step S2 or S4, the specific steps of multi-view matching are:

[0035] Use yolo to detect the bounding box of the object under different viewing angles;

[0036] Use the homography matrix to map the bounding box of the top view to the oblique view;

[0037] The overlapping area and overlapping ratio of the bounding boxes under the top-down and oblique perspectives are calculated to construct a matching matrix, which is used to assist in determining the matching relationship of the bounding boxes under the two perspectives.

[0038] Furthermore, in the step S2 or S4,

[0039] The specific steps to calculate the overlapping area are:

[0040] Convert each object's bounding box into a polygon;

[0041] Calculate the area of ​​the intersection of the bounding box polygon of each oblique view target and the bounding box polygon of the top view target in turn, that is, the overlapping area, to obtain the degree of spatial overlap between the two bounding boxes;

[0042] The specific steps for calculating the overlap ratio are:

[0043] The ratio of the overlapping area of ​​the bounding box polygons of the target under the oblique view and the top view to the total area of ​​the target bounding box is calculated to evaluate the similarity of the bounding boxes under different view angles.

[0044] An animal behavior detection system based on multi-view semi-supervised training,

[0045] The squint submodule is used to obtain the squint data of the target and process it to obtain squint supervision data and squint pseudo labels, and use the squint supervision data to train and optimize the teacher model;

[0046] The bird's-eye view submodule is used to infer the squint view data using multi-view matching through the teacher model based on a semi-supervised learning framework, generating bird's-eye view supervision data and bird's-eye view pseudo labels;

[0047] The transfer training submodule combines the knowledge transfer mechanism of the teacher-student model and adopts transfer learning methods to train the student model to integrate the features of the bird's-eye view and the oblique view. It also realizes knowledge transfer and model optimization through pseudo-labels and supervised data.

[0048] The detection submodule is used to detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

[0049] The beneficial effects of the present invention are:

[0050] 1. The present invention provides an animal behavior detection method and system based on multi-view semi-supervised training. Based on a semi-supervised learning framework and combined with the knowledge transfer mechanism of the teacher-student model, the teacher model with a squinting perspective is used to generate pseudo-label data, which is mapped to a bird's-eye view through a homography matrix. This helps the student model with a bird's-eye view learn more target feature information, thereby improving the accuracy of animal behavior detection.

[0051] 2. This invention utilizes a multi-view video matching mechanism and a teacher-student semi-supervised learning framework. By adapting the training strategy, it better integrates information from different viewpoints and efficiently integrates multi-view data through phased optimization. A dynamic pseudo-label filtering mechanism introduces confidence filtering to filter out low-quality or noisy pseudo-labels, ensuring that the student model only uses reliable pseudo-label signals during training, thereby improving the model's learning effectiveness and performance.

[0052] 3. This invention uses a multi-view video matching mechanism to optimize the target identity matching process based on overlap area and overlap ratio, further reducing ID switching caused by occlusion. This mechanism effectively reduces ID switching errors caused by target occlusion by more accurate matching matrix calculation and leveraging complementary information between multiple views.

[0053] 3. The proposed teacher-student model semi-supervised learning framework combines limited labeled and unlabeled data to generate high-quality pseudo-labels through the teacher model, reducing reliance on labeled data. Furthermore, a weighted combination of labeled data loss and pseudo-label loss improves the model's generalization and detection accuracy in insufficiently labeled data.

[0054] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is a flow chart of an embodiment of the present invention.

[0057] Figure 2 2 is a diagram of the semi-supervised learning framework structure of an embodiment of the present invention.

[0058] Figure 3 4 is a pseudo-label image of a squint perspective according to an embodiment of the present invention.

[0059] Figure 4This is a pseudo-label image from a top-down perspective according to an embodiment of the present invention.

[0060] Figure 5 4 is a framework diagram of a multi-view matching algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] Example 1

[0063] See also Figure 1 and Figure 2 ,The specific steps of an animal behavior detection method based on multi-view semi-supervised training are as follows:

[0064] An animal behavior detection method and system based on multi-view semi-supervised training are used to improve the accuracy of animal behavior detection.

[0065] The technical solution adopted by the present invention to solve the above technical problems is: an animal behavior detection method based on multi-view semi-supervised training, comprising the following steps:

[0066] S1: Taking the animal behavior to be tested as the target, obtain the target's strabismus data and process it to obtain strabismus supervision data and strabismus pseudo-labels, and use the strabismus supervision data to train and optimize the teacher model;

[0067] S2: Based on a semi-supervised learning framework, the teacher model uses multi-view matching to infer the squint data and generate the top-down supervision data and pseudo-labels.

[0068] S3: Combining the knowledge transfer mechanism of the teacher-student model, a transfer learning method is used to train the student model to integrate the features of the bird's-eye view and the oblique view. Knowledge transfer and model optimization are achieved through pseudo-labels and supervised data.

[0069] S4: Detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

[0070] Furthermore, in step S1, the specific steps are:

[0071] S11: Acquire squint angle data from multiple video input devices at different angles;

[0072] S12: Manually label the strabismus view data to generate strabismus view supervision data;

[0073] S13: Train the teacher model using the squint-view supervision data;

[0074] S14: Use the trained teacher model to infer the squint view data and generate squint view pseudo labels;

[0075] S15: Manually check and correct the squint pseudo-labels, add the checked and corrected squint pseudo-labels to the squint supervision dataset, and optimize the performance of the teacher model through multiple rounds of iterations.

[0076] Furthermore, in step S15, the specific steps of manually checking and correcting the squint angle pseudo-label are as follows:

[0077] Perform geometric verification on the squint pseudo-label to check whether the position and size of the target box meet the expectations. If not, perform manual correction.

[0078] The squint pseudo-labels are filtered by confidence to ensure that the quality standards of squint supervision data are met.

[0079] According to the above scheme, in step S2, the specific steps of generating the bird's-eye view supervision data are as follows:

[0080] Map the squint view data to the top view through the homography matrix to generate preliminary pseudo labels;

[0081] Manually check the preliminary pseudo-labels to verify the accuracy and completeness of the target boxes and category labels, and perform manual corrections if they are not satisfied;

[0082] The checked and corrected pseudo-labels are added to the overhead supervision dataset.

[0083] According to the above solution, in step S2, the specific steps of generating a top-view pseudo label are as follows:

[0084] Use the trained teacher model to infer the squint view data and generate squint view pseudo labels;

[0085] The squint perspective pseudo label is mapped to the top view perspective through the homography matrix to generate the top view pseudo label.

[0086] In step S2, the robustness of the mapping frame is improved by reducing the mapping frame before mapping and expanding the mapping frame after mapping.

[0087] In step S3, the specific steps are:

[0088] S31: The student model is initialized and trained using the top-down perspective supervision data;

[0089] S32: Freeze the low-level features of the student model, train only the high-level features, and use the top-down pseudo-labels for optimization;

[0090] S33: Unfreeze some low-level features of the student model and continue fine-tuning the student model using the top-down supervised data. Switch the optimizer from AdamW to NAdam, reduce the learning rate to ensure training stability, and return the weight of the loss function to normal, ensuring that training focuses on feature extraction of the supervised data.

[0091] S34: Freeze the first 18 layers of the student model and use the weighted loss function to perform knowledge transfer fine-tuning training on the student model using the oblique view supervision data;

[0092] S35: Unfreeze 5 of the first 18 layers of the student model, restore the weights of the loss function, and switch the optimizer to NAdam; fine-tune the student model using the bird's-eye view supervised data to achieve final convergence on the bird's-eye view data.

[0093] See also Figure 5 In step S2 or S4, the specific steps of multi-view matching are:

[0094] Use yolo to detect the bounding box of the object under different viewing angles;

[0095] Use the homography matrix to map the bounding box of the top view to the oblique view;

[0096] The overlapping area and overlapping ratio of the bounding boxes under the top-down and oblique perspectives are calculated to construct a matching matrix, which is used to assist in determining the matching relationship of the bounding boxes under the two perspectives.

[0097] Furthermore, in step S2 or S4,

[0098] The specific steps to calculate the overlapping area are:

[0099] Convert each object's bounding box into a polygon;

[0100] Calculate the area of ​​the intersection of the bounding box polygon of each oblique view target and the bounding box polygon of the top view target in turn, that is, the overlapping area, to obtain the degree of spatial overlap between the two bounding boxes;

[0101] The specific steps for calculating the overlap ratio are:

[0102] The ratio of the overlapping area of ​​the bounding box polygons of the target under the oblique view and the top view to the total area of ​​the target bounding box is calculated to evaluate the similarity of the bounding boxes under different view angles.

[0103] This embodiment is based on a semi-supervised learning framework, combined with the knowledge transfer mechanism of the teacher-student model, using the teacher model of the oblique perspective to generate pseudo-label data, and mapping it to the bird's-eye view through the homography matrix, helping the student model of the bird's-eye view to learn more target feature information, thereby achieving the function of improving the accuracy of animal behavior detection.

[0104] Example 2

[0105] The steps of this embodiment are the same as those of embodiment 1, except that each step is applied to a specific example. Specifically, the following steps are included:

[0106] S1: Collecting supervision data and training teacher models

[0107] S11: Collect and label data

[0108] We capture both oblique and overhead perspective video data of the animal's behavior from multiple camera angles, ensuring both diversity and richness. We manually annotate the oblique perspective videos to generate high-quality supervisory data, providing training samples for the teacher model.

[0109] S12: Training the Teacher Model

[0110] The teacher model is trained using labeled squint-view supervision data.

[0111] S13: Optimizing the Iterative Teacher Model

[0112] The trained teacher model is used to infer unlabeled strabismus video data and generate preliminary pseudo-labels. The generated pseudo-labels are manually inspected and adjusted to ensure their accuracy. This includes geometric verification of the generated pseudo-labels to ensure the position and size of the target boxes meet expectations. The pseudo-labels are then filtered using confidence scores to ensure they meet the quality standards of the supervised data.

[0113] The corrected pseudo labels are added to the supervised dataset, and the performance of the teacher model is optimized through multiple rounds of iterations.

[0114] S2: Generate bird's-eye view supervision data and bird's-eye view pseudo labels

[0115] The top-down perspective data is processed in two parts:

[0116] A portion (primarily manually generated, in small batches) generates preliminary pseudo-labels based on the supervised data from the squint view at the corresponding time point, mapped to the bird's-eye view using a homography matrix. The high-quality pseudo-labels generated are manually inspected to verify the accuracy and completeness of the object bounding boxes and category labels. This high-quality bird's-eye view pseudo-label data is then added to the bird's-eye view supervised dataset for subsequent training.

[0117] The other part (mainly model generation, large batch data) directly uses the teacher model trained with squint perspective to infer the squint perspective pseudo label ( Figure 3 ), and then mapped to the top view through the homography matrix to generate the mapping pseudo label ( Figure 3 and Figure 4 dashed box). Figure 3 For the oblique perspective, Figure 4 It can be seen that the pseudo-label of the oblique view is mapped to the top-down view through the mapping operation and hits the corresponding pig.

[0118] During mapping, the following operations are performed to improve the robustness of the mapping box. Specifically, the mapping box before mapping (that is, the pseudo-labels generated by the model) is reduced, the mapping operation is performed, and then the mapping box after the mapping operation is expanded.

[0119] S3: Training the student model

[0120] The student model training process is divided into five stages. The core goal is to effectively integrate the features of the bird's-eye view and the oblique view through transfer learning methods, and to achieve knowledge transfer and model optimization through pseudo-labels and supervised data.

[0121] S31: Phase 1: Complete training of top-down view data (student model initialized with supervised data)

[0122] In the first stage, the student model is initialized using supervised data from a bird's-eye view. This supervised data is generated by mapping the supervised data from a sideways view through a homography matrix and combining it with manual adjustments. This stage aims to enable the student model to quickly learn basic target features from a bird's-eye view and establish initial feature extraction capabilities.

[0123] S32: Second stage: Freeze low-level features and train using top-view data (student model uses semi-supervised pseudo-labels)

[0124] The first 15 low-level features of the student model are frozen, and only high-level features are trained. Pseudo-labeled data from the bird's-eye view is used for optimization. Pseudo-labels (including object bounding boxes and class labels) are generated by the teacher model after inferring the oblique view data using a homography matrix. The goal of this stage is to use semi-supervised learning to enable the student model to further adapt the high-level features of the bird's-eye view data based on its existing features.

[0125] S33: The third stage: Unfreeze some low-level features and continue training with overhead supervision data (supervisory data fine-tuning)

[0126] Unfreeze five of the first 15 layers of the student model and continue fine-tuning the model using supervised data from the bird's-eye view. Unfreezing the underlying features allows the student model to further adapt to the feature changes introduced by the bird's-eye view data, thereby improving overall model performance. During this phase, the optimizer switches from AdamW to NAdam, and the learning rate is reduced to ensure training stability. The weights in the loss function are restored to normal, ensuring that training focuses on extracting features from the supervised data.

[0127] S34: The fourth stage: freeze 18 layers, weight loss, and use the supervised data from the oblique perspective for knowledge transfer fine-tuning training

[0128] The first 18 layers of the student model are frozen, and the model is fine-tuned using the supervised data from the oblique view using a weighted loss function. The key to this stage is to allow the student model to gradually adapt to the feature differences between oblique and overhead views using the supervised data from the oblique view, while retaining the feature learning results from the overhead data.

[0129] S35: Stage 5: Unfreeze five layers, switch the recovery weights to NAdam, and fine-tune using the top-down supervised data (supervised data fine-tuning)

[0130] Unfreeze five of the first 18 layers, restore the loss function weights, and switch to the NAdam optimizer. Finally, fine-tune the model using supervised data from the bird's-eye view, allowing the student model to achieve final convergence on the bird's-eye view data. This phase of training enables the student model to demonstrate robustness and generalization capabilities on both bird's-eye and oblique view data.

[0131] S4: Detect the target through the trained teacher-student model and use multi-view matching for reasoning to reduce ID switching and ensure identity consistency;

[0132] To address the problem of target identity switching caused by severe occlusion or insufficient view coverage in existing tracking methods under a single viewpoint, this embodiment uses a multi-view matching mechanism to optimize the target identity matching process based on overlap area and overlap ratio, further reducing ID switching caused by occlusion. This mechanism effectively reduces erroneous identity switching in the event of target occlusion by more precise matching matrix calculation and leveraging complementary information between multiple views.

[0133] In order to achieve target matching between the top and tilted camera perspectives, the overlap of the bounding boxes under the two perspectives is calculated to measure the degree of correlation between the targets in different perspectives. The algorithm process is as follows: Figure 5 shown.

[0134] S41: Detect bounding boxes at different viewing angles using yolo.

[0135] S42: homography estimation: using the homography matrix to map the bounding box of the top view to the oblique view;

[0136] A homography is a 3×3 matrix that describes a plane-to-plane perspective transformation. It represents the projective relationship between two plane coordinate systems. Any two images on the same plane are related by a homography. Its core formula and solution principle are as follows:

[0137] Assume that the coordinates of a point on the source plane are (x, y) and the coordinates of the corresponding point on the target plane are (x′, y′). The basic formula of the homography matrix is:

[0138]

[0139] Where is a 3×3 matrix representing the transformation between planes:

[0140]

[0141] Expanding the above homogeneous coordinate formula, we can obtain two constraint equations:

[0142]

[0143] The linear constraints are obtained by clearing the denominator:

[0144]

[0145] For each set of corresponding points (x, y)↔(x′, y′), two linear equations can be formed. In order to express it in matrix form, the homogeneous equation is introduced:

[0146]

[0147] Where A is a 2n×9 coefficient matrix; h is the form of the homography matrix to be solved expanded into a vector, that is:

[0148]

[0149] For a single point pair (x,y)↔(x′,y′), the two equations corresponding to it can be written as matrices:

[0150]

[0151] By stacking all corresponding n points, we can form a complete matrix A. To solve the homogeneous equations, we need to satisfy:

[0152]

[0153] This embodiment is implemented through singular value decomposition (SVD). Matrix A is decomposed by SVD:

[0154]

[0155] The last column of V (the singular vector corresponding to the smallest singular value) is h. Reshaping h into a 3×3 matrix yields the homography matrix H. Because the homography matrix is ​​defined in homogeneous coordinates, its absolute value is not unique. Normalization is usually required, constraining H[2,2]=1.

[0156] S43: Multi-view matching; constructing a matching matrix by calculating the overlapping area and overlapping ratio to assist in determining the matching relationship of the bounding boxes under two views.

[0157] To accurately calculate the overlap area, we first convert the bounding boxes for each camera's view into polygons. This allows us to leverage the geometric properties of polygons for accurate overlap area calculation, rather than relying solely on the intersection of rectangles. Specifically, for each object bounding box in the top and oblique view cameras, we define its four vertices as the vertices of a polygon, making it a closed polygon.

[0158] For each oblique camera object, its bounding box polygon will be sequentially calculated with each bounding box polygon of the top camera to determine the overlap area. The overlap area is the area of ​​the intersection of the two polygons. The formula for calculating their overlap area is:

[0159]

[0160] Where P1 is the polygon formed by the top view bounding box, and P2 is the polygon formed by the oblique view bounding box.

[0161] After obtaining the area of ​​the intersection region through geometric calculation, the degree of spatial overlap of the two bounding boxes can be obtained, which provides a basis for subsequent matching.

[0162] In addition to the overlapping area, the concept of overlap ratio is also introduced to further measure the confidence of the match. For each target bounding box in the oblique view camera, its overlap ratio is defined as the ratio of the overlapping area of ​​the bounding box with the top view bounding box to the total area of ​​the target bounding box. Assume that the area of ​​a target bounding box in the oblique view is A, and its overlapping area with a target bounding box in the top view is , then the overlap ratio R can be defined as:

[0163]

[0164] This ratio measures the extent to which the bounding box from the oblique camera is overlapped by the bounding box from the top camera, allowing for the assessment of bounding box similarity between different viewpoints. Compared to using only the overlap area, the overlap ratio can more effectively describe the overlap relationship between objects from different viewpoints, especially when the objects vary in size. The overlap ratio provides a more adaptable matching metric.

[0165] After calculating the overlap area and overlap ratio between the top-view and oblique-view cameras, multi-view object matching is performed based on this information. This process aims to find the best match for each object under both views by maximizing the overlap area and overlap ratio.

[0166]

[0167] This embodiment utilizes a multi-view video matching mechanism and a teacher-student semi-supervised learning framework. By adapting the training strategy, it better integrates information from different viewpoints and efficiently integrates multi-view data through phased optimization. A dynamic pseudo-label filtering mechanism introduces confidence filtering to filter out low-quality or noisy pseudo-labels, ensuring that the student model only uses reliable pseudo-label signals during training, thereby improving the model's learning effectiveness and performance.

[0168] This embodiment uses a multi-view video matching mechanism to optimize the target identity matching process based on overlap area and overlap ratio, further reducing ID switching caused by occlusion. This mechanism effectively reduces erroneous identity switching in the event of target occlusion by more accurate matching matrix calculation and leveraging complementary information between multiple views.

[0169] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0170] Example 3

[0171] This embodiment is used to implement the principles of the above method embodiment to construct an animal behavior detection system based on multi-view semi-supervised training, including a squint perspective submodule, a top-view perspective submodule, a transfer training submodule and a detection submodule.

[0172] The squint submodule is used to obtain the squint data of the target and process it to obtain squint supervision data and squint pseudo labels, and use the squint supervision data to train and optimize the teacher model;

[0173] The bird's-eye view submodule is used to infer the squint view data using multi-view matching through the teacher model based on a semi-supervised learning framework, generating bird's-eye view supervision data and bird's-eye view pseudo labels;

[0174] The transfer training submodule combines the knowledge transfer mechanism of the teacher-student model and adopts transfer learning methods to train the student model to integrate the features of the bird's-eye view and the oblique view. It also realizes knowledge transfer and model optimization through pseudo-labels and supervised data.

[0175] The detection submodule is used to detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

[0176] Each sub-module is mainly used to implement each step of the method embodiment, which will not be described in detail here.

[0177] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, or two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0178] This embodiment also includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory stores a computer program, and when the program is executed by the processor, the processor performs the steps of an animal behavior detection method based on multi-view semi-supervised training.

[0179] This embodiment also provides a computer-readable storage medium having executable instructions stored thereon. When the instructions are executed by a processor, the processor implements an animal behavior detection method based on multi-view semi-supervised training.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware.

[0181] Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0182] This application is described with reference to the flowcharts of the method and computer program product according to Example 1 of the application and the block diagram of the device (system) according to Example 3. It should be understood that each process or block in the flowchart or block diagram, as well as combinations of processes or blocks in the flowchart or block diagram, can be implemented by computer program instructions.

[0183] These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 a process or multiple processes or boxes Figure 1 An animal behavior detection system based on multi-view semi-supervised training based on the functions specified in a box or multiple boxes.

[0184] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes or boxes Figure 1 The steps of an animal behavior detection method based on multi-view semi-supervised training are specified in a box or multiple boxes.

[0186] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A method for animal behavior detection based on multi-view semi-supervised training, characterized by: The following steps are involved: S1: Taking the animal behavior to be tested as the target, obtain the target's strabismus data and process it to obtain strabismus supervision data and strabismus pseudo-labels, and use the strabismus supervision data to train and optimize the teacher model; S2: Based on a semi-supervised learning framework, the teacher model uses multi-view matching to infer the squint data and generate the top-down supervision data and pseudo-labels. S3: Combining the knowledge transfer mechanism of the teacher-student model, a transfer learning method is used to train the student model to integrate the features of the bird's-eye view and the oblique view. Knowledge transfer and model optimization are achieved through pseudo-labels and supervised data. The specific steps are as follows: S31: The student model is initialized and trained using the top-down perspective supervision data; S32: Freeze the low-level features of the student model, train only the high-level features, and use the top-down pseudo-labels for optimization; S33: Unfreeze some low-level features of the student model and continue fine-tuning the student model using the top-down supervised data. Switch the optimizer from AdamW to NAdam, reduce the learning rate to ensure training stability, and return the weight of the loss function to normal, ensuring that training focuses on feature extraction of the supervised data. S34: Freeze the first 18 layers of the student model and use the weighted loss function to perform knowledge transfer fine-tuning training on the student model using the oblique view supervision data; S35: Unfreeze 5 of the first 18 layers of the student model, restore the weights of the loss function, and switch the optimizer to NAdam. Fine-tune the student model using the bird's-eye view supervised data, allowing the student model to achieve final convergence on the bird's-eye view data. S4: Detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

2. The animal behavior detection method based on multi-view semi-supervised training according to claim 1, characterized in that: In the step S1, the specific steps are: S11: Acquire squint angle data from multiple video input devices at different angles; S12: Manually label the strabismus view data to generate strabismus view supervision data; S13: Train the teacher model using the squint-view supervision data; S14: Use the trained teacher model to infer the squint view data and generate squint view pseudo labels; S15: Manually check and correct the squint pseudo-labels, add the checked and corrected squint pseudo-labels to the squint supervision dataset, and optimize the performance of the teacher model through multiple rounds of iterations.

3. The animal behavior detection method based on multi-view semi-supervised training according to claim 2, characterized in that: In step S15, the specific steps of manually checking and correcting the squint angle pseudo-label are as follows: Perform geometric verification on the squint pseudo-label to check whether the position and size of the target box meet the expectations. If not, perform manual correction. The squint pseudo-labels are filtered by confidence to ensure that the quality standards of squint supervision data are met.

4. The animal behavior detection method based on multi-view semi-supervised training according to claim 1, characterized in that: In step S2, the specific steps of generating the bird's-eye view supervision data are as follows: Map the squint view data to the top view through the homography matrix to generate preliminary pseudo labels; Manually check the preliminary pseudo-labels to verify the accuracy and completeness of the target boxes and category labels, and perform manual corrections if they are not satisfied; The checked and corrected pseudo-labels are added to the overhead supervision dataset.

5. The animal behavior detection method based on multi-view semi-supervised training according to claim 1, characterized in that: In step S2, the specific steps of generating a top-view pseudo label are as follows: Use the trained teacher model to infer the squint view data and generate squint view pseudo labels; The squint perspective pseudo label is mapped to the top view perspective through the homography matrix to generate the top view pseudo label.

6. The animal behavior detection method based on multi-view semi-supervised training according to claim 4 or 5, characterized in that: In the step S2, the robustness of the mapping frame is improved by reducing the mapping frame before mapping and expanding the mapping frame after mapping.

7. The animal behavior detection method based on multi-view semi-supervised training according to claim 1, characterized in that: In step S2 or S4, the specific steps of multi-view matching are: Use yolo to detect the bounding box of the object under different viewing angles; Use the homography matrix to map the bounding box of the top view to the oblique view; The overlapping area and overlapping ratio of the bounding boxes under the top-down and oblique perspectives are calculated to construct a matching matrix, which is used to assist in determining the matching relationship of the bounding boxes under the two perspectives.

8. The animal behavior detection method based on multi-view semi-supervised training according to claim 7, characterized in that: In the step S2 or S4, The specific steps to calculate the overlapping area are: Convert each object's bounding box into a polygon; Calculate the area of ​​the intersection of the bounding box polygon of each oblique view target and the bounding box polygon of the top view target in turn, that is, the overlapping area, to obtain the degree of spatial overlap between the two bounding boxes; The specific steps for calculating the overlap ratio are: The ratio of the overlapping area of ​​the bounding box polygons of the target under the oblique view and the top view to the total area of ​​the bounding box of the target is calculated to evaluate the similarity of the bounding boxes under different view angles.

9. A system using the animal behavior detection method based on multi-view semi-supervised training according to any one of claims 1 to 8, characterized in that: The squint submodule is used to obtain the squint data of the target and process it to obtain squint supervision data and squint pseudo labels, and use the squint supervision data to train and optimize the teacher model; The bird's-eye view submodule is used to infer the squint view data using multi-view matching through the teacher model based on a semi-supervised learning framework, generating bird's-eye view supervision data and bird's-eye view pseudo labels; The transfer training submodule combines the knowledge transfer mechanism of the teacher-student model and adopts transfer learning methods to train the student model to integrate the features of the bird's-eye view and the oblique view. It also realizes knowledge transfer and model optimization through pseudo-labels and supervised data. The detection submodule is used to detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

Images