Animal behavior detection method and system based on multi-view semi-supervised training

Abstract

According to the animal behavior detection method and system based on multi-view semi-supervised training provided by the invention, based on a semi-supervised learning framework, a multi-view video matching mechanism and a teacher-student semi-supervised learning framework are adopted, and information from different views is better fused by changing a training strategy; multi-view data are efficiently fused through staged optimization; confidence coefficient screening is introduced through a dynamic pseudo label filtering mechanism, low-quality or noise pseudo labels are filtered, and it is ensured that a student model only uses reliable pseudo label signals in the training process, so that the learning effect and performance of the model are improved; according to the method, false label data is generated by using a teacher model of an oblique view angle, the false label data is mapped to an overlook view angle through a homography matrix, a student model of the overlook view angle is helped to learn more target feature information, identity error switching under the condition of target shielding is effectively reduced through more accurate matching matrix calculation and by using complementary information among multiple view angles, and the user experience is improved. The function of improving the accuracy of animal behavior detection is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an animal behavior detection method and system based on multi-view semi-supervised training. Background Art

[0002] Existing technologies improve the accuracy and robustness of animal behavior monitoring by combining video information from different perspectives. In multi-view monitoring, traditional methods use the geometric relationship between perspectives to match target identities to reduce target identity switching (IDSW). When the occlusion problem is serious, target detection consistency remains challenging. However, the disadvantage of existing methods is that they lack the ability to fuse multi-view data. Although existing technologies can fuse video data from different perspectives, they are still limited in the presence of occlusion or perspective deviation, affecting the robustness of target detection. Pseudo-label noise often leads to the degradation of model training, especially when the confidence of unlabeled data is low. The quality of pseudo-labels directly affects the training effect of the model. Noisy pseudo-labels may introduce erroneous information, thereby affecting the detection accuracy and the generalization ability of the model.

[0003] In the case of data labeling difficulties, existing technologies use a small amount of labeled data and a large amount of unlabeled data through a semi-supervised learning framework to improve the detection ability of the model. Pseudo-label generation technology uses unlabeled data for training, and improving the quality of pseudo-labels and model generalization remains a challenge. Although existing technologies use multi-view fusion and target identity matching to alleviate the target identity switching problem, the target identity switching (IDSW) problem still occurs frequently in the case of severe occlusion or insufficient view coverage. This is because the matching information of the target under different viewpoints is insufficient, or the matching mechanism is not accurate enough, resulting in the model being unable to correctly maintain target consistency.

[0004] Most current multi-view monitoring methods rely on a large amount of labeled data for training. However, when labeled data is scarce, the generalization ability of the model will be significantly reduced. Even with the use of transfer learning or data augmentation technology, it is still difficult to achieve ideal detection results in an environment with insufficient annotations. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an animal behavior detection method and system based on multi-view semi-supervised training, so as to improve the accuracy of animal behavior detection.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: an animal behavior detection method based on multi-view semi-supervised training, comprising the following steps: S1: Taking the animal behavior to be tested as the target, obtain the target's squint data and process it to obtain squint supervision data and squint pseudo labels, and use the squint supervision data to train and optimize the teacher model; S2: Based on the semi-supervised learning framework, the teacher model uses multi-view matching to infer the squint data and generate the bird’s-eye view supervision data and bird’s-eye view pseudo labels; S3: Combined with the knowledge transfer mechanism of the teacher-student model, the transfer learning method is used to train the student model to integrate the features of the bird's-eye view and the oblique view, and the knowledge transfer and model optimization are realized through pseudo-labels and supervised data; S4: Detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

[0007] According to the above scheme, in step S1, the specific steps are: S11: Acquire squint viewing angle data from multiple video input devices at different angles; S12: manually label the squint view data to generate squint view supervision data; S13: Train the teacher model using the squint-view supervised data; S14: Use the trained teacher model to infer the squint view data and generate squint view pseudo labels; S15: Manually check and correct the squint pseudo-labels, add the checked and corrected squint pseudo-labels to the squint supervision dataset, and optimize the performance of the teacher model through multiple rounds of iterations.

[0008] Furthermore, in step S15, the specific steps of manually checking and correcting the squint angle pseudo label are as follows: Perform geometric verification on the pseudo-label of the squint view to check whether the position and size of the target box meet the expectations. If not, make manual corrections. The squint pseudo-labels are filtered by confidence to ensure that the quality standards of squint supervision data are met.

[0009] According to the above scheme, in step S2, the specific steps of generating the bird's-eye view supervision data are as follows: Map the squint view data to the top view through the homography matrix to generate preliminary pseudo labels; Manually check the preliminary pseudo-labels to verify the accuracy and completeness of the target boxes and category labels. If they are not satisfied, they will be manually corrected. The checked and corrected pseudo-labels are added to the top-view supervision dataset.

[0010] According to the above scheme, in step S2, the specific steps of generating a pseudo label of a top-down perspective are as follows: Use the trained teacher model to infer the squint data and generate squint pseudo labels; The squint perspective pseudo label is mapped to the overhead perspective through the homography matrix to generate the overhead perspective pseudo label.

[0011] Furthermore, in the step S2, the robustness of the mapping frame is improved by reducing the mapping frame before mapping and expanding the mapping frame after mapping.

[0012] According to the above scheme, in step S3, the specific steps are: S31: The student model is initialized and trained using bird’s-eye view supervision data; S32: Freeze the low-level features of the student model, train only the high-level features, and use the top-down pseudo-labels for optimization; S33: Unfreeze some low-level features of the student model, and continue to fine-tune the student model using the top-down supervision data; switch the optimizer from AdamW to NAdam, reduce the learning rate to ensure training stability, and restore the weight of the loss function to normal, ensuring that the training focuses on feature extraction of the supervision data; S34: Freeze the first 18 layers of the student model, and use the weighted loss function to perform knowledge transfer fine-tuning training on the student model using the oblique view supervision data; S35: Unfreeze 5 of the first 18 layers of the student model, restore the weights of the loss function, and switch the optimizer to NAdam; use the bird’s-eye view supervision data to fine-tune the student model and allow the student model to achieve final convergence on the bird’s-eye view data.

[0013] According to the above scheme, in step S2 or S4, the specific steps of multi-view matching are: Use yolo to detect the bounding box of the object under different viewing angles; Use the homography matrix to map the bounding box of the top-down view to the oblique view; The overlapping area and overlapping ratio of the bounding boxes under the top-down perspective and the oblique perspective are calculated to construct a matching matrix, and the matching relationship of the bounding boxes under the two perspectives is determined with the assistance of the matching matrix.

[0014] Furthermore, in the step S2 or S4, The specific steps to calculate the overlapping area are: Convert each object's bounding box into a polygon; Calculate the area of ​​the intersection of the bounding box polygon of each oblique view target and the bounding box polygon of the top view target in turn, that is, the overlapping area, to obtain the degree of overlap of the two bounding boxes in space; The specific steps to calculate the overlap ratio are: The ratio of the overlapping area of ​​the bounding box polygons of the target under the oblique viewing angle and the top-down viewing angle to the total area of ​​the target bounding box is calculated to evaluate the similarity of the bounding boxes under different viewing angles.

[0015] An animal behavior detection system based on multi-view semi-supervised training, The squint perspective submodule is used to obtain the squint perspective data of the target and process it to obtain squint perspective supervision data and squint perspective pseudo labels, and use the squint perspective supervision data to train and optimize the teacher model; The bird’s-eye view submodule is used to infer the squint view data using multi-view matching through the teacher model based on a semi-supervised learning framework, and generate bird’s-eye view supervision data and bird’s-eye view pseudo labels; The transfer training submodule is used to combine the knowledge transfer mechanism of the teacher-student model, adopt the transfer learning method, train the student model to integrate the features of the bird's-eye view and the oblique view, and realize knowledge transfer and model optimization through pseudo-labels and supervision data; The detection submodule is used to detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

[0016] The beneficial effects of the present invention are: 1. The present invention provides an animal behavior detection method and system based on multi-view semi-supervised training, which is based on a semi-supervised learning framework and combines the knowledge transfer mechanism of the teacher-student model. The teacher model of the oblique perspective is used to generate pseudo-label data, which is mapped to the bird's-eye view through the homography matrix. This helps the student model of the bird's-eye view to learn more target feature information, thereby achieving the function of improving the accuracy of animal behavior detection.

[0017] 2. The present invention adopts a multi-view video matching mechanism and a teacher-student semi-supervised learning framework, which better integrates information from different viewpoints by changing the training strategy and efficiently integrates multi-view data by phased optimization. The dynamic pseudo-label filtering mechanism introduces confidence screening to filter low-quality or noisy pseudo-labels, ensuring that the student model only uses reliable pseudo-label signals during training, thereby improving the learning effect and performance of the model.

[0018] 3. The present invention uses a multi-view video matching mechanism to optimize the target identity matching process based on the overlapping area and overlapping ratio, further reducing ID switching caused by occlusion. This mechanism effectively reduces identity error switching in the case of target occlusion by more accurate matching matrix calculation and using complementary information between multiple perspectives.

[0019] 3. The teacher-student model semi-supervised learning framework proposed in this invention combines limited labeled data and unlabeled data to generate high-quality pseudo labels through the teacher model, reducing the dependence on labeled data. At the same time, by weighted combination of labeled data loss and pseudo label loss, the generalization ability and detection accuracy of the model in the case of insufficient labeling are improved.

[0020] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a flow chart of an embodiment of the present invention.

[0023] Figure 2 It is a structural diagram of a semi-supervised learning framework according to an embodiment of the present invention.

[0024] Figure 3 It is a pseudo label image of the squint viewing angle according to an embodiment of the present invention.

[0025] Figure 4 This is a pseudo-label image from a top-down perspective according to an embodiment of the present invention.

[0026] Figure 5 It is a framework diagram of the multi-view matching algorithm of an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0028] Example 1 See also Figure 1 and Figure 2 ,The specific steps of an animal behavior detection method based on multi-view semi-supervised training are as follows: An animal behavior detection method and system based on multi-view semi-supervised training are used to improve the accuracy of animal behavior detection.

[0029] The technical solution adopted by the present invention to solve the above technical problems is: an animal behavior detection method based on multi-view semi-supervised training, comprising the following steps: S1: Taking the animal behavior to be tested as the target, obtain the target's squint data and process it to obtain squint supervision data and squint pseudo labels, and use the squint supervision data to train and optimize the teacher model; S2: Based on the semi-supervised learning framework, the teacher model uses multi-view matching to infer the squint data and generate the bird’s-eye view supervision data and bird’s-eye view pseudo labels; S3: Combined with the knowledge transfer mechanism of the teacher-student model, the transfer learning method is used to train the student model to integrate the features of the bird's-eye view and the oblique view, and the knowledge transfer and model optimization are realized through pseudo-labels and supervised data; S4: Detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

[0030] Furthermore, in step S1, the specific steps are: S11: Acquire squint viewing angle data from multiple video input devices at different angles; S12: manually label the squint view data to generate squint view supervision data; S13: Train the teacher model using the squint-view supervised data; S14: Use the trained teacher model to infer the squint view data and generate squint view pseudo labels; S15: Manually check and correct the squint pseudo-labels, add the checked and corrected squint pseudo-labels to the squint supervision dataset, and optimize the performance of the teacher model through multiple rounds of iterations.

[0031] Furthermore, in step S15, the specific steps of manually checking and correcting the squint angle pseudo label are as follows: Perform geometric verification on the pseudo-label of the squint view to check whether the position and size of the target box meet the expectations. If not, make manual corrections. The squint pseudo-labels are filtered by confidence to ensure that the quality standards of squint supervision data are met.

[0032] According to the above scheme, in step S2, the specific steps of generating the bird's-eye view supervision data are as follows: Map the squint view data to the top view through the homography matrix to generate preliminary pseudo labels; Manually check the preliminary pseudo-labels to verify the accuracy and completeness of the target boxes and category labels. If they are not satisfied, they will be manually corrected. The checked and corrected pseudo-labels are added to the top-view supervision dataset.

[0033] According to the above scheme, in step S2, the specific steps of generating a pseudo label of a top-down perspective are as follows: Use the trained teacher model to infer the squint data and generate squint pseudo labels; The squint perspective pseudo label is mapped to the overhead perspective through the homography matrix to generate the overhead perspective pseudo label.

[0034] In step S2, the robustness of the mapping frame is improved by reducing the mapping frame before mapping and expanding the mapping frame after mapping.

[0035] In step S3, the specific steps are: S31: The student model is initialized and trained using bird’s-eye view supervision data; S32: Freeze the low-level features of the student model, train only the high-level features, and use the top-down pseudo-labels for optimization; S33: Unfreeze some low-level features of the student model, and continue to fine-tune the student model using the top-down supervision data; switch the optimizer from AdamW to NAdam, reduce the learning rate to ensure training stability, and restore the weight of the loss function to normal, ensuring that the training focuses on feature extraction of the supervision data; S34: Freeze the first 18 layers of the student model, and use the weighted loss function to perform knowledge transfer fine-tuning training on the student model using the oblique view supervision data; S35: Unfreeze 5 of the first 18 layers of the student model, restore the weights of the loss function, and switch the optimizer to NAdam; use the bird’s-eye view supervision data to fine-tune the student model and allow the student model to achieve final convergence on the bird’s-eye view data.

[0036] See also Figure 5 In step S2 or S4, the specific steps of multi-view matching are: Use yolo to detect the bounding box of the object under different viewing angles; Use the homography matrix to map the bounding box of the top-down view to the oblique view; The overlapping area and overlapping ratio of the bounding boxes under the top-down perspective and the oblique perspective are calculated to construct a matching matrix, and the matching relationship of the bounding boxes under the two perspectives is determined with the assistance of the matching matrix.

[0037] Further, in step S2 or S4, The specific steps to calculate the overlapping area are: Convert each object's bounding box into a polygon; Calculate the area of ​​the intersection of the bounding box polygon of each oblique view target and the bounding box polygon of the top view target in turn, that is, the overlapping area, to obtain the degree of overlap of the two bounding boxes in space; The specific steps to calculate the overlap ratio are: The ratio of the overlapping area of ​​the bounding box polygons of the target under the oblique viewing angle and the top-down viewing angle to the total area of ​​the target bounding box is calculated to evaluate the similarity of the bounding boxes under different viewing angles.

[0038] This embodiment is based on a semi-supervised learning framework, combined with the knowledge transfer mechanism of the teacher-student model, and uses the teacher model of the oblique perspective to generate pseudo-label data, which is mapped to the overhead perspective through the homography matrix, helping the student model of the overhead perspective to learn more target feature information, thereby achieving the function of improving the accuracy of animal behavior detection.

[0039] Example 2 The steps of this embodiment are the same as those of embodiment 1, except that each step is applied to a specific example. Specifically, the following steps are included: S1: Collecting supervision data and training teacher models S11: Collect and annotate data Obtain video data of the animal's behavior from the side-view and top-view perspectives from multiple camera devices at different angles to ensure that the collected data is diverse and rich. Manually annotate the collected side-view videos to generate high-quality supervision data and provide training samples for the teacher model.

[0040] S12: Training the Teacher Model The teacher model is trained using annotated squint supervision data.

[0041] S13: Optimizing the Iterative Teacher Model The trained teacher model is used to infer the unlabeled squint video data and generate preliminary pseudo labels. The generated pseudo label data is manually checked and adjusted to ensure the accuracy of the pseudo labels, including geometric verification of the generated pseudo label data to check whether the position and size of the target box meet the expectations; the pseudo labels are screened by confidence to ensure that they meet the quality standards of the supervised data.

[0042] The corrected pseudo labels are added to the supervised dataset, and the performance of the teacher model is optimized through multiple rounds of iterations.

[0043] S2: Generate bird’s-eye view supervision data and bird’s-eye view pseudo labels The top-down perspective data is processed in two parts: A part (mainly manual, small batch data) generates preliminary pseudo labels based on the supervised data of the corresponding time point of the oblique view, mapped to the overhead view through the homography matrix. Manual inspection is performed on the high-quality data of the preliminary pseudo labels to verify the accuracy and completeness of the target box and category labels. The high-quality overhead pseudo-label data is added to the overhead supervision dataset for subsequent training.

[0044] The other part (mainly model generation, large batch data) directly uses the teacher model trained with squint perspective to infer the squint perspective pseudo label ( Figure 3 The solid line frame), and then mapped to the top view through the homography matrix to generate the mapping pseudo label ( Figure 3 and Figure 4 dashed box). Figure 3 For the oblique viewing angle, Figure 4 It can be seen that the pseudo-label of the oblique view is mapped to the top-down view through the mapping operation and hits the corresponding pig.

[0045] When mapping, the following operations are performed to improve the robustness of the mapping box. Specifically, the mapping box before mapping (that is, the pseudo-labels generated by the model) is reduced, the mapping operation is performed, and then the mapping box after the mapping operation is expanded.

[0046] S3: Training the student model The student model training process is divided into five stages. The core goal is to effectively integrate the features of the bird's-eye view and the oblique view through transfer learning methods, and to achieve knowledge transfer and model optimization through pseudo-labels and supervised data.

[0047] S31: Phase 1: Complete training of bird’s-eye view data (student model initialized with supervised data) In the first stage, the student model is initialized with the supervised data from the bird's-eye view. The supervised data from the bird's-eye view is generated by mapping the supervised data from the side-view view through a homography matrix and combining manual adjustments. This stage aims to allow the student model to quickly learn the basic target features of the bird's-eye view and establish initial feature extraction capabilities.

[0048] S32: Stage 2: Freeze low-level features and train with top-down data (semi-supervised pseudo-labels are used for student models) Freeze the first 15 low-level features of the student model, train only the high-level features, and use the pseudo-label data of the bird's-eye view for optimization. The pseudo-labels are generated by the teacher model after reasoning on the squint data through homography matrix mapping (including the target bounding box and category label). The goal of this stage is to use semi-supervised learning to allow the student model to further adapt the high-level features of the bird's-eye view data based on the existing features.

[0049] S33: The third stage: unfreeze some low-level features and continue to use the top-view supervised data for training (supervised data fine-tuning) Unfreeze 5 of the first 15 layers of the student model and continue fine-tuning the model using the supervised data from the top view. By unfreezing the underlying features, the student model can further adapt to the feature changes brought by the top view data, thereby improving the overall model performance. The optimizer is switched from AdamW to NAdam at this stage, and the learning rate is reduced to ensure training stability. The weights of the loss function are restored to normal to ensure that the training focuses on feature extraction of the supervised data.

[0050] S34: Stage 4: Freeze 18 layers, weight loss, and use supervised data from the oblique perspective for knowledge transfer fine-tuning training Freeze the first 18 layers of the student model, and use the weighted loss function to fine-tune the model with knowledge transfer using the supervised data from the squint perspective. The core of this stage is to allow the student model to gradually adapt to the feature differences between squint and bird's-eye view through the supervised data from the squint perspective, while retaining the feature learning results of the bird's-eye view data.

[0051] S35: Stage 5: Unfreeze five layers, switch recovery weights to NAdam, and fine-tune using top-down supervised data (supervised data fine-tuning) Unfreeze 5 of the first 18 layers, restore the weights of the loss function, and switch the optimizer to NAdam. Use the supervised data from the top-down perspective to perform final fine-tuning training on the model, allowing the student model to achieve final convergence on the top-down data. Through this stage of training, the student model can show good robustness and generalization ability on both top-down and oblique perspective data.

[0052] S4: Detect the target through the trained teacher-student model and use multi-view matching for reasoning to reduce ID switching and ensure identity consistency; In view of the problem of target identity switching caused by severe occlusion or insufficient view coverage in existing tracking methods under a single view, this embodiment adopts a multi-view matching mechanism to optimize the target identity matching process based on the overlapping area and overlap ratio, further reducing ID switching caused by occlusion. This mechanism effectively reduces the erroneous identity switching in the case of target occlusion by more accurate matching matrix calculation and using complementary information between multiple views.

[0053] In order to achieve target matching between the top and tilted cameras, the overlap of the bounding boxes under the two perspectives is calculated to measure the degree of correlation between the targets in different perspectives. The algorithm flow is as follows: Figure 5 shown.

[0054] S41: Detect bounding boxes at different viewing angles through Yolo, S42: homography estimation: using the homography matrix to map the bounding box of the top-down view to the oblique view; The homography matrix is ​​a 3×3 matrix that describes the perspective transformation from plane to plane. It represents the projective relationship between two plane coordinate systems. Any two images on the same plane are related by homography. Its core formula and solution principle are as follows: Assume that the coordinates of the point on the source plane are (x, y) and the coordinates of the corresponding point on the target plane are (x′, y′). The basic formula of the homography matrix is:

[0055] Where is a 3×3 matrix representing the transformation from plane to plane:

[0056] Expanding the above homogeneous coordinate formula, we can get two constraint equations:

[0057] The linear constraints are obtained by clearing the denominator:

[0058] For each set of corresponding points (x, y) ↔ (x′, y′), two linear equations can be formed. In order to express it in matrix form, the homogeneous equation is introduced:

[0059] Where A is a 2n×9 coefficient matrix; h is the form of the homography matrix to be solved expanded into a vector, that is:

[0060] For a single point pair (x,y)↔(x′,y′), the two equations corresponding to it can be written as matrices:

[0061] By stacking all corresponding n points, the complete matrix A is formed. To solve the homogeneous system of equations, it is necessary to satisfy:

[0062] This embodiment is implemented by singular value decomposition (SVD). Perform SVD decomposition on matrix A:

[0063] The last column of V (the singular vector corresponding to the minimum singular value) is h. Reshape h into a 3×3 matrix to obtain the homography matrix H. Since the homography matrix is ​​defined in homogeneous coordinates, the absolute value of the matrix is ​​not unique. It is usually necessary to normalize the matrix and constrain H[2,2]=1.

[0064] S43: multi-view matching; constructing a matching degree matrix by calculating the overlapping area and the overlapping ratio, so as to assist in determining the matching relationship of the bounding boxes under the dual views.

[0065] In order to accurately calculate the overlapping area, the bounding box of each camera view is first converted into a polygon. In this way, the geometric properties of polygons can be used for accurate overlapping area calculation, rather than relying solely on the intersection of rectangles. Specifically, for each target bounding box in the top view camera and the oblique view camera, its four vertices are defined as the vertices of the polygon, making it a closed polygon.

[0066] For each object in the oblique view camera, its bounding box polygon will be calculated in turn with each bounding box polygon in the top view camera to calculate the overlapping area. The overlapping area is the area of ​​the intersection of the two polygons. The calculation formula for their overlapping area is:

[0067] Where P1 is the polygon formed by the top view bounding box, and P2 is the polygon formed by the oblique view bounding box.

[0068] After obtaining the area of ​​the intersection region through geometric calculation, the degree of spatial overlap of the two bounding boxes can be obtained, thereby providing a basis for subsequent matching.

[0069] In addition to the overlapping area, the concept of overlap ratio is introduced to further measure the confidence of the match. For each target bounding box in the oblique view camera, its overlap ratio is defined as the ratio of the overlapping area between the bounding box and the top view bounding box to the total area of ​​the target bounding box. Assume that the area of ​​a target bounding box in the oblique view is A, and its overlapping area with a target bounding box in the top view is , then the overlap ratio R can be defined as:

[0070] This ratio measures how much the bounding box in the oblique view camera is covered by the bounding box in the top view camera, so that the similarity of bounding boxes under different viewpoints can be evaluated. Compared with using only the overlapping area, the overlap ratio can more effectively describe the target overlap relationship between viewpoints, especially when there are differences in the sizes of different targets, the overlap ratio provides a more adaptive matching metric.

[0071] After calculating the overlap area and overlap ratio between the top view and the oblique view cameras, multi-view object matching is performed based on this information. This process aims to find the best match for each object under two views by maximizing the overlap area and overlap ratio.

[0072]

[0073] This embodiment adopts a multi-view video matching mechanism and a teacher-student semi-supervised learning framework, which better integrates information from different viewpoints by changing the training strategy and efficiently integrates multi-view data by phased optimization. The dynamic pseudo-label filtering mechanism introduces confidence screening to filter low-quality or noisy pseudo-labels, ensuring that the student model only uses reliable pseudo-label signals during training, thereby improving the learning effect and performance of the model.

[0074] This embodiment uses a multi-view video matching mechanism to optimize the target identity matching process based on the overlapping area and overlapping ratio, further reducing ID switching caused by occlusion. This mechanism uses more accurate matching matrix calculation and complementary information between multiple perspectives to effectively reduce identity error switching in the case of target occlusion.

[0075] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0076] Example 3 This embodiment is used to implement the principle of the above method embodiment to construct an animal behavior detection system based on multi-view semi-supervised training, including a squint perspective submodule, a top-down perspective submodule, a transfer training submodule and a detection submodule.

[0077] The squint perspective submodule is used to obtain the squint perspective data of the target and process it to obtain squint perspective supervision data and squint perspective pseudo labels, and use the squint perspective supervision data to train and optimize the teacher model; The bird’s-eye view submodule is used to infer the squint view data using multi-view matching through the teacher model based on a semi-supervised learning framework, and generate bird’s-eye view supervision data and bird’s-eye view pseudo labels; The transfer training submodule is used to combine the knowledge transfer mechanism of the teacher-student model, adopt the transfer learning method, train the student model to integrate the features of the bird's-eye view and the oblique view, and realize knowledge transfer and model optimization through pseudo-labels and supervision data; The detection submodule is used to detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

[0078] Each sub-module is mainly used to implement each step of the method embodiment, which will not be described in detail here.

[0079] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0080] This embodiment also includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of an animal behavior detection method based on multi-view semi-supervised training.

[0081] This embodiment also provides a computer-readable storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor implements an animal behavior detection method based on multi-view semi-supervised training.

[0082] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware.

[0083] Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0084] The present application is described with reference to the flowchart of the method and computer program product according to Embodiment 1 of the present application and the block diagram of the device (system) according to Embodiment 3. It should be understood that each process or box in the flowchart or block diagram, and the combination of processes or boxes in the flowchart or block diagram can be implemented by computer program instructions.

[0085] These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process Figure 1 A process or multiple processes or boxes Figure 1 An animal behavior detection system based on multi-view semi-supervised training based on the functions specified in a box or multiple boxes.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes or boxes Figure 1 The steps of an animal behavior detection method based on multi-view semi-supervised training are specified in a box or multiple boxes.

[0088] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. An animal behavior detection method based on multi-view semi-supervised training, characterized in that: The following steps are involved: S1: Taking the animal behavior to be tested as the target, obtain the target's squint data and process it to obtain squint supervision data and squint pseudo labels, and use the squint supervision data to train and optimize the teacher model; S2: Based on the semi-supervised learning framework, the teacher model uses multi-view matching to infer the squint data and generate the bird’s-eye view supervision data and bird’s-eye view pseudo labels; S3: Combined with the knowledge transfer mechanism of the teacher-student model, the transfer learning method is used to train the student model to integrate the features of the bird's-eye view and the oblique view, and the knowledge transfer and model optimization are realized through pseudo-labels and supervised data; S4: Detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

2. The method for animal behavior detection based on multi-view semi-supervised training according to claim 1, characterized in that: In the step S1, the specific steps are: S11: Acquire squint viewing angle data from multiple video input devices at different angles; S12: manually label the squint view data to generate squint view supervision data; S13: Train the teacher model using the squint-view supervised data; S14: Use the trained teacher model to infer the squint view data and generate squint view pseudo labels; S15: Manually check and correct the squint pseudo-labels, add the checked and corrected squint pseudo-labels to the squint supervision dataset, and optimize the performance of the teacher model through multiple rounds of iterations.

3. The method for animal behavior detection based on multi-view semi-supervised training according to claim 2, characterized in that: In step S15, the specific steps of manually checking and correcting the squint angle pseudo-label are as follows: Perform geometric verification on the pseudo-label of the squint view to check whether the position and size of the target box meet the expectations. If not, make manual corrections. The squint pseudo-labels are filtered by confidence to ensure that the quality standards of squint supervision data are met.

4. The method for animal behavior detection based on multi-view semi-supervised training according to claim 1, characterized in that: In step S2, the specific steps of generating the bird's-eye view supervision data are as follows: Map the squint view data to the top view through the homography matrix to generate preliminary pseudo labels; Manually check the preliminary pseudo-labels to verify the accuracy and completeness of the target boxes and category labels. If they are not satisfied, they will be manually corrected. The checked and corrected pseudo-labels are added to the top-view supervision dataset.

5. The method for animal behavior detection based on multi-view semi-supervised training according to claim 1, characterized in that: In step S2, the specific steps of generating the top-view pseudo label are: Use the trained teacher model to infer the squint data and generate squint pseudo labels; The squint perspective pseudo label is mapped to the overhead perspective through the homography matrix to generate the overhead perspective pseudo label.

6. The animal behavior detection method based on multi-view semi-supervised training according to claim 4 or 5, characterized in that: In the step S2, the robustness of the mapping frame is improved by reducing the mapping frame before mapping and expanding the mapping frame after mapping.

7. The method for animal behavior detection based on multi-view semi-supervised training according to claim 1, characterized in that: In the step S3, the specific steps are: S31: The student model is initialized and trained using bird’s-eye view supervision data; S32: Freeze the low-level features of the student model, train only the high-level features, and use the top-down pseudo-labels for optimization; S33: Unfreeze some low-level features of the student model, and continue to fine-tune the student model using the top-down supervision data; switch the optimizer from AdamW to NAdam, reduce the learning rate to ensure training stability, and restore the weight of the loss function to normal, ensuring that the training focuses on feature extraction of the supervision data; S34: Freeze the first 18 layers of the student model, and use the weighted loss function to perform knowledge transfer fine-tuning training on the student model using the oblique view supervision data; S35: Unfreeze 5 of the first 18 layers of the student model, restore the weights of the loss function, and switch the optimizer to NAdam; use the bird’s-eye view supervision data to fine-tune the student model and allow the student model to achieve final convergence on the bird’s-eye view data.

8. The method for animal behavior detection based on multi-view semi-supervised training according to claim 1, characterized in that: In step S2 or S4, the specific steps of multi-view matching are: Use yolo to detect the bounding box of the object under different viewing angles; Use the homography matrix to map the bounding box of the top-down view to the oblique view; The overlapping area and overlapping ratio of the bounding boxes under the top-down perspective and the oblique perspective are calculated to construct a matching matrix, and the matching relationship of the bounding boxes under the two perspectives is determined with the assistance of the matching matrix.

9. The method for animal behavior detection based on multi-view semi-supervised training according to claim 8, characterized in that: In the step S2 or S4, The specific steps to calculate the overlapping area are: Convert each object's bounding box into a polygon; Calculate the area of ​​the intersection of the bounding box polygon of each oblique view target and the bounding box polygon of the top view target in turn, that is, the overlapping area, to obtain the degree of overlap of the two bounding boxes in space; The specific steps to calculate the overlap ratio are: The ratio of the overlapping area of ​​the bounding box polygons of the target under the oblique viewing angle and the top-down viewing angle to the total area of ​​the target bounding box is calculated to evaluate the similarity of the bounding boxes under different viewing angles.

10. An animal behavior detection system based on multi-view semi-supervised training, characterized in that: The squint perspective submodule is used to obtain the squint perspective data of the target and process it to obtain squint perspective supervision data and squint perspective pseudo labels, and use the squint perspective supervision data to train and optimize the teacher model; The bird’s-eye view submodule is used to infer the squint view data using multi-view matching through the teacher model based on a semi-supervised learning framework, and generate bird’s-eye view supervision data and bird’s-eye view pseudo labels; The transfer training submodule is used to combine the knowledge transfer mechanism of the teacher-student model, adopt the transfer learning method, train the student model to integrate the features of the bird's-eye view and the oblique view, and realize knowledge transfer and model optimization through pseudo-labels and supervision data; The detection submodule is used to detect the target through the trained teacher-student model and use multi-view matching reasoning to ensure the identity consistency of the target.

Images