Dairy cow bathing method and apparatus

Abstract

This invention relates to a method and apparatus for medicated bathing of dairy cows. The method includes: Step 1: acquiring an original image of the udder region of a dairy cow using an image acquisition device; Step 2: inputting the color image into a trained teat detection model to identify and segment the two-dimensional pixel coordinates of the teat bounding box; Step 3: extracting depth values ​​from the neighborhood of the teat bounding box based on a depth image, and converting the two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system; Step 4: converting the three-dimensional coordinates in the camera coordinate system into a target pose in the robot arm base coordinate system, and driving a robot arm equipped with a medicated bath actuator to move according to the target pose to perform the medicated bathing operation. This invention enables precise, reliable, and automated medicated bathing of dairy cows, improving work efficiency and eliminating operational safety hazards.

Description

Technical Field

[0001] This invention belongs to the field of intelligent animal husbandry technology, specifically relating to a method and apparatus for medicated bathing dairy cows. Background Technology

[0002] Dairy cows undergo medicated baths before and after milking, known as pre-bath and post-bath baths. The pre-bath disinfects and cleans the teats, reducing mastitis and the number of bacteria in the milk. The post-bath seals the teat ducts, sterilizes, repairs the teat skin, and prevents external pathogens from entering after milking.

[0003] Traditional dairy cow medicated bathing relies heavily on manual operation, resulting in low efficiency, high costs, stress on cows, and safety hazards for workers. Existing automated medicated bathing systems mostly use machine vision 2D images to identify the medicated area. Teats, being small, soft, and highly similar targets, are easily missed under the default feature scale design of conventional detectors. 2D images cannot directly obtain depth information, requiring the detection results to be converted into a 3D target pose that can be executed by a robotic arm. Traditional hand-eye calibration requires manually recording the robotic arm pose, a cumbersome process with significant error accumulation, making it difficult to promote in farm operations.

[0004] Therefore, there is an urgent need to develop a dairy cow medicated bath program that is specifically designed for milking cows on milking parlor rotary machines, taking into account both accuracy and safety. Summary of the Invention

[0005] The purpose of this invention is to provide a dairy cow medicated bath solution that balances accuracy and safety.

[0006] To achieve the above objectives, the present invention provides a method for medicated bathing dairy cows, comprising: Step 1: Acquire raw images of the cow's udder region using an image acquisition device. The raw images include color and depth images of the udder region. Step 2: Input the color image into the trained nipple detection model to identify and segment the two-dimensional pixel coordinates of the nipple bounding box. The nipple detection model is an improved YOLOv11n network, which includes three feature layers from P2 to P4. The detection head of the improved YOLOv11n network is a dynamic head. Step 3: Extract depth values ​​from the neighborhood of the nipple bounding box based on the depth image, and convert the two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system; Step 4: Convert the three-dimensional coordinates in the camera coordinate system to the target pose in the robot arm base coordinate system, and drive the robot arm with the drug bath actuator to move according to the target pose to perform the drug bath operation.

[0007] Preferably, the cow stands on the milking turntable. In step 1, the milking turntable triggers a limit switch located at the medicated bath station, thereby activating the image acquisition device to acquire images.

[0008] Preferably, the method further includes obtaining training data for the nipple detection model through the following steps: The sample images are acquired using an image acquisition device. The sample images include color images of dry and wet teats of multiple breeds and dairy cows at different lactation stages, during the pre- and post-drinking process and when the cows are standing. The nipple boundary is marked in the sample image using a rectangular bounding box; The labeled sample images are subjected to random rotation, random translation, random brightness and contrast jitter, mosaic augmentation, copy-paste augmentation, and Gaussian noise addition, and are divided into training set, validation set and test set in a ratio of 7:2:1.

[0009] Preferably, the dynamic detection head includes a scale-aware attention (SAA) module, which adaptively weights the feature maps output by the P2, P3, and P4 feature layers through the following steps: Global average pooling is performed on feature maps at each scale to obtain channel-level descriptors; The channel-level descriptors at each scale are concatenated to obtain a joint vector; Based on the joint vector, weight factors corresponding to feature maps at each scale are generated through two fully connected layers and a sigmoid function. The feature maps at each scale are weighted and summed according to the weighting factors to obtain a multi-scale fused feature map.

[0010] Preferably, the dynamic detection head further includes a spatial-aware attention (SPAA) module, which is used to perform the following steps: Deformable convolution is performed on the multi-scale fused feature map to obtain a target deformation adaptive offset feature map; The target deformation adaptive offset feature map is convolved pointwise and normalized using the Sigmoid function to obtain a spatial attention mask; The spatial attention mask is multiplied pixel-by-pixel with the multi-scale fused feature map to obtain the spatially enhanced feature map.

[0011] Preferably, the improved YOLOv11n network outputs nipple category confidence and bounding box coordinates based on the spatial augmentation feature map, and the total loss function is: L_total = L_cls + λ 1 · L_box + λ 2 · L_dfl in L_cls For classifying losses, L_box For bounding box regression loss, L_dfl λ1 and λ2 are the distribution focus loss and the equilibrium hyperparameters.

[0012] Preferably, step 3 includes: The center point coordinates of the nipple bounding box are determined based on the two-dimensional pixel coordinates of the nipple bounding box. The range of the neighborhood is determined based on the coordinates of the center point, and the neighborhood is a rectangle with a predetermined number of pixels centered on the center point; Based on the depth image, determine the depth value of each pixel in the neighborhood; Remove outlier points in the neighborhood whose depth values ​​exceed a preset range, and extract the median depth of the remaining pixels; Based on the device parameters of the image acquisition device, the center point coordinates and the median depth are converted into three-dimensional coordinates in the camera coordinate system according to the following formula: X_cam = (u_c - c_x)·Z / f_x

[0013] Y_cam = (v_c - c_y)·Z / f_y

[0014] Z_cam = Z

[0015] in, X_cam , Y_cam , Z_cam Representing the camera coordinate system X , Y , Z coordinate, u_c , v_c The center points are respectively X , Y coordinate, f_x, f_y Indicates image acquisition device X , Y Directional focal length, c_x , c_y This indicates the coordinates of the principal point of the image acquisition device.

[0016] Preferably, in step 4, the three-dimensional coordinates of the nipple in the camera coordinate system are converted into the target pose in the robot arm base coordinate system through the following steps: Solve for the transformation matrix from the camera coordinate system to the robot arm end effector coordinate system. ; The target pose of the nipple in the coordinate system of the robot arm base is calculated using the following formula. :

[0017] in, express The inverse matrix, This represents the transformation matrix from the robot arm's end effector coordinate system to the robot arm's base coordinate system. This represents the three-dimensional coordinates of the nipple in the camera coordinate system.

[0018] Preferably, the image acquisition device is a TOF depth camera, and the transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system is solved through the following steps: The camera is connected to the end of the robotic arm, and the robotic arm is controlled to move the camera to multiple poses in the space, simultaneously acquiring the corresponding calibration board image and the pose of the end of the robotic arm, wherein the calibration board is fixed in the workspace of the robotic arm. Corner detection is performed on the calibration board image, and the transformation matrix from the calibration board coordinate system to the camera coordinate system is solved. ; The pose of the robotic arm's end effector is transformed to obtain the transformation matrix from the robotic arm's end effector coordinate system to the robotic arm's base coordinate system. ; Construct the system of equations AX=XB, where A represents the transformation matrix of the robotic arm end effector between different poses, B represents the transformation matrix of the camera when observing the same calibration plate in different poses, and X is the transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system. The system of equations was solved using the Danilidis dual quaternion algorithm to obtain the transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system. .

[0019] Another aspect of the present invention provides a dairy cow medicated bath device, comprising: An image acquisition device is used to acquire raw images of the udder region of a dairy cow, the raw images including color and depth images of the udder region, and the image acquisition device is located at the end of a robotic arm; The processor is used to perform the following steps: The color image is input into a trained nipple detection model to identify and segment the two-dimensional pixel coordinates of the nipple bounding box. The nipple detection model is an improved YOLOv11n network, which includes three feature layers from P2 to P4. The detection head of the improved YOLOv11n network is a dynamic head. Based on the depth image, depth values ​​are extracted from the neighborhood of the nipple bounding box, and the two-dimensional pixel coordinates are converted into three-dimensional coordinates in the camera coordinate system, and then into three-dimensional pose in the robot arm base coordinate system. The robotic arm, at its end, is also equipped with a bath actuator; The controller drives the robotic arm to move according to the target pose to perform the medicated bath operation.

[0020] The beneficial effects of this invention are as follows: 1. The teat detection model adopts an improved YOLOv11n network to enhance the accuracy and robustness of two-dimensional teat detection in dairy cows. Based on two-dimensional detection, depth information is integrated to achieve accurate acquisition of the three-dimensional coordinates of the teat.

[0021] 2. The nipple detection model of this invention reconstructs a feature pyramid for small targets, specifically cow nipples. Addressing the small size of cow nipples, the original P5 large-target feature layer of YOLOv11n is removed, and a high-resolution P2 feature layer based on a shallow backbone network is added, allowing the model to focus on small target perception. This improves the accuracy of small target detection while reducing model size, balancing accuracy and lightweight design. Furthermore, the nipple detection model employs a DynamicHead detection head with a multi-attention mechanism, integrating SAA and SPAA modules. The SAA module adaptively weights feature responses at various scales, while the SPAA module dynamically adjusts the receptive field sampling position based on deformable convolution. The synergistic effect of these two modules significantly improves nipple detection accuracy under complex lighting conditions and varying nipple reflectivity.

[0022] 3. Precise control of image acquisition timing using limit switch triggering shortens response time and reduces acquisition delays caused by continuous turntable movement. This trigger-based acquisition method performs image acquisition and subsequent processing only when the cow is in position, reducing the overall computation cycle and avoiding processing invalid frames during idle periods.

[0023] 4. By using eye-in-hand calibration, the three-dimensional coordinates of the nipple in the camera coordinate system are transformed to the coordinate system of the robotic arm base, providing a target pose that can be directly executed for the automated drug bath robotic arm.

[0024] 5. Automated and precise medicated bathing eliminates the need for human contact with dairy cows throughout the process, thus eliminating safety hazards for operators.

[0025] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0026] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same parts in the exemplary embodiments of the invention.

[0027] Figure 1 This is a flowchart of a method for bathing dairy cows according to an exemplary embodiment of the present invention; Figure 2 This is a schematic diagram of the feature pyramid structure of a nipple detection model provided according to an exemplary embodiment of the present invention; Figure 3 This is a schematic diagram of the operation flow of the scale-aware attention module of the dynamic detection head according to an exemplary embodiment of the present invention. Figure 4 This is a schematic diagram of the operation flow of the spatial perception attention module of the dynamic detection head provided according to an exemplary embodiment of the present invention. Detailed Implementation

[0028] Preferred embodiments of the invention will now be described in more detail. While preferred embodiments of the invention are described below, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0029] This invention addresses the issue of insufficient teat localization accuracy in automated pre- and post-milking teat bathing scenarios for dairy cows. It improves the 2D teat detection performance in complex pasture environments by reconstructing a feature pyramid for small targets and enhancing the multi-attention detection head of the YOLOv11n detection network. Furthermore, it integrates a TOF depth camera to acquire the 3D coordinates of the teat and employs a dual quaternion hand-eye calibration method to transform these coordinates into the robotic arm's base coordinate system, providing the teat bathing robotic arm with a directly executable target pose.

[0030] Please refer to Figure 1 , Figure 1 A flowchart of a method for medicated bathing of dairy cows provided in an embodiment of the present invention.

[0031] The method of this embodiment can be applied to the pre-milking and post-milking teat bathing stations in large-scale dairy farms. In this scenario, image acquisition equipment and a robotic arm equipped with a teat bathing actuator can work together to perform automated teat bathing operations on cows before and after milking. The method includes the following steps 1 to 4.

[0032] Step 1: Acquire the original image of the cow's udder region using an image acquisition device. The original image includes a color image and a depth image of the udder region.

[0033] In this embodiment, the image acquisition device can be a Time-of-Flight (TOF) depth camera. The TOF depth camera can simultaneously output RGB color frames and depth frames in a single acquisition. The two images are factory-calibrated with rigid bodies, ensuring a pixel-by-pixel correspondence; that is, each pixel in the color image can find a corresponding depth value in the depth image. The color image is used for subsequent two-dimensional detection of the teat target, while the depth image provides distance information for calculating the three-dimensional coordinates of the teat. The TOF depth camera can be mounted on the end effector of a robotic arm to acquire images of the udder region from below and behind the cow. The camera's mounting angle and shooting distance can be adjusted according to the spatial layout of the milking parlor.

[0034] In the continuous operation scenarios of large-scale dairy farms, the medicated bathing station needs to automatically initiate image acquisition when the cows arrive to avoid continuous acquisition and redundant calculations during non-target periods. The following describes a trigger-based acquisition method based on the milking turntable and limit switches.

[0035] In some embodiments, the cow stands on the milking turntable. In step 1, the milking turntable triggers a limit switch located at the medicated bath station, thereby activating the image acquisition device to acquire images.

[0036] A milking carousel is a rotatable ring platform where cows stand inside the carousel's enclosure with their tails facing outwards. As the carousel rotates, they sequentially pass through the pre-milking, milking, and post-milking stations. A limit switch is a contact-type position detection element that outputs a switching signal when mechanically contacted by an external trigger. Specifically, the limit switch can be installed on the drive chain of the milking equipment. When the carousel, carrying the cow, rotates to the corresponding position in the milking station, the trigger on the carousel contacts the limit switch and generates an electrical signal. This electrical signal wakes up the image acquisition module, controlling the TOF depth camera to perform a synchronous acquisition. The limit switch can use a mechanical contact triggering method, with a response time that can be controlled to within 5ms, thereby reducing acquisition delays caused by continuous carousel movement. This trigger-based acquisition method allows the system to perform image acquisition and subsequent processing only when the cow is in position, reducing the overall computation cycle and avoiding processing invalid frames during idle periods.

[0037] Step 2: Input the color image into the trained nipple detection model to identify and segment the two-dimensional pixel coordinates of the nipple bounding box. The nipple detection model is an improved YOLOv11n network, which includes three feature layers from P2 to P4. The detection head of the improved YOLOv11n network is a dynamic head.

[0038] The nipple detection model used in this embodiment uses YOLOv11n as the basic detector and improves it in terms of both the feature pyramid level and the detection head to meet the detection requirements of small targets like nipples in a pasture medicated bath scenario. The overall network structure can be divided into three parts: the backbone, the neck of the feature pyramid, and the improved detection head.

[0039] The standard YOLOv11n network's detection head outputs feature maps at three scales: P3, P4, and P5. Taking an input image resolution of 640×640 pixels as an example, the stride of the P3 feature map is 8, corresponding to a spatial size of 80×80; the stride of the P4 feature map is 16, corresponding to 40×40; and the stride of the P5 feature map is 32, corresponding to 20×20. The P5 feature layer has a large receptive field, suitable for detecting targets that occupy a large proportion of the image. However, in a medicinal bath scene, cow teats appear as small cylindrical targets in the image, with an area typically not exceeding 1% of the image area, classifying it as a small target detection task. The P5 feature layer provides insufficient spatial resolution for such small targets and introduces considerable redundant computation.

[0040] The improved scheme in this embodiment adjusts the feature pyramid to three feature layers: P2, P3, and P4. Specifically, the P5 feature layer is removed, and a new P2 feature layer based on a shallow layer of the backbone network is added. The stride of the P2 feature layer is 4, providing twice the resolution in each spatial dimension compared to the P3 feature layer, and the total number of pixels in the feature map is four times that of the P3 feature layer, which can preserve more contour and texture details of small targets. After removing the P5 feature layer, the number of model parameters can be reduced by, for example, 10.91%, reducing the model size and inference time. The neck feature fusion path can adopt a Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) structure to transfer multi-scale semantic information between the three feature layers P2, P3, and P4, so that the P2 feature layer can obtain semantic richness while maintaining spatial refinement.

[0041] refer to Figure 2 This diagram illustrates the feature pyramid structure of the nipple detection model in this embodiment. The backbone network retains the C3k2 and SPPF structures, with improvements focused on the output layers of the feature pyramid. Specifically, the P5 feature layer is removed, and a new high-resolution P2 feature layer is added. The stride of the P2 feature layer is 4, and the feature map size is 1 / 4 of the input image, specifically designed for the detection of small nipples. The improved three feature maps P2, P3, and P4 correspond to the detection tasks of small, medium, and large targets, respectively, and the target scale coverage more closely matches the actual size distribution of nipples in the medicinal bath scene. Figure 2As shown, the P2 feature layer can completely preserve the contour and edge details of the nipple, providing more refined spatial information for subsequent detection heads. Meanwhile, removing the P5 feature layer reduces the number of model parameters, helping to maintain lightweight deployment capabilities.

[0042] Regarding the detection head, this embodiment replaces the standard YOLOv11n decoupling head with a dynamic detection head. The dynamic detection head may include a scale-aware attention module and a spatial-aware attention module. These two modules sequentially process the feature maps output from feature layers P2 to P4, adaptively weighting the features from both the scale and spatial dimensions. The processing procedures of each module are described below.

[0043] In some implementations, the dynamic detection head includes an SAA module, which adaptively weights the feature maps output by the P2, P3, and P4 feature layers through the following steps: performing global average pooling on the feature maps at each scale to obtain channel-level descriptors; concatenating the channel-level descriptors at each scale to obtain a joint vector; generating weight factors corresponding to the feature maps at each scale based on the joint vector through two fully connected (FC) layers and a sigmoid function; and weighting and summing the feature maps at each scale according to the weight factors to obtain a multi-scale fused feature map.

[0044] The SAA module can adaptively adjust the contribution of feature maps at different scales. In the nipple detection scenario, feature maps at scales P2, P3, and P4 correspond to receptive field sizes of different sizes, and the response intensity to the nipple target varies at each scale. (Reference) Figure 3 The processing flow of the SAA module is as follows: Global Average Pooling (GAP) is performed on the feature maps at scales P2, P3, and P4. The spatial dimensions of the feature maps at the three scales are 1 / 4, 1 / 8, and 1 / 16 of the input image, respectively, and the number of channels is C. The mean is calculated along the height and width spatial dimensions of the feature maps, compressing the spatial features of each scale into a channel-level descriptor vector. The channel-level descriptors of the three scales P2, P3, and P4 are concatenated along the channel dimensions to form a joint vector of dimension 1×1×3C. This joint vector can simultaneously contain channel statistical information of the three scales. The joint vector is input into two fully connected layers for dimensionality transformation. The first fully connected layer FC1 compresses the dimension from 3C to C, introducing a non-linear transformation through the ReLU activation function, enabling the network to learn the non-linear relationship between channels at each scale. The second fully connected layer FC2 further compresses the dimension from C to 3, corresponding to one scalar output for each of the three scales P2, P3, and P4. The output is mapped to the range of 0 to 1 using the Sigmoid function, generating the weighting factors α_ for feature maps at each scale (P2, P3, and P4). l( l (∈{P2, P3, P4}). Since the three feature maps P2, P3, and P4 have different spatial dimensions, before performing weighted summation, feature maps P3 and P4 can be upsampled to the same spatial dimension as P2 using bilinear interpolation. Then, weighted summation is performed according to the weight factors of each scale to obtain the multi-scale fused feature map. F_fused Its spatial size is 1 / 4 of the input image, and the number of channels is C. Multi-scale fusion feature map. F_fused It is then fed into the SPAA module for further processing of the spatial dimensions.

[0045] Through the above processing, the SAA module can adaptively adjust the weights of features in each layer according to the scale distribution of the nipple target in the input image, so that the detection model can aggregate the information of the best resolution layer when the nipple target has multiple poses and distance differences, thereby improving the detection accuracy.

[0046] In some implementations, the dynamic detection head also includes a SPAA module, which performs the following steps: performing deformable convolution on the multi-scale fused feature map, performing pointwise convolution on the target deformation adaptive offset feature map and normalizing it using the Sigmoid function to obtain a spatial attention mask; and multiplying the spatial attention mask pixel-wise with the multi-scale fused feature map to obtain a spatial enhancement feature map.

[0047] The SPAA module can further extract spatial target information from the multi-scale fused feature map output by the SAA module. The SPAA module's processing consists of two stages. The first stage is deformable convolution. Unlike standard convolution, which uses a fixed regular grid for sampling positions, deformable convolution can learn a set of two-dimensional offsets for each sampling point, allowing the sampling position of the convolution kernel to be adaptively adjusted according to the input features. The kernel size of deformable convolution can be 3×3. In the nipple detection scenario, due to changes in the cow's posture, udder shape, and shooting angle, the nipple can exhibit different deformations and orientations in the image. Deformable convolution, through adaptive offset, can make the sampling area fit the actual contour shape of the nipple, outputting a target deformation adaptive offset feature map. The second stage is spatial attention weighting. The offset feature map is compressed from multi-channel features into a single-channel importance heatmap through pointwise convolution (1×1 convolution). After normalization by the Sigmoid function, a spatial attention mask is obtained. The value of each spatial position in this mask reflects the probability that the position contains target information. Multiplying the spatial attention mask element-wise with the multi-scale fused feature map output by the SAA module yields a spatially enhanced feature map. This involves using deformable convolutions and pointwise convolutions to generate the spatial attention mask, which is then applied to the original fused feature map to enhance the nipple region signal. Through this processing, the SPAA module can suppress the feature response of the background region, enhance the feature representation of the nipple target region, reduce background noise interference, and improve detection recall and average precision.

[0048] refer to Figure 4 It illustrates the internal processing flow of the SPAA module from a data perspective. The input to the SPAA module is the multi-scale fused feature map output by the SAA module. F_fused Its spatial size is 1 / 4 of the input image, and the number of channels is C. F_ fused After entering the SPAA module, it processes data along two paths. The first path is the attention mask generation path. F_fused First, a 3×3 deformable convolution is applied. The sampling position of the convolution kernel can be adaptively shifted according to the circular contour and edge shape of the nipple to extract the target deformation features. The output spatial size of the deformable convolution remains 1 / 4 of the input image, and the number of channels remains C. Then, a 1×1 pointwise convolution compresses the number of channels from C to 1, generating a single-channel importance heatmap. The value of each pixel in this heatmap reflects the contribution of that spatial location to nipple detection. After the heatmap is normalized to the range of 0 to 1 using the Sigmoid function, a spatial mask M is obtained. Regions close to 1 in the mask indicate that a nipple target may exist at that location, while regions close to 0 indicate that the location is background. The second path is a bypass direct connection path. (The text abruptly ends here.) Figure 4 The original input is shown by the dashed line on the right. F_fusedIt can directly reach the pixel-by-pixel multiplication stage without going through deformable convolution and pointwise convolution processing. The spatial mask M is directly connected to the bypass. F_fused After pixel-by-pixel multiplication, the feature signals of the nipple region are preserved because the mask weights are close to 1, while the feature signals of the background region are suppressed because the mask weights are close to 0, resulting in a spatially enhanced feature map. F_out . F_out Spatial size and number of channels and input F_fused To maintain consistency, the image size is 1 / 4 × C, and then it is fed into the detection head for nipple category determination and location coordinate regression.

[0049] After processing by the SPAA module, the spatially enhanced feature map can be fed into the classification and regression branches of the detection head, respectively. The classification branch outputs the nipple category confidence score, and the regression branch outputs the bounding box coordinates.

[0050] In some implementations, the improved YOLOv11n network outputs nipple class confidence and bounding box coordinates based on spatially enhanced feature maps, with the total loss function being: L_total = L_cls + λ 1 · L_box + λ 2 · L_dfl in L_cls For classifying losses, L_box For bounding box regression loss, L_dfl λ1 and λ2 are the distribution focus loss and the equilibrium hyperparameters.

[0051] The total loss function used in the model training phase consists of three terms. L_cls The classification loss measures the deviation between the model's predicted class confidence and the true label, and can be calculated using the binary cross-entropy with logits loss (BCE). L_box The bounding box regression loss measures the positional and shape deviations between the predicted and ground truth bounding boxes. It can be calculated using the Complete Intersection over Union (CIoU) loss. CIoU, in addition to considering the center distance and aspect ratio difference between the predicted and ground truth boxes, provides a more refined gradient signal compared to the standard IoU loss. L_dflThe Distribution Focal Loss (L_dfl) is used for finer-grained regression of the positions of each edge of the bounding box. L_dfl models the coordinates of each edge of the bounding box as a discrete probability distribution. By constraining the probabilities of the nearest integer positions to the left and right of the target location, it concentrates the predicted distribution near the true boundary, thereby improving the localization accuracy of the bounding box edges. λ1 and λ2 are balancing hyperparameters used to adjust the relative contributions of the three losses to the total loss. During training, λ1 can be set to, for example, 7.5, and λ2 can be set to, for example, 1.5, giving the bounding box regression loss a higher weight in the total loss, prompting the model to prioritize learning accurate bounding box localization. Through joint optimization of the classification loss, bounding box regression loss, and distribution focal loss, the model can simultaneously learn accurate class discrimination and precise bounding box localization.

[0052] The detection performance of a nipple detection model is closely related to the quality and diversity of its training data. The following describes the process of acquiring and processing the training data.

[0053] In some implementations, the training data for the nipple detection model is obtained through the following steps: acquiring sample images using an image acquisition device, including dry and wet color images of nipples of multiple breeds and different lactation stages during the pre- and post-drinking stages, when the cows are standing; marking the nipple boundaries in the sample images using rectangular bounding boxes; performing random rotation, random translation, random brightness and contrast jitter, mosaic augmentation, copy-paste augmentation, and Gaussian noise addition on the marked sample images, and dividing them into training, validation, and test sets in a 7:2:1 ratio.

[0054] Sample images can be acquired during the pre- and post-milking teat bath stages. These stages involve disinfecting and protecting the teats before and after milking, respectively. During these baths, the cows are standing with their teats naturally drooping, and the imaging conditions are consistent with actual teat bathing operations. The acquired sample images can cover multiple breeds (e.g., Holstein, Jersey, etc.) and cows at different lactation stages, while also incorporating various lighting conditions (e.g., normal indoor lighting, strong overhead lighting, low light, etc.) and various occlusion scenarios (e.g., teat occlusion between teats, tail occlusion, etc.). The teat is considered dry before and after milking, with no residual liquid on the surface. It is considered wet after milking, with different liquids adhering to the teat surface due to various reasons. Including images under both dry and wet conditions allows the model to adapt to changes in appearance such as teat surface reflection and droplet occlusion.

[0055] During the annotation stage, a rectangular bounding box can be used to annotate the bounding rectangle of each nipple. All visible nipples in each image need to be annotated.

[0056] During the data augmentation phase, various augmentation operations can be applied to the labeled sample images to expand the diversity of the dataset. Random rotation applies random angles to the images, simulating teat angle shifts caused by changes in cow posture. Random translation randomly shifts the images horizontally and vertically, simulating teats appearing in different positions within the image. Random brightness and contrast jitter (e.g., variations of ±30%) simulates imaging differences under different lighting conditions and at different times in the pasture. Mosaic Augmentation stitches four training images together, exposing the model to more target distribution combinations within the same training sample, improving its ability to perceive small targets. Copy-Paste Augmentation extracts and pastes labeled teat regions from one image into another, increasing the target density of a single image and alleviating the problem of imbalanced teat instance numbers. Gaussian noise addition superimposes random noise following a Gaussian distribution onto the image, enhancing the model's robustness to noise in low-light imaging conditions by a TOF camera.

[0057] The enhanced dataset can be divided into training, validation, and test sets in a 7:2:1 ratio. The training set is used for optimizing and updating model parameters, the validation set is used for hyperparameter tuning and overfitting monitoring during training, and the test set is used to evaluate the model's final detection performance. During the training phase, a stochastic gradient descent (SGD) optimizer can be used, with an initial learning rate set to, for example, 0.01. This learning rate is then gradually reduced using a cosine annealing scheduling strategy, allowing for more refined parameter tuning in the later stages of training.

[0058] Step 3: Extract depth values ​​from the neighborhood of the nipple bounding box based on the depth image, and convert the two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system.

[0059] After obtaining the two-dimensional pixel coordinates of the nipple bounding box in step 2, it is necessary to combine the distance information from the depth image to convert the two-dimensional planar position of the nipple into three-dimensional spatial coordinates. Because TOF depth cameras may be affected by factors such as object edge scattering and multipath reflection during actual acquisition, the depth values ​​of some pixels may exhibit abnormal deviations. Therefore, outliers need to be removed when extracting depth values ​​to obtain robust depth estimates. The specific methods for depth value extraction and coordinate transformation are described below.

[0060] In some implementations, step 3 includes: determining the center point coordinates of the nipple bounding box based on its two-dimensional pixel coordinates; determining the range of the neighborhood based on the center point coordinates, wherein the neighborhood is a rectangle with a predetermined number of pixels centered at the center point; determining the depth value of each pixel within the neighborhood based on the depth image; removing outliers in the neighborhood whose depth values ​​exceed a preset range, and extracting the median depth of the remaining pixels; and converting the center point coordinates and the median depth into three-dimensional coordinates in the camera coordinate system according to the device parameters of the image acquisition device using the following formula: X_cam = (u_c - c_x)·Z / f_x

[0061] Y_cam = (v_c - c_y)·Z / f_y

[0062] Z_cam = Z

[0063] in, X_cam , Y_cam , Z_cam Representing the camera coordinate system X , Y , Z coordinate, u_c , v_c The center point X , Y coordinate, f_x, f_y Indicates the focal length of the image acquisition device. c_x , c_y This indicates the principal point of the image acquisition device.

[0064] The nipple bounding box output in step 2 is represented by the pixel coordinates of its top-left and bottom-right corners. Based on these coordinates, the center point coordinates (u_c, v_c) of the bounding box can be calculated. This center point approximates the position of the nipple on the image plane. A rectangular region of a predetermined number of pixels is selected as the depth extraction neighborhood, centered on this center point. The neighborhood size can be set to, for example, 11×11 pixels. Selecting a neighborhood instead of a single center pixel for depth extraction avoids accidental deviations in depth values ​​caused by noise at a single point.

[0065] Within a defined neighborhood, the depth value of each pixel is read from the depth image. The depth value output by the TOF depth camera is in millimeters, representing the distance from the camera to the corresponding scene point. Since pixels in the bounding box edge region may fall on the transition zone between the nipple and the background, causing abrupt depth changes or scattering anomalies, outlier removal of depth values ​​within the neighborhood is necessary. Specifically, an effective depth range can be set, for example, from 300mm to 2000mm, and depth values ​​exceeding this range are considered invalid and removed. The lower limit of this range corresponds to the minimum working distance between the camera and the nipple, and the upper limit corresponds to the maximum shooting distance the camera can reach in the bath position. After removing outliers, the median of the depth values ​​of the remaining valid pixels is taken to obtain a robust depth estimate Z. The median is less sensitive to outliers than the mean, further suppressing the interference of residual outliers on depth estimation.

[0066] After obtaining the center point coordinates (u_c, v_c) and the depth estimate Z, the pixel coordinates can be converted into 3D coordinates in the camera coordinate system based on the pinhole camera model. The pinhole camera model describes the geometric mapping relationship of a 3D point projected onto the imaging plane through a lens. The conversion formula is as follows: X_cam = (u_c - c_x)·Z / f_x

[0067] Y_cam = (v_c - c_y)·Z / f_y

[0068] Z_cam = Z

[0069] Where f_x and f_y are the focal lengths of the image acquisition device in the horizontal and vertical directions, respectively, and c_x and c_y are the principal point coordinates of the image acquisition device, i.e., the pixel positions of the intersection of the optical axis and the imaging plane. These parameters are collectively referred to as camera intrinsic parameters and can be obtained from the factory calibration of the TOF depth camera. After calculation using the above formula, the three-dimensional coordinates of the nipple in the camera coordinate system can be obtained. P_camera = ( X_cam, Y_cam, Z_cam ).

[0070] Step 4: Convert the 3D coordinates in the camera coordinate system to the target pose in the robot arm base coordinate system, and drive the robot arm with the bath actuator to move according to the target pose to perform the bath operation.

[0071] The 3D coordinates obtained in step 3 are referenced to the camera coordinate system, while the motion planning of the robotic arm requires its base coordinate system as a reference. Therefore, the 3D coordinates of the nipple in the camera coordinate system need to be transformed into the target pose in the robotic arm base coordinate system through coordinate system transformation. This coordinate system transformation involves two sets of transformation matrices: one set is the transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system, which can be solved in advance through hand-eye calibration; the other set is the transformation matrix from the robotic arm end effector coordinate system to the robotic arm base coordinate system, which can be calculated in real time by the robotic arm's forward kinematics. The coordinate transformation method and hand-eye calibration process are introduced below.

[0072] In some implementations, step 4 involves converting the nipple's three-dimensional coordinates in the camera coordinate system to the target pose in the robot arm base coordinate system through the following steps: solving for the transformation matrix from the camera coordinate system to the robot arm end effector coordinate system. Calculate the target pose of the nipple in the coordinate system of the robot arm base using the following formula. : ,in, express The inverse matrix, This represents the transformation matrix from the robot arm's end effector coordinate system to the robot arm's base coordinate system. This represents the three-dimensional coordinates of the nipple in the camera coordinate system.

[0073] The above coordinate transformation formula maps the three-dimensional coordinates of the nipple from the camera coordinate system to the robot arm base coordinate system through two matrix transformations. The transformation matrix from the camera coordinate system to the robot arm end effector coordinate system. The inverse matrix represents the rigid body transformation from the camera coordinate system to the robot arm's end effector coordinate system. Since the TOF depth camera is fixed to the robot arm's end effector in an eye-in-hand manner, the relative pose between the camera and the end effector remains unchanged after installation. It can be used repeatedly after being calibrated once by hand and eye. This is the transformation matrix from the robot arm's end-effector coordinate system to the robot arm's base coordinate system. This matrix can be calculated in real-time by the robot arm's forward kinematics based on the current angles of each joint. P_camera Multiply by left in sequence and This allows us to obtain the target pose of the nipple in the coordinate system of the robot arm base. P_ base .

[0074] In some implementations, the image acquisition device is a TOF depth camera. The transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system is solved through the following steps: the camera is connected to the end effector of the robotic arm, and the robotic arm is controlled to move the camera to multiple poses in space, simultaneously acquiring the corresponding calibration board image and the pose of the robotic arm end effector, wherein the calibration board is fixed in the workspace of the robotic arm; corner detection is performed on the calibration board image, and the transformation matrix from the calibration board coordinate system to the camera coordinate system is solved. The pose of the robotic arm's end effector is transformed to obtain the transformation matrix from the robotic arm's end effector coordinate system to the robotic arm's base coordinate system. A system of equations AX = XB is constructed, where A represents the transformation matrix of the robotic arm's end effector between different poses, B represents the transformation matrix corresponding to the camera observing the same calibration plate in different poses, and X is the transformation matrix from the camera coordinate system to the robotic arm's end effector coordinate system. The system of equations is solved using the Danilidis dual quaternion algorithm to obtain the transformation matrix from the camera coordinate system to the robotic arm's end effector coordinate system. .

[0075] The purpose of hand-eye calibration is to determine the rigid body transformation relationship between the camera coordinate system and the robot arm's end effector coordinate system. During calibration, a checkerboard calibration plate can be fixed in a stable position within the robot arm's workspace, keeping the calibration plate stationary within the camera's field of view. The robot arm is controlled to move the end effector camera to multiple different poses within the space; the number of poses can be set to, for example, no fewer than 12 sets, to provide sufficient constraint information. At each pose, the RGB image of the calibration plate and the pose data of the robot arm's end effector are recorded simultaneously.

[0076] Corner detection is performed on the calibration board images acquired at various poses, and the pixel coordinates of the checkerboard corner points are extracted. Based on the corner pixel coordinates and the known physical coordinates of the corner points on the calibration board, the transformation matrix from the calibration board coordinate system to the camera coordinate system at each acquisition pose can be solved using the Perspective-n-Point (PnP) algorithm. Simultaneously, the pose data of the robotic arm's end effector at various poses is transformed to obtain the transformation matrix from the robotic arm's end effector coordinate system to the robotic arm's base coordinate system.

[0077] Based on the above data, the hand-eye calibration equation system AX=XB can be constructed. Here, A is the relative transformation matrix of the robotic arm's end effector between two adjacent acquisition poses, which can be derived from the respective transformations of the two poses. Divide to obtain B, which is the relative transformation matrix of the camera when observing the same calibration board in the two corresponding poses. This matrix can be obtained from the respective transformations of the two poses. The result is obtained by division. X is the transformation matrix to be solved from the camera coordinate system to the robot arm end effector coordinate system. Each pair of adjacent poses can generate a set of equations AX=XB, and multiple sets of poses can be combined to form an overdetermined set of equations.

[0078] This embodiment employs the Danilidis dual quaternion algorithm to solve the aforementioned system of equations. Dual quaternions can represent rotation and translation as a single eight-dimensional parameter, allowing the rotation and translation parameters to be solved jointly in the same optimization process. Specifically, the rotation and translation components of the transformation matrix X are encoded as the real and dual parts of the dual quaternion, respectively. The AX=XB equation system is transformed into a linear system of equations with respect to the dual quaternion parameters. The least-squares solution of this linear system of equations is obtained through Singular Value Decomposition (SVD), yielding the dual quaternion representation of X, which is then restored to a 4×4 homogeneous transformation matrix. Compared to the traditional Tsai step-by-step solution method, which calculates rotation first and then translation, the Danilidis dual quaternion algorithm eliminates the cumulative effect of rotation errors propagating to translation parameters in step-by-step solutions, thus improving calibration accuracy.

[0079] Obtain the target pose P_base Then, the robotic arm controller can... P_base The robot arm, equipped with a medicated spray actuator, is guided to move to the corresponding position on the teat to perform the medicated spraying operation. The medicated spray actuator can spray the medicated solution onto the teat, completing the teat disinfection and protection before and after milking.

[0080] The following section uses a typical application scenario to illustrate the complete processing procedure and actual effect of this solution in a large-scale ranch.

[0081] In the milking parlor of a large-scale dairy farm, multiple cows stand in their respective enclosures within the milking carousel. Before and after milking, they pass through the medicated bath station as the carousel rotates. When the carousel carries a cow, either before or after milking, to the medicated bath station, a trigger on the carousel engages a limit switch. The limit switch outputs an electrical signal within, for example, 5ms, activating the image acquisition module. A TOF depth camera mounted at the end of a robotic arm then simultaneously acquires one RGB color image and one depth image from directly behind and below the cow. The two images have a pixel-by-pixel correspondence.

[0082] Color images are fed into an improved YOLOv11n nipple detection model deployed on a field computing device. This model processes the color images using a three-layer feature pyramid (P2 to P4) and a DynamicHead detection head, outputting the two-dimensional pixel coordinates of the bounding boxes for each nipple. Because the P2 feature layer retains one-quarter of the spatial resolution of the input image, the model has a strong ability to perceive small targets such as nipples. In actual testing, compared to the original YOLOv11n, the improved model achieves improvements in detection accuracy (e.g., 2.1%), recall (e.g., 2.5%), and average precision (mAP@0.5) (e.g., 2.2%). Simultaneously, removing the P5 feature layer reduces the number of model parameters by (e.g., 10.91%), improving inference speed and meeting the real-time processing requirements of continuous rotary table operation.

[0083] After obtaining the bounding boxes of each nipple, depth values ​​can be extracted from the center neighborhood of each bounding box. Using an 11×11 pixel neighborhood as a range, outliers with depth values ​​outside the valid range are removed, and the median is taken to obtain a robust depth estimate for each nipple. Combined with the intrinsic parameters of the TOF camera, the center pixel coordinates and depth values ​​of each nipple are converted into 3D coordinates in the camera coordinate system using a pinhole camera model.

[0084] The 3D coordinates are then transformed into the target pose in the robot arm base coordinate system through a pre-calibrated hand-eye transformation matrix and the robot arm's forward kinematics transformation. The hand-eye transformation matrix is ​​pre-solved using the Danilidis dual quaternion algorithm, and the rotation and translation parameters are jointly calibrated in the same optimization process. The calibration operation can be completed in, for example, within 15 minutes using a visualization toolbox, which is significantly shorter than the traditional manual method, which takes, for example, 60 minutes. In actual testing, the maximum error of the hand-eye calibration can be controlled within, for example, 4.4 mm, and 84.09% of the calibration error is concentrated in the range of 0.3 mm to 2.6 mm.

[0085] After receiving the target pose, the robotic arm controller plans the motion trajectory and drives the teat spray actuator to move to the corresponding teat position, sequentially spraying the teats with medication. In actual testing, the maximum error of the system's output teat 3D positioning can be controlled within, for example, 5.1 mm, and 85.25% of the positioning error is concentrated in the range of 1.6 mm to 3.6 mm, meeting the accuracy requirements of teat spraying operations. Because the teat spray actuator can accurately aim at the teats to spray medication, workers do not need to have close contact with the cows during the entire teat spraying process, eliminating the safety risk of being kicked by the cows during manual operation.

[0086] In summary, this solution improves the accuracy of 2D nipple detection by reconstructing a feature pyramid for small targets and using a multi-attention detection head. It then combines this with a TOF depth camera to acquire 3D coordinates, which are transformed to the robotic arm's base coordinate system via dual quaternion hand-eye calibration. This achieves an automated drug bathing process from image acquisition to execution. While maintaining a lightweight model, this solution balances positioning accuracy and operational safety, making it suitable for continuous drug bathing operations in large-scale ranches.

[0087] Another embodiment of the present invention also provides a dairy cow medicated bath device, comprising: An image acquisition device is used to acquire raw images of the cow's udder region. The raw images include color and depth images of the udder region. The image acquisition device is located at the end of a robotic arm. The processor is used to perform the following steps: A color image is input into a trained nipple detection model to identify and segment the two-dimensional pixel coordinates of the nipple bounding box. The nipple detection model is an improved YOLOv11n network, which includes three feature layers from P2 to P4. The detection head of the improved YOLOv11n network is a dynamic head. Depth values ​​are extracted from the neighborhood of the nipple bounding box based on the depth image, and the two-dimensional pixel coordinates are converted into three-dimensional coordinates in the camera coordinate system, and then converted into the target pose in the robot arm base coordinate system. The robotic arm has a medicated bath actuator at its end. The controller drives the robotic arm to move according to the target pose and perform the medicated bath operation.

[0088] Other details of this embodiment are as described above and will not be repeated here.

[0089] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for medicated bathing dairy cows, characterized in that, include: Step 1: Acquire raw images of the cow's udder region using an image acquisition device. The raw images include color and depth images of the udder region. Step 2: Input the color image into the trained nipple detection model to identify and segment the two-dimensional pixel coordinates of the nipple bounding box. The nipple detection model is an improved YOLOv11n network, which includes three feature layers from P2 to P4. The detection head of the improved YOLOv11n network is a dynamic head. Step 3: Extract depth values ​​from the neighborhood of the nipple bounding box based on the depth image, and convert the two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system; Step 4: Convert the three-dimensional coordinates in the camera coordinate system to the target pose in the robot arm base coordinate system, and drive the robot arm with the drug bath actuator to move according to the target pose to perform the drug bath operation.

2. The method for medicated bathing dairy cows according to claim 1, characterized in that, The cow stands on the milking turntable. In step 1, the milking turntable triggers the limit switch located at the medicated bath station, thereby activating the image acquisition device to acquire images.

3. The method for medicated bathing dairy cows according to claim 1, characterized in that, It also includes obtaining the training data for the nipple detection model through the following steps: The sample images are acquired using an image acquisition device. The sample images include color images of dry and wet teats of multiple breeds and dairy cows at different lactation stages, during the pre- and post-drinking process and when the cows are standing. The nipple boundary is marked in the sample image using a rectangular bounding box; The labeled sample images are subjected to random rotation, random translation, random brightness and contrast jitter, mosaic augmentation, copy-paste augmentation, and Gaussian noise addition, and are divided into training set, validation set and test set in a ratio of 7:2:

1.

4. The method for medicated bathing dairy cows according to claim 1, characterized in that, The dynamic detection head includes a scale-aware attention module, which adaptively weights the feature maps output by the P2, P3, and P4 feature layers through the following steps: Global average pooling is performed on feature maps at each scale to obtain channel-level descriptors; The channel-level descriptors at each scale are concatenated to obtain a joint vector; Based on the joint vector, weight factors corresponding to feature maps at each scale are generated through two fully connected layers and a sigmoid function. The feature maps at each scale are weighted and summed according to the weighting factors to obtain a multi-scale fused feature map.

5. The method for medicated bathing dairy cows according to claim 4, characterized in that, The dynamic detection head also includes a spatial awareness attention module, which is used to perform the following steps: Deformable convolution is performed on the multi-scale fused feature map to obtain a target deformation adaptive offset feature map; The target deformation adaptive offset feature map is convolved pointwise and normalized using the Sigmoid function to obtain a spatial attention mask; The spatial attention mask is multiplied pixel-by-pixel with the multi-scale fused feature map to obtain the spatially enhanced feature map.

6. The method for medicated bathing dairy cows according to claim 5, characterized in that, The improved YOLOv11n network outputs nipple category confidence and bounding box coordinates based on the spatial augmentation feature map, and the total loss function is: L_total = L_cls + λ 1 · L_box + λ 2 · L_dfl in L_cls For classifying losses, L_box For bounding box regression loss, L_dfl λ1 and λ2 are the distribution focus loss and the equilibrium hyperparameters.

7. The method for medicated bathing dairy cows according to claim 1, characterized in that, Step 3 includes: The center point coordinates of the nipple bounding box are determined based on the two-dimensional pixel coordinates of the nipple bounding box. The range of the neighborhood is determined based on the coordinates of the center point, and the neighborhood is a rectangle with a predetermined number of pixels centered on the center point; Based on the depth image, determine the depth value of each pixel in the neighborhood; Remove outlier points in the neighborhood whose depth values ​​exceed a preset range, and extract the median depth of the remaining pixels; Based on the device parameters of the image acquisition device, the center point coordinates and the median depth are converted into three-dimensional coordinates in the camera coordinate system according to the following formula: X_cam = (u_c - c_x)·Z / f_x Y_cam = (v_c - c_y)·Z / f_y Z_cam = Z in, X_cam , Y_cam , Z_cam Representing the camera coordinate system X , Y , Z coordinate, u_c , v_c The center points are respectively X , Y coordinate, f_x、f_y Indicates image acquisition device X , Y Directional focal length, c_x , c_y This indicates the principal point coordinates of the image acquisition device.

8. The method for medicated bathing dairy cows according to claim 1, characterized in that, In step 4, the three-dimensional coordinates of the nipple in the camera coordinate system are converted into the target pose in the robot arm base coordinate system through the following steps: Solve for the transformation matrix from the camera coordinate system to the robot arm's end effector coordinate system. ; The target pose of the nipple in the coordinate system of the robot arm base is calculated using the following formula. : in, express The inverse matrix, This represents the transformation matrix from the robot arm's end effector coordinate system to the robot arm's base coordinate system. This represents the three-dimensional coordinates of the nipple in the camera coordinate system.

9. The method for medicated bathing dairy cows according to claim 8, characterized in that, The image acquisition device is a TOF depth camera. The transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system is solved through the following steps: The camera is connected to the end of the robotic arm, and the robotic arm is controlled to move the camera to multiple poses in the space, simultaneously acquiring the corresponding calibration board image and the pose of the end of the robotic arm, wherein the calibration board is fixed in the workspace of the robotic arm. Corner detection is performed on the calibration board image, and the transformation matrix from the calibration board coordinate system to the camera coordinate system is solved. ; The pose of the robotic arm's end effector is transformed to obtain the transformation matrix from the robotic arm's end effector coordinate system to the robotic arm's base coordinate system. ; Construct the system of equations AX=XB, where A represents the transformation matrix of the robotic arm end effector between different poses, B represents the transformation matrix of the camera when observing the same calibration plate in different poses, and X is the transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system. The system of equations was solved using the Danilidis dual quaternion algorithm to obtain the transformation matrix from the camera coordinate system to the robotic arm end effector coordinate system. .

10. A dairy cow medicated bath device, characterized in that, include: An image acquisition device is used to acquire raw images of the udder region of a dairy cow, the raw images including color and depth images of the udder region, and the image acquisition device is located at the end of a robotic arm; The processor is used to perform the following steps: The color image is input into a trained nipple detection model to identify and segment the two-dimensional pixel coordinates of the nipple bounding box. The nipple detection model is an improved YOLOv11n network, which includes three feature layers from P2 to P4. The detection head of the improved YOLOv11n network is a dynamic head. Based on the depth image, depth values ​​are extracted from the neighborhood of the nipple bounding box, and the two-dimensional pixel coordinates are converted into three-dimensional coordinates in the camera coordinate system, and then converted into the target pose in the robot arm base coordinate system. The robotic arm, at its end, is also equipped with a bath actuator; The controller drives the robotic arm to move according to the target pose to perform the medicated bath operation.

Images