A method and a robot device for recognizing and recording the state of a cow's teat by fusing 2D and 3D vision

Abstract

This invention relates to the fields of livestock farming automation and machine vision technology, and particularly to a method and robotic device for identifying and recording the state of dairy cow teats by integrating 2D and 3D vision. The method includes: acquiring three-dimensional image information of the teat region, generating spatial pose information and object numbering relationships; performing directional image acquisition and appearance analysis to obtain appearance state results; combining 3D point clouds for fitting and projection, performing target matching, spatial fusion, and data fusion to generate a comprehensive state result, and then generating a medicated bath path trajectory through consistency screening; executing medicated bath control, and re-acquiring images after execution to verify the medicated liquid coverage result; finally, associating and storing the comprehensive state result, medicated bath execution result, and medicated liquid coverage result as a historical record. This invention, through the fusion of 2D and 3D information and closed-loop verification of execution effects, significantly improves the accuracy of teat state recognition and the quality of medicated bath coverage, achieving traceable individual health management.

Description

Technical Field

[0001] This invention relates to the fields of livestock farming automation and machine vision technology, and in particular to a method and robotic device for recognizing and recording the state of dairy cow teats by integrating 2D and 3D vision. Background Technology

[0002] In the field of livestock farming automation and machine vision technology, existing solutions typically revolve around a medicated bath station, image acquisition device, point cloud acquisition device, control system, and actuator. By acquiring image information or three-dimensional image information of the nipple region, the nipple position is identified, an execution trajectory is generated, and the medicated bath process is completed. However, these solutions have limitations such as the separation between the apparent state result and the geometric structure result, the instability of the object numbering relationship, and the lack of verification of the liquid coverage result after the medicated bath is executed.

[0003] Existing methods often rely on separate image recognition paths or separate 3D point cloud processing paths, or perform overall pose correction before local image acquisition. In the case of medicinal bath execution, these methods are prone to problems such as confusion in the correspondence between single nipple objects and inability to review the liquid coverage status, making it difficult to achieve stable implementation of comprehensive status results and historical record addition.

[0004] To address the joint processing of apparent state results and geometric structure results after medicated bathing, existing technologies generally suffer from insufficient coordination in areas such as time synchronization, coordinate system mapping, target matching, spatial fusion, data fusion, consistency screening of verification sub-modules, control command sending, and associated storage. This makes it difficult to establish a consistent workflow at the medicated bathing station, from the generation of 3D point cloud data, establishment of single teat objects, acquisition of single teat object image information, generation of apparent state results, generation of geometric structure results, generation of comprehensive state results, acquisition of medicated bathing results, and appending of historical records. Consequently, the correspondence between the same single teat object and the processing of the front and rear wheels becomes unclear, and the medicated bathing coverage result after medicated bathing is difficult to continuously correspond with the contour feature information, classification labels, projected coordinate sets, and medicated bathing path trajectory before execution. This affects the continuous use and subsequent processing of dairy cow teat state recognition and recording in breeding operations. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for identifying and recording the state of cow teats by integrating 2D and 3D vision, comprising:

[0006] S100. Obtain three-dimensional image information of the nipple region, perform spatial pose information generation and object numbering relationship establishment processing to obtain three-dimensional point cloud data, spatial pose information and object numbering relationship;

[0007] The three-dimensional image information refers to image information acquired by a depth camera or a binocular camera that reflects the spatial distribution relationship of the nipple region, including depth information, spatial hierarchy information, and positional relationship with the camera coordinate system corresponding to each acquisition position in the nipple region; the three-dimensional point cloud data refers to a set of point clouds converted from the three-dimensional image information, and each point in the point cloud set corresponds one-to-one with the position in the camera coordinate system.

[0008] S200. Based on the spatial pose information and object numbering relationship, perform classification label generation and center coordinate sequence verification to obtain the apparent state result;

[0009] S300. Based on the apparent state results, target matching, spatial fusion and data fusion processing are performed to obtain the medicinal bath path trajectory;

[0010] S400. Based on the medicinal bath path trajectory, control commands are sent and medicinal bath execution is performed, and image acquisition and medicinal liquid coverage result verification are performed to obtain the historical record and updated object number relationship.

[0011] Furthermore, the process of generating spatial pose information and establishing object numbering relationships includes:

[0012] The spatial pose information generation process includes: based on the three-dimensional point cloud data, performing point cloud boundary cleaning and discrete point removal on each single nipple object, generating direction information according to the extension direction of the center coordinate sequence, and generating deflection angle information in combination with the point cloud boundary distribution, forming spatial pose information containing 3D coordinates, the direction corresponding to the center coordinate sequence, and deflection angle information.

[0013] The object numbering relationship establishment process includes: assigning a unique object number to each single nipple object according to the corresponding order of the spatial position of each single nipple object in the world coordinate system and the current shooting position, and recording the object number together with the center coordinate sequence, so that the object number maintains the same calling standard for the same single nipple object in subsequent image acquisition, target matching, path trajectory generation and history record appending processes.

[0014] Furthermore, the process of generating classification labels and verifying the center coordinate sequence includes:

[0015] The classification label generation process includes: labeling and recording the appearance state of a single nipple object;

[0016] The tagged records include color anomaly location markers, contour boundary anomaly markers, and drug liquid coverage location markers;

[0017] The center coordinate sequence verification process includes: checking the correspondence between the center position relationship corresponding to the contour feature information and the center coordinate sequence; when the two correspond continuously, the object numbering relationship remains unchanged and the classification label is written under the object numbering relationship; when the two are not continuous, the single nipple object is marked as an object to be re-image acquired.

[0018] The classification labels and verified contour feature information are merged into an appearance state result with object numbering relationships.

[0019] Furthermore, the preceding processes for target matching, spatial fusion, and data fusion processing also include:

[0020] Point cloud processing, fitting, and projection coordinate set generation are performed, specifically including:

[0021] Read the corresponding point cloud sets in the 3D point cloud data one by one according to the object number relationship, and perform boundary cleaning, discrete point removal and continuous point retention for each point cloud set;

[0022] Centerline fitting, contour boundary fitting, and spatial pose information fitting are performed along the center coordinate sequence direction to generate length information, diameter information, axis direction, tilt angle information, and contour boundary information. The above information together constitute the geometric structure result.

[0023] Based on the coordinate system mapping relationship, the fitted point cloud set is used to generate a projected coordinate set, and the projected coordinate set and the object number relationship are written into the cache.

[0024] Furthermore, the process of target matching, spatial fusion, and data fusion includes:

[0025] First, read the object number relationship in the appearance state result and the object number relationship in the geometric structure result. Then, perform target matching item by item according to the same object number relationship. The target matching first calls the center position relationship between the projection coordinate set and the contour feature information for the first matching, then calls the contour boundary and the contour boundary information for the second matching, and finally calls the stable correspondence between the confidence information and the length information and the diameter information for the third matching.

[0026] When all three matches remain within the allowed range, it is determined that the current appearance result and the geometric structure result belong to the same single nipple object. Then, the spatial fusion process is entered. The spatial fusion process first aligns the contour boundary in the appearance result with the projection coordinate set in the geometric structure result, and then constrains the distribution range of the classification label in the alignment result so that the classification label always falls within the contour boundary range of the single nipple object.

[0027] The length information, diameter information, axis direction, tilt angle information, and contour boundary information are written into the same object numbering relationship to form a comprehensive status result.

[0028] Furthermore, the process of forming the comprehensive state result includes:

[0029] Perform consistency screening and path trajectory generation for the verification submodules, specifically including:

[0030] Read the comprehensive status results one by one according to the object number relationship, and perform consistency checks on four conditions: whether the confidence information is within the allowable range, whether the contour boundary is continuously corresponding to the projected coordinate set, whether the length information, diameter information and axis direction are stable, and whether the classification label position falls within the contour boundary range. Mark the object number relationship that meets all conditions as a pass object, and mark the object that only partially fails to meet the conditions as a secondary precise positioning object.

[0031] For objects, a medicinal bath path trajectory is generated based on their spatial pose information, center coordinate sequence, and contour boundary.

[0032] The medicated bath path trajectory includes the execution order of a single nipple object, the spraying order, the terminal movement trajectory, and the connection relationship between each segment of the path trajectory.

[0033] Furthermore, the process of sending control commands and executing the medicated bath includes:

[0034] The medicated bath path trajectory is decomposed into control instructions that can be called segment by segment by the robot equipment. These instructions are sent sequentially to the end effector according to the object numbering relationship, so that the robot equipment can complete the continuous actions of positioning, stopping, turning and medicated bath treatment according to the control instructions. The object numbering relationship, path trajectory calling time, and end effector completion status are recorded simultaneously and combined into the medicated bath execution result.

[0035] Furthermore, the process of image acquisition and drug coverage result verification includes:

[0036] After the robot device completes the medicine bath processing corresponding to a certain object numbering relationship, it immediately calls the 2D vision module to perform near-field image acquisition on the object numbering relationship, obtains the image information after execution, and then processes the image information after execution according to the image preprocessing, color extraction and morphological processing path. Finally, it performs medicine liquid coverage result verification based on the contour boundary position retained before execution and the color distribution position in the current image.

[0037] The verification of the liquid coverage result includes: checking whether the liquid coverage position in the current image falls within the boundary range of the contour before execution, and checking whether the liquid coverage position is consistent with the end movement range of the liquid bath path trajectory. When the liquid coverage position is continuous, the boundary correspondence is stable, and the coverage range falls within the contour boundary of the current object numbering relationship, the verification is deemed successful. When the liquid coverage position deviates from the contour boundary or the coverage range is obviously insufficient, the object numbering relationship is marked as an object to be precisely located a second time, and the liquid coverage result is generated.

[0038] Furthermore, the process of obtaining the relationship between historical records and object numbers includes:

[0039] The classification label, confidence information, length information, diameter information, axis direction, tilt angle information, and contour boundary information in the comprehensive status result are read according to the object number relationship. Then, the bath execution result and liquid coverage result under the same object number relationship are read, and field alignment processing is performed to generate an object-level record.

[0040] The object-level record includes object number relationships, comprehensive status results, bath execution results, liquid coverage results, path trajectory call time, and current round processing conclusions. When the current object number relationship already exists in the history record, the current round object-level record is appended to the end of the history record corresponding to the object number relationship. When the current object number relationship appears for the first time in the history record, a new history record corresponding to the object number relationship is created, and finally the history record and object number relationship are obtained.

[0041] Furthermore, a cow teat robot device integrating 2D and 3D vision includes: a 3D image information acquisition module, a single teat object creation module, a single teat object image information acquisition module, an appearance state result generation module, a geometric structure result generation module, a comprehensive state result generation module, and a bathing execution and historical record addition module; the modules are connected in sequence to implement the method described in any of the above-mentioned embodiments.

[0042] The key innovations of this invention include:

[0043] (1) By acquiring three-dimensional image information of the nipple region and generating three-dimensional point cloud data, spatial separation and center coordinate sequence generation are performed based on the three-dimensional point cloud data to obtain a single nipple object. Then, spatial pose information is generated and object numbering relationship is established based on the single nipple object. Based on the spatial pose information and object numbering relationship, image information of the single nipple object is obtained, forming a continuous processing link from three-dimensional point cloud data to appearance state result.

[0044] (2) Based on the apparent state result and the geometric structure result, target matching, spatial fusion and data fusion are performed to obtain the comprehensive state result. Based on the comprehensive state result, the consistency screening of the verification sub-module and the path trajectory generation are performed so that the medicinal bath path trajectory corresponds to the comprehensive state result.

[0045] (3) Based on the results of the medicinal bath, image acquisition and medicinal liquid coverage result verification are performed to obtain the medicinal liquid coverage result. Based on the medicinal liquid coverage result and the comprehensive status result, the related storage and historical records are added so that the medicinal liquid coverage result, the comprehensive status result and the object number relationship are in the same record link.

[0046] The following are its main beneficial effects:

[0047] (1) In view of the problem that the image information of the nipple region and the three-dimensional image information are processed separately in the existing solution and the correspondence of single nipple objects is unclear, the present invention establishes the relationship between single nipple objects, spatial pose information and object numbering in a continuous manner, so that the image information of single nipple objects and three-dimensional point cloud data are called under the same object numbering relationship, and the source of the appearance state result and the source of the geometric structure result are kept in correspondence, which is suitable for continuous processing scenarios in the bath station.

[0048] (2) In view of the problem that the apparent state results and geometric structure results are separated in the existing scheme, the present invention forms a comprehensive state result through target matching, spatial fusion and data fusion, so that the contour feature information, classification label, confidence information and length information, diameter information, axis direction, tilt angle information, contour boundary and projection coordinate set are completed in the same processing link. The comprehensive state result can be directly used for consistency screening and path trajectory generation in the verification submodule.

[0049] (3) In view of the problem that the existing solution lacks verification of the liquid coverage result after the execution of the medicinal bath, the present invention makes the liquid coverage result correspond to the medicinal bath path trajectory, contour feature information and comprehensive status result before execution by image acquisition and liquid coverage result verification based on the execution result of the medicinal bath. The processing status after the execution of the medicinal bath can be continued to be written into the subsequent link.

[0050] (4) In view of the problem that the path trajectory of the medicine bath in the existing scheme is mostly generated by a single image information or a single spatial information, the present invention performs consistency screening of the comprehensive state result by verifying the sub-module and then generates the path trajectory of the medicine bath, so that the path trajectory of the medicine bath is based on the processing of the appearance state result and the geometric structure result together, and the medicine bath execution result is continuous with the preceding recognition link.

[0051] (5) In view of the problem that the recorded content is separated from the execution process and the subsequent processing lacks a continuous basis in the existing scheme, the present invention, through associated storage and historical record addition, makes the relationship between the drug liquid coverage result, the comprehensive status result and the object number continuously retained in the same historical record, and the recording relationship of the same single teat object in the front and rear wheel processing can be continued, which facilitates the continuous use and subsequent processing of dairy cow teat status identification and recording in breeding operations. Attached Figure Description

[0052] Figure 1 A flowchart illustrating a method for recognizing and recording the state of a cow's teats that integrates 2D and 3D vision, provided as an embodiment of this application;

[0053] Figure 2 This is a structural block diagram of a cow teat robot device that integrates 2D and 3D vision, provided as an embodiment of this application. Detailed Implementation

[0054] Example 1: Refer to Figure 1 This is a flowchart illustrating a method for recognizing and recording the state of a cow's teats that integrates 2D and 3D vision, provided by an embodiment of the present invention. The process may include at least steps S100-S400:

[0055] S100. Obtain three-dimensional image information of the nipple region, perform spatial pose information generation and object numbering relationship establishment processing to obtain three-dimensional point cloud data, spatial pose information and object numbering relationship;

[0056] S200. Based on the spatial pose information and object numbering relationship, perform classification label generation and center coordinate sequence verification to obtain the apparent state result;

[0057] S300. Based on the apparent state results, target matching, spatial fusion and data fusion processing are performed to obtain the medicinal bath path trajectory;

[0058] S400. Based on the medicinal bath path trajectory, control commands are sent and medicinal bath execution is performed, and image acquisition and medicinal liquid coverage result verification are performed to obtain the historical record and updated object number relationship.

[0059] Step S100 includes at least steps S110-S130:

[0060] S110. Obtain the three-dimensional image information of the nipple region, perform three-dimensional point cloud data generation processing, and obtain three-dimensional point cloud data.

[0061] Specifically, the input source for this section is the teat area in the medicated bath station, and the execution entities are the 3D vision module and the data processing unit connected to it. The 3D vision module is a data acquisition unit set at a fixed position in the medicated bath station. The 3D vision module includes a depth camera or a binocular camera, used to acquire three-dimensional image information of the teat area when the cow enters the medicated bath station and is within the image acquisition range.

[0062] The three-dimensional image information refers to the image information acquired by the depth camera or binocular camera that reflects the spatial distribution relationship of the nipple region. Its content includes the depth information, spatial hierarchy information and positional relationship with the camera coordinate system corresponding to each acquisition position in the nipple region.

[0063] The three-dimensional point cloud data refers to the point cloud set obtained by converting the three-dimensional image information. Each point in the point cloud set corresponds one-to-one with the position in the camera coordinate system and serves as the basic input for subsequent spatial separation and point cloud processing.

[0064] During operation, after the control system starts the current round of processing at the bath station, it calls the joint calibration and hand-eye calibration results completed in the previous step, and calls the camera intrinsic parameter matrix and distortion parameters to correct the original three-dimensional image information acquired by the depth camera or binocular camera. When using a binocular camera, the left and right images are first matched according to the binocular matching and correction relationship, and then the depth information is formed by combining the shooting position. When using a depth camera, the depth information is directly read and matched with the camera coordinate system.

[0065] Subsequently, the data processing unit converts the corrected 3D image information into the 3D point cloud data, and during the conversion process, it removes discrete points and invalid points that are significantly outside the acquisition range of the nipple region, while retaining the point cloud set that continuously corresponds to the nipple region.

[0066] Understandably, in one engineering embodiment, the 3D vision module is mounted on a fixed bracket below the tawny bathing station, with the acquisition direction facing the teat area. When a cow enters the tawny bathing station and stops at the predetermined photographing position, the control system triggers a three-dimensional image information acquisition, and then immediately executes the three-dimensional point cloud data generation process to generate a set of 3D point cloud data corresponding to the current photographing position. This 3D point cloud data is recorded in this section as the output field name "3D Point Cloud Data," which is directly used as the input field of S120, and continues to be saved as the basic input for point cloud processing and fitting processing in the subsequent S310.

[0067] S120. Based on the three-dimensional point cloud data, perform spatial separation and center coordinate sequence generation processing to obtain a single nipple object.

[0068] Specifically, the input source for this section is the three-dimensional point cloud data output by S110, and the execution entity is the point cloud processing module. Spatial separation refers to separating adjacent but independent point cloud sets within the nipple region according to spatial continuity, contour boundary continuity, and centerline distribution relationships within the point cloud set corresponding to the three-dimensional point cloud data.

[0069] The central coordinate sequence refers to a set of 3D coordinates arranged sequentially along the extension direction of a single nipple object, from the end closest to the breast to the end furthest from the breast. This set of 3D coordinates is used to represent the changing relationship of the central position of a single nipple object.

[0070] The single nipple object refers to an object unit that, after spatial separation, has an independent point cloud boundary, a continuous central coordinate sequence, and can independently participate in subsequent image acquisition and target matching processing.

[0071] During runtime, the point cloud processing module first reads the three-dimensional point cloud data, limits the range of the point cloud set, removes point cloud parts that are obviously unrelated to the nipple region, and then searches for continuously distributed regions in the remaining point cloud set. For regions that are interconnected but have different boundaries, the point cloud processing module separates them again according to the change position of the boundary and the change position of the center line direction, thereby forming multiple independent candidate regions.

[0072] Furthermore, the point cloud processing module performs center coordinate sequence generation processing on each candidate region. Specifically, it extracts continuous 3D coordinates according to the spatial extension order of the candidate region from the near end to the far end, and records the continuous 3D coordinates as the center coordinate sequence of the candidate region. When a candidate region has many boundary interruptions, discontinuous center coordinate sequence, or too small point cloud set, the point cloud processing module marks the candidate region as an invalid region and does not output it as the single nipple object.

[0073] For candidate regions that have passed the above processing, the point cloud processing module solidifies them into the single nipple object and keeps the relationship between its corresponding center coordinate sequence and the point cloud boundary unchanged, so that it can be called again in subsequent spatial pose information generation and processing.

[0074] Understandably, in the engineering embodiment, if there are four separable nipple point cloud regions in the current acquisition image of the medicated bath station, the point cloud processing module performs spatial separation and center coordinate sequence generation processing on the four regions respectively to generate four single nipple objects; if the center coordinate sequence of one region is interrupted due to occlusion, then the region is not output in this round of processing, the control system records the situation and waits for subsequent re-acquisition.

[0075] The single nipple object is recorded as the output field name "single nipple object" in this section and used as the input field of S130; at the same time, the center coordinate sequence corresponding to the single nipple object is also retained, which is used as the object positioning basis when S210 calls the image acquisition position.

[0076] S130. Based on the single nipple object, perform spatial pose information generation and object numbering relationship establishment processing to obtain spatial pose information and object numbering relationship.

[0077] Specifically, the input source for this section is the single nipple object output by S120, and the execution entities are the point cloud processing module and the control system. The spatial pose information refers to the positional and directional relationships of the single nipple object in the camera coordinate system, end coordinate system, tool coordinate system, or world coordinate system, and its content includes the 3D coordinates of the single nipple object, the direction corresponding to the center coordinate sequence, and the deflection angle information.

[0078] The object numbering relationship refers to the sequential correspondence of multiple single nipple objects in the same round of processing. This sequential correspondence is used to maintain the consistency of the calling criteria for the same single nipple object in subsequent image acquisition, target matching, path trajectory generation, and historical record appending processes.

[0079] During runtime, the point cloud processing module performs spatial pose information generation processing on each of the single nipple objects. First, it reads the point cloud boundary and center coordinate sequence of the single nipple object. Then, based on the hand-eye calibration results and coordinate system mapping relationship, it transforms the single nipple object from the camera coordinate system to the coordinate relationship called by the control system. Subsequently, the point cloud processing module generates the orientation information of the single nipple object based on the extension direction of the center coordinate sequence, and generates the corresponding deflection angle information in combination with the point cloud boundary distribution, finally forming the complete spatial pose information.

[0080] Furthermore, the control system establishes the object numbering relationship according to the corresponding order of the spatial position of each single nipple object in the world coordinate system and the current shooting position. Specifically, it first reads the spatial pose information of each single nipple object, then assigns an object number according to a fixed sorting rule, and records the object number together with the center coordinate sequence of the single nipple object. In subsequent processing, as long as the object numbering relationship is not updated, the same object number always corresponds to the same single nipple object.

[0081] Understandably, in the engineering embodiment, the control system generates four sets of spatial pose information for the four single teat objects output in this round, and establishes a set of object numbering relationships. When the cow undergoes a significant change in body position in the medicated bath station, causing the three-dimensional image information to be re-acquired, the control system re-executes S110 to S130 and replaces the previous round's object numbering relationship with the latest object numbering relationship, so that the object numbering relationship received in S210 always corresponds to the single teat object in the current round.

[0082] Therefore, the field output in this section is named "spatial pose information and object number relationship". This output is directly used as the input for "obtaining the spatial pose information and object number relationship" in S210. At the same time, the object number relationship is also called by S310 as the input for "obtaining the three-dimensional point cloud data and the object number relationship". The spatial pose information continues to participate in the image acquisition position control in S200.

[0083] In summary, this step improves upon the previous one by advancing the processing results of the 3D vision module from general 3D image information of the nipple region to information on individual nipple objects, spatial pose information, and object numbering relationships. This ensures that subsequent steps in S210 receive input with object correspondences, rather than undifferentiated nipple region inputs. Consequently, subsequent image acquisition, target matching, and historical record appending revolve around the same single nipple object, resulting in a more complete processing chain and clearer correspondences between individual nipple objects.

[0084] Step S200 includes at least steps S210-S230:

[0085] S210. Obtain the spatial pose information and object number relationship, perform image acquisition, time synchronization and coordinate system mapping relationship calling processing to obtain single nipple object image information.

[0086] Specifically, the input sources for this section are the spatial pose information and object numbering relationship output by S130, and the execution entities are the control system, the 2D vision module, and the image information processor.

[0087] The image acquisition refers to performing directional acquisition on a single nipple object according to the corresponding 3D coordinates, deflection angle information and shooting position in the spatial pose information, rather than performing uniform acquisition on the entire nipple region.

[0088] The time synchronization refers to corresponding the image acquisition time of the 2D vision module with the three-dimensional image information acquisition time of the preceding 3D vision module, and retaining the acquisition results under the same object numbering relationship within the allowed time range.

[0089] The coordinate system mapping relationship call refers to retrieving the correspondence between the camera coordinate system, end coordinate system, tool coordinate system and world coordinate system from the results of the preceding joint calibration and hand-eye calibration, and converting the spatial pose information of the single nipple object into the shooting position and shooting path that the 2D vision module can directly call.

[0090] During operation, the control system first reads the object numbering relationship, generates a photographing position for each single nipple object corresponding to each object number, and then controls the robot to move the 2D vision module to the corresponding photographing position. When the 2D vision module is installed on the end effector, the control system simultaneously calls the pose correction result to ensure that the acquisition direction of the 2D vision module corresponds to the centerline direction of the single nipple object. When the 2D vision module is set at a fixed position in the bathing station, the image information processor extracts the corresponding region of interest (ROI) in the current image based on the spatial pose information.

[0091] Furthermore, in this step, the time synchronization uses the object numbering relationship as the calling index, that is, a single nipple object under the same object numbering relationship only calls the latest spatial pose information corresponding to it; if the control system determines that the cow's body position in the bathing station has changed, or if the 2D vision module finds that the deflection angle information exceeds the allowable range after reaching the shooting position, it will not continue the current image acquisition, but will record the abnormal state corresponding to the object numbering relationship, and re-call S110 to S130 to generate new spatial pose information and object numbering relationship.

[0092] Understandably, in a practical engineering embodiment, a fixed 3D vision module and a 2D vision module installed near the spraying mechanism are set in the medicated bath station; after S130 has output the four object numbering relationships, the control system first calls the coordinate system mapping relationship for the first single nipple object in the object numbering relationship to obtain the first image position, and then the 2D vision module completes the image acquisition of the first single nipple object, and then completes the image acquisition of the remaining single nipple objects in sequence.

[0093] For each image acquisition, the image information processor writes the image information and its corresponding object number relationship into the cache, and records the result as the output field name "single nipple object image information". The single nipple object image information is directly called as the input field of S220, while retaining its object number relationship for subsequent S230 and S320 to perform corresponding processing of appearance state results and geometric structure results.

[0094] S220. Based on the image information of the single nipple object, perform image preprocessing, color extraction, and morphological processing to obtain contour feature information and confidence information.

[0095] Specifically, the input source for this section is the single nipple object image information output by S210, and the execution entities are the image processing module and the image information processor. The single nipple object image information is image information containing object numbering relationships, image capture location, and time synchronization results, used for subsequent image processing focused solely on this single nipple object. Image preprocessing refers to performing brightness adjustment, boundary cropping, noise removal, and distortion correction on the single nipple object image information to make the boundaries of the single nipple object in the image clearer. Color extraction refers to reading the color distribution information of the single nipple object surface from the single nipple object image information and separately marking the locations with significant color changes.

[0096] The morphological processing refers to performing closing operations, boundary smoothing, and local connectivity processing on the image after image preprocessing and color extraction to remove scattered noise and preserve the continuous contour of the single nipple object. During runtime, the image processing module first reads an image of the single nipple object based on the object numbering relationship preserved in S210, and then completes image preprocessing according to the camera intrinsic matrix and distortion parameters. After image preprocessing, the image processing module performs color extraction based on the region of interest corresponding to the single nipple object, separating locations with significant color changes from their surrounding areas to form a color distribution record. Subsequently, the image processing module performs morphological processing synchronously on the boundary regions in the color distribution record and the original image, performing closing operations on broken edges, removing local noise points, and preserving continuous boundaries.

[0097] Furthermore, after completing morphological processing, the image processing module performs contour extraction on the boundary of the single nipple object and generates corresponding confidence information based on the continuity of the contour boundary, the concentration of color distribution, and the image clarity. The contour feature information refers to the boundary position, boundary continuity, and center position relationship obtained after contour extraction; the confidence information refers to the image processing module's record of the reliability of the contour extraction result.

[0098] If the image processing module finds that the current single nipple object image information has large-area occlusion, severe boundary interruption, or mismatch between the color extraction area and the object number relationship, the confidence information corresponding to the single nipple object is marked as low confidence, and the anomaly is written into the temporary record of the object number relationship for use by S230 when performing center coordinate sequence verification.

[0099] Understandably, in the actual engineering embodiment, after the 2D vision module completes near-field acquisition of the first single nipple object, the image processing module first performs image preprocessing on the image, then extracts the color distribution on the surface of the single nipple object from the image, and then removes surrounding hair and background noise through morphological processing, finally obtaining complete contour feature information and its corresponding confidence information; if the second single nipple object has blurred image edges due to the deviation of the shooting position, the image processing module records the confidence information of the object as a low state and waits for further verification in S230.

[0100] After this step, the contour feature information and confidence information are recorded as the output field name "contour feature information and confidence information". This output is directly called as the input field of S230. At the same time, the contour feature information continues to be used as the boundary reference before execution when the drug coverage result is checked in S420.

[0101] S230. Based on the contour feature information and confidence information, perform classification label generation and center coordinate sequence verification to obtain the appearance state result.

[0102] Specifically, the input sources for this section are the contour feature information and confidence information output by S220, and the object numbering relationship and center coordinate sequence formed in S130 are used as corresponding references. The execution entities are the image processing module, the image information processor and the control system.

[0103] The classification label generation refers to labeling and recording the appearance state of a single nipple object based on the boundary shape, color distribution, and local change positions in the contour feature information. The center coordinate sequence verification refers to checking the correspondence between the center position relationship corresponding to the contour feature information and the center coordinate sequence under the same object numbering relationship in S130 to determine whether the current image acquisition result still belongs to the same single nipple object. The appearance state result refers to the object-level output result composed of the classification label and the verified contour feature information, under the premise that the object numbering relationship remains unchanged.

[0104] During runtime, the image processing module first reads the contour feature information and confidence information under the same object numbering relationship, and performs classification label generation processing according to the preset classification label rules; the classification label rules include at least color anomaly location markers, contour boundary anomaly markers, and liquid coverage location markers, so that the image processing module forms a structured appearance record of the current single nipple object.

[0105] Subsequently, the image information processor calls the center coordinate sequence stored in S130 to check the center position relationship in the current contour feature information. If the center position relationship corresponding to the current contour feature information corresponds continuously to the center coordinate sequence, the object numbering relationship remains unchanged, and the current classification label is written under the object numbering relationship. If the two are not continuous, or the current confidence information is lower than the calling threshold, the control system marks the single nipple object as an object to be re-image acquired, and can trigger S210 to re-execute image acquisition.

[0106] Furthermore, this step does not directly modify the object numbering relationship, but maintains a stable correspondence between the same single nipple object in S100 and S200 through center coordinate sequence verification. Thus, when S320 subsequently calls the appearance state result and the geometric structure result for target matching, the appearance state result already has a clear object numbering relationship and is no longer an independent image detection result.

[0107] Understandably, in the actual engineering embodiment, after the image processing module generates a classification label for the contour feature information of the first single teat object, it calls the center coordinate sequence of the first single teat object for verification; when the verification is passed, the classification label, contour feature information and confidence information are merged into the appearance state result of the first single teat object; when the current contour boundary of the third single teat object no longer corresponds to the original center coordinate sequence due to the slight movement of the cow, the control system does not output the appearance state result of the object, but returns to S210 to re-execute image acquisition.

[0108] After this step, the apparent state result is recorded as the output field name "apparent state result" and is directly used by S320 as an input field; at the same time, the correspondence between the apparent state result and the object number is retained and used by S430 for associated storage and historical record appending.

[0109] In summary, this step further consolidates the single nipple object image information obtained in S210 into an apparent state result with object numbering relationships. A center coordinate sequence check is added before output, ensuring that the subsequent S320 receives input with confirmed object correspondence. Compared to existing processing chains that rely solely on image contours or object detection boxes for direct execution, this step maintains a continuous correspondence between the image processing result and preceding spatial pose information. The preceding and following call relationships for single nipple objects are more stable, and the source of subsequent comprehensive state results is clearer.

[0110] Step S300 includes at least steps S310-S330:

[0111] S310. Obtain the relationship between the three-dimensional point cloud data and the object number, perform point cloud processing, fitting and projection coordinate set generation processing to obtain the geometric structure result.

[0112] Specifically, the input sources for this section are the 3D point cloud data output by S110 and the object numbering relationship output by S130, and the execution entities are the point cloud processing module and the data processing unit.

[0113] The point cloud processing refers to performing object-level processing on the spatially separated point cloud set, so that each object number corresponds to an independent point cloud set, without mixing the entire nipple region. The fitting refers to continuously fitting the shape orientation, centerline distribution, and boundary extension relationship of the single nipple object based on its point cloud boundary, center coordinate sequence, and spatial pose information under the same object numbering relationship, ensuring the geometric position of the single nipple object in the 3D point cloud data remains intact. The projected coordinate set refers to the coordinate set obtained by projecting the fitted single nipple object point cloud set onto the image processing path according to the invoked coordinate system mapping relationship. This coordinate set maintains the same object numbering relationship when performing target matching with the subsequent appearance state results in S320.

[0114] During runtime, the point cloud processing module first reads the corresponding point cloud sets in the 3D point cloud data one by one according to the object numbering relationship, and then performs boundary cleaning, discrete point removal, and continuous point retention processing on each point cloud set. For point cloud sets with many boundary interruptions, the point cloud processing module first calls the previous center coordinate sequence for supplementation and verification, and then performs fitting processing. For point cloud sets with low point cloud density but clear object numbering relationships, the point cloud processing module retains the centerline direction and boundary position of the point cloud set, and does not directly remove it, but marks it as an object to be screened by the subsequent verification submodule.

[0115] Furthermore, the fitting process revolves around at least three aspects: first, fitting the centerline of the single nipple object along the direction of the center coordinate sequence; second, fitting the contour boundary of the single nipple object along the boundary position; and third, fitting the spatial pose information between the proximal and distal ends of the single nipple object. Through the above fitting process, the point cloud processing module generates the length information, diameter information, axial direction, tilt angle information, and contour boundary information of the single nipple object.

[0116] The above information together constitutes the minimum set in the geometric structure result. Among them, the length information, diameter information, and axis direction are the core parameters used by the present invention for subsequent spatial fusion and path trajectory generation. The tilt angle information and contour boundary information are preferred parameters. When the shooting position changes significantly or the single nipple object is deflected, it continues to participate in subsequent processing.

[0117] Subsequently, the point cloud processing module generates a corresponding set of projected coordinates from the fitted point cloud set based on the coordinate system mapping relationship between the camera coordinate system and the image coordinate system, and writes the set of projected coordinates and the object number relationship into the cache. If the set of projected coordinates under the same object number relationship exceeds the valid range of the previous single nipple object image information, the data processing unit records the anomaly and retains the object number relationship, and hands it over to S320 to continue to perform target matching and spatial fusion judgment.

[0118] Understandably, in an operational engineering embodiment, after completing S210 to S230 within the medicated bath station, the system has obtained the apparent state results of four single nipple objects. At this time, the point cloud processing module continues to retrieve the point cloud sets corresponding to the four object number relationships from the three-dimensional point cloud data output by S110, performs fitting processing on each set of point cloud data and generates a set of projected coordinates. Among them, the point cloud set of the first single nipple object has continuous boundaries, and the point cloud processing module directly generates length information, diameter information and axis direction. The point cloud set of the second single nipple object has a slight interruption due to local occlusion. The point cloud processing module calls the previous center coordinate sequence and continues to complete the fitting processing. The point cloud set of the third single nipple object has a large deflection, so the tilt angle information and contour boundary information are retained for subsequent S330 calls.

[0119] After processing in this section, the geometric structure result is recorded as an output field name and directly used as input for the "Geometric Structure Result" in S320. Although the projection coordinate set is included in the geometric structure result, it is read separately when S320 performs target matching and spatial fusion. At the same time, it can still be used as a spatial boundary reference before execution when the liquid coverage result is checked in S420.

[0120] S320. Based on the apparent state result and the geometric structure result, target matching, spatial fusion and data fusion processing are performed to obtain a comprehensive state result.

[0121] Specifically, the input sources for this section are the apparent state result output by S230 and the geometric structure result output by S310, and the execution entities are the data fusion module, the image information processor, and the point cloud processing module. The target matching refers to, under the condition that the object numbering relationship remains unchanged, matching the contour feature information, classification label, and confidence information in the apparent state result with the projection coordinate set, length information, diameter information, axis direction, and contour boundary in the geometric structure result, thereby determining whether the apparent state result and the geometric structure result originate from the same single nipple object.

[0122] The spatial fusion refers to, after target matching is successful, placing the contour boundaries in the image processing path and the spatial pose information in the point cloud processing path under the same object numbering relationship for positional correspondence, so that the boundary positions in the appearance state results and the boundary positions in the geometric structure results form a unified relationship. The data fusion refers to merging the classification labels, confidence information, length information, diameter information, axis direction, tilt angle information, and contour boundary information under the same object numbering relationship into a set of object-level records.

[0123] During runtime, the data fusion module first reads the object numbering relationship from the appearance state result, then reads the object numbering relationship from the geometric structure result, and performs target matching item by item according to the same object numbering relationship. During target matching, the data fusion module prioritizes the center position relationship between the projected coordinate set and the contour feature information for the first matching, then calls the contour boundary and contour boundary information for the second matching, and finally calls the stable correspondence between the confidence information and the length and diameter information for the third matching. When all three matchings remain within the allowable range, the data fusion module determines that the current appearance state result and the current geometric structure result belong to the same single nipple object and enters spatial fusion processing. When the first matching passes but the second matching fails, the data fusion module does not directly delete the current object numbering relationship, but retains it and records it as an object to be verified, for further judgment by the consistency screening in the verification submodule of S330.

[0124] Furthermore, the spatial fusion process is not simply a data overlay. Instead, it first aligns the contour boundaries in the apparent state result with the projected coordinate sets in the geometric structure result. Then, it constrains the distribution range of the classification labels within this alignment result, ensuring that the labels always fall within the contour boundary of the single nipple object. Subsequently, the data fusion module writes the length, diameter, axial direction, and tilt angle information into the same object numbering relationship, forming a continuous record at the object level. The resulting comprehensive state result is not a single detection box result, nor a separate image processing result or point cloud processing result, but rather a comprehensive output that binds the apparent state result and the geometric structure result according to the same object numbering relationship.

[0125] Understandably, in a practical engineering embodiment, if the appearance result of the first single nipple object shows clear boundaries and stable classification labels, and the geometric structure result of the first single nipple object shows that the set of projected coordinates is consistent with the boundary range, and the length and diameter information are continuous, then the data fusion module will directly fuse the two into a comprehensive state result of the first single nipple object; if the appearance result of the second single nipple object has relatively clear boundaries, but the set of projected coordinates in the geometric structure result is slightly offset, then the data fusion module will first record the object numbering relationship and enter the object state to be verified, and will not directly discard it in this section; if the classification label of the third single nipple object shows that there is a drug coverage location mark, and the outline boundary and axis direction in the geometric structure result are stable, then the data fusion module will complete the full fusion under the same object numbering relationship.

[0126] After processing in this section, the comprehensive status result is recorded as an output field name and directly used as input for the "Comprehensive Status Result" in S330. At the same time, the object number relationship in the comprehensive status result continues to be used by S430 for associated storage and historical record appending.

[0127] S330. Based on the comprehensive status results, perform consistency screening and path trajectory generation processing for the verification submodule to obtain the medicinal bath path trajectory.

[0128] Specifically, the input source for this section is the comprehensive state result output by S320, and the execution entities are the verification submodule, the control system, and the image information processor. The consistency screening of the verification submodule refers to performing a unified check on the object numbering relationship, confidence information, contour boundary, projected coordinate set, length information, diameter information, axis direction, and classification label within the comprehensive state result, thereby distinguishing between single nipple objects that can directly enter the medicated bath and those that require re-image acquisition or repositioning. The path trajectory generation process refers to generating the robot's execution path, spraying sequence, and end effector trajectory based on the comprehensive state result that has passed the consistency screening of the verification submodule, according to the object numbering relationship, spatial pose information, and contour boundary, and uniformly defining this result as the medicated bath path trajectory.

[0129] During runtime, the verification submodule first reads the comprehensive status results one by one according to the object number relationship, and performs consistency screening based on four conditions: First, whether the confidence information is within the allowable range; Second, whether the contour boundary maintains a continuous correspondence with the projection coordinate set; Third, whether the length information, diameter information and axis direction remain stable within the spatial pose information of the same single nipple object; Fourth, whether the position of the classification label falls within the contour boundary range of the single nipple object.

[0130] When all four conditions are met, the verification submodule marks the object numbering relationship as a passed object; when only some conditions are not met, the verification submodule marks the object numbering relationship as a secondary precise positioning object and sends the record to the control system; when multiple conditions are not met at the same time, the control system marks the object numbering relationship as a re-image acquisition object and subsequently returns to S210 to call the single nipple object image information.

[0131] Furthermore, for objects passing through, the control system performs path trajectory generation processing based on their spatial pose information, center coordinate sequence, and contour boundary. In this processing, the control system first determines the execution order of single nipple objects based on object numbering relationships, then determines the orientation relationship of the spraying mechanism based on axis direction and tilt angle information, and subsequently determines the coverage range of the path trajectory based on contour boundary and length information. The execution paths corresponding to each passing object are then spliced ​​together into a continuous medicated bath path trajectory. For objects requiring secondary precise positioning, the control system retains the current object numbering relationships and overall state results, but does not directly enter the medicated bath execution. Instead, it triggers local repositioning based on their spatial pose information and re-enters the consistency screening of the verification submodule after repositioning.

[0132] Understandably, in a practical engineering embodiment, if the first and third objects among the four single nipple objects pass the consistency screening, the control system first generates the corresponding spraying sequence based on the center coordinate sequence of the first and third objects, and then forms two segments of medicated bath path trajectory based on their contour boundaries; the second object is marked as a secondary precise positioning object because there is a deviation between the projected coordinate set and the contour boundary; the fourth object is marked as a re-image acquisition object because the confidence information and classification label position are both abnormal, and waits to return to S210 to re-call the single nipple object image information.

[0133] Therefore, the field output in this section is named "medicinal bath path trajectory". This output is directly called as the input field of "acquiring the medicinal bath path trajectory" in S410. At the same time, it is retained by the control system through the object numbering relationship of the object, the secondary precise positioning object and the re-image acquisition object, for use by S420 and S430.

[0134] In summary, this step achieves the following technical benefits: Instead of directly generating path trajectories when the apparent state or geometric structure results are valid on their own, it performs object-level filtering through consistency screening in the verification submodule after the comprehensive state results are formed. This results in a more complete input source for the path trajectory. Compared to existing processing chains that directly enter job control based solely on image contours or single spatial positioning, this step incorporates object numbering relationships, contour boundaries, projection coordinate sets, and classification labels into the same verification chain. This makes the correspondence between the bath path trajectory and the single nipple object clearer, and the subsequent bath execution and historical record addition are more seamlessly integrated.

[0135] Step S400 includes at least steps S410-S430:

[0136] S410. Obtain the path trajectory of the medicinal bath, send control commands and perform medicinal bath execution processing to obtain the medicinal bath execution result.

[0137] Specifically, the input source for this section is the medicated bath path trajectory output by S330, and the execution entities are the control system, the control command sending module, and the robot device. The medicated bath path trajectory is an execution trajectory generated according to object numbering relationships, spatial pose information, center coordinate sequence, and contour boundaries. Its content includes the execution order of a single nipple object, the arrival relationship corresponding to the photographing position, the motion planning relationship in the end-effector coordinate system, and the connection relationship between each segment of the path trajectory. The control command sending refers to the control system decomposing the medicated bath path trajectory into control commands that can be called segment by segment by the robot device, and sending them sequentially to the end effector according to the object numbering relationship. The medicated bath execution refers to the robot device completing continuous actions of positioning, stopping, turning, and medicated bath treatment according to the control commands.

[0138] During operation, the control system first reads the object numbering relationship in the medicated bath path trajectory and establishes an execution queue according to the object numbering relationship. Then, the control command sending module sends the first segment of the medicated bath path trajectory to the robot device, so that the end effector moves to the position corresponding to the first single nipple object. Then, the orientation relationship of the end effector is adjusted according to the axial direction and tilt angle information of the object, so that the medicated bath treatment corresponds to the contour boundary of the single nipple object.

[0139] Furthermore, when the control system detects that an object number relationship is marked as a secondary precise positioning object in S330, the current control command is not sent directly into the bath processing. Instead, the relocation path is first invoked to recalibrate the position relationship of the end effector, and then the control system rereads the path trajectory corresponding to the object number relationship and continues execution. When an object number relationship is marked as a re-image acquisition object, the control system skips the bath processing corresponding to the object number relationship and only sends control commands to the other passing objects.

[0140] Understandably, in an operational engineering embodiment, the control system first sends a control command to the first nipple object, and the robot device arrives near the first image position according to the first path trajectory. Then, it performs medicated bath treatment along the contour boundary range recorded in the path trajectory. After completion, it continues to call the second path trajectory to enter the next nipple object. If the second nipple object corresponds to a secondary precise positioning object in S330, the control system first triggers local repositioning based on its spatial pose information, and then continues to call the second path trajectory.

[0141] Throughout the execution process, the control system synchronously records the object number relationship, path trajectory call time, and end effector completion status, and merges them into an output field name "medicinal bath execution result". The medicinal bath execution result is directly called by "medicinal bath execution result" in S420, while the object number relationship and path trajectory call time are retained for S430 to perform associated storage and correspond to the comprehensive status result.

[0142] S420. Based on the results of the medicinal bath, perform image acquisition and medicinal liquid coverage result verification processing to obtain the medicinal liquid coverage result.

[0143] Specifically, the input source for this section is the medicated bath execution result output by S410, and the execution entities are the 2D vision module, the image processing module, the image information processor, and the control system. The image acquisition in this section refers to post-execution image acquisition, meaning that after the medicated bath processing is completed, image information is acquired again for a single nipple object corresponding to the same object numbering relationship, rather than re-executing a unified acquisition of the entire nipple region.

[0144] The verification of the drug coverage result refers to checking the color distribution, contour boundary and drug coverage position in the image after execution against the contour feature information, classification label and drug bath path trajectory formed in S220 and S230 before execution, so as to determine whether the drug coverage range of the current single nipple object falls within the area corresponding to the contour boundary.

[0145] During operation, after the robot completes the medicinal bath treatment corresponding to a certain object numbering relationship, the control system immediately calls the 2D vision module to perform near-field image acquisition on the object numbering relationship and sends the acquired image information to the image processing module. The image processing module processes the image information after execution according to the image preprocessing, color extraction and morphological processing path in S220, and then performs medicinal liquid coverage result verification based on the contour boundary position retained before execution and the color distribution position in the current image.

[0146] Furthermore, during the verification of the liquid coverage result, the image information processor first checks whether the liquid coverage position in the current image falls within the boundary range of the previous contour, and then checks whether the liquid coverage position is consistent with the end movement range of the liquid bath path trajectory. When the liquid coverage position is continuous, the boundary correspondence is stable, and the coverage range falls within the contour boundary of the current object numbering relationship, the image processing module determines that the object numbering relationship passes the liquid coverage result verification. When the liquid coverage position deviates from the contour boundary or the coverage range is significantly insufficient, the image processing module marks the object numbering relationship as an object to be precisely located a second time and sends the corresponding record to the control system.

[0147] Understandably, in the engineering embodiment, after the first single nipple object completes the medicated bath treatment, the 2D vision module acquires the image information of the first single nipple object again. The image processing module extracts the medicated liquid coverage position in the image and compares the position with the contour boundary before execution. If the medicated liquid coverage position is consistent with the contour boundary, it is retained as a pass state. If the medicated liquid coverage position of the third single nipple object is only located in part of the contour boundary, the image processing module records it as an insufficient coverage state and hands it over to the control system to decide whether to perform secondary precise positioning.

[0148] After processing in this section, the liquid coverage result is recorded as the output field name "liquid coverage result". This output is directly called by "liquid coverage result" in S430. At the same time, the object number relationship and coverage status in the liquid coverage result can also be fed back to the control system for relocation reference when calling the next round of liquid bath path trajectory.

[0149] S430. Based on the drug liquid coverage result and the comprehensive status result, perform associated storage and historical record appending processing to obtain the relationship between historical records and object numbers.

[0150] Specifically, the input sources for this section are the drug solution coverage result output by S420 and the comprehensive status result output by S320. The execution entities are the control system, the data processing unit, and the associated storage module. The associated storage refers to binding the comprehensive status result, the drug bath execution result, and the drug solution coverage result under the same object numbering relationship, forming a traceable object-level record. The historical record appending refers to appending the current binding result to the existing historical records of the single nipple object in chronological order under the existing object numbering relationship, rather than overwriting existing records.

[0151] During operation, the control system first reads the classification label, confidence information, length information, diameter information, axis direction, tilt angle information, and contour boundary information from the comprehensive status result according to the object number relationship. Then, it reads the liquid coverage result under the same object number relationship and the liquid bath execution result already retained in S410. Subsequently, the data processing unit performs field alignment processing to ensure that the comprehensive status result, liquid bath execution result, and liquid coverage result corresponding to the same object number relationship are in one-to-one correspondence.

[0152] Furthermore, the associated storage module generates an object-level record for each object number relationship. The object-level record includes at least the object number relationship, the overall status result, the bath execution result, the liquid coverage result, the path trajectory call time, and the current round processing conclusion. When the current object number relationship already exists in the history record, the associated storage module appends the current round object-level record to the end of the history record corresponding to the object number relationship, while retaining the previous record unchanged. When the current object number relationship appears for the first time in the history record, the associated storage module creates a new history record corresponding to the object number relationship.

[0153] For records marked as objects requiring secondary precise positioning in S420, the control system synchronously writes an overwrite status flag when adding historical records, so that when S210 is called in the next round, the control system can directly call the latest spatial pose information and object numbering relationship based on the object numbering relationship, and continue to perform image acquisition, time synchronization and coordinate system mapping relationship calling processing.

[0154] Understandably, in a complete engineering embodiment, the first single nipple object forms the first object-level record in this round of processing. The object-level record contains the comprehensive status result, the bath execution result, and the liquid coverage result corresponding to the first object number relationship. If the third single nipple object has an insufficient coverage state, the coverage state and processing conclusion are additionally retained when the historical record is appended, and it is used as the priority calling object for the next round of processing in the control system.

[0155] After processing in this section, the historical records and object number relationships are recorded as the output field name "historical records and object number relationships". The historical records are used to continuously save additional records under the same object number relationship, and the object number relationships are continued to be called by S210, so that subsequent image acquisition still revolves around the same single nipple object, thereby forming a closed-loop processing link between S100 and S400.

[0156] Summary of the technical effects of this step: This step associates the post-execution drug coverage result with the pre-execution comprehensive status result under the same object numbering relationship for storage. Instead of treating the drug bath execution as an independent endpoint, it unifies the pre-identification, execution process, and post-execution verification of a single nipple object into the historical record. Compared to existing methods that only save job completion information or single detection results, this step allows for continuous appending of records for the same single nipple object across different rounds. Subsequent S210 calls to the object numbering relationship directly connect to the previous processing link, resulting in a more complete processing procedure.

[0157] Example 2: Figure 2 This diagram illustrates a structural block diagram of a cow teat robot device integrating 2D and 3D vision according to an embodiment of the present invention. Figure 2As shown, the structure may include:

[0158] The 3D image information acquisition module 01 is used to acquire 3D image information of the teat region and generate 3D point cloud data. Specifically, the 3D image information acquisition module receives acquisition input from the teat region in the bathing station. The acquisition input is entered into the data processing unit by a 3D vision acquisition component set at a fixed position in the bathing station. The data processing unit performs continuous image acquisition of the teat region according to the acquisition sequence in the camera coordinate system, and then performs correction, invalid point removal, and point cloud transformation on the acquired 3D image information to form 3D point cloud data corresponding to the current acquisition time. The 3D image information acquisition module is triggered when the cow enters the bathing station and reaches the acquisition range. When the cow's body position changes beyond the current acquisition range, the current round of processing is stopped and acquisition input is received again. The 3D point cloud data is passed as the output object of this module to the single teat object establishment module as the basic input for spatial separation, center coordinate sequence generation, spatial pose information generation, and object numbering relationship establishment. At the same time, the 3D point cloud data is also passed to the geometric structure result generation module as the basic input for point cloud processing, fitting, and projection coordinate set generation.

[0159] The single nipple object creation module 02, connected to the 3D image information acquisition module, is used to perform spatial separation, center coordinate sequence generation, spatial pose information generation, and object numbering relationship establishment based on the 3D point cloud data, thereby obtaining spatial pose information and object numbering relationships. Specifically, the single nipple object creation module receives 3D point cloud data output from the 3D image information acquisition module, performs spatial separation on the 3D point cloud data according to spatial continuity, sequentially generates center coordinate sequences for each separated point cloud set, generates corresponding spatial pose information based on the center coordinate sequences and point cloud boundaries, and establishes object numbering relationships within the same round of acquisition. The spatial separation corresponds to the independent division of single nipple objects, the center coordinate sequence corresponds to the arrangement of the center positions of single nipple objects along the extension direction, the spatial pose information corresponds to the positional relationship of single nipple objects in the camera coordinate system, end coordinate system, tool coordinate system, and world coordinate system, and the object numbering relationship corresponds to the sequential relationship of each single nipple object in the current round of processing. When the boundary of the separated point cloud set is interrupted, the center coordinate sequence is discontinuous, or the object order conflicts, the single nipple object establishment module records the abnormal state of the current object numbering relationship and retains the object numbering relationship established in the previous round. The spatial pose information and object numbering relationship are passed as the output objects of this module to the single nipple object image information acquisition module, serving as the input objects for image acquisition, time synchronization, and coordinate system mapping relationship calls. At the same time, the object numbering relationship is also passed to the geometric structure result generation module, serving as the calling index for the one-to-one correspondence between 3D point cloud data and single nipple objects.

[0160] The single nipple object image information acquisition module 03, connected to the single nipple object establishment module, is used to perform image acquisition, time synchronization, and coordinate system mapping based on the spatial pose information and object numbering relationship to obtain single nipple object image information. Specifically, the single nipple object image information acquisition module receives the spatial pose information and object numbering relationship output from the single nipple object establishment module, and calls the acquisition path and shooting position configuration in the control system to perform image acquisition on each single nipple object corresponding to each object numbering relationship. The image acquisition revolves around the single nipple object and does not uniformly capture the entire nipple region; the time synchronization corresponds to the time correspondence between the current single nipple object image information and the preceding 3D point cloud data; the coordinate system mapping relationship call corresponds to the process of mapping the spatial pose information to the image acquisition position and the region of interest in the image. During the image acquisition process, the single nipple object image information acquisition module reads the spatial pose information under the current object numbering relationship, and then sequentially checks the acquisition position, acquisition direction, and acquisition time. When the acquisition position deviates from the current spatial pose information, the control system stops the current acquisition and re-calls the object numbering relationship. The single nipple object image information is passed to the appearance state result generation module as the output object of this module, and serves as the input object for image preprocessing, color extraction, morphological processing, classification label generation, and center coordinate sequence verification. At the same time, the single nipple object image information is also called again in the medicine bath execution and history record appending module as the post-execution image input of the medicine liquid coverage result verification stage.

[0161] The appearance state result generation module 04, connected to the single nipple object image information acquisition module, is used to perform image preprocessing, color extraction, morphological processing, classification label generation, and center coordinate sequence verification based on the single nipple object image information to obtain the appearance state result. Specifically, the appearance state result generation module receives the single nipple object image information output from the single nipple object image information acquisition module. The image processing module and image information processor sequentially perform image preprocessing, color extraction, and morphological processing on the single nipple object image information. Then, classification labels are generated based on the processed contour feature information, and center coordinate sequence verification is performed in conjunction with the center coordinate sequence formed by the single nipple object establishment module. The image preprocessing corresponds to image boundary smoothing, noise removal, and distortion correction; the color extraction corresponds to reading the surface color distribution of the single nipple object; the morphological processing corresponds to performing closing operations and contour smoothing on the image boundaries; the classification labels correspond to the structured marking of the appearance state of the single nipple object; and the center coordinate sequence verification corresponds to checking the consistency between the image side center position relationship and the 3D point cloud side center coordinate sequence. When the image boundary is blurred, the classification label position is off, or the center coordinate sequence verification fails, the apparent state result generation module retains the current object numbering relationship and sends a re-acquisition request to the single nipple object image information acquisition module. The apparent state result is passed to the comprehensive state result generation module as the output object of this module, serving as the input object for target matching, spatial fusion, and data fusion. Simultaneously, the apparent state result is also passed to the medicinal bath execution and historical record appending module, serving as the preceding record object for post-execution medicinal liquid coverage result verification and associated storage.

[0162] The geometric structure result generation module 05, connected to the 3D image information acquisition module and the single nipple object establishment module, is used to acquire the 3D point cloud data and the object numbering relationship, perform point cloud processing, fitting, and projection coordinate set generation to obtain the geometric structure result. Specifically, the geometric structure result generation module receives the 3D point cloud data output from the 3D image information acquisition module and the object numbering relationship output from the single nipple object establishment module, reads the corresponding point cloud set one by one according to the object numbering relationship, and the point cloud processing module performs point cloud processing, fitting, and projection coordinate set generation on the point cloud set. The point cloud processing corresponds to boundary cleaning, discrete point removal, and continuous point retention; the fitting corresponds to geometric arrangement around the center coordinate sequence, axis direction, length information, diameter information, tilt angle information, and contour boundary; and the projection coordinate set generation corresponds to mapping the current point cloud set to the position set in the image processing path. When the point cloud set is missing, the contour boundary is interrupted, or the object numbering relationship is discontinuous, the geometric structure result generation module records the point cloud abnormal state of the current object numbering relationship and maintains the object numbering relationship for subsequent consistency filtering. The geometric structure result is passed to the comprehensive state result generation module as the output object of this module, and serves as the input object for target matching, spatial fusion, and data fusion. At the same time, the projection coordinate set and contour boundary in the geometric structure result are also called by the medicine bath execution and history record appending module as the spatial reference before execution when verifying the medicine liquid coverage result.

[0163] The comprehensive state result generation module 06, connected to the apparent state result generation module and the geometric structure result generation module, is used to perform target matching, spatial fusion, data fusion, consistency screening of the verification submodule, and path trajectory generation based on the apparent state result and the geometric structure result to obtain the medicinal bath path trajectory. Specifically, the comprehensive state result generation module receives the apparent state result output from the apparent state result generation module and the geometric structure result output from the geometric structure result generation module. The data fusion module first performs target matching according to the object number relationship, and then performs spatial fusion and data fusion on the matched objects to form a comprehensive state result. Subsequently, the verification submodule performs consistency screening on the comprehensive state result, and the control system generates the medicinal bath path trajectory based on the screening result. The target matching corresponds to the object correspondence processing between the contour feature information in the appearance state result, the projection coordinate set in the classification label and geometric structure result, and the contour boundary. The spatial fusion corresponds to the unified processing of the two-dimensional boundary and the three-dimensional position relationship. The data fusion corresponds to merging the classification label, length information, diameter information, axis direction, tilt angle information, and contour boundary into a single nipple object record. The consistency screening of the verification submodule corresponds to the consistency check of each field in the comprehensive state result. The path trajectory generation corresponds to generating the execution order and motion path for the object number relationship that has passed the screening. When the target matching fails, the consistency screening fails, or the object boundary conflicts, the comprehensive state result generation module sends the corresponding object number relationship to the single nipple object image information acquisition module or the bath execution and history record appending module for re-acquisition, repositioning, or recording processing. The bath path trajectory is passed to the bath execution and history record appending module as the output object of this module, serving as the direct input for control command sending and bath execution. At the same time, the comprehensive state result is also synchronously passed to the bath execution and history record appending module as the preceding data object for subsequent association storage and history record appending.

[0164] The medicated bath execution and historical record appending module 07 is connected to the comprehensive status result generation module, the single nipple object image information acquisition module, and the appearance status result generation module. It is used to acquire the medicated bath path trajectory, send control commands and execute the medicated bath, obtain the medicated bath execution result, and perform image acquisition and medicated liquid coverage result verification based on the medicated bath execution result to obtain the medicated liquid coverage result. It is also used to perform associated storage and historical record appending based on the medicated liquid coverage result and the comprehensive status result to obtain the relationship between historical records and object numbers. Specifically, the bath execution and history record appending module receives the bath path trajectory and comprehensive status result output from the comprehensive status result generation module, receives the single nipple object image information call relationship output from the single nipple object image information acquisition module, and receives the appearance status result call relationship output from the appearance status result generation module. First, the control system and control command sending module send control commands according to the object number relationship to drive the robot device to complete the bath execution and form the bath execution result. Then, the image acquisition path corresponding to the single nipple object image information acquisition module is called to acquire images of the single nipple object again after execution. The image processing module combines the appearance status result and comprehensive status result retained before execution to perform liquid coverage result verification and form liquid coverage result. Finally, the associated storage module writes the comprehensive status result, bath execution result, and liquid coverage result into the same record according to the object number relationship and performs history record appending to form the history record and object number relationship. When the bath execution and history record appending module encounters interruptions, failures to verify the liquid coverage result, or object numbering conflicts, it first retains the current object numbering relationship and the current round of records, and then sends the corresponding call information back to the comprehensive status result generation module or the single nipple object image information acquisition module for path trajectory reconstruction or image re-acquisition processing. The history records and object numbering relationships are returned as output objects of this module to the single nipple object image information acquisition module and the single nipple object establishment module, serving as the basis for the next round of image acquisition, object numbering relationship calls, and record connection, thereby completing the data and control closed loops between the modules.

Claims

1. A method for recognizing and recording the state of cow teats by integrating 2D and 3D vision, characterized in that, include: S100. Obtain three-dimensional image information of the nipple region, perform spatial pose information generation and object numbering relationship establishment processing to obtain three-dimensional point cloud data, spatial pose information and object numbering relationship; The three-dimensional image information refers to image information acquired by a depth camera or a binocular camera that reflects the spatial distribution relationship of the nipple region, including depth information, spatial hierarchy information and positional relationship with the camera coordinate system corresponding to each acquisition position in the nipple region. The three-dimensional point cloud data refers to the point cloud set obtained by converting the three-dimensional image information, and each point in the point cloud set corresponds one-to-one with the position in the camera coordinate system. S200. Based on the spatial pose information and object numbering relationship, perform classification label generation and center coordinate sequence verification to obtain the apparent state result; S300. Based on the apparent state results, target matching, spatial fusion and data fusion processing are performed to obtain the medicinal bath path trajectory; S400. Based on the medicinal bath path trajectory, control commands are sent and medicinal bath execution is performed, and image acquisition and medicinal liquid coverage result verification are performed to obtain the historical record and updated object number relationship.

2. The method according to claim 1, characterized in that, The process of generating spatial pose information and establishing object numbering relationships includes: The spatial pose information generation process includes: based on the three-dimensional point cloud data, performing point cloud boundary cleaning and discrete point removal on each single nipple object, generating direction information according to the extension direction of the center coordinate sequence, and generating deflection angle information in combination with the point cloud boundary distribution, forming spatial pose information containing 3D coordinates, the direction corresponding to the center coordinate sequence, and deflection angle information. The object numbering relationship establishment process includes: assigning a unique object number to each single nipple object according to the corresponding order of the spatial position of each single nipple object in the world coordinate system and the current shooting position, and recording the object number together with the center coordinate sequence, so that the object number maintains the same calling standard for the same single nipple object in subsequent image acquisition, target matching, path trajectory generation and history record appending processes.

3. The method of claim 2, wherein, The process of generating classification labels and verifying the center coordinate sequence includes: The classification label generation process includes: labeling and recording the appearance state of a single nipple object; The tagged records include color anomaly location markers, contour boundary anomaly markers, and drug liquid coverage location markers; The center coordinate sequence verification process includes: checking the correspondence between the center position relationship corresponding to the contour feature information and the center coordinate sequence; when the two correspond continuously, the object numbering relationship remains unchanged and the classification label is written under the object numbering relationship; when the two are not continuous, the single nipple object is marked as an object to be re-image acquired. The classification labels and verified contour feature information are merged into an appearance state result with object numbering relationships.

4. The method of claim 2, wherein, The preceding processes for target matching, spatial fusion, and data fusion processing also include: Point cloud processing, fitting, and projection coordinate set generation are performed, specifically including: Read the corresponding point cloud sets in the 3D point cloud data one by one according to the object number relationship, and perform boundary cleaning, discrete point removal and continuous point retention for each point cloud set; Centerline fitting, contour boundary fitting, and spatial pose information fitting are performed along the center coordinate sequence direction to generate length information, diameter information, axis direction, tilt angle information, and contour boundary information. The above information together constitute the geometric structure result. Based on the coordinate system mapping relationship, the fitted point cloud set is used to generate a projected coordinate set, and the projected coordinate set and the object number relationship are written into the cache.

5. The method of claim 4, wherein, The process of target matching, spatial fusion, and data fusion includes: First, read the object number relationship in the appearance state result and the object number relationship in the geometric structure result. Then, perform target matching item by item according to the same object number relationship. The target matching first calls the center position relationship between the projection coordinate set and the contour feature information for the first matching, then calls the contour boundary and the contour boundary information for the second matching, and finally calls the stable correspondence between the confidence information and the length information and the diameter information for the third matching. When all three matches remain within the allowed range, it is determined that the current appearance result and the geometric structure result belong to the same single nipple object. Then, the spatial fusion process is entered. The spatial fusion process first aligns the contour boundary in the appearance result with the projection coordinate set in the geometric structure result, and then constrains the distribution range of the classification label in the alignment result so that the classification label always falls within the contour boundary range of the single nipple object. The length information, diameter information, axis direction, tilt angle information, and contour boundary information are written into the same object numbering relationship to form a comprehensive status result.

6. The method of claim 5, wherein, The process of forming the comprehensive state result includes: Perform consistency screening and path trajectory generation for the verification submodules, specifically including: Read the comprehensive status results one by one according to the object number relationship, and perform consistency checks on four conditions: whether the confidence information is within the allowable range, whether the contour boundary is continuously corresponding to the projected coordinate set, whether the length information, diameter information and axis direction are stable, and whether the classification label position falls within the contour boundary range. Mark the object number relationship that meets all conditions as a pass object, and mark the object that only partially fails to meet the conditions as a secondary precise positioning object. For objects, a medicinal bath path trajectory is generated based on their spatial pose information, center coordinate sequence, and contour boundary. The medicated bath path trajectory includes the execution order of a single nipple object, the spraying order, the terminal movement trajectory, and the connection relationship between each segment of the path trajectory.

7. The method of claim 1, wherein, The process of sending control commands and executing the medicated bath includes: The medicated bath path trajectory is decomposed into control instructions that can be called segment by segment by the robot equipment. These instructions are sent sequentially to the end effector according to the object numbering relationship, so that the robot equipment can complete the continuous actions of positioning, stopping, turning and medicated bath treatment according to the control instructions. The object numbering relationship, path trajectory calling time, and end effector completion status are recorded simultaneously and combined into the medicated bath execution result.

8. The method of claim 7, wherein, The process of image acquisition and drug coverage result verification includes: After the robot device completes the medicine bath processing corresponding to a certain object numbering relationship, it immediately calls the 2D vision module to perform near-field image acquisition on the object numbering relationship, obtains the image information after execution, and then processes the image information after execution according to the image preprocessing, color extraction and morphological processing path. Finally, it performs medicine liquid coverage result verification based on the contour boundary position retained before execution and the color distribution position in the current image. The verification of the liquid coverage result includes: checking whether the liquid coverage position in the current image falls within the boundary range of the contour before execution, and checking whether the liquid coverage position is consistent with the end movement range of the liquid bath path trajectory. When the liquid coverage position is continuous, the boundary correspondence is stable, and the coverage range falls within the contour boundary of the current object numbering relationship, the verification is deemed successful. When the liquid coverage position deviates from the contour boundary or the coverage range is obviously insufficient, the object numbering relationship is marked as an object to be precisely located a second time, and the liquid coverage result is generated.

9. The method of claim 5, wherein, The process of obtaining the relationship between historical records and object numbers includes: The classification label, confidence information, length information, diameter information, axis direction, tilt angle information, and contour boundary information in the comprehensive status result are read according to the object number relationship. Then, the bath execution result and liquid coverage result under the same object number relationship are read, and field alignment processing is performed to generate an object-level record. The object-level record includes object number relationships, comprehensive status results, bath execution results, liquid coverage results, path trajectory call time, and current round processing conclusions. When the current object number relationship already exists in the history record, the current round object-level record is appended to the end of the history record corresponding to the object number relationship. When the current object number relationship appears for the first time in the history record, a new history record corresponding to the object number relationship is created, and finally the history record and object number relationship are obtained.

10. A robotic device for producing cow teats that integrates 2D and 3D vision, characterized in that, include: The system comprises a 3D image information acquisition module, a single nipple object creation module, a single nipple object image information acquisition module, an appearance state result generation module, a geometric structure result generation module, a comprehensive state result generation module, and a bath execution and history record appending module; these modules are connected in sequence to implement the method described in any one of claims 1-9.

Images