Visual, language and behavior linkage grassland robot patrol inspection and driving integrated method

Abstract

The present application relates to the technical field of multi-modal interaction, in particular to a visual, language and behavior linkage grassland robot patrol and driving integrated method, comprising the following steps: acquiring a cow group image, labeling coordinates, collecting distances, inputting a density model, generating a patrol target sequence, driving a robot to collect signs, judging abnormalities, generating an abnormal individual identification set, screening a monitoring target, inputting a visual question and answer, outputting a next patrol position coordinate, fusing a terrain semantic to calculate a patrol path motion instruction set, completing detection calculation deviation adjustment, generating a sound and light output, and generating a driving control instruction set, in the present application, by extracting the panoramic features of the grassland cattle herd and semantic labeling, the position and posture and density are converted into structured information, the semantic patrol sequence is generated by combining text and vision, target adaptive update and real-time response are realized, infrared depth comparison is used to identify signs of abnormality, image semantic matching is used to optimize the path, terrain semantic and motion parameters are fused to improve obstacle avoidance and adaptability, and task continuity and reliable identification are ensured.

Description

Technical Field

[0001] This invention relates to the field of multimodal interaction technology, and in particular to an integrated method for grassland robot inspection and herding that combines vision, language and behavior. Background Technology

[0002] Multimodal interaction technology refers to a comprehensive technical system that integrates multiple perception and expression modes such as vision, language, and behavior to enable machine systems to understand and respond to complex environmental information. This field involves core technologies such as target detection and recognition in computer vision, semantic understanding and generation in natural language processing, and path planning and execution in robot motion control. Through multimodal information fusion, machines can achieve comprehensive perception and intelligent decision-making of the environment, playing an important role in application scenarios such as intelligent robots, human-computer interaction, and autonomous driving. With the development of deep learning technology, multimodal interaction is gradually evolving from single perception to cross-modal collaboration, enabling machines to make judgments and take actions by comprehensively utilizing information from multiple senses, just like humans.

[0003] Traditional grassland robot patrol and driving away refers to the operation method of using a robot platform to monitor specific areas and drive away target animals in the grassland environment. This technology mainly addresses the animal management problem in grassland ecological protection. The traditional method uses a monocular camera to collect images and then extracts animal features through a convolutional neural network for classification and identification. Then, the robot is controlled to move along a fixed trajectory according to a preset coordinate point sequence. At the same time, a pre-recorded sound file is played or a buzzer is activated to emit a single frequency sound wave signal to drive away the animals. This method relies on manually labeled image samples to train the classifier. After obtaining the robot's position using a GPS positioning module, it navigates point by point according to a stored waypoint list. When driving away animals, the audio player is directly triggered to output a fixed audio waveform.

[0004] Traditional grassland robot patrol and herding operations rely on a single visual channel for image acquisition and target recognition, lacking multi-dimensional representation of individual position and posture changes. This results in inaccurate judgment of the distribution of inspected objects, image classification methods are constrained by human samples, and recognition errors are prone to occur when scene lighting and viewing angles change. Fixed trajectory navigation ignores real-time environmental differences, and the robot's movement path cannot be flexibly adjusted according to terrain features, leading to gaps in the coverage area. The herding process uses preset sound wave outputs and fails to adjust according to animal behavior feedback, resulting in unstable response effects. When targets are dense or the terrain is significantly undulating, the continuity of inspection and operational safety are affected. Summary of the Invention

[0005] To address the technical problems of traditional grassland robot inspection and herding, which rely on a single visual channel for image acquisition and target recognition, lacking multi-dimensional representation of individual position and posture changes, resulting in inaccurate judgment of the distribution of inspected objects, image classification methods constrained by manual samples and prone to recognition deviations when scene lighting and viewing angles change, fixed trajectory navigation ignoring real-time environmental differences, robot movement paths unable to flexibly adjust according to terrain features leading to gaps in coverage, and the herding process using preset sound wave outputs that fail to adjust based on animal behavioral feedback, resulting in unstable response effects, and affecting inspection continuity and operational safety when targets are dense or terrain is significantly undulating, this invention provides an integrated method for grassland robot inspection and herding that combines vision, language, and behavior.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an integrated method for inspecting and driving away grassland robots based on vision, language, and behavior, comprising the following steps:

[0007] S1: Acquire grassland cattle distribution images through panoramic vision sensors, perform individual segmentation and localization on the grassland cattle distribution images, extract individual spatial coordinates, posture direction angles and group distribution density parameters, input the scene description statement and the visual feature vector composed of individual spatial coordinates, posture direction angles and group distribution density parameters into the natural language processing model, perform semantic association calculation, and generate inspection target sequence;

[0008] S2: Call the inspection target sequence to drive the robot to move to the individual position, collect eye temperature value, body length-to-width ratio and standing posture angle through infrared temperature measurement and depth vision, perform threshold comparison judgment, and generate an abnormal individual identifier set when the value exceeds the limit;

[0009] S3: Based on the set of abnormal individual identifiers, filter the monitoring targets, obtain visual information on the distribution of the surrounding cattle herds, input the images of the surrounding individual locations and the target screening question text into a visual question answering model for semantic matching, and output the coordinates of the next inspection location.

[0010] S4: Plan a path based on the coordinates of the next inspection position, monitor the terrain slope and obstacle distribution characteristics, weight and fuse the boundary point set characteristics of the obstacle area with the target guidance parameters of the next inspection position coordinates, and calculate the turning angle sequence and speed adjustment sequence through a visual navigation algorithm to obtain the inspection path motion instruction set.

[0011] As a further embodiment of the present invention, the inspection target sequence includes an individual identification number, a priority weight value, and a task scheduling index; the abnormal individual identifier set includes an abnormal individual number, an abnormal state category, and a risk assessment level; the next inspection location coordinates include a target spatial point, geographic reference coordinates, and a neighborhood aggregation identifier; and the inspection path motion instruction set includes a speed control parameter group, a path node sequence, and a direction angle set.

[0012] As a further aspect of the present invention, the specific steps of S1 are as follows:

[0013] S101: Acquire grassland cattle herd distribution images through panoramic vision sensors, analyze pixel grayscale distribution and perform region clustering segmentation, retrieve boundary point coordinates and reconstruct individual circumscribed contour matrices, calculate the horizontal and vertical coordinates of the center point and serialize them to generate individual spatial coordinate sets.

[0014] S102: Based on the individual spatial coordinate set, retrieve the pixel direction vectors within the individual region and statistically analyze the angle distribution. Calculate the angle difference based on the main direction angle and the long axis of the contour. Calculate the Euclidean distance between individuals and perform aggregation comparison on the angle differences of neighboring individuals based on the distance value to generate an individual posture parameter set.

[0015] S103: Extract visual feature vectors based on the individual posture parameter set, obtain scene description statements and input them into the natural language processing model, perform text visual feature alignment on semantic entity vectors and visual feature vectors and calculate semantic similarity, analyze priority indicators and adjust sequence weights, and generate inspection target sequences.

[0016] As a further aspect of the present invention, the specific steps of S2 are as follows:

[0017] S201: Obtain the individual location information of the inspection target sequence identifier, drive the robot to move to the coordinate position, collect the surface temperature distribution through infrared thermometry, calculate the weighted average value of the infrared pixel energy of the eye and convert the temperature parameter to generate an eye temperature value dataset.

[0018] S202: Based on the eye temperature dataset, three-dimensional structural data frames are acquired through depth vision, the coordinates of the body surface boundary and skeleton nodes are extracted, the body length-to-width ratio is calculated, and the posture angle is obtained according to the center of gravity axis to generate a morphological posture feature matrix.

[0019] S203: Based on the morphological and posture feature matrix and the eye temperature value dataset, call the threshold comparison unit to perform difference judgment on the feature parameters and the threshold benchmark set, filter out the individual indexes that deviate from the threshold range and aggregate them to generate an abnormal individual identifier set.

[0020] As a further aspect of the present invention, the specific steps of S3 are as follows:

[0021] S301: Obtain the set of abnormal individual identifiers, collect visual information on the distribution of cattle herds around the individual's location area, match the outline boundary pixel array with the spatial coordinate point cloud, calculate the Euclidean distance and compare it with the herd density threshold, cluster the distance areas below the threshold, and generate a distribution map of the surrounding individual locations.

[0022] S302: Based on the surrounding individual location distribution map, a visual question answering model is used to parse the target question text to obtain semantic vectors, extract visual feature vector embedding values ​​and calculate cosine similarity, compare the results with semantic matching thresholds, aggregate the centroid coordinates of regions greater than the thresholds, and form a set of aggregated coordinates of monitoring targets.

[0023] S303: Based on the aggregated coordinate set of the monitored target, compare the difference in cattle distribution density between the candidate area and the non-target area, calculate the directional offset angle and extract the offset points whose offset angles conform to the stable interval, set the offset point position parameters as the moving reference point, and generate the coordinates of the next inspection position.

[0024] As a further aspect of the present invention, the calculation of Euclidean distance and comparison with the population density threshold refers to: based on the synchronously acquired depth information, converting the two-dimensional pixel coordinates of the contour boundary pixel array into three-dimensional spatial coordinates, calculating the Euclidean distance between each three-dimensional coordinate point after conversion, and obtaining the spatial distance matrix.

[0025] The population density threshold is determined based on a range of values ​​obtained through statistical fitting of the average visible distance between individuals in different terrain environments.

[0026] As a further aspect of the present invention, the specific steps of S4 are as follows:

[0027] S401: Obtain the coordinates of the next inspection location and the terrain point cloud data, calculate the elevation difference value of adjacent sampling points, compare the elevation difference value according to the slope determination threshold, mark the area above the threshold as the obstacle area and record the index position, and generate a terrain slope and obstacle distribution dataset.

[0028] S402: Based on the terrain slope and obstacle distribution dataset, call the obstacle area index and inspection location coordinates to perform spatial mapping, filter the boundary point set according to the Euclidean distance parameter, and perform weighted fusion of the boundary point set features of the obstacle area with the target guidance parameter of the next inspection location coordinates to obtain the path reference feature set;

[0029] S403: Call the path reference feature set, use the visual navigation algorithm to calculate the azimuth change value and speed adjustment sequence between path nodes, and perform time-series integration operation to obtain the inspection path motion instruction set.

[0030] As a further aspect of the present invention, the value of the slope determination threshold is adaptively adjusted based on the statistical distribution characteristics of the elevation difference values ​​in the terrain point cloud data;

[0031] The weighted fusion of the boundary point set features of the obstacle area and the target guidance parameters of the next inspection position coordinates means that by assigning obstacle avoidance weight factors and target guidance weight factors to the boundary point set features of the obstacle area and the target guidance parameters of the next inspection position coordinates respectively, and performing a weighted summation operation, a path reference feature set is obtained.

[0032] As a further aspect of the present invention, the method further includes step S5:

[0033] S5: Based on the inspection path motion instruction set, complete the detection task, call the preset geographical coordinates of the target area to obtain the current cattle herd distribution status, calculate the deviation value of the direction angle between the herd centroid and the center point of the target area, adjust the orientation deflection angle of the sound and light device, the robot heading correction angle and the robot movement speed control amount, and generate the driving control instruction set.

[0034] The driving control command set includes audio-visual output parameters, movement control quantities, and deviation correction coefficients.

[0035] As a further aspect of the present invention, the specific steps of S5 are as follows:

[0036] S501: Based on the inspection path motion instruction set, perform pose sampling on the geographic coordinate points of the detection task area, call the pixel clustering coordinates in the cattle herd scattering image frame sequence, calculate the partition density difference and determine the central tendency direction, and generate the cattle herd scattering state matrix.

[0037] S502: Based on the cattle herd distribution state matrix, extract individual position vectors and calculate the herd centroid coordinates, obtain the coordinates of the center point of the target area to be driven, calculate the angle between the two direction vectors and compare it with the reference direction angle, and generate a herd centroid offset angle parameter set.

[0038] S503: Call the group centroid offset angle parameter set, combine the orientation deflection angle control parameters of the acoustic and optical device, the robot heading correction parameters and the robot moving speed control amount, perform proportional adjustment calculation on the angle difference and correct the direction and velocity vector, and generate a driving control command set.

[0039] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0040] In this invention, panoramic feature extraction and semantic annotation are performed on grassland cattle herd images to transform individual positions, postures, and distribution densities into structured semantic information. This enables the determination of inspection targets to have contextual relevance and dynamic judgment capabilities. Based on the joint analysis of textual and visual features, a semantically driven inspection sequence is generated, achieving adaptive updates and real-time responses for target selection. During movement, cross-comparison of infrared and depth data is used to identify abnormal physical signs, giving the status judgment continuous monitoring attributes. By matching surrounding group images with semantic questions to optimize path selection, the logical consistency of inspection sequence and regional connection is ensured. Path fusion calculation is performed by combining terrain semantics and motion parameters to improve obstacle avoidance sensitivity and terrain adaptability during movement, thereby maintaining the continuity of task execution and the reliability of identification decisions in complex environments. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the accompanying drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the steps of the present invention;

[0043] Figure 2 This is a detailed schematic diagram of S1 of the present invention;

[0044] Figure 3 This is a detailed schematic diagram of S2 of the present invention;

[0045] Figure 4 This is a detailed schematic diagram of S3 of the present invention;

[0046] Figure 5 This is a detailed schematic diagram of S4 of the present invention;

[0047] Figure 6 This is a detailed schematic diagram of S5 of the present invention. Detailed Implementation

[0048] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0049] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0050] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0051] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0052] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0053] Please see Figure 1 This invention provides an integrated method for inspecting and driving away livestock using a grassland robot that combines vision, language, and behavior, comprising the following steps:

[0054] S1: Acquire grassland cattle distribution images through panoramic vision sensors, perform individual segmentation and localization on the grassland cattle distribution images, extract individual spatial coordinates, posture direction angles and group distribution density parameters, input the scene description statement and the visual feature vector composed of individual spatial coordinates, posture direction angles and group distribution density parameters into the natural language processing model, perform semantic association calculation, and generate inspection target sequence;

[0055] S2: Call the inspection target sequence to drive the robot to move to the individual position, collect eye temperature value, body length-to-width ratio and standing posture angle through infrared temperature measurement and depth vision, perform threshold comparison judgment, and generate an abnormal individual identifier set when the value exceeds the limit;

[0056] S3: Based on the set of abnormal individual identifiers, filter monitoring targets, obtain visual information on the distribution of surrounding cattle, input the images of surrounding individual locations and the target screening question text into a visual question answering model for semantic matching, and output the coordinates of the next inspection location;

[0057] S4: Plan the path based on the coordinates of the next inspection position, monitor the terrain slope and obstacle distribution characteristics, weight and fuse the boundary point set characteristics of the obstacle area with the target guidance parameters of the next inspection position coordinates, and calculate the turning angle sequence and speed adjustment sequence through the visual navigation algorithm to obtain the inspection path motion instruction set;

[0058] S5: Based on the inspection path motion instruction set, complete the detection task, call the preset geographical coordinates of the target area to obtain the current cattle herd distribution status, calculate the deviation value of the direction angle between the herd centroid and the center point of the target area, adjust the orientation deflection angle of the sound and light device, the robot heading correction angle and the robot movement speed control amount, and generate the driving control instruction set.

[0059] The inspection target sequence includes individual identification number, priority weight value and task scheduling index; the abnormal individual identifier set includes abnormal individual number, abnormal status category and risk assessment level; the next inspection location coordinates include target spatial point, geographic reference coordinates and neighborhood aggregation identifier; the inspection path motion instruction set includes speed control parameter group, path node sequence and direction angle set; and the driving control instruction set includes sound and light output parameters, movement control quantity and deviation correction coefficient.

[0060] Please see Figure 2 The specific steps of S1 are as follows:

[0061] S101: Acquire grassland cattle herd distribution images through panoramic vision sensors, analyze pixel grayscale distribution and perform region clustering segmentation, retrieve boundary point coordinates and reconstruct individual circumscribed contour matrices, calculate the horizontal and vertical coordinates of the center point and serialize them to generate individual spatial coordinate sets.

[0062] A 1920×1080 pixel image of the distribution of cattle on a grassland was acquired using a panoramic vision sensor. First, a pixel located at coordinates (100, 150) was converted to grayscale. Its original three-channel color values ​​were R:55, G:65, B:70. The grayscale value was calculated as 55×0.299+65×0.587+70×0.114, resulting in a grayscale value of 62.58. This process was repeated for all pixels in the image. Next, the grayscale value of each individual cattle was used for identification. The grayscale range is determined based on the statistical distribution of grayscale values ​​of 500 cattle samples in the original image data. The 5% and 95% quantiles of the distribution are taken as the lower and upper limits of the range, respectively, which are [30, 75]. The grayscale range of grassland is set to [80, 125]. Then, pixel-level region clustering is performed on the image. For example, starting from the top left corner (0, 0) pixel, the image is scanned line by line. When the pixel with coordinates (401, 502) is detected with a grayscale value of 58, it is within the cattle recognition range. Within [30, 75], it is marked as the initial point of cluster region A, and its eight adjacent pixels are detected. If the gray values ​​of adjacent pixels also fall within this interval, for example, the gray value of (401, 503) is 61, then the point is assigned to cluster region A, and this process is repeated until the gray values ​​of adjacent pixels are no longer within this interval. At this point, the segmentation of an independent individual cow region is completed. Then, the coordinates of the boundary points of cluster region A are retrieved and recorded. The condition for a pixel to be determined as a boundary point is that its... The individual cattle belongs to cluster region A, but at least one of its eight neighboring pixels has a gray value that does not belong to the range [30, 75]. For example, pixel (410, 515) has a gray value of 63, while its neighboring pixel (410, 516) has a gray value of 95. Therefore, (410, 515) is recorded as a boundary point. By traversing cluster region A, a set containing the coordinates of 200 boundary points is obtained. Then, the outer contour matrix of the individual cattle is reconstructed based on this set. The specific steps involve obtaining the coordinates of these boundary points. The minimum value of the x-coordinate among the coordinates With the maximum value and the minimum value of the ordinate. With the maximum value The outline of the circumscribed rectangle is constructed. Then, the x and y coordinates of the center point of the outline are calculated. The x coordinate is calculated as 440.5 by (401+480) / 2, and the y coordinate is calculated as 522 by (502+542) / 2. Thus, the center point coordinates of individual 1 are (440.5, 522.0). The same process is used to obtain the center coordinates of individual 2 as (610.0, 550.0) and the coordinates of individual 3 as (490.5, 625.0). Finally, the center points are serialized and encoded, with individual 1 encoded as ID1, individual 2 encoded as ID2, and individual 3 encoded as ID3, generating the individual spatial coordinate set {(ID1, 440.5, 522.0), (ID2, 610.0, 550.0), (ID3, 490.5, 625.0)}.

[0063] S102: Based on the individual spatial coordinate set, retrieve the pixel direction vectors within the individual region and statistically analyze the angle distribution to determine the main direction angle of the pixel gradient; determine the major axis direction angle of the individual's circumscribed contour based on the aspect ratio of the circumscribed rectangle; calculate the difference between the main direction angle of the pixel gradient and the major axis direction angle as the individual's pose tilt angle; perform aggregation comparison on the individual pose tilt angles of neighboring individuals based on the Euclidean distance between individuals to generate an individual pose parameter set;

[0064] Based on {(ID1, 440.5, 522.0), (ID2, 610.0, 550.0), (ID3, 490.5, 625.0)}, firstly, for the bounding rectangular contour region where the ID1 individual is located, i.e., the range defined by coordinates (401, 502) to (480, 542), the direction vector of the pixels within this region is retrieved. Specifically, for any pixel P(x, y) within the region, its gray-level gradient G in the horizontal direction is... x It is obtained by subtracting the gray value of its left neighbor from the gray value of its right neighbor. For example, if the gray value of the left pixel (429, 520) of P(430, 520) is 50, and the gray value of the right pixel (431, 520) is 60, then G is obtained. x =60-50=10, its gray-level gradient G in the vertical direction y It is obtained by subtracting the gray value of the adjacent pixel above it from the gray value of the pixel below it. For example, if the gray value of the pixel above it (430, 519) is 52, and the gray value of the pixel below it (430, 521) is 58, then G is obtained. y =58-52=6, and the angle of the direction vector of this pixel is then determined by... Right now The calculated angle is approximately 30.96 degrees. Traversing the 3120 pixels within the ID1 region yields a set of 3120 angle values. Statistical distribution analysis of these angle values ​​divides the range from 0 to 180 degrees into 18 angle intervals, each 10 degrees wide. The results show that the 30-40 degree interval has the most pixels (890). Therefore, the median of this interval, 35 degrees, is determined as the principal orientation angle of the ID1 individual. Next, its long axis is calculated. The width of its circumscribed rectangle is 480-401=79 pixels, and its height is 542-502=40 pixels. Since the width is greater than the height, the long axis is horizontal, and its angle is 0 degrees. The difference between the principal orientation angle and the long axis is then calculated. For an ID1 individual, this difference is |35-0|=35 degrees. Similarly, the angle difference for ID2 individuals is calculated to be 85 degrees, and for ID3 individuals, it is 25 degrees. Finally, the Euclidean distance between the individuals is calculated. The distance between ID1 and ID2 is... The calculation result is Similarly, the distance between ID1 and ID3 was calculated to be 116.14 pixels, and the distance between ID2 and ID3 was 138.86 pixels. Then, based on a preset proximity distance threshold, the angular differences between neighboring individuals were aggregated and compared. This threshold was set with reference to the safe social distance of cattle in a grassland environment, and was 1.5 times the average body length of cattle. If the average body length of cattle in the image is approximately 100 pixels, then the threshold is set to 150 pixels. According to the calculation, the distance between ID1 and ID3 (116.14 < 150) and the distance between ID2 and ID3 (138.86 < 150), therefore ID1 and ID3, and ID2 and ID3 are determined to be neighboring individuals. The distance between ID1 and ID2 (171.81 > 15) is also considered. 0 indicates that the individual is not a neighboring individual. In the aggregation comparison, the angle difference of ID1 (35 degrees) is compared with the angle difference of ID3 (25 degrees). The difference is 10 degrees, which is less than the preset posture coordination judgment threshold of 20 degrees. This indicates that the postures of ID1 and ID3 are relatively consistent and they are in the same resting or eating state. This association information is recorded. However, the angle difference of ID2 (85 degrees) is 60 degrees different from the angle difference of ID3 (25 degrees), which is greater than 20 degrees. The postures are inconsistent. Finally, the angle difference of each individual and the relationship with neighboring individuals are integrated to generate the individual posture parameter set {(ID1, 35, [ID3]), (ID2, 85, [ID3]), (ID3, 25, [ID1, ID2])}.

[0065] S103: Extract visual feature vectors based on individual posture parameter sets, obtain scene description statements and input them into a natural language processing model, perform text visual feature alignment on semantic entity vectors and visual feature vectors and calculate semantic similarity, analyze priority indicators and adjust sequence weights, and generate inspection target sequences.

[0066] Based on {(ID1, 35, [ID3]), (ID2, 85, [ID3]), (ID3, 25, [ID1, ID2])}, a visual feature vector is first extracted for each individual. This vector consists of three dimensions: the coordinates of the individual's center point, the pose parameter (angle difference), and the number of neighboring individuals. For example, the visual feature vector of ID1 is [440.5, 522.0, 35, 1], the visual feature vector of ID2 is [610.0, 550.0, 85, 1], and the visual feature vector of ID3 is [490.5, 625.0, 25, 2]. Simultaneously, the system acquires a scenario description from the user, such as "find and prioritize driving away cows that are standing alone and out of the herd in the distance," and inputs this statement into a natural language processing model for parsing. The model decomposes the statement into key semantic entities: "distant," "out of the herd," and "standing," converting each entity into a corresponding semantic vector. Then, it performs text-visual feature alignment between the semantic entity vectors and the visual feature vectors of each individual cow. Specifically, it calculates the correlation between the dimension values ​​of the visual feature vectors and the semantic entities. For the semantic term "distant," its correlation is calculated with the second dimension of the visual feature vector. The values ​​(vertical axis) are positively correlated. Values ​​in the [0, 360] range are considered near, [361, 720] are considered medium, and [721, 1080] are considered far. For the semantic meaning of "outlier," the correlation is negatively correlated with the fourth dimension value (number of neighboring individuals). The outlier degree is set to 1.0 when the number of neighboring individuals is 0, 0.5 when it is 1, and 0.1 when it is 2 or more. For the semantic meaning of "standing," the correlation is positively correlated with the third dimension value (angle difference). An individual's posture tilt angle is considered low when it is in the [0, 30] degree range, and its correlation with the semantic meaning of "standing" is 0.1. Values ​​in the [31, 6] degree range are considered high when it is in the [31, 6] degree range. At [0] degrees, it is a medium posture tilt level, and its correlation with the semantics of 'standing' is 0.5. At [61, 90] degrees, it is a high posture tilt level, and its correlation with the semantics of 'standing' is 1.0. The correlation is used for inspection priority ranking and is not used as the final judgment basis for abnormal posture category. Then, the overall semantic similarity between each individual and the description statement is calculated. The sequence weight is adjusted by analyzing the priority index. This weight is determined according to the word order and modifiers in the user input statement. For example, "prioritize driving away" makes the weight of "standing" and "separated from the group" higher than "far away". The weight of "standing" is set to W. stand =0.5, the weight of "outlier" is W alone =0.4, the weight of "far away" is W far =0.1. Next, calculate the priority score for each cow. Taking ID2 as an example, its vertical coordinate of 550.0 belongs to medium distance, corresponding to a score of 0.5 for the "far away" index. The number of neighboring individuals is 1, corresponding to a score of 0.5 for the "out-of-group" index. The angle difference of 85 degrees belongs to standing posture, corresponding to a score of 1.0 for the "standing" index. Its final priority score is 0.5 × W. far +0.5×W alone+1.0×W stand =0.5×0.1+0.5×0.4+1.0×0.5=0.05+0.2+0.5=0.75,

[0067] Table 1: Inspection Target Priority Assessment Table

[0068] <![CDATA[ID1]]> 0.5 0.5 0.5 0.50 <![CDATA[ID2]]> 0.5 0.5 1.0 0.75 <![CDATA[ID3]]> 0.5 0.1 0.1 0.14

[0069] As shown in Table 1, by performing the same calculation process on individuals, the final priority of ID1 is 0.50 and the final priority of ID3 is 0.14. Comparing the final priority scores of individuals, ID2 has the highest score of 0.75, followed by ID1, and ID3 has the lowest score. The individuals are arranged in descending order of scores to generate the inspection target sequence [ID2, ID1, ID3].

[0070] Please see Figure 3 The specific steps of S2 are as follows:

[0071] S201: Obtain the individual location information of the inspection target sequence identifier, drive the robot to move to the coordinate position, collect the surface temperature distribution through infrared thermometry, calculate the weighted average value of the infrared pixel energy of the eye and convert it into temperature parameters to generate an eye temperature value dataset.

[0072] The robot obtains the location information (610.0, 550.0) of the first individual ID2 in the inspection target sequence [ID2, ID1, ID3]. It then moves the robot at a speed of 0.5 meters per second to a position 3 meters away from the target individual. The infrared thermal imager mounted on the robot's gimbal, with a detector resolution of 320×240, is aimed at the face of individual ID2 to measure its temperature, acquiring a frame of infrared image containing the individual's surface temperature distribution. In this image, the bullseye region is located based on a pre-trained facial key point recognition model. This region corresponds to a 20×20 pixel rectangle in the image coordinate system, with its upper left corner coordinates (150, 100). The infrared image of this region is then calculated. The weighted average of the infrared energy is calculated as follows: A 3×3 pixel matrix at the center of the eye region is selected as the core calculation area. The weight of the central pixel (160, 110) is set to 0.3, the weights of its four adjacent pixels (above, below, left, and right) are 0.12, and the weights of its four diagonally adjacent pixels are 0.07. The weighting is based on the principle that the closer a pixel is to the center of the eyeball, the less its temperature reading is affected by the moist area at the corner of the eye, and the more accurately it reflects the true body temperature. The infrared energy values ​​of the nine pixels in the core area are [258, 260, 259, 265, 278, 266, 261, 263, 262]. The weighted average is calculated as follows:

[0073] 278×0.3+(265+266+261+263)×0.12+(258+259+262+262)×0.07=83.4+126.6+72.87=282.87.

[0074] The energy-weighted average is then converted into actual temperature parameters using the sensor's built-in calibration function. The conversion coefficient of this function is obtained by calibrating the sensor under a standard blackbody radiation source environment before it leaves the factory. Assuming the conversion relationship is temperature value = energy value × 0.1 + 10.613, the eye temperature of ID2 is 282.87 × 0.1 + 10.613 = 38.9 degrees Celsius. The robot continues to move to the positions of ID1 and ID3 according to the sequence, repeating the above acquisition and calculation process. The eye temperature of ID1 is measured to be 38.5 degrees Celsius and the eye temperature of ID3 is measured to be 39.8 degrees Celsius, generating an eye temperature value dataset {(ID2, 38.9), (ID1, 38.5), (ID3, 39.8)}.

[0075] S202: Based on the eye temperature dataset, three-dimensional structural data frames are acquired through depth vision, the coordinates of the body surface boundary and skeleton node are extracted, the body length-to-width ratio is calculated, and the posture angle is obtained according to the center of gravity axis to generate a morphological posture feature matrix.

[0076] Based on the eye temperature dataset {(ID2, 38.9), (ID1, 38.5), (ID3, 39.8)}, the robot continues to collect a frame of 3D structural data for individual ID2 in the sequence using a depth vision sensor. This data frame is a 3D point cloud containing 307,200 points, used to describe the three-dimensional morphology of the cattle. First, the body surface boundary is extracted from this point cloud data. Specifically, a 3D voxel grid is constructed, the point cloud data is placed into the grid, and the voxels containing data points are traversed. If at least one of the six neighboring voxels of a certain voxel is empty, then the points in that voxel are identified as boundary points, thus obtaining a set of body surface boundary points consisting of 51,200 points. Simultaneously, the coordinates of skeletal nodes are extracted by analyzing the curvature changes in the set of boundary points on the body surface. Regions with curvature higher than a preset value of 0.8 are identified as candidate joint areas. For example, significant curvature changes are observed in the head, limb joints, and rump of cattle, thus locating the head node coordinates as (-0.9, 0.1, 1.3) meters, the rump node coordinates as (1.2, 0.0, 1.2) meters, the left forehoof node as (-0.7, 0.4, 0.1) meters, and the right hindhoof node as (1.0, -0.3, 0.1) meters. Next, the body length-to-width ratio of the individual is calculated. The body length is defined as the three-dimensional Euclidean distance between the head node and the rump node, calculated as follows:

[0077] rice;

[0078] Body width was defined as the distance between corresponding nodes of the left and right scapulae, measured to be 0.85 meters. The body length-to-width ratio was 2.10 / 0.85≈2.47. Then, the posture angles were calculated based on the center of gravity axis, which is defined as the vector connecting the head node and the rump node. The angle between the projection of this vector onto the horizontal plane and the direction of the cow's movement is the heading angle, while the angle between this vector and the horizontal plane is the pitch angle. The pitch angle was calculated as arcsin((1.2-1.3) / 2.10)≈-2.74 degrees. This negative value indicates that the rump is slightly higher than the head. The same operation was performed on individuals in the sequence, and the obtained data were integrated to generate a morphological posture feature matrix.

[0079] Table 2: Individual Morphological and Postural Characteristics

[0080] <![CDATA[ID1]]> 2.15 2.53 −1.50 <![CDATA[ID2]]> 2.10 2.47 −2.74 <![CDATA[ID3]]> 2.30 3.11 5.20

[0081] Table 2 lists the morphological and posture characteristics obtained after three-dimensional structural analysis of individual cattle in the inspection sequence.

[0082] S203: Based on the morphological and pose feature matrix and the eye temperature value dataset, call the threshold comparison unit to perform difference judgment on the feature parameters and the threshold benchmark set, filter out the individual indexes that deviate from the threshold range and aggregate them to generate an abnormal individual identifier set;

[0083] Based on the generated morphological and posture feature matrix and eye temperature dataset, a threshold comparison unit is invoked. This unit has a pre-set threshold benchmark set established with reference to statistical data of healthy cattle. Specifically, the normal range benchmark value for eye temperature is set to [38.0, 39.5] degrees Celsius, which is determined based on the 95% confidence interval of data obtained from monitoring 1000 healthy Angus cattle under non-stress conditions for 7 consecutive days. The normal range benchmark value for body length-to-width ratio is set to [2.20, 2.80], which is set according to the adult cattle body size standard database. Exceeding this range indicates emaciation or abdominal distension. The normal standing range benchmark value for posture pitch angle is set to [-5.0, 5.0] degrees. Excessive head raising or lowering of cattle indicates respiratory diseases or discomfort. Subsequently, a difference judgment is performed between the feature parameters of each individual and the corresponding threshold benchmark set. First, individual ID1 is evaluated. Its eye temperature of 38.5 degrees Celsius falls within the range of [38.0, 39.5], with a difference of 0. Its body length-to-width ratio of 2.53 falls within the range of [2.20, 2.80], with a difference of 0. Its pitch angle of -1.50 degrees falls within the range of [-5.0, 5.0], with a difference of 0. Therefore, all parameters of ID1 are within the threshold range. Next, individual ID2 is evaluated. Its eye temperature is 38.9 degrees Celsius, its body length-to-width ratio is 2.47, and its pitch angle is -2.74 degrees. After comparing each parameter with the baseline set, none of these parameters exceed the threshold range. Finally, individual ID3 is evaluated. Its eye temperature is 39.8 degrees Celsius, exceeding the upper limit of 39.5 degrees Celsius by 0.3 degrees Celsius. Its body length-to-width ratio is 3.11, exceeding the upper limit of 2.80 by 0.31 degrees. Its pitch angle is 5.20 degrees, exceeding the upper limit of 5.0 degrees by 0.20 degrees. During this process, individual indices with at least one parameter deviating from the threshold range are selected. In this case, the eye temperature, body length-to-width ratio, and attitude pitch angle of individual ID3 all deviated from the normal range. Therefore, its index ID3 was filtered out. The filtered individual indexes that deviated were aggregated and put into a set. If there were other individuals, such as ID5, that were also detected as abnormal, their indexes were also added to the set. In this inspection, only ID3 was detected as abnormal, and finally, the abnormal individual identifier set {ID3} was generated.

[0084] Please see Figure 4 The specific steps of S3 are as follows:

[0085] S301: Obtain the set of abnormal individual identifiers, collect visual information on the distribution of the surrounding cattle herd in the individual location area, match the outline boundary pixel array with the spatial coordinate point cloud, calculate the Euclidean distance and compare it with the herd density threshold, cluster the distance area below the threshold, and generate a distribution map of the surrounding individual locations.

[0086] The robot retrieves the set of abnormal individual identifiers {ID3} generated in the previous inspection cycle. It then collects visual information and 3D spatial coordinate point cloud data of the surrounding cattle herd within a 20-meter radius centered on the location of ID3 (490.5, 625.0). From the acquired 1920×1080 pixel visual image, the robot identifies and segments the remaining four cattle individuals (ID4, ID5, ID6, and ID7), extracting their contour boundary pixel arrays. For example, the contour of ID4 consists of 255 pixels, with one pixel having 2D coordinates of (350, 610). By querying the synchronously acquired spatial coordinate point cloud data, the robot matches this 2D pixel coordinate with the depth information, obtaining its 3D spatial coordinates as (-5.2, 8.5, 1.1) meters. Using this method, the identification... The individual contour boundary pixel arrays are all converted into 3D spatial coordinate point clouds, and the 3D coordinates of the geometric center point of each individual are calculated, resulting in ID3 (0.0, 0.0, 1.2) meters, which is used as the reference origin; ID4 (-5.2, 8.5, 1.1) meters; ID5 (-4.9, 9.1, 1.2) meters; ID6 (10.1, -2.5, 1.1) meters; and ID7 (12.3, -1.8, 1.3) meters. Next, the 3D Euclidean distance between the abnormal individual ID3 and its surrounding individuals is calculated. For example, the distance between ID3 and ID4 is:

[0087] rice;

[0088] Similarly, the distances between ID3 and ID5 were calculated to be 10.3 meters, between ID3 and ID6 to be 10.4 meters, and between ID3 and ID7 to be 12.4 meters. These calculated distances were then compared to a population density threshold, which was set based on the typical social distance of cattle, taking 1.2 times the average nearest neighbor distance of 100 cattle in a naturally grazing state. If the average nearest neighbor distance is 9 meters, then the threshold is set to 9 × 1.2 = 10.8 meters. Comparing the calculated distances with 10.8 meters, the distances between ID3 and ID4, ID5, and ID6 were all less than 10.8 meters, while the distance to ID7 (12.4 meters) was greater than 10.8 meters. Based on this, the areas with distances below the threshold were clustered, with ID3, ID4, ID5, and ID6 grouped into a tight cluster, while ID7 was considered an outlier. A distribution map of the surrounding individuals was generated, clearly marking the area of ​​the tight cluster and the relative coordinates of the cattle.

[0089] S302: Based on the distribution map of surrounding individuals, a visual question answering model is used to parse the target question text to obtain semantic vectors, extract visual feature vector embedding values ​​and calculate cosine similarity, compare the results with semantic matching thresholds, aggregate the centroid coordinates of regions greater than the thresholds, and form an aggregated coordinate set of monitoring targets.

[0090] Based on the generated distribution map of surrounding individuals, including tightly clustered individuals and outlier ID7, the robot receives a new target question text: "Find a water source near the cattle herd." First, it processes the sentence using the text parsing part of a visual question-answering model, converting the core term "water source" into a 128-dimensional semantic vector V using word embedding technology. text Its vector values ​​are, for example, [0.12, -0.34, ..., 0.58]. Meanwhile, the model's visual parsing part processes the surrounding environment image acquired by the robot's camera, dividing the image into multiple 16×16 grid regions, extracting the visual features of each region, and converting it into a similarly... The embedding value of a dimensional visual feature vector. For example, a region near coordinates (800, 950) in an image exhibits blue reflection and specific texture, and its visual feature vector is V. visual1 =[0.15, -0.29, ..., 0.61], while the visual feature vector of another grassland area is V. visual2 =[0.75, 0.11, ..., -0.23], then, calculate the text semantic vector. The cosine similarity to the visual feature vector of each image region is calculated by dividing the dot product of the two vectors by the product of their magnitudes. For example, V text With V visual1 The cosine similarity calculation result is 0.91, while the similarity with V is... visual2 The cosine similarity was 0.23. The calculated similarity result was then compared with a preset semantic matching threshold. This threshold was determined by testing the model on a dataset containing 5000 "image-question-answer" pairs, selecting the minimum similarity value that would achieve a 90% recognition accuracy, and was set to 0.85. The region similarity calculation result was then compared with 0.85. visual1 The similarity score is 0.91, which is greater than 0.85. Therefore, this region is determined to be semantically matched with "water source". visual2 Regions with a similarity of 0.23 are considered mismatched (below 0.85). This process is then applied to the image regions within the field of view, filtering out regions with a similarity greater than 0.85. Assume a total of [number missing] regions are filtered out. Three adjacent matching regions with centroid coordinates of (810, 960), (825, 958), and (818, 975) are aggregated. The average coordinate value of these three regions is calculated, that is, the horizontal coordinate is (810+825+818) / 3=817.7 and the vertical coordinate is (960+958+975) / 3=964.3, forming the aggregated coordinate set of the monitoring target {(817.7, 964.3)}.

[0091] S303: Based on the aggregated coordinate set of the monitored target, compare the difference in cattle distribution density between the candidate area and the non-target area, calculate the directional offset angle and extract the offset points whose offset angles conform to the stable interval, set the offset point position parameters as the moving reference point, and generate the coordinates of the next inspection position.

[0092] Based on the generated aggregated coordinate set of the monitored target {(817.7, 964.3)}, the robot needs to plan the next movement reference point. First, within the current field of view, the area centered on the aggregated coordinate point of the monitored target is designated as the candidate area, and the remaining area is designated as the non-target area. The spatial gradient values ​​of the coordinate points in the two areas are compared. The spatial gradient value is calculated by randomly selecting 100 coordinate points in each area and calculating the cattle distribution density of the points in three-dimensional space. Since there are no cattle in the candidate area (water source), its... The average density is 0. However, in the non-target area (where the cattle herd is located), according to data from S301, there are 4 cattle within a radius of 10.8 meters, with a density of approximately 4 / (π×10.8²)≈0.011 cattle / square meter. There is a significant density difference between the two, i.e., the spatial gradient value. Next, the directional offset angle between the vector pointing from the robot's current position (0,0) to the monitored target aggregation coordinate point (817.7, 964.3) and the vector pointing to the non-target area (the center of the cattle herd) is calculated. To avoid directly crossing the cattle herd, the robot needs to... To find a direction that both faces the target and avoids the cattle, the current position is used as the vertex, and the aggregated coordinates of the monitored target are used as one edge. In a 360-degree direction centered on the cattle, a virtual path is generated every 5 degrees as another edge. The angle between the virtual path and the direction of the monitored target is calculated, i.e., the direction offset angle. Offset points whose offset angles meet the stable interval are extracted. The stable interval is set with reference to the kinematic constraints of robots and the safe approach principle in animal behavior, and is set to [20, 45] degrees. That is, the direction of travel chosen by the robot needs to deviate from the direction of directly rushing to the water source by 20 to 45 degrees, and approach in a roundabout way. Assuming that there is a point 5 meters away from the current position of the robot in the direction of offset angle of 30 degrees, there are no cattle within a 2-meter radius of the point, and the spatial gradient value is 0, the offset condition is met, so the offset point is selected. Finally, the position parameters of this offset point that meets the conditions, such as its relative coordinates (4.33, 2.50) in the robot coordinate system, are set as the moving reference point, and the coordinates of the next inspection position (4.33, 2.50) are generated.

[0093] Please see Figure 5 The specific steps of S4 are as follows:

[0094] S401: Obtain the coordinates of the next inspection location and terrain point cloud data, calculate the elevation difference value of adjacent sampling points, compare the elevation difference value according to the slope judgment threshold, mark the area above the threshold as the obstacle area and record the index position, and generate a terrain slope and obstacle distribution dataset.

[0095] The system acquires the robot's next inspection position coordinates (4.33, 2.50) and terrain point cloud data obtained from a 20-meter area scanned by LiDAR. This data contains 50,000 3D coordinate points. First, for ease of calculation, the point cloud data is projected onto a 2D grid with the robot's current position as the origin. The grid size is 20×20 meters, and each grid cell is 0.5×0.5 meters. The average elevation of the points within each grid cell is taken as the elevation value of that cell. Next, the elevation difference between adjacent grid cells is calculated. For example, for grid cell A (coordinates 2.0, ... The calculated average elevation of grid cell 1.5 is 0.45 meters. The average elevation of its adjacent grid cell B (coordinates 2.5, 1.5) in the positive X-axis direction is 0.68 meters, and the horizontal distance between the centers of the two cells is 0.5 meters. Therefore, the elevation difference between them is 0.68 - 0.45 = 0.23 meters, corresponding to a slope of arctan(0.23 / 0.5), which is approximately 24.7 degrees. Subsequently, the calculated slope value is compared with a preset slope judgment threshold, which is set based on the physical performance parameters of the robot's drive wheels and the average friction coefficient of the grass environment. The specific setup process is as follows: Consulting the robot's technical manual, we learned that its maximum climbing angle on dry asphalt surfaces is 20 degrees. Considering that the friction coefficient in a grassy environment is about 40% lower than that on asphalt surfaces, the safe climbing angle is adjusted downwards proportionally and a safety margin of 2 degrees is added, i.e., 20 × (1 - 0.4) - 2 = 10 degrees. Therefore, the slope judgment threshold is set to 10 degrees. The previously calculated slope value of 24.7 degrees is compared with this threshold of 10 degrees. Since 24.7 degrees is higher than 10 degrees, the area containing grid cells A and B is marked as an obstacle area, and its index in the grid map is recorded. For example, a location is assigned the index number Obs1, and the coordinate range of the grid cells it covers is [(2.0, 1.5), (2.5, 1.5)]. The entire grid map is traversed, and the process of calculating the elevation difference and comparing the slope is repeated for adjacent cells. Areas with a slope of less than 10 degrees are marked as passable areas, areas with a slope between 10 and 15 degrees are marked as warning areas requiring slowing down, and areas with a slope greater than 15 degrees are uniformly marked as impassable obstacle areas. Finally, the slope calculation values, area type labels, and index positions of the grid cells are integrated to generate a terrain slope and obstacle distribution dataset.

[0096] S402: Based on the terrain slope and obstacle distribution dataset, spatial mapping is performed by calling the obstacle area index and inspection location coordinates. The set of boundary points is filtered according to the Euclidean distance parameter. The features of the boundary point set of the obstacle area are weighted and fused with the target guidance parameter of the next inspection location coordinate to obtain the path reference feature set.

[0097] Based on the generated terrain slope and obstacle distribution dataset, which contains the obstacle region index Obs1 and its coordinate range [(2.0, 1.5), (2.5, 1.5)], spatial mapping is performed using this obstacle region index and the inspection position coordinates (4.33, 2.50) determined in the previous step. This marks the obstacle region on the robot navigation map, forming a virtual no-entry zone. Next, the set of boundary points surrounding Obs1 is filtered according to the Euclidean distance parameter. Specifically, the boundary points of Obs1 are first determined, i.e., the center points of the passable grid cells adjacent to the obstacle region. For example, grid cells (1.5, 1.5) and (3.0, 1.5) are both boundary points of Obs1. Then, a filtering distance parameter is set, which is 1.5 times the robot's width as the safe passage distance. If the robot width is 0.8 meters, then this parameter is 1.2 meters. The distance from the robot's current position is calculated... The system calculates a straight path from position (0, 0) to target position (4.33, 2.50) and filters out boundary points whose perpendicular distance to this path is less than 1.2 meters. For example, boundary point (3.0, 1.5) is 0.9 meters from the path, less than 1.2 meters, and is therefore retained in the boundary point set. Another boundary point (1.5, 2.5) is 1.5 meters from the path, greater than 1.2 meters, and is discarded. This results in a set of 5 key boundary points. The features of these 5 boundary points are then weighted and fused with the semantic parameters of the target position. The features of each boundary point are its coordinate position and risk value. The risk value is inversely proportional to its distance from the actual edge of the obstacle; the closer the distance, the higher the risk value. For example, the risk value of boundary point (3.0, 1.5) is 0.8. The semantic parameter of the target position (4.33, 2.50) is its attraction value, set to 1.0. The fusion weight is set according to task priority, and the obstacle avoidance weight W... obs Set to 0.6, goal-oriented weight W tar The weight is set to 0.4. The basis for this weighting is to reach the target as quickly as possible while ensuring safety. Therefore, the obstacle avoidance weight is slightly higher. A comprehensive evaluation value is calculated for each boundary point. Evaluation value = W tar × (1 / distance to target) - W obs × Risk value, for example, for the boundary point (3.0, 1.5), its distance to the target is The value of the meter is 0.4×(1 / 1.66)-0.6×0.8=0.24-0.48=-0.24. This calculation is performed on 5 boundary points to obtain the path reference feature set {(BP1, -0.24), (BP2, -0.15), (BP3, 0.05), (BP4, -0.18), (BP5, -0.31)}.

[0098] S403: Call the path reference feature set, calculate the azimuth change value and speed adjustment sequence between path nodes by the visual navigation algorithm, and perform time-series integration operation to obtain the inspection path motion instruction set;

[0099] The path reference feature set {(BP1, -0.24), (BP2, -0.15), (BP3, 0.05), (BP4, -0.18), (BP5, -0.31)} is invoked and processed by the path planning module in the visual navigation unit. This module searches for a path from the robot's current position (0, 0) to the target position (4.33, 2.50) on a virtual grid map. During the search, the passage cost of each grid is determined by whether it is an obstacle and its evaluation value in the reference feature set. The higher the evaluation value of the reference point, the more likely it is to be blocked. The lower the travel cost, the better. An optimal path consisting of a series of path nodes is obtained through search, for example: (0, 0) → (1.0, 0.5) → (2.0, 1.0) → (3.2, 1.8) → (4.33, 2.50). Then, the azimuth angle change of the travel direction represented by the lines connecting adjacent nodes in the path is calculated. The azimuth angle of the first path segment (0, 0) → (1.0, 0.5) is arctan(0.5 / 1.0), approximately 26.6 degrees. The azimuth angle of the second path segment (1.0, 0.5) → (2.0, 1.0) is also arctan(0.5 / 1.0), approximately 26.6 degrees. The azimuth change of the second segment is 0 degrees, calculated as tan((1.0-0.5) / (2.0-1.0)). The azimuth of the third segment ((2.0, 1.0) → (3.2, 1.8) is approximately 33.7 degrees, calculated as arctan((1.8-1.0) / (3.2-2.0)). Therefore, the azimuth change is 33.7-26.6=7.1 degrees. Based on this, a sequence of azimuth change values ​​[0.0, 7.1, ...] is obtained. Simultaneously, a speed adjustment sequence is generated based on the azimuth change values ​​and node positions, with the speed adjustment rule set as follows: When the azimuth change is less than 5 degrees, the speed is set to the maximum cruising speed of 0.8 m / s. When the change is between 5 and 15 degrees, the speed is adjusted to 0.5 m / s. When the change is greater than 15 degrees, the speed is reduced to 0.2 m / s. For the second path segment, since the azimuth change is 0 degrees, the speed is set to 0.8 m / s. For the third path segment, since the azimuth change is 7.1 degrees, the speed is set to 0.5 m / s. Finally, a timing integration calculation is performed to convert the path segment length, set speed, and steering angle into low-level control commands with timestamps. For example, the length of the second path segment is... The first path segment is approximately 1.44 meters long, requiring 1.12 / 0.8 = 1.4 seconds. Therefore, the generated instruction is {timestamp: t1, duration: 1.4 seconds, linear velocity: 0.8 m / s, angular velocity: 0 radians / s}. The second path segment is approximately 1.44 meters long, requiring 1.44 / 0.5 = 2.88 seconds. The corresponding angular velocity is 7.1 degrees / 2.88 seconds ≈ 2.47 degrees / s. Therefore, the generated instruction is {timestamp: t1+1.4, duration: 2.88 seconds, linear velocity: 0.5 m / s, angular velocity: 0.043 radians / s}. Integrating the instructions from the path nodes yields the inspection path motion instruction set.

[0100] Please see Figure 6 The specific steps of S5 are as follows:

[0101] S501: Based on the inspection path motion instruction set, perform pose sampling on the geographic coordinate points of the detection task area, call the pixel clustering coordinates in the cattle herd scattering image frame sequence, calculate the partition density difference and determine the central tendency direction, and generate the cattle herd scattering state matrix.

[0102] Based on the generated inspection path motion instruction set, the robot moves along the planned path and performs pose sampling on the geographic coordinates of the inspection task area during the movement. Specifically, it samples every 2 meters or every 3 seconds, recording the current GPS coordinates, heading angle, and pitch angle. Simultaneously, it calls the top wide-angle camera to capture a 1920×1080 pixel image of the cattle herd distribution. In the continuously acquired image frame sequence, each image is processed. First, the image is divided into 16 4×4 grids, each grid being 480×270 pixels. Then, image segmentation technology is used to identify individual cattle in the image, and the pixel cluster coordinates of each cattle's outline are extracted, i.e., the pixel coordinates of its geometric center point are calculated. For example, in the image frame acquired at time t1, 4 cattle are identified, and their pixel cluster coordinates are (200, 300), (250, 350), (8 ...800, 300), (800, 300), (800, 300), (800, 300), (800, 300), (800, 300), (800, 300), (800, 300), (800, (600), (850, 620) are mapped to 16 partitioned grids. The number of cattle in each partition is counted to obtain the partition density. The partition (top left corner) contains 2 cattle, the partition contains 2 cattle, and the density of the other partitions is 0. Then the partition density difference is calculated by calculating the density standard deviation of the non-zero density partitions. Here, the standard deviation is 0, which means that the cattle herd is clustered in two groups. Then the central tendency direction is determined. This process is to calculate the average value of the pixel coordinates of the cattle and obtain the centroid coordinates of the cattle herd in the image as ((200+250+800+850) / 4, (300+350+600+620) / 4)=(525, 467.5). The direction vector from the image center (960, 540) to this centroid coordinate is the central tendency direction. The robot pose, partition density, and central tendency direction vector at each sampling time are integrated to generate the cattle herd scatter state matrix.

[0103] Table 3 Example of cattle herd distribution state matrix

[0104] <![CDATA[t1]]> (10.5，8.2) 2 2 (−435，−72.5) <![CDATA[t2]]> (12.3，9.1) 1 3 (−390，−55.0) <![CDATA[t3]]> (14.1，10.0) 0 4 (−310，−20.0)

[0105] As shown in Table 3, this table records the robot's position at different sampling times, the distribution density of the cattle in the key areas, and the direction of the cattle's concentration trend in the robot's field of vision. It can be used to dynamically analyze the movement and aggregation behavior of the cattle.

[0106] S502: Based on the cattle herd dispersion state matrix, extract the individual position vector and calculate the herd centroid coordinates, obtain the coordinates of the center point of the target area to be driven, calculate the angle between the two direction vectors and compare it with the reference direction angle, and generate a herd centroid offset angle parameter set.

[0107] Based on the cattle herd dispersion state matrix at time t3, which shows that the herd has been completely concentrated within the partition, the robot extracts the individual position vectors of the four cattle in the image at that time. By combining the pixel coordinates with depth sensor data, it converts them into three-dimensional coordinates in the robot's current position (14.1, 10.0) coordinate system, namely P1 (5.1, 3.2, 0) meters, P2 (5.3, 3.5, 0) meters, P3 (5.0, 3.8, 0) meters, and P4 (5.5, 3.6, 0) meters. The centroid coordinates of the herd formed by these four cattle are then calculated. The calculation process involves adding the values ​​of each individual on multiple coordinate axes and taking the average. The centroid X coordinate = (5.1 + 5.3 + 5.0 + 5.5) / 4 = 5.225 meters, and the centroid Y coordinate = (3.2 + 3.5 + 3.8 + 3.6) / 4 = 3.525 meters, thus obtaining the herd centroid coordinates. The target area is marked as (5.225, 3.525) meters. Simultaneously, the system obtains the coordinates of the preset target area's center point, which is the pasture's watering area, located at (50.0, 45.0) meters on the global map. The robot converts these global coordinates to relative coordinates in its current coordinate system, obtaining the target point coordinates as (35.9, 35.0) meters. Next, the angle between the two direction vectors—the group's center of mass and the target area's center point—is calculated. The first vector points from the robot's current position to the group's center of mass, at (5.225, 3.525), and the second vector points from the robot's current position to the target point, at (35.9, 35.0). The dot product of the two vectors is 5.225 × 35.9 + 3.525 × 35.0 = 187.5775 + 123.375 = 310.952. The magnitudes of the two vectors are... and The cosine of the included angle is 310.9525 / (6.30×50.14)≈0.984, corresponding to an included angle of approximately 10.2 degrees. This included angle is then compared with the reference direction angle, which is set to 0 degrees. The reference direction angle is set based on the ideal driving state where the robot, the centroid of the herd, and the target point are in a straight line, at which point the included angle is 0. The comparison process involves calculating the difference between the actual included angle and the reference angle, i.e., 10.2-0=10.2 degrees. This value represents the degree of deviation of the herd centroid from the ideal driving path. This deviation angle value is stored in a set to generate the herd centroid deviation angle parameter set {10.2}.

[0108] S503: Call the group centroid offset angle parameter set, combine the orientation deflection angle control parameters of the acoustic and light device, the robot heading correction parameters and the robot moving speed control amount, perform proportional adjustment calculation on the angle difference and correct the direction and velocity vector, and generate the driving control instruction set;

[0109] The herd centroid offset angle parameter set {10.2} is invoked. This value of 10.2 degrees represents the angle by which the herd centroid deviates from the ideal driving route. The robot needs to correct its behavior by combining the angle control parameters of its onboard audio-visual device with the movement speed control value. The current orientation of the audio-visual device is the robot's forward direction, i.e., 0 degrees. The current value of the movement speed control value is the set cruising speed of 0.6 m / s. First, a proportional adjustment calculation is performed on the angle difference. The purpose of this calculation is to calculate the amount of correction required for the angle of the audio-visual device and the robot's movement direction, and to set a proportional coefficient K for angle correction. p The value is set based on experimental data on the sensitivity of cattle to sound and light stimuli, and is set to 0.7. This coefficient means that the adjustment range is 70% of the deviation each time. The angle that the sound and light device needs to be adjusted is 10.2 × 0.7 = 7.14 degrees. Since the centroid of the cattle herd is to the right of the target direction, the sound and light device needs to be deflected to the right by 7.14 degrees. At the same time, the robot's own movement direction also needs to be corrected, and a heading correction proportional coefficient K is set. d The value is set based on the robot's motion performance and the friction of the ground, and is set to 0.4. The angle that the robot's heading needs to be corrected is 10.2 × 0.4 = 4.08 degrees. The robot also adjusts its heading to the right by 4.08 degrees. Then the velocity vector is corrected. When the offset angle is greater than 5 degrees, the robot needs to decelerate for more precise adjustment. The speed adjustment rule is set as: new speed = original speed × (1 - offset angle / 100). Then the corrected speed is 0.6 × (1 - 10.2 / 100) = 0.6 × 0.898 ≈ 0.54 m / s. The calculated deflection angle of the sound and light device, the robot's heading correction angle, and the corrected movement speed are integrated to generate the driving control command set {sound and light deflection: 7.14 degrees, heading correction: 4.08 degrees, target speed: 0.54 m / s}.

[0110] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for integrating vision, language, and behavior in grassland robot inspection and herding, characterized in that: Includes the following steps: S1: Acquire grassland cattle distribution images through panoramic vision sensors, perform individual segmentation and localization on the grassland cattle distribution images, extract individual spatial coordinates, posture direction angles and group distribution density parameters, input the scene description statement and the visual feature vector composed of individual spatial coordinates, posture direction angles and group distribution density parameters into the natural language processing model, perform semantic association calculation, and generate inspection target sequence; S2: Call the inspection target sequence to drive the robot to move to the individual position, collect eye temperature value, body length-to-width ratio and standing posture angle through infrared temperature measurement and depth vision, perform threshold comparison judgment, and generate an abnormal individual identifier set when the value exceeds the limit; S3: Based on the set of abnormal individual identifiers, filter the monitoring targets, obtain visual information on the distribution of the surrounding cattle herds, input the images of the surrounding individual locations and the target screening question text into a visual question answering model for semantic matching, and output the coordinates of the next inspection location. S4: Plan the path according to the coordinates of the next inspection position, monitor the terrain slope and obstacle distribution characteristics, weight and fuse the boundary point set characteristics of the obstacle area with the target guidance parameters of the next inspection position coordinates, and calculate the turning angle sequence and speed adjustment sequence through the visual navigation algorithm to obtain the inspection path motion instruction set; S5: Based on the inspection path motion instruction set, complete the detection task, call the preset geographical coordinates of the target area to obtain the current cattle herd distribution status, calculate the deviation value of the direction angle between the herd centroid and the center point of the target area, adjust the orientation deflection angle of the sound and light device, the robot heading correction angle and the robot movement speed control amount, and generate the driving control instruction set. The driving control command set includes audio-visual output parameters, movement control quantities, and deviation correction coefficients.

2. The integrated method for grassland robot inspection and herding based on vision, language, and behavior linkage as described in claim 1, characterized in that, The inspection target sequence includes an individual identification number, a priority weight value, and a task scheduling index; the abnormal individual identifier set includes an abnormal individual number, an abnormal state category, and a risk assessment level; the next inspection location coordinates include a target spatial point, geographic reference coordinates, and a neighborhood aggregation identifier; and the inspection path motion instruction set includes a speed control parameter group, a path node sequence, and a direction angle set.

3. The integrated method for grassland robot inspection and herding based on vision, language, and behavior linkage as described in claim 1, characterized in that, The specific steps of S1 are as follows: S101: Acquire grassland cattle herd distribution images through panoramic vision sensors, analyze pixel grayscale distribution and perform region clustering segmentation, retrieve boundary point coordinates and reconstruct individual circumscribed contour matrices, calculate the horizontal and vertical coordinates of the center point and serialize them to generate individual spatial coordinate sets. S102: Based on the individual spatial coordinate set, retrieve the pixel direction vectors within the individual region and statistically analyze the angle distribution to determine the pixel gradient principal direction angle; determine the major axis direction angle of the individual's circumscribed contour according to the aspect ratio of the circumscribed rectangle; calculate the difference between the pixel gradient principal direction angle and the major axis direction angle as the individual's pose tilt angle; perform aggregation comparison on the individual pose tilt angles of neighboring individuals based on the Euclidean distance between individuals to generate an individual pose parameter set; S103: Extract visual feature vectors based on the individual posture parameter set, obtain scene description statements and input them into the natural language processing model, perform text visual feature alignment on semantic entity vectors and visual feature vectors and calculate semantic similarity, analyze priority indicators and adjust sequence weights, and generate inspection target sequences.

4. The integrated method for grassland robot inspection and herding based on vision, language, and behavior linkage as described in claim 3, characterized in that, The specific steps of S2 are as follows: S201: Obtain the individual location information of the inspection target sequence identifier, drive the robot to move to the coordinate position, collect the surface temperature distribution through infrared thermometry, calculate the weighted average value of the infrared pixel energy of the eye and convert the temperature parameter to generate an eye temperature value dataset. S202: Based on the eye temperature dataset, three-dimensional structural data frames are acquired through depth vision, the coordinates of the body surface boundary and skeleton nodes are extracted, the body length-to-width ratio is calculated, and the posture angle is obtained according to the center of gravity axis to generate a morphological posture feature matrix. S203: Based on the morphological and posture feature matrix and the eye temperature value dataset, call the threshold comparison unit to perform difference judgment on the feature parameters and the threshold benchmark set, filter out the individual indexes that deviate from the threshold range and aggregate them to generate an abnormal individual identifier set.

5. The integrated method for grassland robot inspection and herding based on vision, language, and behavior linkage as described in claim 4, characterized in that, The specific steps for S3 are as follows: S301: Obtain the set of abnormal individual identifiers, collect visual information on the distribution of cattle herds around the individual's location area, match the outline boundary pixel array with the spatial coordinate point cloud, calculate the Euclidean distance and compare it with the herd density threshold, cluster the distance areas below the threshold, and generate a distribution map of the surrounding individual locations. S302: Based on the surrounding individual location distribution map, a visual question answering model is used to parse the target question text to obtain semantic vectors, extract visual feature vector embedding values ​​and calculate cosine similarity, compare the results with semantic matching thresholds, aggregate the centroid coordinates of regions greater than the thresholds, and form a set of aggregated coordinates of monitoring targets. S303: Based on the aggregated coordinate set of the monitored target, compare the difference in cattle distribution density between the candidate area and the non-target area, calculate the directional offset angle and extract the offset points whose offset angles conform to the stable interval, set the offset point position parameters as the moving reference point, and generate the coordinates of the next inspection position.

6. The integrated method for grassland robot inspection and herding based on vision, language, and behavior linkage as described in claim 5, characterized in that, The calculation of Euclidean distance and comparison with the population density threshold refers to: based on the synchronously acquired depth information, converting the two-dimensional pixel coordinates of the contour boundary pixel array into three-dimensional spatial coordinates, calculating the Euclidean distance between each three-dimensional coordinate point after conversion, and obtaining the spatial distance matrix. The population density threshold is determined based on a range of values ​​obtained through statistical fitting of the average visible distance between individuals in different terrain environments.

7. The integrated method for grassland robot inspection and herding based on vision, language, and behavior linkage as described in claim 5, characterized in that, The specific steps of S4 are as follows: S401: Obtain the coordinates of the next inspection location and the terrain point cloud data, calculate the elevation difference value of adjacent sampling points, compare the elevation difference value according to the slope determination threshold, mark the area above the threshold as the obstacle area and record the index position, and generate a terrain slope and obstacle distribution dataset. S402: Based on the terrain slope and obstacle distribution dataset, call the obstacle area index and inspection location coordinates to perform spatial mapping, filter the boundary point set according to the Euclidean distance parameter, and perform weighted fusion of the boundary point set features of the obstacle area with the target guidance parameter of the next inspection location coordinates to obtain the path reference feature set; S403: Call the path reference feature set, use the visual navigation algorithm to calculate the azimuth change value and speed adjustment sequence between path nodes, and perform time-series integration operation to obtain the inspection path motion instruction set.

8. The integrated method for grassland robot inspection and herding based on vision, language, and behavior linkage as described in claim 7, characterized in that, The slope determination threshold value is adaptively adjusted based on the statistical distribution characteristics of the elevation difference values ​​in the terrain point cloud data; The weighted fusion of the boundary point set features of the obstacle area and the target guidance parameters of the next inspection position coordinates means that by assigning obstacle avoidance weight factors and target guidance weight factors to the boundary point set features of the obstacle area and the target guidance parameters of the next inspection position coordinates respectively, and performing a weighted summation operation, a path reference feature set is obtained.

9. The integrated method for grassland robot inspection and herding based on vision, language, and behavior linkage as described in claim 1, characterized in that, The specific steps of S5 are as follows: S501: Based on the inspection path motion instruction set, perform pose sampling on the geographic coordinate points of the detection task area, call the pixel clustering coordinates in the cattle herd scattering image frame sequence, calculate the partition density difference and determine the central tendency direction, and generate the cattle herd scattering state matrix. S502: Based on the cattle herd distribution state matrix, extract individual position vectors and calculate the herd centroid coordinates, obtain the coordinates of the center point of the target area to be driven, calculate the angle between the two direction vectors and compare it with the reference direction angle, and generate a herd centroid offset angle parameter set. S503: Call the group centroid offset angle parameter set, combine the orientation deflection angle control parameters of the acoustic and optical device, the robot heading correction parameters and the robot moving speed control quantity, perform proportional adjustment calculation on the included angle deviation value and correct the direction and velocity vector, and generate a driving control command set.

Images