Lawn semantic segmentation path planning method and system, and lawn mower

Abstract

The application belongs to the technical field of intelligent mowing equipment, computer vision and mobile robot path planning, and discloses a lawn semantic segmentation path planning method, a system and a mower. Visible light and near-infrared images are synchronously collected and time-matched, and multispectral data is generated through distortion correction and spatial alignment. Semantic segmentation is realized through machine learning technology, and model output is the probability of lawn, hardened ground, obstacles and other area categories. The ground semantic grid is constructed in combination with camera calibration parameters and mower pose projection. A passing cost map is generated relying on semantic categories, uncertainty and safety boundaries, the lawn workable area is extracted, the coverage path segment is generated by matching the mowing width, the work path is obtained by cost routing and splicing, and multiple frames of semantic results are aligned to the unified ground coordinate during work. Scene changes are detected and local re-planning is performed, and the pruning effect is determined in combination with the front and rear images and the actuator load, and the area to be worked is updated.

Description

Technical Field

[0001] This invention belongs to the technical fields of intelligent lawn mowing equipment, computer vision, artificial intelligence and mobile robot path planning, specifically involving a lawn semantic segmentation path planning method, system and lawn mower. Background Technology

[0002] Autonomous lawn mowing equipment needs to define its operating area within an outdoor environment where lawns, paved surfaces, green boundaries, trees, rocks, flower beds, drainage facilities, and temporary obstacles coexist. Within this area, it must create a mowing path that covers the lawn. Outdoor lawn environments are affected by factors such as sunlight direction, cloud cover, tree shadows, lawn moisture content, grass species, grass density, dead grass cover, and seasonal variations. When determining lawn areas solely based on visible light color, shaded lawns may be identified as non-lawn areas, and areas resembling artificial carpets, green plastic, or moss may also be identified as lawn areas.

[0003] Near-infrared (NIR) images provide vegetation reflectance information that differs from visible light images. Using both visible and NIR images for lawn identification can supplement spectral information beyond color and texture. However, visible light and NIR imaging devices may have different resolutions, fields of view, lens distortions, and installation locations. Without proper temporal and spatial alignment, the same ground location may correspond to different pixel positions, and directly stitching the two types of images can easily create misalignment features at lawn boundaries and obstacle edges.

[0004] Existing lawn visual recognition methods typically output lawn masks, lawn boundaries, or classification results for grass versus non-grass. Pixel-level segmentation results are still in image coordinates, while the lawnmower's walking mechanism needs to receive the target position and heading in ground coordinates. Without camera calibration and lawnmower pose projection, the segmentation results cannot be directly used as executable paths.

[0005] Lawn mowing is a regional coverage task. Point-to-point path search can only obtain the traversable path from the starting point to the destination, and cannot guarantee that the mowing machine will cover the entire workable area of ​​the lawn. If the current position of the lawnmower is directly set as the starting point and the centroid of the lawn area is set as the destination, the generated path only solves the arrival problem, and cannot solve the problems of forming coverage path segments, accessing coverage path segments, connecting adjacent coverage path segments, and re-mowing missed areas.

[0006] Semantic segmentation results may exhibit high uncertainty in areas such as lawn boundaries, shaded regions, and areas with small obstacles. Simply converting the class with the highest probability to a fixed passage cost will miss the instability in judgment reflected when the probabilities of different classes are close. Path planning results may approach uncertain boundaries, and path smoothing may cause the smoothed path to deviate from the original search path and enter obstacle regions.

[0007] The lawnmower moves continuously during operation. When directly comparing the segmentation results of adjacent image frames, changes in camera viewpoint can cause changes in pixel positions, potentially resulting in significant differences even if the ground scene remains unchanged. Grass swaying, short-term shadows, and single-frame recognition errors can also trigger unnecessary path replanning.

[0008] The fact that a lawnmower passes through a planned area does not mean that the mowing actuator has actually mowed that area. Wheel slippage, path tracking deviation, actuator stoppage, fallen grass, or temporary obstruction by obstacles can all cause missed mowing. Calculating coverage solely based on the theoretical travel trajectory is insufficient to reflect the actual completion status of the operation. Summary of the Invention

[0009] The purpose of this invention is to provide a method, system, and lawnmower for lawn semantic segmentation path planning, which solves the problems of visible light single information being affected by outdoor light, unclear temporal and spatial relationships of bispectral images, semantic segmentation results not being directly converted into ground operation paths, point-to-point paths not being able to achieve lawn coverage, dynamic changes easily triggering replanning, and theoretical path coverage not reflecting the actual mowing status.

[0010] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0011] Grassland semantic segmentation path planning methods include:

[0012] Visible light imaging devices and near-infrared imaging devices installed on the lawnmower are used to acquire visible light images and near-infrared images of the lawn scene with a time correspondence.

[0013] Based on the calibration parameters of the visible light imaging device and the near-infrared imaging device, distortion correction and spatial alignment are performed on the visible light image and the near-infrared image of the lawn scene to form multispectral input data.

[0014] By inputting multispectral data into a semantic segmentation model, the category information of image pixels belonging to lawn areas, hardened ground areas, obstacle areas, and other areas can be obtained.

[0015] Based on the calibration parameters and the lawnmower pose, the category information is projected onto the ground coordinates to form a ground semantic grid;

[0016] A ground access cost map is generated based on the semantic categories in the ground semantic grid, and the workable area of ​​the lawn is determined from the ground semantic grid;

[0017] Based on the mower's mowing working width, a coverage path segment is formed within the lawn's workable area. A transfer path is generated between adjacent coverage path segments based on the ground traffic cost map. The coverage path segments and transfer paths are then connected to form the mowing operation path.

[0018] The mowing path is converted into the target position and target heading that the mower can execute, and the walking mechanism and mowing execution mechanism of the mower are controlled to perform the mowing operation according to the mowing path.

[0019] As a preferred technical solution of the present invention, the formation of multispectral input data includes: adding acquisition timestamps to the visible light image and the near-infrared image of the lawn scene, and selecting image frames whose timestamp difference is not greater than the time pairing threshold as multispectral image pairs;

[0020] Based on the intrinsic parameters of the visible light imaging device, the intrinsic parameters of the near-infrared imaging device, and the relative extrinsic parameters between the visible light imaging device and the near-infrared imaging device, distortion correction and coordinate transformation are performed respectively.

[0021] The spatially aligned red, green, blue, and near-infrared channels are stacked according to their pixel positions to form a four-channel multispectral input tensor.

[0022] As a preferred embodiment of the present invention, after forming the multispectral input data, it further includes:

[0023] The reliability of visible light is determined based on the local brightness, local saturation, and local contrast of the visible light image of the lawn scene;

[0024] Near-infrared reliability is determined based on the local response intensity, saturated pixel ratio, and weak response pixel ratio of near-infrared images of a lawn scene.

[0025] The registration reliability is determined based on the local registration residual between the visible light image and the near-infrared image of the lawn scene.

[0026] By utilizing visible light reliability, near-infrared reliability, and registration reliability, the visible light semantic features and near-infrared semantic features are spatially weighted and fused to obtain fused semantic features, which are then input into the decoding processing part of the semantic segmentation model.

[0027] As a preferred embodiment of the present invention, the semantic segmentation model outputs the probability of each image pixel belonging to the grass region, hardened ground region, obstacle region, and other regions; the formation of the ground access cost map includes:

[0028] Project category probabilities onto a ground semantic grid;

[0029] The degree of semantic uncertainty is determined based on the category probability distribution corresponding to the same ground semantic raster unit;

[0030] The basic passage cost is determined based on the semantic category, and the additional uncertainty cost is determined based on the degree of semantic uncertainty.

[0031] The basic access cost and the uncertainty-added cost are combined into a continuous access cost for the ground semantic raster unit.

[0032] As a preferred embodiment of the present invention, the method of forming a ground access cost map further includes:

[0033] The basic safety distance is determined based on the outer contour of the lawnmower body, the outer contour of the mowing actuator, and the positioning error.

[0034] The additional braking distance is determined based on the current speed of the lawnmower, braking response time, and braking deceleration; the dynamic safety distance is determined based on the basic safety distance and the additional braking distance.

[0035] The obstacle area is expanded outward according to the dynamic safety distance, and the outer boundary and inner prohibited operation boundary of the lawn work area are shrunken inward according to the dynamic safety distance, forming a dynamic safety buffer zone;

[0036] Set the dynamic safety buffer as a no-passage zone or a high-passage-cost zone.

[0037] As a preferred embodiment of the present invention, forming a lawn mowing path includes:

[0038] Identify connected regions within the workable area of ​​the lawn;

[0039] The spacing between adjacent cover path segments is determined based on the mowing work width and the allowable overlap width of adjacent work strips;

[0040] Along the candidate coverage direction, intersect each workable area of ​​the connected lawn with the parallel work line to form a coverage path segment;

[0041] Determine the access order of the covered path segments;

[0042] The dynamic weighted A-satellite path search is used to generate transfer paths between adjacent coverage path segments in the ground access cost map. The heuristic weight of the dynamic weighted A-satellite path search is 1.4 in the early stage of the search and decreases linearly to 1.0 as the number of expanded nodes increases.

[0043] The path formed by connecting the covered path segment and the transfer path is smoothed, and collision verification and boundary verification are performed on the smoothed path according to the ground traffic cost map.

[0044] As a preferred embodiment of the present invention, determining the candidate coverage direction includes:

[0045] For multiple candidate coverage directions, separate coverage path segments are generated;

[0046] Calculate the coverage path length, number of turns, transfer path cost, and boundary risk cost for each candidate coverage direction;

[0047] The target coverage direction is selected based on the weighted result of the coverage path length, number of turns, transfer path cost, and boundary risk cost.

[0048] As a preferred embodiment of the present invention, the lawn mowing process further includes:

[0049] Continuously acquire visible light images and near-infrared images of the lawn scene to obtain the category probabilities at consecutive time points;

[0050] Based on the lawnmower pose at each time step, the class probabilities of consecutive time steps are projected onto the same ground coordinates; the consecutive class probabilities of the same ground semantic raster unit are fused in time to form a spatiotemporally aligned ground semantic raster.

[0051] The scene change region is determined based on the difference between the spatiotemporally aligned ground semantic raster at the current moment and the spatiotemporally aligned ground semantic raster at historical moments.

[0052] As a preferred embodiment of the present invention, when the number of consecutive frames in which the scene change area exists reaches a change duration threshold, and the scene change area intersects with the path to be mowed or the distance between the scene change area and the path to be mowed is less than the path influence distance, the affected path interval is determined, and the change duration threshold is 3 frames; the estimated arrival time is determined based on the current position of the lawnmower, its speed, and the position of the scene change area; when the estimated arrival time is less than the response time threshold, local replanning is performed on the affected path interval, and unaffected coverage path segments and transfer paths are retained; after the number of consecutive frames in which the scene change area disappears reaches a change removal threshold, the scene change marker is removed, and the change removal threshold is 5 frames.

[0053] As a preferred embodiment of the present invention, the lawn mowing process further includes: determining the theoretically covered area based on the lawnmower's executed path and the outline of the mowing actuator; acquiring multispectral image change information of the theoretically covered area before and after the mowing operation, and acquiring the load information of the mowing actuator; marking the ground semantic grid cells as trimmed, untrimmed, or with uncertain trimming status based on the multispectral image change information and the load information of the mowing actuator; re-adding continuous ground areas in the untrimmed or uncertain trimming status to the remaining work area; and generating the remaining coverage path based on the updated remaining work area.

[0054] As a preferred embodiment of the present invention, the lawn mowing process further includes: recording at least one execution anomaly event among emergency stop events, manual takeover events, collision protection events, lawn mowing actuator protection events, and continuous wheel slippage events, and recording the ground position corresponding to the execution anomaly event; determining the event impact area centered on the ground position corresponding to the execution anomaly event; increasing the historical risk cost of the event impact area and writing the historical risk cost into the ground passage cost map; and reducing the historical risk cost according to a preset attenuation range after the lawn mower passes through the event impact area without any execution anomalies.

[0055] This invention also discloses a lawn semantic segmentation path planning system for performing the lawn semantic segmentation path planning method described above, comprising:

[0056] A multispectral image acquisition module is used to acquire visible light images and near-infrared images of a lawn scene with a time-corresponding relationship.

[0057] The image alignment module is used to perform distortion correction and spatial alignment on the visible light image and near-infrared image of the lawn scene according to the calibration parameters, so as to form multispectral input data.

[0058] The semantic segmentation module is used to input multispectral input data into the semantic segmentation model to obtain the category information of image pixels belonging to lawn areas, hard ground areas, obstacle areas and other areas;

[0059] The ground semantic raster generation module is used to project category information onto ground coordinates based on calibration parameters and lawnmower pose to form a ground semantic raster.

[0060] The access cost map generation module is used to generate a ground access cost map based on the semantic categories in the ground semantic grid, and to determine the workable area of ​​the lawn from the ground semantic grid;

[0061] The coverage path planning module is used to form coverage path segments within the workable area of ​​the lawn based on the mowing work width, form transfer paths between adjacent coverage path segments based on the ground access cost map, and connect the coverage path segments and transfer paths into the mowing work path.

[0062] The path output module is used to convert the mowing operation path into the target location and target heading that the lawnmower can execute.

[0063] As a preferred technical solution of the present invention, a cross-spectral reliability fusion module is used to perform weighted fusion of visible light semantic features and near-infrared semantic features based on visible light reliability, near-infrared reliability, and registration reliability; a dynamic update module is used to project the category probabilities of consecutive moments onto the same ground coordinates to form a spatiotemporally aligned ground semantic grid, and update the mowing operation path according to the scene change area; an actual mowing state determination module is used to determine the actual mowing state based on the multispectral image change information before and after the mowing operation and the load information of the mowing execution mechanism; and a risk cost correction module is used to correct the historical risk cost in the ground access cost map according to the ground location corresponding to the execution abnormal event.

[0064] Meanwhile, this invention also discloses a lawnmower, including a body, a walking mechanism, a mowing execution mechanism, a visible light imaging device, a near-infrared imaging device, a pose acquisition device, a processor, and a memory; the visible light imaging device and the near-infrared imaging device are disposed on the body and are used to acquire visible light images and near-infrared images of the lawn scene; the pose acquisition device is used to acquire the pose of the lawnmower; the memory stores a computer program; when the processor executes the computer program, it implements the lawn semantic segmentation path planning method described above, and controls the walking mechanism and the mowing execution mechanism according to the mowing operation path.

[0065] As a preferred technical solution of the present invention, the processor determines the theoretically covered area based on the executed path and the outline of the mowing execution mechanism, determines the actual trimming state based on the multispectral image change information of the theoretically covered area before and after the mowing operation and the load information of the mowing execution mechanism, adds the untrimmed area back into the remaining working area, and controls the walking mechanism and the mowing execution mechanism to perform supplementary mowing operations along the remaining covered path.

[0066] As a preferred embodiment of the present invention, after performing a weighted fusion of visible light semantic features and near-infrared semantic features based on spatial location correspondence according to visible light reliability, near-infrared reliability, and registration reliability, the method further includes:

[0067] The information reliability of image pixels is determined based on visible light reliability, near-infrared reliability, and registration reliability, and the information reliability is projected onto the ground semantic grid according to the probability of image pixels belonging to grass area, hard ground area, obstacle area, and other areas.

[0068] Ground semantic grid cells with a probability of grass area not less than 0.60, a probability of obstacle area not less than 0.20, and an information reliability not less than 0.50 are identified as candidate grass workable grid cells. After connecting region aggregation and removal of isolated grid cells with an area smaller than the minimum effective work area, a grass workable area is formed.

[0069] Ground semantic raster cells with information reliability below 0.50 are excluded from coverage path segments, and the passage cost of ground semantic raster cells with information reliability below 0.50 in the ground passage cost map is increased;

[0070] Ground semantic grid cells with information reliability below 0.50 are allowed to participate in the formation of transfer paths only when there is no low-risk transfer path that bypasses ground semantic grid cells with information reliability below 0.50, and the probability of obstacle areas for ground semantic grid cells with information reliability below 0.50 is below 0.20.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] Visible light and near-infrared images are combined for semantic segmentation after temporal pairing and spatial alignment, which can reduce information loss caused by changes in lighting, shadows, and lawn color. The semantic segmentation results are then projected onto ground coordinates using camera calibration relationships and lawnmower pose, forming a ground semantic grid that the lawnmower can use for path planning, thus establishing a coordinate relationship between image recognition results and equipment motion control.

[0073] Different access rules are applied to lawn areas, paved ground areas, obstacle areas, and other areas. Semantic uncertainty is further converted into additional risk costs, avoiding the loss of risk information caused by hard judgment based solely on the highest probability category. Coverage path segments handle lawn mowing tasks, while transfer paths handle coverage path segment connections and obstacle avoidance tasks, preventing the use of single point-to-point paths to replace area coverage paths. Dynamic safety buffers are associated with the lawnmower's outer contour, the mowing actuator's outer contour, and braking distance, enabling image boundaries to be converted into safety boundaries with meanings related to equipment size and motion status. Continuous semantic segmentation results are aligned in the same ground coordinate system before change detection, reducing spurious changes caused by lawnmower movement.

[0074] Feedback on actual mowing status is used to update the remaining work area, identifying areas that the path has traversed but where mowing has not been completed, and generating supplementary mowing paths. The reliability of cross-spectral information can be transferred from the semantic feature fusion stage to the ground semantic grid and path planning stages, enabling high-reliability lawn areas to form covered path segments, excluding low-reliability areas from these segments, and using them only as restricted transfer areas when safety conditions are met, thus balancing lawn coverage capacity and access safety in uncertain areas. Attached Figure Description

[0075] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0076] Figure 1 This is the main flowchart of the lawn semantic segmentation path planning method of the present invention.

[0077] Figure 2 This is a flowchart of the multispectral input data generation process of the present invention.

[0078] Figure 3 is a flowchart of the cross-spectral reliability semantic fusion processing of the present invention.

[0079] Figure 4 is a block diagram of the hardware structure of the lawnmower of the present invention. Detailed Implementation

[0080] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0081] The following is in conjunction with the appendix Figures 1-4 The embodiments of the present invention will be described in detail below.

[0082] The overall processing architecture of this invention uses an image coordinate system, a lawnmower body coordinate system, a ground local coordinate system, and a work map coordinate system to describe data relationships. The image coordinate system has its origin at the top left corner of the image, with the horizontal pixel direction as the first image axis and the vertical pixel direction as the second image axis. The lawnmower body coordinate system is fixed to the body, with the forward direction of the body as the first body axis and the horizontal direction of the body as the second body axis. The ground local coordinate system has its origin at a ground reference point near the current position of the lawnmower. The work map coordinate system is used to accumulate and store the ground semantic raster and executed paths for the entire work area. The multispectral image acquisition module outputs visible light and near-infrared images of the lawn scene with acquisition timestamps. The image alignment module outputs multispectral input data corresponding to the spatial location. The semantic segmentation module outputs the four-category probability corresponding to each image pixel. The ground semantic raster generation module outputs the ground semantic raster. The access cost map generation module outputs the ground access cost map and the workable area of ​​the lawn. The coverage path planning module outputs the coverage path segment, the transfer path, and the lawn mowing operation path. The path output module outputs the target location, target heading, and start / stop status of the mowing actuator.

[0083] Each ground semantic raster cell in the ground semantic raster records at least the ground location, probability of lawn areas, probability of hardened ground areas, probability of obstacle areas, probability of other areas, information reliability, and update time. Ground semantic raster cells may also record semantic uncertainty, basic access cost, boundary risk cost, historical risk cost, actual mowing status, and most recent operation time.

[0084] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further illustrated below with reference to specific implementation examples.

[0085] Example 1: In this example, a visible light imaging device and a near-infrared imaging device are installed at the front of the lawnmower. The visible light imaging device outputs red, green, and blue channel images with a resolution of 1280×720 pixels. The near-infrared imaging device outputs a single-channel near-infrared image with a resolution of 1280×720 pixels, configured with a filter component with a center wavelength of 850nm and a passband width of 40nm. The acquisition frequency of both imaging devices is set to 20Hz. A unified trigger controller sends acquisition trigger signals to both imaging devices and records a microsecond-level acquisition timestamp for each frame of image. The time pairing threshold is set to 10ms; when the difference in acquisition timestamps between two frames is not greater than 10ms, a multispectral image pair is formed; otherwise, the earlier frame is discarded. The installation height is set to 0.55m, the downward tilt angle of the imaging optical axis relative to the horizontal plane is set to 18 degrees, the installation position and angle are kept fixed, and the pitch and roll angles of the machine participate in ground projection compensation.

[0086] Before deployment, the visible light imaging device and the near-infrared imaging device were calibrated using a planar calibration board to obtain intrinsic parameters, distortion parameters, and the rotation and translation relationships between the coordinate systems of the two devices. The image alignment module used the intrinsic parameters and distortion parameters to perform distortion correction, and then mapped the near-infrared image to the visible light image coordinates through rotation and translation relationships. Local registration checks were performed on the mapped images, selecting the boundary between the lawn and hardened ground, the edge of the tree trunk, and the edge of the flower bed as structural edges. The distance between the structural edges of the two types of images was calculated, and local areas exceeding 3 pixels were marked as low registration reliability areas. After spatial alignment, the red, green, blue, and near-infrared channels were stacked at the same pixel positions to form a 720×1280×4 four-channel multispectral input tensor. The semantic segmentation model input size was 512×512 pixels, cropped from the common effective field of view and scaled to 512×512×4 to maintain aspect ratio.

[0087] Furthermore, the red, green, and blue channels undergo brightness range normalization, while the near-infrared channel is normalized based on the effective response range, with values ​​between 0 and 1. In scenes with uneven illumination, contrast-limited adaptive histogram equalization is performed on the visible light brightness component, with a grid size of 8×8 and a local contrast limit parameter of 2.0. The near-infrared channel uses a 3×3 median filter to remove isolated response points, and the visible light channel uses a 5×5 Gaussian filter to suppress noise, with a standard deviation of 1.2. Gaussian filtering may be omitted when image noise is low, and median filtering may be omitted when there are no isolated response points, maintaining spatial correspondence across the four channels in the output.

[0088] Furthermore, an encoder-decoder structure is adopted, with the first layer convolution having 4 input channels, enabling all four channels to participate in feature extraction. The encoding process outputs semantic features at four spatial scales: high-resolution features preserve grass texture, lawn edges, and small obstacle outlines; medium-resolution features represent continuous regions and boundary shapes; and low-resolution features represent the contextual relationships between the lawn and roads, trees, and flower beds. The cross-scale feature fusion process adjusts the channels and sizes, progressively transferring high-level semantic information to the high-resolution features. The decoding process outputs a 512×512×4 class probability tensor, with the four channels corresponding to the lawn region, hardened ground region, obstacle region, and other regions, respectively. The sum of the four class probabilities for each pixel is 1. The lawn region is the area of ​​mowable herbaceous vegetation; the hardened ground region includes paved roads, concrete surfaces, stone paths, and other surfaces that allow movement but should not activate the lawnmower; the obstacle region includes tree trunks, stones, flower bed fencing, and other impassable objects; and other regions include areas with insufficient information, severe occlusion, or areas not covered by training annotations. The semantic segmentation model does not directly output binarized results but retains the four class probabilities.

[0089] The pose acquisition device outputs the lawnmower's two-dimensional position, heading angle, pitch angle, and roll angle in the work map coordinate system, achieved through a combination of satellite positioning, inertial measurement, wheel speed measurement, or visual odometry. When the ground is approximately flat, a homography mapping from image coordinates to ground coordinates is established based on the intrinsic and extrinsic parameters of the visible light imaging device, the lawnmower's pose, and ground plane parameters. Four categories of probabilities are projected onto the local ground coordinates and then transformed back to the work map coordinates. When there is a gentle slope, the pitch and roll angles are used to correct the ground plane orientation. The side length of the ground semantic grid is set to 0.05m. When a grid cell receives multiple pixel projections, they are weighted and accumulated according to projection distance, incident angle, and information reliability. When a new projection is merged with historical projections, the more recent and reliable projection has a higher weight, while historical projections gradually decay.

[0090] Furthermore, in the basic implementation, the basic passage cost is 1 for lawn areas, 6 for hardened ground areas, and 12 for other areas. Obstacle areas are set to prohibit expansion. Lawn areas are used to form covered path segments; hardened ground areas do not form covered path segments but can form transfer paths, in which case the mowing execution mechanism stops working; other areas are only allowed to pass at low speed when there are no low-risk connecting paths and the obstacle probability is below the safety threshold. For lawn workable areas, grid cells with a lawn area probability of not less than 0.60, an obstacle area probability of not less than 0.20, and an information reliability of not less than 0.50 are selected. After connectivity aggregation, isolated areas with an area smaller than the minimum effective work area are deleted, and holes with an obstacle probability of not less than 0.20 are not filled.

[0091] The mowing working width is 0.42m, with an allowable overlap width of 0.04m and a center-to-center distance of 0.38m between adjacent mowing path segments. The workable area of ​​the lawn is recessed along the outline of the mowing machine and the outer contour of the mowing actuator. Parallel work lines with a spacing of 0.38m are generated along the target coverage direction. Line segments are obtained by intersecting these recessed areas; segments with a length of at least 0.50m are retained as mowing path segments, otherwise they are merged into adjacent segments or form local re-mowing paths. A reciprocating access sequence is adopted. When internal obstacles exist, a single parallel work line may form multiple separate mowing path segments. The access order is determined based on the cost of the transfer path.

[0092] In this embodiment, a dynamic weighted A-satellite path search is employed. Nodes correspond to ground semantic grid cells, and the cumulative cost consists of the distance traveled, ground travel cost, and turning cost. The initial heuristic weight is 1.4, which linearly decreases to 1.0 and remains constant as more nodes are added. Termination conditions are: the target endpoint enters the closed set, the open set is empty, or the number of expanded nodes reaches the computational budget. In the latter two cases, the access order is reselected for the search.

[0093] Path smoothing and safety verification are as follows:

[0094] After connecting the covered path segment and the transfer path, third-order B-spline smoothing is performed on the discrete inflection points in the transfer path, while the covered path segment remains a straight line or the original curve. Resampling is performed at 0.05m intervals, and the ground access cost map is queried for each sampling point. Smoothed segments entering obstacle areas, dynamic safety buffer zones, or areas outside the workable lawn area are deemed invalid, and the path is reverted to its original state or smoothed again after adding boundary constraints. If the curvature exceeds the allowable value, a turning transition point is added or the local speed is reduced. The path output module generates a target position sequence at 0.10m intervals and determines the target heading. The covered path segment corresponds to the working state of the mowing actuator, and the transfer path switches to a stopped state when located on hardened ground.

[0095] Furthermore, the semantic segmentation model is trained as follows:

[0096] Training data was collected from various lawn locations under different lighting conditions, weather conditions, and lawn statuses, including direct sunlight on sunny days, shade, cloud cover, low light in the early morning, withered grass, fallen leaves, water accumulation and reflection, and damp hardened surfaces. Each dataset includes visible light images, near-infrared images, timestamps, calibration parameters, and manually labeled masks. Labeling categories are lawn areas, hardened surface areas, obstacle areas, and other areas. Areas covered in shadow but with confirmed mowing vegetation are labeled as lawn; areas with sparse fallen leaves that allow passage are labeled as lawn; completely occluded areas are labeled as other; low stones or other objects that may damage the mowing actuators are labeled as obstacles; paved roads are labeled as hardened surfaces; and areas that are severely blurred, have saturated reflections, or are occluded, making category determination impossible, are labeled as other. Training, validation, and test sets are divided by lawn location, with adjacent frames not spanning multiple sets. During training, synchronous geometric augmentation is performed on multispectral image pairs. All four channels maintain spatial correspondence using the same cropping, rotation, scaling, and flipping parameters. Brightness, contrast, and color temperature perturbations are applied to the visible light channel, while gain and local noise perturbations are applied to the near-infrared channel. The maximum spatial registration perturbation displacement is 2 pixels. The training loss consists of weighted cross-entropy loss, region overlap loss, and boundary loss. The optimizer's initial learning rate is 0.001. Training stops and the optimal parameters are saved when the validation set's overall performance index fails to improve for 12 consecutive rounds. During model inference, the four-channel input undergoes the same normalization, and the output probabilities of the four categories are not binarized. When the near-infrared device is temporarily unavailable, the model switches to a degradation mode or fills the near-infrared channel with the training data mean, while increasing the base passage cost in other regions and reducing the maximum driving speed.

[0097] Furthermore, the details of the hierarchical generation of the covered path segments and transfer paths are as follows:

[0098] An eight-neighborhood connectivity analysis is performed on the workable area of ​​the lawn, and each connected area forms an independent work sub-region. Areas smaller than 0.30m² do not enter the automatic cover path and can be marked as manual treatment areas or low-speed mowing areas. When the width of the narrow passage caused by internal obstacles is less than the safe passage width of the lawnmower, the areas on both sides are treated as different sub-regions.

[0099] In a workable lawn area measuring 18.0m long and 12.0m wide, with a mowing work width of 0.42m, an overlap width of 0.04m, and a center-to-center spacing of 0.38m, arranging work strips along the width direction can form 31 complete coverage path segments and boundary compensation segments. The start and end points of the coverage path segments are located outside the dynamic safety buffer zone. When crossing internal obstacles, they split into sub-path segments on both sides of the buffer zone and are assigned access sequences. The access sequence is arranged in a reciprocal manner according to the parallel work line numbers, with odd and even numbers in opposite directions. When multiple separate coverage path segments exist, an endpoint connection graph is constructed, where nodes are endpoints and edge weights are estimated transfer path costs. Starting from the coverage path segment closest to the current location and with a higher mowing priority, the next segment with a lower connection cost is selected while maintaining the overall reciprocal order.

[0100] The candidate movement directions for the dynamic weighted A-star path search include forward, backward, left, right, and four diagonals. The cumulative cost of each node includes movement distance, continuous passage cost, and turning cost. When passing through hardened ground areas, the path output module issues a stop command to the mowing actuator before entering, and restarts after re-entering the lawn. Coverage rate is calculated by dividing the intersection area of ​​the theoretical sweeping area and the workable lawn area by the total workable area. Repeat coverage rate is calculated by dividing the repeat coverage area by the total workable area. The stopping condition for coverage path planning is that the remaining uncovered continuous area is all less than the minimum re-mowing area, or the remaining area cannot be reached under dynamic safety constraints; unreachable areas are output separately.

[0101] Furthermore, the joint selection method for coverage directions is as follows: candidate coverage directions are generated at 15-degree intervals from 0 degrees to 165 degrees, and a coverage path segment and an estimated transfer path are generated for each candidate direction. The comprehensive evaluation value is calculated using the following formula:

[0102] ;

[0103] in For direction The overall evaluation value; , , , These are the weights for coverage path length, number of turns, boundary risk cost, and transfer path cost, respectively. , , , Directions The corresponding normalized coverage path length, normalized turning number, normalized boundary risk cost, and normalized transfer path cost.

[0104] For example, weight , , , The values ​​were set to 0.35, 0.25, 0.25, and 0.15 respectively. In a set of candidate direction data, the normalized coverage path length for the 0-degree direction was 0.82, the normalized number of turns was 0.65, the normalized boundary risk cost was 0.35, the normalized transfer path cost was 0.55, and the comprehensive evaluation value was 0.620; the normalized coverage path length for the 30-degree direction was 0.76, the normalized number of turns was 0.40, the normalized boundary risk cost was 0.22, the normalized transfer path cost was 0.38, and the comprehensive evaluation value was 0.478; the normalized coverage path length for the 60-degree direction was 0.88, the normalized number of turns was 0.55, the normalized boundary risk cost was 0.20, the normalized transfer path cost was 0.45, and the comprehensive evaluation value was 0.563. The 30-degree direction with the lowest comprehensive evaluation value was selected. When there are many candidate directions, they can be first filtered according to the main axis direction of the workable area of ​​the lawn to reduce the computational load.

[0105] Example 2: This example adds a cross-spectral reliability fusion module to Example 1. In specific implementation, the visible light image of the lawn scene is divided into a 32×32 pixel local region, and the average brightness, brightness standard deviation, saturated pixel ratio, dark pixel ratio and local gradient intensity are calculated.

[0106] When the average brightness is too low and the proportion of dark pixels is too high, local areas may be in deep shadow or underexposed; when the average brightness is too high and the proportion of saturated pixels is too high, they may be in strong reflection or overexposed; when the local gradient intensity is too low, there is a lack of usable texture. These indicators are combined to form a visible light reliability between 0 and 1. For near-infrared reliability, the average response intensity, response standard deviation, proportion of saturated pixels, and proportion of weak response pixels are calculated for each near-infrared local region. A high saturation ratio or a high weak response ratio reduces reliability, while a response intensity within the effective range and clear local structural edges increase reliability. For registration reliability, after global spatial alignment, visible light brightness edges and near-infrared structural edges are extracted. The average nearest distance is calculated for two sets of edges within the same local region. Registration reliability is 1 when the distance is no more than 1 pixel, decreases with increasing distance from 1 to 3 pixels, and is set to low reliability when the distance is greater than 3 pixels.

[0107] The semantic segmentation model sets up visible light encoding branches and near-infrared encoding branches, which output visible light semantic features and near-infrared semantic features with the same spatial size, respectively. The visible light fusion weights are jointly determined by the visible light reliability and registration reliability, while the near-infrared fusion weights are jointly determined by the near-infrared reliability and registration reliability, and are weighted sequentially by spatial location after normalization. When the registration reliability is below 0.30, direct fusion is not performed; instead, the branch with higher reliability is selected for output or the location is marked as having low information reliability.

[0108] In a set of local input data, the visible light reliability is 0.25, the near-infrared reliability is 0.80, and the registration reliability is 1.00. The normalized visible light fusion weight is approximately 0.238, and the near-infrared fusion weight is approximately 0.762. When the corresponding visible light semantic feature value is 0.31 and the near-infrared semantic feature value is 0.74, the weighted fusion semantic feature value is approximately 0.638, while the fusion semantic feature value using fixed equal-weight fusion is 0.525. The difference between the weighted fusion semantic feature value and the proximity infrared semantic feature value is approximately 0.102, and the difference between the fixed equal-weight fusion semantic feature value and the proximity infrared semantic feature value is 0.215.

[0109] Furthermore, the degree of semantic uncertainty is calculated using the following formula:

[0110] ;

[0111] in For the first The degree of semantic uncertainty of each ground semantic raster unit; This grid cell belongs to a category The probability of; These correspond to the lawn area, the paved surface area, the obstacle area, and other areas, respectively. It is the natural logarithm. When the probabilities of the four classes are concentrated in a single class... When close to 0, close to uniform distribution Close to 1.

[0112] In a set of ground semantic raster data, the probability of a lawn area is 0.70, the probability of a hardened ground area is 0.10, the probability of an obstacle area is 0.15, and the probability of other areas is 0.05. Substituting these values ​​into the calculation yields... The probability is approximately 0.660. Although the lawn area has the highest probability, the obstacle area still has a probability of 0.15. Path planning cannot treat the corresponding grid cell as the same low-risk area as the high-confidence lawn.

[0113] The cost of continuous passage is calculated using the following formula:

[0114] ;

[0115] in For the first The cost of continuous passage through a grid cell; Basic transit costs; Weights for semantic uncertainty; The degree of semantic uncertainty; For boundary risk weights; For border risks; Historical risk weights; This comes at the cost of historical risks.

[0116] Boundary risk is determined based on the distance from the grid cell to the boundary of the nearest obstacle area, other areas, or the workable area of ​​the lawn; the smaller the distance, the higher the risk. In a set of calculated data, the basic access cost is 1. It is 0.660. It is 3. It is 0.40. It is 2. It is 0.20. The probability is 1.5, and the continuous passage cost is approximately 4.080. Although the highest probability category of the corresponding grid cell is the lawn area, the continuous passage cost is still higher than that of the high-confidence lawn area, so the transfer path will preferentially choose the area with lower uncertainty.

[0117] The dynamic safety distance calculation considers a lawnmower body width of 0.56m, with the mowing actuator's outline not exceeding the body's lateral outline, a half-width of 0.28m, and a maximum lateral positioning error of 0.08m. When the lawnmower travels at 0.50m / s, the additional lateral safety distance corresponding to the control response and braking process is taken as 0.17m, and the dynamic safety distance is 0.53m. When the ground semantic grid side length is 0.05m, the obstacle area expands by 11 grid cells along the outer side, and the workable lawn area boundary contracts by 11 grid cells along the inner side. When the speed decreases to 0.20m / s, the additional braking distance decreases, and the dynamic safety distance decreases accordingly, but is not less than the sum of the body's half-width and the maximum positioning error. After the dynamic safety buffer is generated, the coverage path segment is generated, and after path smoothing is completed, all sampling points re-query the dynamic safety buffer.

[0118] Furthermore, the reliability of cross-spectral information is transferred from the feature fusion stage to the ground semantic raster and used to determine whether ground semantic raster units can form coverage path segments or transfer paths.

[0119] In the first image region, the visible light image is affected by tree shadows, resulting in a decrease in local average brightness and an increase in the proportion of dark pixels; the visible light reliability is determined to be 0.25. The near-infrared image does not saturate, its local response intensity is within the effective range, and structural edges are clear; the near-infrared reliability is determined to be 0.80. The average distance between the visible light and near-infrared structural edges is no greater than 1 pixel, and the registration reliability is 1.00. After normalization based on visible light and near-infrared reliability, the visible light fusion weight is approximately 0.238, and the near-infrared fusion weight is approximately 0.762. The visible light semantic feature value is 0.31, the near-infrared semantic feature value is 0.74, and the weighted fused semantic feature value is approximately 0.638. Under fixed equal-weight fusion conditions, the fused semantic feature value is 0.525. Reliability-weighted fusion makes the fused semantic features closer to the more reliable near-infrared semantic features in the current image region, reducing the impact of low-reliability visible light information under shadow conditions on the semantic judgment of the lawn.

[0120] In the second image region, both the visible light and near-infrared images can detect ground edges, but the average distance between the two sets of structural edges is greater than 3 pixels, and the registration reliability is less than 0.30. The cross-spectral reliability fusion module does not perform direct weighted fusion of the visible light semantic features and near-infrared semantic features of the second image region. Instead, it selects the coding branch with higher reliability for output and marks the second image region as a low-information-reliability location.

[0121] The ground semantic raster generation module projects category probabilities onto ground coordinates based on camera calibration parameters and lawnmower pose, while simultaneously projecting information reliability onto corresponding raster cells. Raster cells corresponding to the first image region satisfying a lawn region probability of at least 0.60, an obstacle region probability of at least 0.20, and information reliability of at least 0.50 are identified as candidate lawn-operable raster cells, participating in the formation of lawn-operable areas and coverage path segments. A ground semantic raster cell corresponding to the second image region has a lawn region probability of 0.70, a hardened ground region probability of 0.10, an obstacle region probability of 0.15, and other region probabilities of 0.05. If only the semantic category with the highest probability is selected, it will be identified as a lawn region, with a corresponding basic passage cost of 1. However, because its cross-spectral local registration reliability is below 0.30 and its information reliability is below 0.50, it does not meet the formation conditions for candidate lawn-operable raster cells and is excluded from the lawn-operable area, thus not forming a coverage path segment.

[0122] Although the raster cell corresponding to the second image region has a probability of 0.70 for a lawn region, the probabilities of the four categories are dispersed. According to the aforementioned semantic uncertainty calculation method, its semantic uncertainty is approximately 0.660. With a basic passage cost of 1... , , , , Under these conditions, the continuous passage cost is approximately 4.080. The passage cost map generation module further increases the passage cost of low-information-reliability raster cells based on the continuous passage cost.

[0123] The coverage path planning module performs the following checks: If a low-risk transfer path exists that can bypass the second image region, the low-risk transfer path is selected, and the grid cell corresponding to the second image region is not used; if no low-risk transfer path exists that can bypass the second image region, and the probability of the obstacle region corresponding to the second image region is less than 0.20, the grid cell corresponding to the second image region is allowed to participate in the formation of the transfer path; regardless of whether the formation of the transfer path is allowed, the grid cell is not used to form a coverage path segment. Therefore, the second image region only serves as a limited connection in path planning and does not serve as a lawn mowing operation.

[0124] The coverage path planning module connects and aggregates candidate lawn workable grid cells with an information reliability of not less than 0.50 to form lawn workable areas. Based on a mowing working width of 0.42m and an allowable overlap width of 0.04m between adjacent work zones, the center-to-center spacing of coverage path segments is set to 0.38m. The coverage path planning module generates parallel coverage path segments only within lawn workable areas. When a parallel work line intersects the second image area, which is not included in the lawn workable area, the coverage path segment splits into two sub-coverage path segments on either side of the second image area. The coverage path planning module searches for transfer paths between the two sub-coverage path segments based on the ground access cost map. If a low-risk transfer path bypasses the second image area, it is bypassed. If no low-risk transfer path exists and the probability of an obstacle area in the second image area is less than 0.20, the transfer path is allowed to pass through the second image area, but the second image area is not included in the theoretical mowing coverage area.

[0125] Under fixed fusion and hard-class path planning conditions, the visible light and near-infrared features of the first image region affected by shadows are fused with equal weights. Although the second image region has a large registration deviation, the lawn region, with a maximum class probability of 0.70, participates in forming the lawn workable area according to the basic passage cost of 1. The coverage path segment directly passes through the second image region, failing to distinguish ground areas with seemingly clear semantic categories but unreliable imaging registration. In this embodiment, the first image region uses highly reliable near-infrared information to compensate for low-illuminance visible light information, allowing shadowed lawns to still enter the lawn workable area. The second image region, due to its low registration reliability, is excluded from the coverage path segment but is not directly set as a permanent obstacle. It can still serve as a restricted transfer area when there is no low-risk alternative connection path and the obstacle area probability meets safety conditions. Thus, the same ground semantic grid unit assumes three different path roles—coverage operation, prohibited coverage, or restricted transfer—based on the reliability of cross-spectral information.

[0126] Example 3: Based on Example 2, during lawn mowing operations, the multispectral image acquisition module acquires multispectral image pairs at a 5Hz update rate, and the class probability at each moment is projected onto the operation map coordinates according to the corresponding pose. The ground semantic raster generation module does not directly compare the segmentation masks in adjacent image coordinates, but compares the class probabilities of the same ground semantic raster unit at different times. The image position changes caused by the lawnmower's movement are eliminated by ground coordinate projection.

[0127] When the same ground semantic raster cell receives class probabilities from multiple time points consecutively, the class probability with higher reliability at the current time point has a higher weight, while the historical class probability decays according to the time interval. When new observations are missing, the historical class probability is retained until the maximum effective time, and the raster cell that exceeds the maximum effective time is converted to other areas or a pending confirmation state.

[0128] When the probability of an obstacle area at the current moment increases by more than the change probability threshold relative to the historical probability of an obstacle area, or when the probability of a lawn area decreases by more than the change probability threshold relative to the historical probability of a lawn area, the corresponding grid cell is marked as a candidate changed grid cell.

[0129] Candidate change grids are aggregated into candidate change regions through connected regions. When the area is less than 0.10m², path replanning is not triggered separately. However, when the candidate change region is within 0.80m in front of the lawnmower, it still enters the safe parking judgment.

[0130] Within a 5-frame observation window, a candidate change region is considered a change region if it appears in at least 3 frames, satisfying the change persistence condition. The change region is removed from the mark if it disappears for 5 consecutive frames. The change removal threshold (5 frames) is greater than the change persistence threshold (3 frames) to prevent change regions from repeatedly appearing and disappearing near the threshold.

[0131] When the changed area intersects with the theoretical sweep strip of the mowing path to be executed, the changed area affects the path to be executed; when the changed area does not intersect with the theoretical sweep strip, but the minimum distance from the changed area to the path to be executed is less than 0.60m, the changed area also affects the path to be executed. The estimated arrival time is calculated based on the path distance from the mower's current position to the changed area and the current travel speed. If the estimated arrival time is less than 4s, the dynamic update module immediately outputs a deceleration command and initiates local replanning; if the estimated arrival time is not less than 4s, the dynamic update module completes local replanning in the background, and switches to the updated path after the updated path is prepared.

[0132] During local replanning, the first and last path points affected by the changed area are determined. Executed paths before the first path point are not included in the replanning, while unaffected coverage path segments and transfer paths after the last path point remain unchanged. Using the safe connection point before the first path point as the local starting point and the safe connection point after the last path point as the local ending point, a local alternative path is generated using dynamic weighted A-star path search in the updated ground access cost map. After collision and boundary checks, the affected path segments are replaced. When a new obstacle completely blocks the current work sub-area, the coverage path planning module regroups the incomplete coverage path segments and searches for transfer paths to another work sub-area. If no safe transfer path exists, the lawnmower stops moving and outputs information about the blocked area.

[0133] In terms of motion compensation, a stationary rectangular obstacle occupies an area of ​​80 pixels wide and 60 pixels high in the previous image frame. The lawnmower's movement causes the rectangular obstacle to shift laterally by 12 pixels in the subsequent image frame. When directly comparing the binary obstacle masks of the two frames, the symmetry difference area of ​​the rectangular region is 1440 pixels. After projecting the obstacles of the two frames onto the same ground coordinate system, the ground positions corresponding to the stationary rectangular obstacle coincide, and the difference area under ideal registration conditions is 0. Ground coordinate comparison can avoid misinterpreting the pixel displacement caused by the lawnmower's own movement as a scene change.

[0134] Example 4: This example is a further optimization based on Example 3. In this example, the path output module records the actual position trajectory of the lawnmower and the status of the mowing actuator.

[0135] When the lawnmower is in operation, it sweeps along its actual pose trajectory based on its outline to form the theoretically covered area. When the lawnmower is stationary, the corresponding trajectory is not included in the theoretically covered area. The theoretically covered area is calculated using the actual pose trajectory instead of the planned path. Changes in the actual trajectory caused by path tracking deviations, temporary obstacle avoidance, and manual intervention are all reflected in the theoretically covered area.

[0136] For areas theoretically covered, multispectral image features before mowing are preserved. After the lawnmower passes, a backward imaging device or a forward imaging device after the lawnmower turns acquires a post-mowing multispectral image. Based on the lawnmower's pose, the pre-mowing and post-mowing multispectral images are projected onto the same ground semantic grid, and changes in lawn texture scale, local directional gradient, visible light color distribution, and near-infrared local response are compared. After mowing, the proportion of elongated textures formed by taller blades typically decreases, and the local surface texture tends to be more uniform.

[0137] The lawnmower controller outputs the blade motor current, blade speed, and protection status. When the blade motor current is higher than the no-load reference and the blade speed remains within the allowable range, the lawnmower is in an effective cutting state; when the blade motor current is close to the no-load reference or the blade speed is lower than the allowable range, the theoretically covered area may not have been effectively cut.

[0138] Ground semantic grid cells are marked as "trimmed" when the mowing actuator is in operation at the corresponding location, the blade speed is within the allowable range, the blade motor load indicates that a cutting process is occurring, and the changes in the multispectral image before and after the mowing operation reach the state change threshold. They are marked as "untrimmed" when the mowing actuator is stopped, the blade stops for protection, or the changes in the image before and after the operation do not reach the state change threshold. If the image is occluded or the load information of the mowing actuator is missing, the state is marked as "uncertain."

[0139] In a set of operational data, the blade motor's no-load reference current was 2.1A. The operational current for the first ground area was 3.4A, with the blade speed within the allowable range. After the operation, the texture intensity decreased by 28% compared to before the operation, and the first ground area was marked as trimmed. The operational current for the second ground area was 2.2A, and after the operation, the texture intensity decreased by 4% compared to before the operation, and the second ground area was marked as untrimmed.

[0140] Adjacent ground semantic raster cells in an untrimmed state or with uncertain trimming states are aggregated into the remaining work area. When the area of ​​the remaining work area is not less than the minimum re-trimming area, the coverage path planning module forms a supplementary coverage path segment within the remaining work area, prioritizing formation along the original coverage direction to avoid adding a large number of turns; if the original coverage direction is affected by changes in obstacles, the coverage direction evaluation is re-executed. After the supplementary coverage path is completed, the actual trimming state determination is performed again. The conditions for completing the mowing operation include: there are no untrimmed areas whose area has reached the minimum re-trimming area; there are no areas with uncertain trimming states whose area has reached the minimum re-trimming area; and inaccessible areas have been individually marked and output.

[0141] Furthermore, the lawnmower controller records execution anomaly events, including emergency stop events, manual takeover events, collision protection events, mowing actuator protection events, and continuous wheel slippage events. Each execution anomaly event records the event type, occurrence time, lawnmower ground position, lawnmower heading, speed, and event resolution method. The risk cost correction module maps execution anomaly events to a ground semantic grid. Collision protection events and mowing actuator protection events correspond to higher risk levels, with an event impact radius set to 0.80m. Manual takeover events require determining their risk level based on the takeover reason: manual takeover used to adjust the work sequence does not increase historical risk costs; manual takeover used to avoid unidentified obstacles increases historical risk costs.

[0142] When a collision protection event occurs for the first time in the same ground area, the historical risk cost of the ground semantic grid cell at the event center increases by 4. The increase in historical risk cost gradually decreases as the distance from the event center increases.

[0143] When the same type of execution anomaly event occurs repeatedly in the same ground area, the historical risk cost continues to increase, but does not exceed the upper limit of the historical risk cost. In subsequent operations, when the lawnmower passes through the event-affected area at low speed without execution anomalies, the historical risk cost decreases by 0.5 for each pass, and stops decreasing after reaching 0. When the event-affected area is not observed again for a long period of time, the historical risk cost decays according to the time period. The historical risk cost does not directly modify the semantic segmentation model parameters, nor does it change the probability threshold of the lawn area in the entire image; it only affects the ground access cost map near the corresponding ground location, making risk correction spatially local.

[0144] In addition, the multispectral image acquisition module can use a single multispectral imaging device that can simultaneously output visible light and near-infrared bands, and each spectral channel still needs to complete channel response correction and spatial correspondence check.

[0145] The semantic segmentation model can employ a dual-encoding branch structure to perform reliable gated fusion at multiple spatial scales, or a four-channel early fusion structure to directly input the same encoding processing part. It can also set a boundary prediction branch to output lawn boundary probabilities for use in boundary risk calculation. The ground semantic grid can use a fixed resolution, or a layered resolution, using smaller grids near the lawnmower and obstacle boundaries, and larger grids further away. The workable area of ​​the lawn can be further decomposed into work units based on connectivity, entrance width, and internal obstacle distribution. Covering paths can be reciprocating, parallel, or hybrid.

[0146] The path output module outputs linear velocity / angular velocity or target velocity / steering angle based on the type of walking mechanism. The dynamic update module can update only the ground semantic grid within a predetermined distance in front of the lawnmower, reducing the update frequency for already trimmed and unchanged areas. Scene change areas include newly added obstacles, obstacle removal, temporary lawn covering, water accumulation on hardened ground, and blocked original paths. Trimming priority is determined by vegetation activity information, texture scale information, historical trimming time, and selectable height information. When the lawnmower execution mechanism load or image change information is unavailable, the actual trimming status determination module identifies areas with uncertain trimming status based on available information. The risk cost correction module can set an expiration period for execution anomalies; if the expiration period expires and there are no repeated execution anomalies, the historical risk cost gradually decays. The lawn cover path planning system can be deployed on the lawnmower's local processor, with model training completed externally and parameters written to memory; alternatively, some processing can be deployed on edge computing devices, switching to a local low-speed safe mode when communication is interrupted.

[0147] Comparative Example 1: Using the same visible light and near-infrared images and encoded semantic features as in Example 2, the difference lies in applying fixed weights of 0.5 to the visible light and near-infrared semantic features, respectively. With a visible light semantic feature value of 0.31 and a near-infrared semantic feature value of 0.74, the fixed-channel fusion result is 0.525. Example 2, based on a visible light reliability of 0.25 and a near-infrared reliability of 0.80, yields a fusion result of approximately 0.638. The difference between the fixed-channel fusion result and the near-infrared semantic feature value is 0.215, while the difference between the fusion result and the near-infrared semantic feature value in Example 2 is approximately 0.102. Under the input condition of localized underexposure in the visible light while maintaining effective near-infrared information, cross-spectral reliability fusion can reduce the proportion of low-reliability visible light information in the fused semantic features.

[0148] Comparative Example 2: The same four-category probabilities of 0.70, 0.10, 0.15, and 0.05 were used to calculate the semantic uncertainty in Example 2. The difference was that only the lawn area with the highest probability was selected as the ground semantic raster unit category, and the basic access cost of 1 was directly used as the final access cost. The access cost of Comparative Example 2 was 1.

[0149] Example 2 calculates a semantic uncertainty of approximately 0.660 and, given boundary and historical risks, a continuous passage cost of approximately 4.080. Compared to Example 2, which cannot represent situations where the obstacle region probability is 0.15 and multiple categories still possess probabilistic quality, Example 2 effectively transmits the dispersion of category probabilities to the path planning process.

[0150] Comparative Example 3: Using the same continuous images as Example 3, the difference is that obstacle masks in adjacent image frames are directly compared. When a stationary rectangular obstacle moves laterally by 12 pixels in an adjacent image frame, Comparative Example 3 obtains a difference area of ​​1440 pixels. After Projecting the obstacle probabilities of two frames onto the same ground coordinates in Example 3, the ground positions corresponding to the stationary rectangular obstacle coincide. Comparative Example 3 includes the change in viewpoint caused by the lawnmower's movement in the scene change calculation, while Example 3 only uses the change in category probability at the same ground position as the basis for determining scene changes.

[0151] Comparative Example 4: Using the same actual driving trajectory and lawn mowing actuator outline as Example 4, the difference is that the theoretically covered area is considered as completed mowing. The workable area of ​​the lawn is 120.0 m². When the theoretically covered area reaches 120.0 m², the work completion rate calculated in Comparative Example 4 is 100%. Post-work inspection revealed that the first missed area was 3.2 m², and the second missed area was 2.4 m². Example 4 marked a 5.6 m² area as unmowed based on multispectral image change information and lawn mowing actuator load information. The actual completed area was 114.4 m², and the completion rate corresponding to the actual mowing state was approximately 95.33%. Example 4 added the 5.6 m² unmowed area back to the remaining work area and generated a supplementary coverage path to avoid equating the path passage directly with mowing completion.

[0152] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0153] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A semantic segmentation path planning method for lawns, characterized in that, include: Visible light imaging devices and near-infrared imaging devices installed on the lawnmower are used to acquire visible light images and near-infrared images of the lawn scene with a time correspondence. Based on the calibration parameters of the visible light imaging device and the near-infrared imaging device, distortion correction and spatial alignment are performed on the visible light image and the near-infrared image of the lawn scene to form multispectral input data. By inputting multispectral data into a semantic segmentation model, the category information of image pixels belonging to lawn areas, hardened ground areas, obstacle areas, and other areas can be obtained; Based on the calibration parameters and the lawnmower's pose, the category information is projected onto the ground coordinates to form a ground semantic grid; A ground access cost map is generated based on the semantic categories in the ground semantic grid, and the workable area of ​​the lawn is determined from the ground semantic grid; Based on the mower's mowing working width, a coverage path segment is formed within the lawn's workable area. A transfer path is generated between adjacent coverage path segments based on the ground traffic cost map. The coverage path segments and transfer paths are then connected to form the mowing operation path. The mowing path is converted into the target position and target heading that the mower can execute, and the walking mechanism and mowing execution mechanism of the mower are controlled to perform the mowing operation according to the mowing path.

2. The lawn semantic segmentation path planning method according to claim 1, characterized in that, The process of generating multispectral input data includes: adding acquisition timestamps to the visible light image and near-infrared image of the lawn scene, and selecting image frames whose timestamp difference is not greater than the time pairing threshold as multispectral image pairs; Based on the intrinsic parameters of the visible light imaging device, the intrinsic parameters of the near-infrared imaging device, and the relative extrinsic parameters between the visible light imaging device and the near-infrared imaging device, distortion correction and coordinate transformation are performed respectively. The spatially aligned red, green, blue, and near-infrared channels are stacked according to their pixel positions to form a four-channel multispectral input tensor.

3. The lawn semantic segmentation path planning method according to claim 1, characterized in that, After generating the multispectral input data, the following are also included: The reliability of visible light is determined based on the local brightness, local saturation, and local contrast of the visible light image of the lawn scene; Near-infrared reliability is determined based on the local response intensity, saturated pixel ratio, and weak response pixel ratio of near-infrared images of a lawn scene. The registration reliability is determined based on the local registration residual between the visible light image and the near-infrared image of the lawn scene. By utilizing visible light reliability, near-infrared reliability, and registration reliability, the visible light semantic features and near-infrared semantic features are spatially weighted and fused to obtain fused semantic features, which are then input into the decoding processing part of the semantic segmentation model.

4. The lawn semantic segmentation path planning method according to claim 1, characterized in that, The semantic segmentation model outputs the probability of each image pixel belonging to the category of lawn region, hardened ground region, obstacle region, and other regions. The formation of a ground access cost map includes: Project category probabilities onto a ground semantic grid; The degree of semantic uncertainty is determined based on the category probability distribution corresponding to the same ground semantic raster unit; The basic passage cost is determined based on the semantic category, and the additional uncertainty cost is determined based on the degree of semantic uncertainty. The basic access cost and the uncertainty-added cost are combined into a continuous access cost for the ground semantic raster unit.

5. The lawn semantic segmentation path planning method according to claim 4, characterized in that, The formation of a ground access cost map also includes: The basic safety distance is determined based on the outer contour of the lawnmower body, the outer contour of the mowing actuator, and the positioning error. The additional braking distance is determined based on the current speed of the lawnmower, braking response time, and braking deceleration; the dynamic safety distance is determined based on the basic safety distance and the additional braking distance. The obstacle area is expanded outward according to the dynamic safety distance, and the outer boundary and inner prohibited operation boundary of the lawn work area are shrunken inward according to the dynamic safety distance, forming a dynamic safety buffer zone; Set the dynamic safety buffer as a no-passage zone or a high-passage-cost zone.

6. The lawn semantic segmentation path planning method according to claim 1, characterized in that, The formation of the lawn mowing path includes: Identify connected regions within the workable area of ​​the lawn; The spacing between adjacent cover path segments is determined based on the mowing work width and the allowable overlap width of adjacent work strips; Along the candidate coverage direction, intersect each workable area of ​​the connected lawn with the parallel work line to form a coverage path segment; Determine the access order of the covered path segments; The dynamic weighted A-satellite path search is used to generate transfer paths between adjacent coverage path segments in the ground access cost map. The heuristic weight of the dynamic weighted A-satellite path search is 1.4 in the early stage of the search and decreases linearly to 1.0 as the number of expanded nodes increases. The path formed by connecting the covered path segment and the transfer path is smoothed, and collision verification and boundary verification are performed on the smoothed path according to the ground traffic cost map.

7. The lawn semantic segmentation path planning method according to claim 6, characterized in that, Determining candidate coverage directions includes: For multiple candidate coverage directions, separate coverage path segments are generated; Calculate the coverage path length, number of turns, transfer path cost, and boundary risk cost for each candidate coverage direction; The target coverage direction is selected based on the weighted result of the coverage path length, number of turns, transfer path cost, and boundary risk cost.

8. The lawn semantic segmentation path planning method according to claim 1, characterized in that, The lawn mowing process also includes: Continuously acquire visible light images and near-infrared images of the lawn scene to obtain the category probabilities at consecutive time points; Based on the lawnmower pose at each time step, the class probabilities of consecutive time steps are projected onto the same ground coordinates; the consecutive class probabilities of the same ground semantic raster unit are fused in time to form a spatiotemporally aligned ground semantic raster. The scene change region is determined based on the difference between the spatiotemporally aligned ground semantic raster at the current moment and the spatiotemporally aligned ground semantic raster at historical moments.

9. A lawn semantic segmentation path planning system, used to execute the lawn semantic segmentation path planning method according to any one of claims 1-8, characterized in that, include: A multispectral image acquisition module is used to acquire visible light images and near-infrared images of a lawn scene with a time-corresponding relationship. The image alignment module is used to perform distortion correction and spatial alignment on the visible light image and near-infrared image of the lawn scene according to the calibration parameters, so as to form multispectral input data. The semantic segmentation module is used to input multispectral input data into the semantic segmentation model to obtain the category information of image pixels belonging to lawn areas, hard ground areas, obstacle areas and other areas; The ground semantic raster generation module is used to project category information onto ground coordinates based on calibration parameters and lawnmower pose to form a ground semantic raster. The access cost map generation module is used to generate a ground access cost map based on the semantic categories in the ground semantic grid, and to determine the workable area of ​​the lawn from the ground semantic grid; The coverage path planning module is used to form coverage path segments within the workable area of ​​the lawn based on the mowing work width, form transfer paths between adjacent coverage path segments based on the ground access cost map, and connect the coverage path segments and transfer paths into the mowing work path. The path output module is used to convert the mowing operation path into the target location and target heading that the lawnmower can execute.

10. A lawnmower, comprising a body, a walking mechanism, a mowing execution mechanism, a visible light imaging device, a near-infrared imaging device, a pose acquisition device, a processor, and a memory; the visible light imaging device and the near-infrared imaging device are disposed on the body and are used to acquire visible light images and near-infrared images of the lawn scene; the pose acquisition device is used to acquire the pose of the lawnmower; the memory stores a computer program; characterized in that: When the processor executes the computer program, it implements the lawn semantic segmentation path planning method according to any one of claims 1 to 8, and controls the walking mechanism and the mowing execution mechanism according to the mowing operation path.

Images