A garden inspection method and system with air-ground collaboration

Abstract

The present invention discloses an air-ground coordinated garden inspection method and system, which belongs to the field of garden inspection technology and is used to solve the technical problems that the existing garden inspection routes and inspection ranges are relatively fixed, the entire garden space cannot be inspected comprehensively, the inspection results cannot be viewed intuitively at any time, and there is a lack of hidden danger prevention measures. The method includes: constructing a digital twin model of the target garden based on digital twin technology; determining the inspection period and inspection path of the day according to the simulation results; controlling the inspection robot to perform daily inspections along the inspection path of the day during the inspection period of the day; detecting the operating environment of the inspection robot in real time, and controlling the inspection drone to perform high-altitude follow-up inspections based on the operating environment; performing non-standard target recognition based on the first video stream transmitted by the inspection robot and the second video stream transmitted by the inspection drone; generating an inspection report based on the results of non-standard target recognition and the results of non-standard target prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of garden inspection, and in particular to an air-ground coordinated garden inspection method and system. Background Art

[0002] Garden inspections primarily focus on landscape architecture, equipment, safety features, and potential pest and disease hazards, personnel safety hazards, and fire hazards. These inspections typically involve daily, periodic, and seasonal inspections to ensure safe garden operations. Currently, garden inspections are primarily conducted manually. Inspectors must follow the inspection system, carefully examining every detail of the garden facilities. Finally, they must prepare an inspection report and submit it to the relevant departments for review.

[0003] As you can imagine, manual inspections not only consume significant manpower and resources, but are also limited by the inspectors' abilities and experience, leading to the possibility of missed inspections and false positives. Furthermore, manual inspections are limited in scope, unable to obtain high-altitude information or information about blind spots, preventing a comprehensive inspection of the garden's conditions. Inspectors are restricted to specific inspection items and routes, making it easy to miss hidden safety hazards.

[0004] Some gardens currently use inspection robots instead of manual inspections. However, these robots also have a limited inspection range and can only assess situations within their field of view. They cannot obtain information about hidden dangers at high altitudes or outside the designated routes. Inspection results are often partial, and the inspection range cannot cover the entire garden space. Furthermore, it is impossible to visually identify where irregularities have occurred. Only after the inspection is completed and an inspection report is generated can the inspection status be determined. Furthermore, the inspection status is only available for that day, and it is impossible to predict potential hidden dangers in the future based on the current inspection situation. Inspection information is often delayed and lacks foresight and preventive measures. Summary of the Invention

[0005] The embodiments of the present invention provide an air-ground coordinated garden inspection method and system, which are used to solve the following technical problems: the existing garden inspection routes and inspection ranges are relatively fixed, it is impossible to conduct comprehensive inspections of the entire garden space, it is impossible to intuitively see the inspection results at any time, and there is a lack of hidden danger prevention measures.

[0006] The embodiment of the present invention adopts the following technical solutions:

[0007] On the one hand, an embodiment of the present invention provides a garden inspection method with air-ground collaboration, the method comprising: constructing a digital twin model of a target garden based on digital twin technology; driving the digital twin model to perform operation simulation using historical operation data of the target garden, and determining the inspection time period and inspection route for the day based on the simulation results;

[0008] During the inspection period of the day, the inspection robots are controlled to perform routine inspections along the inspection route of the day; wherein each inspection robot is equipped with at least one inspection drone;

[0009] Real-time detection of the operating environment of the inspection robot, and control of the inspection drone to perform high-altitude follow-up inspections based on the operating environment;

[0010] Performing non-standard target recognition based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone;

[0011] Extracting garden operation characteristic parameters from the video stream, applying the garden operation characteristic parameters to the digital twin model to perform non-standard target prediction;

[0012] An inspection report is generated based on the results of the irregular target identification and the results of the irregular target prediction.

[0013] In a feasible implementation, a digital twin model of the target garden is constructed based on digital twin technology, specifically including:

[0014] Acquire the garden landform data determined in the target garden planning stage in the database, and construct a three-dimensional garden terrain model based on the garden landform data;

[0015] Using a drone equipped with a depth camera, a plurality of depth images of the garden landscape are obtained; and based on image stitching technology, the plurality of depth images are seamlessly stitched together to obtain an overall depth image of the garden;

[0016] Based on the depth information of each pixel in the overall depth image of the garden, a differential value of each pixel is calculated; if the differential value of any pixel in a preset pixel area centered on the pixel is the largest, the pixel information of the pixel is used to form a set of landscape modeling data to obtain a landscape modeling data set; wherein the pixel information includes at least: pixel coordinates, pixel depth information, and pixel color values ​​in RGB channels;

[0017] Converting pixel coordinates in the landscape modeling data set based on the depth information to obtain three-dimensional coordinates of the landscape modeling data; and constructing a corresponding three-dimensional model of the garden landscape according to the three-dimensional coordinates of the landscape modeling data;

[0018] The three-dimensional model of the garden terrain is combined with the three-dimensional model of the garden landscape to obtain a three-dimensional physical model of the target garden;

[0019] The operating data type of each facility in the target garden is obtained, and each operating data type is set as an initial value to form a data model, and the data model is combined with the three-dimensional physical model to obtain a digital twin model of the target garden.

[0020] In a feasible implementation, the digital twin model is driven by the historical operation data of the target garden to perform simulation operation, and the inspection period and inspection route of the day are determined according to the simulation results, specifically including:

[0021] Obtaining historical values ​​corresponding to the operation data type within a preset historical time period to form the historical operation data;

[0022] Inputting the historical operation data into the digital twin model, driving the digital twin model to continue simulating the operation, and obtaining the operation forecast data for the day; wherein the operation forecast data for the day includes at least one or more of the following: passenger flow forecast data for each time period, passenger flow distribution forecast data for each time period, predicted interaction rate of each tourist destination, and weather forecast data;

[0023] According to the passenger flow forecast data of each time period, the time period with the least passenger flow during the garden business hours on that day is selected as the inspection time period on that day;

[0024] Determine the key inspection targets for the day within the target garden based on the forecasted daily operation data;

[0025] The inspection route for the day is constructed based on the distance relationship between the key inspection targets for the day and the actual road planning.

[0026] In a feasible implementation, based on the daily operation forecast data, determining the key inspection targets for the day in the target garden specifically includes:

[0027] Calculate the predicted passenger flow for each tourist destination in each time period based on the passenger flow forecast data and passenger flow distribution forecast data for each time period;

[0028] Based on expert experience, a weight table of the impact of various weather conditions on the safety of different tourist destinations is developed; and based on the weather forecast data, the weather impact weight of each tourist destination on the day is obtained in the weight table;

[0029] according to Determine the inspection weight of each tourist destination; where W j is the inspection weight of the jth tour target, ω j is the weather impact weight of the day corresponding to the j-th tourist destination. If the weather has no impact on the j-th tourist destination, the weather impact weight of the day is 1; P ijis the predicted passenger flow of the jth tourist destination in the i-th period, P i is the total predicted passenger flow of the target garden in the i-th period, n is the total business hours of the target garden on that day, T ij is the predicted interaction rate of the jth tour target in the i-th period;

[0030] The inspection weight average of all the tourist targets on the day is calculated, and the tourist targets whose inspection weight exceeds the inspection weight average are determined as the key inspection targets on the day.

[0031] In a feasible implementation, real-time detection of the operating environment of the inspection robot and control of the inspection drone to perform high-altitude follow-up inspection based on the operating environment specifically include:

[0032] Monitoring and displaying the inspection route and real-time position of the inspection robot in the digital twin model;

[0033] Based on the real-time position, determine whether there is an obstruction above the inspection route in front of the inspection robot. If so, obtain the maximum height of the obstructing target in front in the digital twin model, and set the flight altitude of the inspection drone according to the maximum height;

[0034] Controlling the inspection drone carried by the inspection robot to take off vertically, and after reaching the flight altitude, following the inspection trajectory of the inspection robot to perform synchronous inspection;

[0035] In the first video stream recorded by the inspection robot, the video stream segment recorded after the inspection drone is started is marked as a synchronous recording segment.

[0036] In a feasible implementation, based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, irregular target recognition is performed, specifically including:

[0037] Based on the timestamps, align the synchronized recording segments in the first video stream with the second video stream frame by frame, and perform frame sampling to obtain a plurality of aligned sampling frame combinations;

[0038] Determining a mapping matrix between the synchronous recording segment and the second video stream according to the position coordinates of the same target in each sampling frame combination;

[0039] Based on the real-time target detection RT-DETR algorithm, a multi-target detection model is constructed;

[0040] During model training, through custom functions As the fitness function of the multi-target detection model; where R p is the number of samples that correctly detect hidden dangers; F p Indicates the number of false alarm samples; FN Indicates the number of missed alarm samples;

[0041] After a preset number of iterative trainings, the optimal parameters of the fitness function are determined as the final parameters of the model to obtain a trained multi-target detection model;

[0042] Identifying irregular targets in the first video stream and the second video stream using the trained multi-target detection model;

[0043] According to the mapping matrix, the position coordinates of the irregular targets identified in the second video stream are mapped and compared with the position coordinates of the irregular targets identified in the first video stream to eliminate duplicate targets.

[0044] In a feasible implementation, extracting garden operation characteristic parameters from the video stream specifically includes:

[0045] The garden operation characteristic parameters in the first video stream and the second video stream are extracted through a feature extraction model; wherein the garden operation parameters include at least one or more of the following: vegetation characteristics of each landscape, appearance characteristics of each facility, road safety characteristics in the inspection route, and sanitation characteristics in the inspection area.

[0046] In a feasible implementation, the garden operation characteristic parameters are applied to the digital twin model to perform irregular target prediction, specifically including:

[0047] Inputting the garden operation characteristic parameters into the digital twin model to simulate the actual garden operation scene;

[0048] Based on the simulation results of the digital twin model, it is predicted that the tourist destinations with a high probability of experiencing irregularities within the preset time period are identified as predicted irregular destinations.

[0049] In a feasible implementation manner, generating an inspection report based on the result of the irregular target identification and the result of the irregular target prediction specifically includes:

[0050] Displaying the identified irregular target at the corresponding position of the digital twin model with a first attribute feature; displaying the predicted irregular target at the corresponding position of the digital twin model with a second attribute feature; wherein the attribute feature includes at least one or more of the following: icon shape, icon color, and warning method;

[0051] During non-inspection periods, several free drones are controlled to conduct supplementary inspections over the garden, obtain a third video stream, and identify irregular targets in it;

[0052] Fill in the detailed information of all identified irregular targets and predicted irregular targets into the preset template, generate the inspection report, and send it to the terminal of the relevant person in charge to complete the daily inspection.

[0053] On the other hand, an embodiment of the present invention further provides an air-ground coordinated garden inspection system, the system comprising:

[0054] An inspection path generation module is used to build a digital twin model of the target garden based on digital twin technology; drive the digital twin model to perform simulation operations using the historical operation data of the target garden, and determine the inspection time period and inspection path for the day based on the simulation results;

[0055] An inspection control module is used to control the inspection robots to perform daily inspections along the inspection route during the inspection period of the day; wherein each inspection robot is equipped with at least one inspection drone; detect the operating environment of the inspection robots in real time, and control the inspection drones to perform high-altitude follow-up inspections based on the operating environment;

[0056] A target recognition and prediction module is used to identify irregular targets based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, and to extract garden operation characteristic parameters from the video streams; the garden operation characteristic parameters are applied to the digital twin model to predict irregular targets;

[0057] The report generation module is used to generate an inspection report based on the results of the irregular target identification and the results of the irregular target prediction.

[0058] Compared with the existing technology, the air-ground coordinated garden inspection method and system provided by the embodiments of the present invention has the following beneficial effects:

[0059] First of all, the present invention deeply combines digital twin technology with intelligent garden inspection. It simulates the garden operation status through the digital twin model, so as to obtain the best inspection time and inspection path of the day. It can generate different inspection routes every day according to the daily passenger flow, passenger flow distribution, and crowded time periods. It overcomes the defects of fixed inspection routes such as inflexible inspection and fixed inspection range, and can greatly improve the flexibility and comprehensiveness of garden inspection.

[0060] The present invention then equips the inspection robot with an inspection drone. When there's no vegetation obstructing the robot's path, the drone rests in the robot's nest. Only when the digital twin system detects vegetation obstructing the robot's inspection route does the drone launch, compensating for the robot's limited field of view and providing dual information from both the sky and the ground. This differs from conventional air-to-ground collaborative working modes. The proposed working mode deactivates the drone when high-altitude inspections are not necessary, reducing unnecessary energy consumption and achieving energy conservation and emission reductions.

[0061] Finally, after identifying irregular targets based on the video streams captured by robots and drones, the present invention also predicts irregular targets within the digital twin system using operational characteristic parameters extracted from the video streams. Through digital twin simulations, the time when a target will evolve into an irregular target can be predicted, enabling proactive prevention and treatment, providing better support for garden maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0063] Figure 1 A flow chart of a garden inspection method for air-ground collaboration provided by an embodiment of the present invention;

[0064] Figure 2 A schematic structural diagram of an air-ground coordinated garden inspection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0066] The embodiment of the present invention provides a garden inspection method with air-ground coordination, such as Figure 1 As shown, the air-ground collaborative garden inspection method specifically includes steps S101-S106:

[0067] S101. Based on digital twin technology, construct a digital twin model of the target garden.

[0068] Specifically, the garden landform data determined in the target garden planning stage is obtained from the database, and a three-dimensional model of the garden terrain is constructed based on the garden landform data.

[0069] Furthermore, a plurality of depth images of the garden landscape are obtained by using a drone equipped with a depth camera, and then based on image stitching technology, the plurality of depth images are seamlessly stitched together to obtain an overall depth image of the garden.

[0070] Furthermore, based on the depth information of each pixel in the overall depth image of the garden, a differential value for each pixel is calculated. If a pixel has the largest differential value within a predetermined pixel region centered on that pixel, the pixel information of that pixel is combined into a set of landscape modeling data, thereby obtaining a landscape modeling data set. The pixel information includes at least: pixel coordinates, pixel depth information, and the pixel's color value in the RGB channels.

[0071] As a feasible implementation method, the calculation formula for the difference value of each pixel in the depth image is:

[0072] Among them, d x,y Represents the depth information of pixel I(x,y), I k (x, y) represents the color value of pixel I(x, y) in the k color channel, e represents the natural constant, σ1 and σ2 represent the scale standard deviation, c1 and c2 represent the length of the unit pixel in the horizontal and vertical directions respectively, and f x Indicates the horizontal focal length of the depth camera, f y Indicates the vertical focal length of the depth camera.

[0073] Furthermore, pixel coordinates in the landscape modeling data set are converted based on the depth information to obtain the three-dimensional coordinates of the landscape modeling data. Based on the three-dimensional coordinates of the landscape modeling data, a corresponding three-dimensional garden landscape model is constructed. The three-dimensional garden terrain model is then combined with the three-dimensional garden landscape model to obtain a three-dimensional physical model of the target garden.

[0074] Furthermore, the operating data type of each facility in the target garden is obtained, and each operating data type is set as an initial value to form a data model, and the data model is combined with the three-dimensional physical model to obtain a digital twin model of the target garden.

[0075] As a feasible implementation, the operational data types of various facilities in the garden may include: electricity consumption data and operational data of each power facility, passenger flow data of each landscape and each building, vegetation type and vegetation standard specification data of each landscape, etc. In the initial state, these data can be set to 0.

[0076] S102. Drive the digital twin model through the historical operation data of the target garden to perform simulation operation simulation, and determine the inspection time period and inspection route for the day based on the simulation results.

[0077] Specifically, historical values ​​corresponding to the operational data types within a preset historical time period are obtained to form historical operational data. This historical operational data is then fed into the digital twin model, which then drives the model to continue simulating operational simulations to generate daily operational forecast data. This daily operational forecast data includes at least one or more of the following: passenger flow forecast data for each time period, passenger flow distribution forecast data for each time period, predicted interaction rates for each tourist destination, and weather forecast data.

[0078] As a feasible implementation, in a garden, tourist destinations primarily include major landscape features, such as rockery, ponds, and bamboo forests, all of which are landscape-type tourist destinations. Tourist destinations also include buildings, such as pavilions, halls, and wooden houses, all of which are architectural-type tourist destinations. Tourist destinations also include interactive facilities, such as benches and swings, all of which are interactive-type tourist destinations. In other words, any place a visitor might visit in a garden can be categorized as a tourist destination.

[0079] Furthermore, based on the predicted passenger flow data for each time period, the period with the lowest passenger flow during the garden's business hours is selected as the daily inspection period. Furthermore, based on the daily operational forecast data, key inspection targets within the target garden are determined for the day. Based on the distance relationship between key inspection targets and actual road planning, the daily inspection route is constructed, with the closest key inspection targets connected by roads as adjacent inspection targets.

[0080] As a feasible implementation method, based on the daily operational forecast data, key inspection targets for the day are determined within the target garden, specifically including:

[0081] Based on the passenger flow forecast data for each time period and the passenger flow distribution forecast data for each time period, the predicted passenger flow for each tourist destination in each time period is calculated. Then, based on expert experience, a weighted table is developed to determine the impact of various weather conditions on the safety of different tourist destinations. Based on the weather forecast data, the weight of the weather impact on each tourist destination on that day is obtained from the weighted table.

[0082] Then according to Determine the inspection weight of each tourist destination; where W j is the inspection weight of the jth tour target, ω j is the weather impact weight of the day corresponding to the j-th tourist destination. If the weather has no impact on the j-th tourist destination, the weather impact weight of the day is 1; P ij is the predicted passenger flow of the jth tourist destination in the i-th period, Pi is the total predicted passenger flow of the target garden in the i-th period, n is the total business hours of the target garden on that day, T ij is the predicted interaction rate of the j-th tour target in the i-th period.

[0083] Finally, the average inspection weight of all tourist targets on that day is calculated, and the tourist targets whose inspection weight exceeds the average inspection weight are determined as the key inspection targets on that day.

[0084] S103: During the daily inspection period, control the inspection robots to perform routine inspections along the daily inspection route; each inspection robot is equipped with at least one inspection drone. The inspection robot's operating environment is detected in real time, and the inspection drone is controlled to perform high-altitude follow-up inspections based on the operating environment.

[0085] Specifically, once the inspection window arrives, the inspection robot will conduct its daily inspections along a designated route. The robot is equipped with at least one inspection drone, along with auxiliary modules such as a lidar obstacle avoidance module and a GPS positioning module. The GPS positioning module transmits the robot's real-time location to the digital twin model, enabling the robot's inspection route and real-time location to be monitored and displayed in the digital twin.

[0086] Furthermore, based on the inspection robot's real-time position, the digital twin model determines whether vegetation is present above the robot's inspection route. If so, the digital twin model determines the maximum height of any obstructing objects ahead and sets the inspection drone's flight altitude accordingly. The drone, carried by the inspection robot, is then controlled to take off vertically. Once it reaches altitude, it then follows the robot's inspection trajectory for a synchronized inspection.

[0087] Furthermore, in the first video stream recorded by the inspection robot, the video stream segment recorded after the inspection drone is started is marked as a synchronous recording segment.

[0088] In one embodiment, when the drone rises to a flight altitude and starts recording video, the inspection robot marks the first mark in the video stream it records. After the drone finishes recording, the inspection robot marks the second mark in the video stream it records. The segment between the two marks is the synchronized recording segment.

[0089] As a feasible implementation, the obstructing object in front of the inspection robot could be vegetation or a building roof. When the digital twin model detects that the inspection robot has reached an unobstructed area, it sends a descent command to the inspection drone, which returns to the inspection robot's nest until the next detection of vegetation or building obstruction.

[0090] S104: Identify irregular targets based on the first video stream sent back by the inspection robot and the second video stream sent back by the inspection drone.

[0091] Specifically, based on timestamps, the synchronized recording segments in the first video stream are aligned frame by frame with the second video stream, and frame sampling is performed to obtain several aligned sampling frame combinations. Then, based on the position coordinates of the same object in each sampling frame combination, a mapping matrix is ​​determined between the synchronized recording segments and the second video stream.

[0092] As a feasible implementation method, in the synchronous recording segment and the second video stream, video alignment is performed according to the timestamp. After alignment, frame sampling is performed in the two video streams respectively, and several image frames are randomly collected, and the timestamps of the image frames collected in the two video streams are the same. The two image frames with the same timestamp are combined into a sampling frame combination, and an identical target is identified in each sampling frame combination, and the position coordinates of this target in the two image frames are obtained. Based on the mapping relationship of the position coordinates of the same target in each sampling frame combination, a mapping matrix is ​​constructed. The mapping matrix can reflect the position conversion relationship between the second video stream and the target in the synchronous recording segment.

[0093] Furthermore, a multi-target detection model is constructed based on the real-time target detection RT-DETR algorithm. During the model training process, the custom function As the fitness function of the multi-target detection model; where R p is the number of samples that correctly detect hidden dangers; F p Indicates the number of false alarm samples; F N Indicates the number of missed alarm samples.

[0094] After a preset number of iterative trainings, the optimal parameters of the fitness function are determined as the final model parameters, thereby obtaining a trained multi-target detection model. The trained multi-target detection model is used to identify irregular targets in the first and second video streams.

[0095] Furthermore, the position coordinates of the irregular targets identified in the second video stream are mapped according to the mapping matrix and compared with the position coordinates of the irregular targets identified in the first video stream to eliminate duplicate targets.

[0096] The present invention calculates the mapping matrix between the video stream collected by the drone and the video stream collected by the robot, eliminates the repeated targets detected by both, can reduce repeated judgments, and avoid repeated labeling in the digital twin system.

[0097] S105, and extracting garden operation characteristic parameters from the video stream, applying the garden operation characteristic parameters to the digital twin model to perform non-standard target prediction.

[0098] Specifically, the garden operation characteristic parameters in the first video stream and the second video stream are extracted through a feature extraction model; wherein the garden operation parameters include at least one or more of the following: vegetation characteristics of each landscape, appearance characteristics of each facility, road safety characteristics in the inspection route, and sanitation characteristics in the inspection area.

[0099] Furthermore, the garden operation characteristic parameters are input into the digital twin model to simulate the actual operation scenario of the garden. Then, based on the simulation results of the digital twin model, the tourist destinations with a high probability of irregularities within the preset time period are predicted and determined as predicted irregular targets.

[0100] In one embodiment, in a garden, some places may not currently be identified as non-standard targets. For example, the height of a piece of vegetation is close to the standard height but has not yet reached the standard height. This situation may evolve into a non-standard target in a very short period of time. At this time, through the simulation of the digital twin system, the time when the target evolves into a non-standard target can be predicted, so as to carry out early prevention and treatment, and provide better support for garden maintenance work.

[0101] S106: Generate an inspection report based on the results of the non-standard target identification and the results of the non-standard target prediction.

[0102] Specifically, the identified non-standard target is displayed at the corresponding position of the digital twin model with a first attribute feature; the predicted non-standard target is displayed at the corresponding position of the digital twin model with a second attribute feature; wherein the attribute feature includes at least one or more of the following: icon shape, icon color, and warning method.

[0103] In one embodiment, the identified non-standard target is displayed in the corresponding position of the digital twin model with a red icon or red text, and the predicted non-standard target is displayed in the corresponding position of the digital twin model with an orange icon or orange text, and the predicted time of evolution into the non-standard target is marked.

[0104] As a feasible implementation, during off-day inspection periods, several drones are controlled to conduct supplementary inspections over the garden, generating a third video stream and identifying irregular targets within it. The detailed information of all irregular targets identified and predicted during the garden's business hours is then entered into a pre-set template to generate an inspection report, which is then sent to the relevant responsible person's terminal, completing the daily inspection.

[0105] The present invention realizes all-round and all-time garden inspection by conducting air-ground coordinated inspection during inspection periods and aerial inspection during non-inspection periods. While replacing manual inspections, it improves the efficiency and scope of garden inspections to an unprecedented level, and plays an important role in promoting the technical development of smart garden inspections and efficient garden inspections.

[0106] In addition, the embodiment of the present invention also provides a garden inspection system with air-ground coordination, such as Figure 2 As shown, the air-ground collaborative garden inspection system 200 specifically includes:

[0107] The inspection path generation module 210 is used to build a digital twin model of the target garden based on digital twin technology; drive the digital twin model to perform simulation operation simulation based on the historical operation data of the target garden, and determine the inspection time period and inspection path of the day according to the simulation results;

[0108] The inspection control module 220 is used to control the inspection robots to perform daily inspections along the inspection route during the inspection period of the day; each inspection robot is equipped with at least one inspection drone; the inspection robot's operating environment is detected in real time, and the inspection drone is controlled to perform high-altitude follow-up inspections based on the operating environment;

[0109] The target recognition and prediction module 230 is used to identify irregular targets based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, and extract garden operation characteristic parameters from the video streams; apply the garden operation characteristic parameters to the digital twin model to predict irregular targets;

[0110] The report generation module 240 is used to generate an inspection report according to the result of the irregular target identification and the result of the irregular target prediction.

[0111] The various embodiments of the present invention are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are simplified. For relevant details, refer to the descriptions of the method embodiments.

[0112] The above description of specific embodiments of the present invention is provided. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0113] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A garden inspection method with air-ground coordination, characterized in that: The method comprises: Based on digital twin technology, a digital twin model of the target garden is constructed; the digital twin model is driven by the historical operation data of the target garden to perform simulation operations, and the inspection period and inspection route of the day are determined based on the simulation results; During the inspection period of the day, the inspection robots are controlled to perform routine inspections along the inspection route of the day; wherein each inspection robot is equipped with at least one inspection drone; Real-time detection of the operating environment of the inspection robot, and control of the inspection drone to perform high-altitude follow-up inspections based on the operating environment; Based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, irregular target recognition is performed, specifically including: Based on the timestamps, align the synchronized recording segments in the first video stream with the second video stream frame by frame, and perform frame sampling to obtain a plurality of aligned sampling frame combinations; Determining a mapping matrix between the synchronous recording segment and the second video stream according to the position coordinates of the same target in each sampling frame combination; Based on the real-time target detection RT-DETR algorithm, a multi-target detection model is constructed; During model training, through custom functions , as the fitness function of the multi-target detection model; wherein, The number of samples that correctly detected hidden dangers; Indicates the number of false alarm samples; Indicates the number of missed alarm samples; After a preset number of iterative trainings, the optimal parameters of the fitness function are determined as the final parameters of the model to obtain a trained multi-target detection model; Identifying irregular targets in the first video stream and the second video stream using the trained multi-target detection model; Mapping the position coordinates of the irregular targets identified in the second video stream according to the mapping matrix, and comparing them with the position coordinates of the irregular targets identified in the first video stream to eliminate duplicate targets; Extracting garden operation characteristic parameters from the video stream, applying the garden operation characteristic parameters to the digital twin model, and performing non-standard target prediction; An inspection report is generated based on the results of the irregular target identification and the results of the irregular target prediction.

2. The air-ground coordinated garden inspection method according to claim 1 is characterized in that: Based on digital twin technology, a digital twin model of the target garden is constructed, specifically including: Acquire the garden landform data determined in the target garden planning stage in the database, and construct a three-dimensional model of the garden terrain based on the garden landform data; Using a drone equipped with a depth camera, several depth images of the garden landscape are acquired. Based on image stitching technology, several sets of depth images are seamlessly stitched together to obtain an overall depth image of the garden. Based on the depth information of each pixel in the overall depth image of the garden, a differential value of each pixel is calculated; if the differential value of any pixel in a preset pixel area centered on the pixel is the largest, the pixel information of the pixel is used to form a set of landscape modeling data to obtain a landscape modeling data set; wherein the pixel information includes at least: pixel coordinates, pixel depth information, and pixel color values ​​in RGB channels; Converting pixel coordinates in the landscape modeling data set based on the depth information to obtain three-dimensional coordinates of the landscape modeling data; and constructing a corresponding three-dimensional model of the garden landscape according to the three-dimensional coordinates of the landscape modeling data; The three-dimensional model of the garden terrain is combined with the three-dimensional model of the garden landscape to obtain a three-dimensional physical model of the target garden; The operating data type of each facility in the target garden is obtained, and each operating data type is set as an initial value to form a data model, and the data model is combined with the three-dimensional physical model to obtain a digital twin model of the target garden.

3. The air-ground coordinated garden inspection method according to claim 2 is characterized in that: The digital twin model is driven by the historical operation data of the target garden to perform simulation operation, and the inspection period and inspection route of the day are determined according to the simulation results, specifically including: Obtaining historical values ​​corresponding to the operation data type within a preset historical time period to form the historical operation data; Inputting the historical operation data into the digital twin model, driving the digital twin model to continue simulating the operation, and obtaining the daily operation forecast data; wherein the daily operation forecast data at least includes: passenger flow forecast data for each time period, passenger flow distribution forecast data for each time period, predicted interaction rate of each tourist destination, and weather forecast data; According to the passenger flow forecast data of each time period, the time period with the least passenger flow during the garden business hours on that day is selected as the inspection time period on that day; Determine the key inspection targets for the day within the target garden based on the forecasted daily operation data; The inspection route for the day is constructed based on the distance relationship between the key inspection targets for the day and the actual road planning.

4. The air-land coordinated garden inspection method according to claim 3 is characterized in that: Based on the daily operational forecast data, determine the key inspection targets for the day within the target garden, specifically including: Calculate the predicted passenger flow for each tourist destination in each time period based on the passenger flow forecast data and passenger flow distribution forecast data for each time period; Based on expert experience, a weight table of the impact of various weather conditions on the safety of different tourist destinations is developed; and based on the weather forecast data, the weather impact weight of each tourist destination on the day is obtained in the weight table; according to , determine the inspection weight of each tourist destination; among them, is the inspection weight of the j-th tour target, is the weather impact weight of the day corresponding to the j-th tourist destination. If the weather has no impact on the j-th tourist destination, the weather impact weight of the day is 1; is the predicted passenger flow of the jth tourist destination in the i-th period, is the total predicted passenger flow of the target garden in the i-th period, n is the total business hours of the target garden on that day, is the predicted interaction rate of the jth tour target in the i-th period; The inspection weight average of all the tourist targets on the day is calculated, and the tourist targets whose inspection weight exceeds the inspection weight average are determined as the key inspection targets on the day.

5. The air-ground coordinated garden inspection method according to claim 1 is characterized in that: Real-time detection of the operating environment of the inspection robot and control of the inspection drone to perform high-altitude follow-up inspections based on the operating environment, specifically including: Monitoring and displaying the inspection route and real-time position of the inspection robot in the digital twin model; Based on the real-time position, determine whether there is an obstruction above the inspection route in front of the inspection robot. If so, obtain the maximum height of the obstructing target in front in the digital twin model, and set the flight altitude of the inspection drone according to the maximum height; Controlling the inspection drone carried by the inspection robot to take off vertically, and after reaching the flight altitude, following the inspection trajectory of the inspection robot to perform synchronous inspection; In the first video stream recorded by the inspection robot, the video stream segment recorded after the inspection drone is started is marked as a synchronous recording segment.

6. The air-ground coordinated garden inspection method according to claim 1 is characterized in that: Extracting garden operation characteristic parameters from the video stream, including: The garden operation characteristic parameters in the first video stream and the second video stream are extracted through a feature extraction model; wherein the garden operation parameters include at least: vegetation characteristics of each landscape, appearance characteristics of each facility, road safety characteristics in the inspection route, and sanitation characteristics in the inspection area.

7. The air-ground coordinated garden inspection method according to claim 6 is characterized in that: Applying the garden operation characteristic parameters to the digital twin model to perform irregular target prediction specifically includes: Inputting the garden operation characteristic parameters into the digital twin model to simulate the actual garden operation scene; Based on the simulation results of the digital twin model, it is predicted that the tourist destinations with a high probability of experiencing irregularities within the preset time period are identified as predicted irregular destinations.

8. The air-ground coordinated garden inspection method according to claim 7 is characterized in that: Generate an inspection report based on the results of the irregular target identification and the irregular target prediction, specifically including: Displaying the identified irregular target at the corresponding position of the digital twin model with a first attribute feature; displaying the predicted irregular target at the corresponding position of the digital twin model with a second attribute feature; wherein the attribute feature includes at least: icon shape, icon color, and warning method; During non-inspection periods, several free drones are controlled to conduct supplementary inspections over the garden, obtain a third video stream, and identify irregular targets in it; Fill in the detailed information of all identified irregular targets and predicted irregular targets into the preset template, generate the inspection report, and send it to the terminal of the relevant person in charge to complete the daily inspection.

9. A garden inspection system with air-ground coordination, characterized in that: The system comprises: An inspection path generation module is used to build a digital twin model of the target garden based on digital twin technology; drive the digital twin model to perform simulation operations using the historical operation data of the target garden, and determine the inspection time period and inspection path for the day based on the simulation results; An inspection control module is used to control the inspection robots to perform daily inspections along the inspection route during the inspection period of the day; wherein each inspection robot is equipped with at least one inspection drone; detect the operating environment of the inspection robots in real time, and control the inspection drones to perform high-altitude follow-up inspections based on the operating environment; The target recognition and prediction module is used to identify irregular targets based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, specifically including: Based on the timestamps, align the synchronized recording segments in the first video stream with the second video stream frame by frame, and perform frame sampling to obtain a plurality of aligned sampling frame combinations; Determining a mapping matrix between the synchronous recording segment and the second video stream according to the position coordinates of the same target in each sampling frame combination; Based on the real-time target detection RT-DETR algorithm, a multi-target detection model is constructed; During model training, through custom functions , as the fitness function of the multi-target detection model; wherein, The number of samples that correctly detected hidden dangers; Indicates the number of false alarm samples; Indicates the number of missed alarm samples; After a preset number of iterative trainings, the optimal parameters of the fitness function are determined as the final parameters of the model to obtain a trained multi-target detection model; Identifying irregular targets in the first video stream and the second video stream using the trained multi-target detection model; Mapping the position coordinates of the irregular targets identified in the second video stream according to the mapping matrix and comparing them with the position coordinates of the irregular targets identified in the first video stream to eliminate duplicate targets; extracting garden operation characteristic parameters from the video stream; applying the garden operation characteristic parameters to the digital twin model to predict irregular targets; The report generation module is used to generate an inspection report based on the results of the irregular target identification and the results of the irregular target prediction.

Images