Air-ground cooperative garden inspection method and system

Abstract

The invention discloses an air-ground cooperative garden inspection method and system, belongs to the technical field of garden inspection, and is used for solving the technical problems that an existing garden inspection route and inspection range are relatively fixed, coverage inspection of the whole garden space cannot be performed, an inspection result cannot be visually seen at any time, and hidden danger prevention means are lacked. The method comprises the following steps: constructing a digital twinborn model of a target garden based on a digital twinborn technology; according to an analogue simulation result, determining a current-day inspection time period and a current-day inspection path; controlling the inspection robot to perform daily inspection along the inspection path of the day in the inspection period of the day; the operation environment of the inspection robot is detected in real time, and the inspection unmanned aerial vehicle is controlled to carry out high-altitude following inspection according to the operation environment; based on a first video stream returned by the inspection robot and a second video stream returned by the inspection unmanned aerial vehicle, non-standard target identification is carried out; and generating an inspection report according to a result of non-standard target identification and a result of non-standard target prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of garden inspection, and in particular to an air-ground coordinated garden inspection method and system. Background Art

[0002] Garden inspection mainly includes inspection of landscape building facilities, equipment facilities, safety facilities, hidden dangers of pests and diseases, hidden dangers of personnel safety, hidden dangers of fire, etc., which usually require daily inspection, regular inspection and seasonal inspection to ensure the safe operation of the garden. At present, garden inspection mainly relies on manual work. Inspectors need to check every detail of garden facilities one by one according to the requirements of the inspection system, and finally form an inspection report and submit it to the relevant departments for verification.

[0003] As you can imagine, manual inspections not only consume a lot of manpower and material resources, but are also limited by the ability and experience of inspectors, which may lead to missed inspections and false inspections. In addition, manual inspections have a limited inspection range, and cannot obtain information from high altitudes and blind spots. They cannot conduct all-round inspections of garden conditions. Inspectors can only inspect according to the set inspection items and inspection routes, which may easily miss some hidden safety hazards.

[0004] At present, some gardens use inspection robots instead of manual inspections, but the inspection range of the inspection robots is also limited. They can only judge the situation within their field of vision, and cannot obtain information on hidden dangers at high altitudes and outside the prescribed routes. The inspection results are relatively one-sided, and the inspection range cannot cover the entire garden space. It is also impossible to intuitively see where irregularities occur. It is necessary to wait until the inspection is completed and an inspection report is formed to know the inspection situation. Moreover, only the inspection situation of the day can be obtained. It is impossible to predict the hidden dangers that may occur in the next period of time based on the current inspection situation. The acquisition of inspection information is lagging and lacks foresight and prevention. Summary of the invention

[0005] The embodiment of the present invention provides an air-ground coordinated garden inspection method and system, which is used to solve the following technical problems: the existing garden inspection routes and inspection ranges are relatively fixed, it is impossible to conduct comprehensive inspections of the entire garden space, it is impossible to intuitively see the inspection results at any time, and there is a lack of hidden danger prevention measures.

[0006] The embodiment of the present invention adopts the following technical solution:

[0007] On the one hand, an embodiment of the present invention provides an air-ground collaborative garden inspection method, the method comprising: constructing a digital twin model of a target garden based on digital twin technology; driving the digital twin model to perform simulation operation simulation through historical operation data of the target garden, and determining the inspection period and inspection path of the day according to the simulation results;

[0008] During the inspection period of the day, the inspection robots are controlled to perform routine inspections along the inspection route of the day; wherein each inspection robot is equipped with at least one inspection drone;

[0009] Detect the operating environment of the inspection robot in real time, and control the inspection drone to perform high-altitude follow-up inspection according to the operating environment;

[0010] Based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, irregular target recognition is performed;

[0011] Extracting garden operation characteristic parameters from the video stream, applying the garden operation characteristic parameters to the digital twin model, and performing irregular target prediction;

[0012] An inspection report is generated according to the result of the irregular target identification and the result of the irregular target prediction.

[0013] In a feasible implementation, a digital twin model of a target garden is constructed based on digital twin technology, specifically including:

[0014] Acquire the garden landform data determined in the target garden planning stage in the database, and construct a three-dimensional model of the garden terrain according to the garden landform data;

[0015] A plurality of depth images of the garden landscape are obtained by using a drone equipped with a depth camera; the plurality of groups of depth images are seamlessly stitched together based on image stitching technology to obtain an overall depth image of the garden;

[0016] Based on the depth information of each pixel in the overall depth image of the garden, the differential value of each pixel is calculated; if the differential value of any pixel in a preset pixel area centered on the pixel is the largest, the pixel information of the pixel is used to form a set of landscape modeling data to obtain a landscape modeling data set; wherein the pixel information at least includes: pixel coordinates, pixel depth information, and color values ​​of the pixel in RGB channels;

[0017] Convert pixel coordinates in the landscape modeling data set based on the depth information to obtain three-dimensional coordinates of the landscape modeling data; construct a corresponding three-dimensional model of the garden landscape according to the three-dimensional coordinates of the landscape modeling data;

[0018] The three-dimensional model of the garden terrain is combined with the three-dimensional model of the garden landscape to obtain a three-dimensional physical model of the target garden;

[0019] The operating data type of each facility in the target garden is obtained, and each operating data type is set as an initial value to form a data model, and the data model is combined with the three-dimensional physical model to obtain a digital twin model of the target garden.

[0020] In a feasible implementation, the digital twin model is driven by the historical operation data of the target garden to perform simulation operation, and the inspection period and inspection route of the day are determined according to the simulation results, specifically including:

[0021] Obtaining historical values ​​corresponding to the operation data type within a preset historical time period to form the historical operation data;

[0022] Input the historical operation data into the digital twin model, drive the digital twin model to continue to perform simulation operation, and obtain the operation forecast data for the day; wherein the operation forecast data for the day includes at least one or more of the following: passenger flow forecast data for each time period, passenger flow distribution forecast data for each time period, predicted interaction rate of each tourist destination, and weather forecast data;

[0023] According to the passenger flow forecast data of each time period, the time period with the least passenger flow during the garden business hours of the day is selected as the inspection time period of the day;

[0024] Determine the key inspection targets for the day in the target garden based on the forecast data of the day's operations;

[0025] Based on the distance relationship between the key inspection targets of the day and the actual road planning, the inspection route of the day is constructed.

[0026] In a feasible implementation, based on the daily operation forecast data, determining the key inspection targets for the day in the target garden specifically includes:

[0027] Calculate the predicted passenger flow of each tourist destination in each time period based on the passenger flow forecast data and passenger flow distribution forecast data of each time period;

[0028] Based on expert experience, a weight table of the impact of various weather conditions on the safety of different tourist destinations is formulated; and according to the weather forecast data, the weather impact weight of each tourist destination on the day is obtained in the weight table;

[0029] according to Determine the inspection weight of each tourist destination; where W j is the inspection weight of the jth tour target, ω j is the weather impact weight of the day corresponding to the jth tourist destination. If the weather has no impact on the jth tourist destination, the weather impact weight of the day is 1; P ijis the predicted passenger flow of the jth tourist destination in the i-th period, P i is the total predicted passenger flow of the target garden in the i-th period, n is the total business hours of the target garden on that day, T ij is the predicted interaction rate of the jth tour target in the i-th period;

[0030] The inspection weight average of all the sightseeing targets on the day is calculated, and the sightseeing targets whose inspection weights exceed the inspection weight average are determined as the key inspection targets on the day.

[0031] In a feasible implementation, real-time detection of the operating environment of the inspection robot and control of the inspection drone to perform high-altitude follow-up inspection according to the operating environment specifically include:

[0032] Monitoring and displaying the inspection route and real-time position of the inspection robot in the digital twin model;

[0033] According to the real-time position, determine whether there is an obstruction above the inspection route in front of the inspection robot. If so, obtain the maximum height of the obstructing target in front in the digital twin model, and set the flight height of the inspection drone according to the maximum height;

[0034] Controlling the inspection drone carried by the inspection robot to take off vertically, and after reaching the flight altitude, following the inspection trajectory of the inspection robot to perform synchronous inspection;

[0035] In the first video stream recorded by the inspection robot, the video stream segment recorded after the inspection drone is started is marked as a synchronous recording segment.

[0036] In a feasible implementation, based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, irregular target recognition is performed, specifically including:

[0037] Based on the timestamp, align the synchronous recording segments in the first video stream with the second video stream frame by frame, and perform frame sampling to obtain a plurality of aligned sampling frame combinations;

[0038] Determine a mapping matrix between the synchronous recording segment and the second video stream according to the position coordinates of the same target in each sampling frame combination;

[0039] Based on the real-time target detection RT-DETR algorithm, a multi-target detection model is constructed;

[0040] During model training, through custom functions As the fitness function of the multi-target detection model; where R p F is the number of samples that correctly detect hidden dangers; p Indicates the number of false alarm samples; FN Indicates the number of missed alarm samples;

[0041] After a preset number of iterative trainings, the optimal parameters of the fitness function are determined as the final parameters of the model, and a trained multi-target detection model is obtained;

[0042] Identify irregular targets in the first video stream and the second video stream by using the trained multi-target detection model;

[0043] According to the mapping matrix, the position coordinates of the irregular targets identified in the second video stream are mapped and compared with the position coordinates of the irregular targets identified in the first video stream to eliminate duplicate targets.

[0044] In a feasible implementation manner, extracting garden operation characteristic parameters from the video stream specifically includes:

[0045] The garden operation characteristic parameters in the first video stream and the second video stream are extracted through a feature extraction model; wherein the garden operation parameters include at least one or more of the following: vegetation characteristics of each landscape, appearance characteristics of each facility, road safety characteristics in the inspection route, and sanitation characteristics in the inspection area.

[0046] In a feasible implementation, the garden operation characteristic parameters are applied to the digital twin model to perform irregular target prediction, specifically including:

[0047] Inputting the garden operation characteristic parameters into the digital twin model to simulate the actual garden operation scene;

[0048] Based on the simulation results of the digital twin model, it is predicted that the tourist destinations with a high probability of experiencing irregular situations within the preset time period are identified as predicted irregular targets.

[0049] In a feasible implementation manner, an inspection report is generated according to the result of the irregular target identification and the result of the irregular target prediction, specifically including:

[0050] Displaying the identified irregular target at the corresponding position of the digital twin model with a first attribute feature; displaying the predicted irregular target at the corresponding position of the digital twin model with a second attribute feature; wherein the attribute feature includes at least one or more of the following: icon shape, icon color, warning method;

[0051] During non-inspection periods of the day, several free drones are controlled to conduct supplementary inspections over the garden, obtain the third video stream, and identify irregular targets in it;

[0052] The detailed information of all the identified irregular targets and predicted irregular targets are filled into the preset template, the inspection report is generated, and sent to the terminal of the relevant person in charge to complete the daily inspection.

[0053] On the other hand, an embodiment of the present invention further provides an air-ground coordinated garden inspection system, the system comprising:

[0054] The inspection path generation module is used to build a digital twin model of the target garden based on the digital twin technology; drive the digital twin model to perform simulation operation simulation through the historical operation data of the target garden, and determine the inspection period and inspection path of the day according to the simulation results;

[0055] The inspection control module is used to control the inspection robot to perform daily inspections along the inspection path of the day during the inspection period of the day; wherein each inspection robot is equipped with at least one inspection drone; real-time detection of the operating environment of the inspection robot, and control the inspection drone to perform high-altitude follow-up inspections according to the operating environment;

[0056] A target recognition and prediction module is used to recognize irregular targets based on the first video stream sent back by the inspection robot and the second video stream sent back by the inspection drone, and to extract garden operation characteristic parameters from the video streams; the garden operation characteristic parameters are applied to the digital twin model to predict irregular targets;

[0057] The report generation module is used to generate an inspection report according to the result of the irregular target identification and the result of the irregular target prediction.

[0058] Compared with the prior art, the air-ground coordinated garden inspection method and system provided by the embodiment of the present invention has the following beneficial effects:

[0059] First of all, the present invention deeply combines digital twin technology with intelligent garden inspection. It simulates the garden operation status through the digital twin model, so as to obtain the best inspection time and inspection path of the day. It can generate different inspection routes every day according to the daily passenger flow, passenger flow distribution, crowded time periods and other differences, overcoming the defects of inflexible inspection of fixed inspection routes and fixed inspection range, and can greatly improve the flexibility and comprehensiveness of garden inspection.

[0060] Then, the present invention carries an inspection drone on the inspection robot. When there is no vegetation blocking the robot, the inspection drone stops in the robot's nest. Only when the digital twin system finds that there is vegetation blocking the inspection route in front of the robot, the drone will rise to work, making up for the lack of viewing angle of the inspection robot and obtaining dual information from the sky and the ground. This is different from the general air-ground collaborative working mode. The working mode proposed by the present invention does not start the drone when there is no need for high-altitude inspection, which can reduce useless energy consumption and achieve energy conservation and emission reduction.

[0061] Finally, after identifying irregular targets based on the video streams collected by robots and drones, the present invention also predicts irregular targets in the digital twin system through the operational characteristic parameters extracted from the video streams. Through the simulation of the digital twin system, it is possible to predict the time when the target evolves into an irregular target, so as to prevent and deal with it in advance, providing better support for garden maintenance work. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0063] Figure 1 A flow chart of a garden inspection method for air-ground collaboration provided by an embodiment of the present invention;

[0064] Figure 2 A schematic structural diagram of an air-ground coordinated garden inspection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0066] The embodiment of the present invention provides a garden inspection method with air-ground coordination, such as Figure 1 As shown, the air-ground collaborative garden inspection method specifically includes steps S101-S106:

[0067] S101. Based on digital twin technology, construct a digital twin model of the target garden.

[0068] Specifically, the garden landform data determined in the target garden planning stage is obtained from the database, and a three-dimensional model of the garden terrain is constructed based on the garden landform data.

[0069] Furthermore, a plurality of depth images of the garden landscape are obtained by using a drone equipped with a depth camera, and then based on the image stitching technology, the plurality of depth images are seamlessly stitched together to obtain an overall depth image of the garden.

[0070] Furthermore, based on the depth information of each pixel in the overall depth image of the garden, the differential value of each pixel is calculated. If the differential value of any pixel in the preset pixel area centered on itself is the largest, the pixel information of the pixel is used to form a set of landscape modeling data to obtain a landscape modeling data set; wherein the pixel information at least includes: pixel coordinates, pixel depth information, and pixel color value in RGB channel.

[0071] As a feasible implementation method, the differential value calculation formula for each pixel in the depth image is:

[0072] Among them, d x,y Represents the depth information of pixel I(x,y), I k (x, y) represents the color value of pixel I(x, y) in the k color channel, e represents the natural constant, σ1 and σ2 represent the scale standard deviation, c1 and c2 represent the length of the unit pixel in the horizontal and vertical directions respectively, and f x Indicates the focal length of the depth camera in the horizontal direction, f y Indicates the vertical focal length of the depth camera.

[0073] Furthermore, the pixel coordinates in the landscape modeling data set are transformed based on the depth information to obtain the three-dimensional coordinates of the landscape modeling data. Then, the corresponding three-dimensional model of the garden landscape is constructed according to the three-dimensional coordinates of the landscape modeling data. The three-dimensional model of the garden terrain is spliced ​​and fused with the three-dimensional model of the garden landscape to obtain the three-dimensional physical model of the target garden.

[0074] Furthermore, the operating data type of each facility in the target garden is obtained, and each operating data type is set as an initial value to form a data model, and the data model is combined with the three-dimensional physical model to obtain a digital twin model of the target garden.

[0075] As a feasible implementation method, the operation data types of various facilities in the garden may include: power consumption data and operation data of various power facilities, passenger flow data of various landscapes and buildings, vegetation type and vegetation standard specification data of various landscapes, etc. In the initial state, these data can be set to 0.

[0076] S102. Drive the digital twin model through the historical operation data of the target garden to perform operation simulation, and determine the inspection period and inspection route for the day based on the simulation results.

[0077] Specifically, the historical values ​​corresponding to the operation data types in the preset historical time period are obtained to form the historical operation data. The historical operation data is input into the digital twin model, and the digital twin model is driven to continue the simulation operation simulation to obtain the operation forecast data for the day. Among them, the operation forecast data for the day includes at least one or more of the following: passenger flow forecast data for each time period, passenger flow distribution forecast data for each time period, predicted interaction rate of each tourist destination, and weather forecast data.

[0078] As a feasible implementation method, in the garden, the various sightseeing targets mainly include the main landscapes, such as rockery, pond, bamboo forest, etc., which are all sightseeing targets of the landscape type. The sightseeing targets also include various buildings, such as pavilions, halls, wooden houses, etc., which are all architectural sightseeing targets. The sightseeing targets also include various interactive facilities, such as benches, swings, etc., which are all interactive sightseeing targets. It can be understood that the places that tourists may visit in the garden can be divided into individual sightseeing targets.

[0079] Furthermore, according to the passenger flow forecast data of each time period, the time period with the least passenger flow during the garden business hours of the day is selected as the inspection period of the day. And according to the operation forecast data of the day, the key inspection targets of the day in the target garden are determined. Based on the distance relationship between the key inspection targets of the day and the actual road planning, the key inspection targets with the closest distance and connected by roads are used as adjacent inspection targets to construct the inspection path of the day.

[0080] As a feasible implementation method, the key inspection targets of the day in the target garden are determined based on the daily operation forecast data, including:

[0081] According to the passenger flow forecast data and passenger flow distribution forecast data of each period, the predicted passenger flow of each tourist destination in each period is calculated. Then, based on expert experience, a weight table of the impact of various weather conditions on the safety of different tourist destinations is formulated, and according to the weather forecast data, the weather impact weight of each tourist destination on the day is obtained in the weight table.

[0082] Then according to Determine the inspection weight of each tourist destination; where W j is the inspection weight of the jth tour target, ω j is the weather impact weight of the day corresponding to the jth tourist destination. If the weather has no impact on the jth tourist destination, the weather impact weight of the day is 1; P ij is the predicted passenger flow of the jth tourist destination in the i-th period, Pi is the total predicted passenger flow of the target garden in the i-th period, n is the total business hours of the target garden on that day, T ij is the predicted interaction rate of the j-th tour target in the i-th period.

[0083] Finally, the average inspection weights of all tourist targets for the day are calculated, and the tourist targets whose inspection weights exceed the average inspection weights are identified as the key inspection targets for the day.

[0084] S103, during the inspection period of the day, control the inspection robot to perform daily inspection along the inspection route of the day; wherein each inspection robot is equipped with at least one inspection drone. Real-time detection of the operating environment of the inspection robot is performed, and according to the operating environment, the inspection drone is controlled to perform high-altitude follow-up inspection.

[0085] Specifically, after arriving at the inspection period of the day, the inspection robot will conduct daily inspections along the designated inspection route of the day. The inspection robot is equipped with at least one inspection drone, as well as auxiliary modules such as the laser radar obstacle avoidance module and the GPS positioning module. The GPS positioning module can transmit the real-time position of the inspection robot to the digital twin model, so that the inspection route and real-time position of the inspection robot can be monitored and displayed in the digital twin model.

[0086] Furthermore, according to the real-time position of the inspection robot, the digital twin model determines whether vegetation appears above the inspection route in front of the inspection robot. If so, the maximum height of the obstructing target in front is obtained in the digital twin model, and the flight altitude of the inspection drone is set according to the maximum height. Then the inspection drone carried by the inspection robot is controlled to take off vertically, and after reaching the flight altitude, it follows the inspection trajectory of the inspection robot for synchronous inspection.

[0087] Furthermore, in the first video stream recorded by the inspection robot, the video stream segment recorded after the inspection drone is started is marked as a synchronous recording segment.

[0088] In one embodiment, when the drone rises to a flight altitude and starts recording video, the inspection robot marks the first mark in the video stream it records. After the drone finishes recording, the inspection robot marks the second mark in the video stream it records. The segment between the two marks is the synchronized recording segment.

[0089] As a feasible implementation method, the obstruction target in front of the inspection robot may be vegetation or the roof of a building. When the digital twin model detects that the inspection robot has reached an unobstructed area, it sends a descent command to the inspection drone, which falls back to the inspection robot's nest until the next vegetation or building obstruction is detected.

[0090] S104, performing non-standard target recognition based on the first video stream sent back by the inspection robot and the second video stream sent back by the inspection drone.

[0091] Specifically, based on the timestamp, the synchronous recording segment in the first video stream is aligned frame by frame with the second video stream, and frame sampling is performed to obtain a plurality of aligned sampling frame combinations. Then, according to the position coordinates of the same target in each sampling frame combination, the mapping matrix between the synchronous recording segment and the second video stream is determined.

[0092] As a feasible implementation method, in the synchronous recording segment and the second video stream, video alignment is performed according to the timestamp. After the alignment, frame sampling is performed in the two video streams respectively, and a number of image frames are randomly collected, and the timestamps of the image frames collected in the two video streams are the same. The two image frames with the same timestamp are combined into a sampling frame combination, and an identical target is identified in each sampling frame combination, and the position coordinates of the target in the two image frames are obtained. A mapping matrix is ​​constructed based on the mapping relationship between the position coordinates of the same target in each sampling frame combination. The mapping matrix can reflect the position conversion relationship between the second video stream and the target in the synchronous recording segment.

[0093] Furthermore, a multi-target detection model is constructed based on the real-time target detection RT-DETR algorithm. As the fitness function of the multi-target detection model; where R p F is the number of samples that correctly detect hidden dangers; p Indicates the number of false alarm samples; F N Indicates the number of missed alarm samples.

[0094] After a preset number of iterative trainings, the optimal parameters of the fitness function are determined as the final parameters of the model to obtain a trained multi-target detection model. The trained multi-target detection model is used to identify irregular targets in the first video stream and the second video stream.

[0095] Furthermore, the position coordinates of the irregular targets identified in the second video stream are mapped according to the mapping matrix and compared with the position coordinates of the irregular targets identified in the first video stream to eliminate duplicate targets.

[0096] The present invention calculates a mapping matrix between the video stream collected by the drone and the video stream collected by the robot, eliminates repeated targets detected by both, can reduce repeated judgments, and avoid repeated labeling in the digital twin system.

[0097] S105, extracting garden operation characteristic parameters from the video stream, applying the garden operation characteristic parameters to the digital twin model, and performing irregular target prediction.

[0098] Specifically, the garden operation characteristic parameters in the first video stream and the second video stream are extracted through a feature extraction model; wherein the garden operation parameters include at least one or more of the following: vegetation characteristics of each landscape, appearance characteristics of each facility, road safety characteristics in the inspection route, and sanitation characteristics in the inspection area.

[0099] Furthermore, the garden operation characteristic parameters are input into the digital twin model to simulate the actual operation scenario of the garden. Then, based on the simulation results of the digital twin model, the tourist targets with a high probability of experiencing irregular situations within a preset time period are predicted and determined as predicted irregular targets.

[0100] In one embodiment, in a garden, some places may not currently be identified as non-standard targets. For example, the height of a piece of vegetation is close to the standard height but has not yet reached the standard height. This situation may evolve into a non-standard target in a very short time. At this time, through the simulation of the digital twin system, the time when the target evolves into a non-standard target can be predicted, so as to take preventive measures and deal with it in advance, providing better support for garden maintenance work.

[0101] S106: Generate an inspection report based on the results of the irregular target identification and the results of the irregular target prediction.

[0102] Specifically, the identified non-standard target is displayed at the corresponding position of the digital twin model with a first attribute feature; the predicted non-standard target is displayed at the corresponding position of the digital twin model with a second attribute feature; wherein the attribute feature includes at least one or more of the following: icon shape, icon color, and warning method.

[0103] In one embodiment, the identified non-standard target is displayed as a red icon or red text at the corresponding position of the digital twin model, and the predicted non-standard target is displayed as an orange icon or orange text at the corresponding position of the digital twin model, and the predicted time of evolution into the non-standard target is marked.

[0104] As a feasible implementation method, during the non-daily inspection period, several free drones are controlled to conduct supplementary inspections over the garden to obtain a third video stream and identify irregular targets. Then, the detailed information of all irregular targets identified and predicted during the garden business hours is filled into the preset template, and an inspection report is generated and sent to the terminal of the relevant person in charge to complete the daily inspection.

[0105] The present invention realizes all-round and all-time garden inspection by conducting air-ground coordinated inspection during inspection periods and aerial inspection during non-inspection periods. While replacing manual inspections, it improves the efficiency and scope of garden inspections to an unprecedented level, and plays an important role in promoting the technical development of smart garden inspections and efficient garden inspections.

[0106] In addition, the embodiment of the present invention also provides a garden inspection system with air-ground coordination, such as Figure 2 As shown, the air-ground coordinated garden inspection system 200 specifically includes:

[0107] The inspection path generation module 210 is used to construct a digital twin model of the target garden based on the digital twin technology; drive the digital twin model to perform simulation operation simulation through the historical operation data of the target garden, and determine the inspection period and inspection path of the day according to the simulation results;

[0108] The inspection control module 220 is used to control the inspection robot to perform daily inspections along the inspection path of the day during the inspection period of the day; wherein each inspection robot is equipped with at least one inspection drone; detect the operating environment of the inspection robot in real time, and control the inspection drone to perform high-altitude follow-up inspections according to the operating environment;

[0109] The target recognition and prediction module 230 is used to recognize irregular targets based on the first video stream sent back by the inspection robot and the second video stream sent back by the inspection drone, and extract garden operation characteristic parameters from the video streams; apply the garden operation characteristic parameters to the digital twin model to predict irregular targets;

[0110] The report generation module 240 is used to generate an inspection report according to the result of the irregular target identification and the result of the irregular target prediction.

[0111] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0112] The above describes specific embodiments of the present invention. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0113] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. A garden inspection method with air-ground coordination, characterized in that: The method comprises: Based on digital twin technology, a digital twin model of the target garden is constructed; the digital twin model is driven to perform simulation operation through the historical operation data of the target garden, and the inspection period and inspection route of the day are determined according to the simulation results; During the inspection period of the day, the inspection robots are controlled to perform routine inspections along the inspection route of the day; wherein each inspection robot is equipped with at least one inspection drone; Detect the operating environment of the inspection robot in real time, and control the inspection drone to perform high-altitude follow-up inspection according to the operating environment; Based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, irregular target recognition is performed; Extracting garden operation characteristic parameters from the video stream, applying the garden operation characteristic parameters to the digital twin model, and performing irregular target prediction; An inspection report is generated according to the result of the irregular target identification and the result of the irregular target prediction.

2. The air-ground coordinated garden inspection method according to claim 1 is characterized in that: Based on digital twin technology, a digital twin model of the target garden is constructed, including: Acquire the garden landform data determined in the target garden planning stage in the database, and construct a three-dimensional model of the garden terrain according to the garden landform data; A plurality of depth images of the garden landscape are obtained by using a drone equipped with a depth camera; the plurality of groups of depth images are seamlessly stitched together based on image stitching technology to obtain an overall depth image of the garden; Based on the depth information of each pixel in the overall depth image of the garden, the differential value of each pixel is calculated; if the differential value of any pixel in a preset pixel area centered on the pixel is the largest, the pixel information of the pixel is used to form a set of landscape modeling data to obtain a landscape modeling data set; wherein the pixel information at least includes: pixel coordinates, pixel depth information, and color values ​​of the pixel in RGB channels; Convert pixel coordinates in the landscape modeling data set based on the depth information to obtain three-dimensional coordinates of the landscape modeling data; construct a corresponding three-dimensional model of the garden landscape according to the three-dimensional coordinates of the landscape modeling data; The three-dimensional model of the garden terrain is combined with the three-dimensional model of the garden landscape to obtain a three-dimensional physical model of the target garden; The operating data type of each facility in the target garden is obtained, and each operating data type is set as an initial value to form a data model, and the data model is combined with the three-dimensional physical model to obtain a digital twin model of the target garden.

3. The air-ground coordinated garden inspection method according to claim 2 is characterized in that: The digital twin model is driven by the historical operation data of the target garden to perform simulation operation, and the inspection period and inspection route of the day are determined according to the simulation results, specifically including: Obtaining historical values ​​corresponding to the operation data type within a preset historical time period to form the historical operation data; Input the historical operation data into the digital twin model, drive the digital twin model to continue to perform simulation operation, and obtain the operation forecast data for the day; wherein the operation forecast data for the day includes at least one or more of the following: passenger flow forecast data for each time period, passenger flow distribution forecast data for each time period, predicted interaction rate of each tourist destination, and weather forecast data; According to the passenger flow forecast data of each time period, the time period with the least passenger flow during the garden business hours of the day is selected as the inspection time period of the day; Determine the key inspection targets for the day in the target garden based on the forecast data of the day's operations; Based on the distance relationship between the key inspection targets of the day and the actual road planning, the inspection route of the day is constructed.

4. The air-ground coordinated garden inspection method according to claim 3 is characterized in that: According to the forecast data of the day's operation, the key inspection targets of the day in the target garden are determined, including: Calculate the predicted passenger flow of each tourist destination in each time period based on the passenger flow forecast data and passenger flow distribution forecast data of each time period; Based on expert experience, a weight table of the impact of various weather conditions on the safety of different tourist destinations is formulated; and according to the weather forecast data, the weather impact weight of each tourist destination on the day is obtained in the weight table; according to Determine the inspection weight of each tourist destination; where W j is the inspection weight of the jth tour target, ω j is the weather impact weight of the day corresponding to the jth tourist destination. If the weather has no impact on the jth tourist destination, the weather impact weight of the day is 1; P ij is the predicted passenger flow of the jth tourist destination in the i-th period, P i is the total predicted passenger flow of the target garden in the i-th period, n is the total business hours of the target garden on that day, T ij is the predicted interaction rate of the jth tour target in the i-th period; The inspection weight average of all the sightseeing targets on the day is calculated, and the sightseeing targets whose inspection weights exceed the inspection weight average are determined as the key inspection targets on the day.

5. The air-ground coordinated garden inspection method according to claim 1 is characterized in that: Real-time detection of the operating environment of the inspection robot, and control of the inspection drone to perform high-altitude follow-up inspection according to the operating environment, specifically including: Monitoring and displaying the inspection route and real-time position of the inspection robot in the digital twin model; According to the real-time position, determine whether there is an obstruction above the inspection route in front of the inspection robot. If so, obtain the maximum height of the obstructing target in front in the digital twin model, and set the flight height of the inspection drone according to the maximum height; Controlling the inspection drone carried by the inspection robot to take off vertically, and after reaching the flight altitude, following the inspection trajectory of the inspection robot to perform synchronous inspection; In the first video stream recorded by the inspection robot, the video stream segment recorded after the inspection drone is started is marked as a synchronous recording segment.

6. The air-ground coordinated garden inspection method according to claim 1 is characterized in that: Based on the first video stream returned by the inspection robot and the second video stream returned by the inspection drone, irregular target recognition is performed, specifically including: Based on the timestamp, align the synchronous recording segments in the first video stream with the second video stream frame by frame, and perform frame sampling to obtain a plurality of aligned sampling frame combinations; Determine a mapping matrix between the synchronous recording segment and the second video stream according to the position coordinates of the same target in each sampling frame combination; Based on the real-time target detection RT-DETR algorithm, a multi-target detection model is constructed; During model training, through custom functions As the fitness function of the multi-target detection model; where R p F is the number of samples that correctly detect hidden dangers; p Indicates the number of false alarm samples; F N Indicates the number of missed alarm samples; After a preset number of iterative trainings, the optimal parameters of the fitness function are determined as the final parameters of the model, and a trained multi-target detection model is obtained; Identify irregular targets in the first video stream and the second video stream by using the trained multi-target detection model; According to the mapping matrix, the position coordinates of the irregular targets identified in the second video stream are mapped and compared with the position coordinates of the irregular targets identified in the first video stream to eliminate duplicate targets.

7. The air-ground coordinated garden inspection method according to claim 1 is characterized in that: Extracting garden operation characteristic parameters from the video stream, including: The garden operation characteristic parameters in the first video stream and the second video stream are extracted through a feature extraction model; wherein the garden operation parameters include at least one or more of the following: vegetation characteristics of each landscape, appearance characteristics of each facility, road safety characteristics in the inspection route, and sanitation characteristics in the inspection area.

8. The air-ground coordinated garden inspection method according to claim 7 is characterized in that: Applying the garden operation characteristic parameters to the digital twin model to perform irregular target prediction specifically includes: Inputting the garden operation characteristic parameters into the digital twin model to simulate the actual garden operation scene; Based on the simulation results of the digital twin model, it is predicted that the tourist destinations with a high probability of experiencing irregular situations within the preset time period are identified as predicted irregular targets.

9. The air-ground coordinated garden inspection method according to claim 8 is characterized in that: Generate an inspection report based on the result of the irregular target identification and the result of the irregular target prediction, specifically including: Displaying the identified irregular target at the corresponding position of the digital twin model with a first attribute feature; displaying the predicted irregular target at the corresponding position of the digital twin model with a second attribute feature; wherein the attribute feature includes at least one or more of the following: icon shape, icon color, warning method; During non-inspection periods of the day, several free drones are controlled to conduct supplementary inspections over the garden, obtain the third video stream, and identify irregular targets in it; The detailed information of all the identified irregular targets and predicted irregular targets are filled into the preset template, the inspection report is generated, and sent to the terminal of the relevant person in charge to complete the daily inspection.

10. A garden inspection system with air-ground coordination, characterized in that: The system comprises: The inspection path generation module is used to build a digital twin model of the target garden based on the digital twin technology; drive the digital twin model to perform simulation operation simulation through the historical operation data of the target garden, and determine the inspection period and inspection path of the day according to the simulation results; The inspection control module is used to control the inspection robot to perform daily inspections along the inspection path of the day during the inspection period of the day; wherein each inspection robot is equipped with at least one inspection drone; real-time detection of the operating environment of the inspection robot, and control the inspection drone to perform high-altitude follow-up inspections according to the operating environment; A target recognition and prediction module is used to recognize irregular targets based on the first video stream sent back by the inspection robot and the second video stream sent back by the inspection drone, and to extract garden operation characteristic parameters from the video streams; the garden operation characteristic parameters are applied to the digital twin model to predict irregular targets; The report generation module is used to generate an inspection report according to the result of the irregular target identification and the result of the irregular target prediction.

Images