Deep learning-based botanical garden maintenance condition identification method and system

Abstract

The embodiment of the invention provides a plant garden maintenance condition identification method and system based on deep learning, and the method comprises the steps: obtaining a plurality of video streams in different directions of a to-be-detected region, and synthesizing each frame of image corresponding to each video stream into a four-dimensional matrix through a cat method of a pytorch frame; inputting the four-dimensional matrix into a pre-trained botanical garden detection model, wherein each frame of image obtains a corresponding number of target detection frames; sequentially caching video contents of first preset time for the videos in all directions, detecting whether a target event exists in a target detection frame of each frame of image in the current video of the first preset time, if so, marking the image of the target event as a target event effective frame of a corresponding category, judging whether a storage condition is met, and if so, executing the next step; if yes, storing corresponding and / or video information and the like in the current first time and uploading the information to the service terminal. According to the invention, efficient identification of grazing behaviors, human damage behaviors and watering maintenance behaviors can be realized, and process videos can be stored.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of deep learning target detection and image classification, and in particular to a method, system, electronic device, and computer-readable storage medium for identifying the maintenance status of a botanical garden based on deep learning. Background Art

[0002] As the world pays more and more attention to ecological and environmental protection, the protection of the ecosystem around hydropower stations has become one of the most important issues. The construction and operation of hydropower stations may have an impact on the surrounding vegetation and ecological environment. In order to maintain this fragile balance, it is imperative to develop a method that can monitor and protect the rare botanical gardens of hydropower stations. Grazing and human destruction pose severe challenges to vegetation and soil health. Therefore, it is particularly urgent to develop a systematic, automated, and intelligent method to monitor and identify grazing, human destruction, watering and other activities to provide protection for botanical gardens. With its breakthrough in image recognition, deep learning methods provide a disruptive means to replace manual automatic monitoring. Summary of the invention

[0003] The purpose of the embodiments of the present disclosure is to provide a method, system, electronic device and computer-readable storage medium for identifying the maintenance status of a botanical garden based on deep learning, so as to solve the aforementioned problems existing in the prior art.

[0004] In order to achieve the above objectives, the technical solutions adopted by the embodiments of the present disclosure are as follows:

[0005] On the one hand, the embodiment of the present disclosure proposes a method for identifying the maintenance status of a botanical garden based on deep learning, the method comprising: obtaining multiple monitoring video streams of different directions of the area to be detected, and synthesizing the frame images of each video stream at the same time into a four-dimensional matrix using the cat method of the pytorch framework;

[0006] The four-dimensional matrix is ​​input into a pre-trained botanical garden detection model, and each frame image obtains a corresponding number of target detection frames; wherein the botanical garden detection model is obtained based on YOLOv5 network training, and the targets include: animals, personnel and water splashes, the personnel include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering;

[0007] The video content of the first preset time is sequentially cached for each directional video, and whether the target detection frame of each frame image in the current first preset time has at least one type of target event, and if so, the image with the target event is simultaneously marked as a valid frame of the corresponding category of target event, wherein the target event includes: animals, non-maintenance personnel, and watering water flow;

[0008] Determine whether the ratio of the number of valid frames of the same target event detected within the current first preset time in each orientation video to the number of detected frames is greater than or equal to the first preset threshold. If so, there is a target event of the corresponding type within the current first preset time in the corresponding orientation video, save the corresponding orientation photos and / or videos, event type, event occurrence time and location within the current first preset time and upload them to the service terminal.

[0009] Optionally, the detecting whether a target event exists in a target detection frame of each frame image in the video of the current first preset time, and if so, marking images with at least one type of target event as valid frames of the corresponding target event at the same time, includes:

[0010] If an animal exists in each of the target detection frames, the image containing the animal is marked as a valid animal frame;

[0011] If there is a person in each of the target detection frames, the image containing the person is input into a pre-trained image classification model to obtain a classification result of a maintenance person or a non-maintenance person, and the image belonging to the non-maintenance person is marked as a valid frame of the non-maintenance person;

[0012] If there is water splash in each of the target detection frames, the height and width of the water splash detection frame are obtained, and the classification result of natural water flow or watering water flow is obtained according to the height and width of the water splash detection frame, and the image belonging to the watering water flow is marked as a valid frame of the watering water flow; wherein, the image classification model is obtained based on YOLOv5 network training.

[0013] Optionally, before saving the corresponding position photo and / or video, event type, event time and event location within the current first preset time and uploading to the service terminal, the method further includes:

[0014] Determine whether the time difference between the occurrence time of the target event within the current first preset time and the end time of the previous target event of the same type is greater than the reference time threshold. If so, the target event within the current first preset time is a new target event, and the relevant data of the new target event is saved and sent;

[0015] If not, record the end time of the target event within the current relevant preset time.

[0016] Optionally, the image classification model training process includes:

[0017] Extract human images from the COCO dataset and extract maintenance staff images from any botanical garden surveillance scene video or photo, mark the human images and maintenance staff images, mark maintenance staff as staff, and human images as non-staff; the basis for marking maintenance staff and non-staff is whether the clothing is maintenance clothing;

[0018] The labeled maintenance staff pictures and human body pictures are input into the YOLOv5 image classification network to be trained as training sets to obtain a trained image classification model.

[0019] Optionally, the training process of the botanical garden detection model includes:

[0020] Extract pictures from the COCO dataset that contain any one or more of the five categories of sheep, cows, horses, dogs, and people, and retain the label information of the corresponding category; among them, the category label of people is person, and the category labels of sheep, cows, horses, and dogs are all animal;

[0021] Collect water splash images as a water splash dataset and mark water as the water splash label;

[0022] The pictures with labels of sheep, cow, horse, dog, human and water splash are respectively input into the YOLOv5 network to be trained as training sets to obtain the trained botanical garden detection model.

[0023] Optionally, obtaining the classification results of natural water flow and watering water flow according to the height and width of the water splash detection frame includes:

[0024] Calculate the ratio of the width w and height h of the splash detection frame k=w / h;

[0025] It is determined whether the ratio k of the width w and height h of the water splash detection frame is less than a preset threshold k0. If so, the water splash is a natural water flow; if not, the water splash is a water flow for watering.

[0026] Another aspect of the present disclosure is to provide a method for identifying the maintenance status of a botanical garden based on deep learning, wherein the method comprises:

[0027] Obtain multiple surveillance video streams from different directions of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix;

[0028] The four-dimensional matrix is ​​input into a pre-trained botanical garden detection model, and each frame image obtains a corresponding number of target detection frames; wherein the botanical garden detection model is obtained based on YOLOv5 network training, and the targets include: animals, personnel and water splashes, the personnel include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering;

[0029] Perform real-time detection on the target detection frame of each frame image in each direction video to see if there is at least one type of target event. If so, mark the image with the target event as a valid frame of the corresponding target event type, wherein the target events include: animals, non-maintenance personnel, and watering water flow;

[0030] A preset number of valid frame time queues of the same target event are saved in chronological order for each orientation video, and it is determined whether the time difference between the first frame time and the last frame time of the valid frame time queue of the same target event is less than a second preset time. If so, there is a target event of the corresponding type in the current valid frame time queue of the corresponding orientation video, and the corresponding orientation photos and / or videos, event type, event occurrence time and location are saved for a certain length of time from the next moment after the last frame time of the current valid frame time queue, and uploaded to the service terminal.

[0031] Optionally, the step of performing real-time detection on the target detection frame of each frame image in each orientation video to determine whether at least one type of target event exists, and if so, marking the image with the target event as a valid frame of the corresponding target event type, includes:

[0032] If an animal exists in each of the target detection frames, the image containing the animal is marked as a valid animal frame;

[0033] If there is a person in each of the target detection frames, the image containing the person is input into a pre-trained image classification model to obtain a classification result of maintenance personnel or non-maintenance personnel, and the image belonging to the non-maintenance personnel is marked as a non-maintenance personnel valid frame; wherein the image classification model is trained based on the YOLOv5 network.

[0034] If there is water splash in each of the target detection frames, the height and width of the water splash detection frame are obtained, and the classification result of natural water flow or watering water flow is obtained according to the height and width of the water splash detection frame, and the image belonging to the watering water flow is marked as a valid frame of the watering water flow;

[0035] Optionally, before the method starts to save the corresponding orientation photos and / or videos, event types, event occurrence time and occurrence location for a certain period of time from the moment after the last frame time of the current valid frame time queue, and uploads them to the service terminal, the method further includes:

[0036] Determine whether the time difference between the occurrence time of the current target event and the end time of the previous target event of the same type is greater than the reference time threshold. If so, the current target event valid frame time queue is the new target event, and data is saved and sent from the next moment of the current valid frame time queue;

[0037] If not, record the time of the current target event valid frame time queue.

[0038] Optionally, the image classification model training process includes:

[0039] Extract human images from the COCO dataset and extract maintenance staff images from any botanical garden surveillance scene video or photo, mark the human images and maintenance staff images, mark maintenance staff as staff, and human images as non-staff; the basis for marking maintenance staff and non-staff is whether the clothing is maintenance clothing;

[0040] The labeled maintenance staff pictures and human body pictures are input into the YOLOv5 image classification network to be trained as training sets to obtain a trained image classification model.

[0041] Optionally, the training process of the botanical garden detection model includes:

[0042] Extract pictures from the COCO dataset that contain any one or more of the five categories of sheep, cows, horses, dogs, and people, and retain the label information of the corresponding category; among them, the category label of people is person, and the category labels of sheep, cows, horses, and dogs are all animal;

[0043] Collect water splash images as a water splash dataset and mark water as the water splash label;

[0044] The pictures with labels of sheep, cow, horse, dog, human and water splash are respectively input into the YOLOv5 network to be trained as training sets to obtain the trained botanical garden detection model.

[0045] Optionally, obtaining the classification results of natural water flow and watering water flow according to the height and width of the water splash detection frame includes:

[0046] Calculate the ratio of the width w and height h of the splash detection frame k=w / h;

[0047] It is determined whether the ratio k of the width w and height h of the water splash detection frame is less than a preset threshold k0. If so, the water splash is a natural water flow; if not, the water splash is a water flow for watering.

[0048] Another aspect of the present disclosure is to provide a system for identifying the maintenance status of a botanical garden based on deep learning, the system comprising:

[0049] The first acquisition module is used to acquire multiple surveillance video streams at different locations of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix;

[0050] A first detection module is used to input the four-dimensional matrix into a pre-trained botanical garden detection model, and obtain a corresponding number of target detection frames for each frame image; wherein the botanical garden detection model is obtained based on YOLOv5 network training, the targets include: animals, people and water splashes, the people include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering;

[0051] A first marking module is used to sequentially cache the video content of the first preset time for each directional video, detect whether there is at least one type of target event in the target detection frame of each frame image in the current first preset time, and if so, mark the image with the target event as a valid frame of the corresponding category of target event, wherein the target event includes: animals, non-maintenance personnel and watering water flow;

[0052] The first judgment module is used to judge whether the ratio of the number of valid frames of the same target event detected within the current first preset time in each orientation video to the number of detected frames is greater than or equal to a first preset threshold. If so, there is a target event of the corresponding type within the current first preset time in the corresponding orientation video, and the corresponding orientation photos and / or videos, event type, event occurrence time and location within the current first preset time are saved and uploaded to the service terminal.

[0053] Another aspect of the present disclosure is to provide a system for identifying the maintenance status of a botanical garden based on deep learning, the system comprising:

[0054] The second acquisition module is used to obtain multiple monitoring video streams at different locations of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix;

[0055] The second detection module is used to input the four-dimensional matrix into a pre-trained botanical garden detection model, and each frame image obtains a corresponding number of target detection frames; wherein the botanical garden detection model is obtained based on YOLOv5 network training, the targets include: animals, people and water splashes, the people include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering;

[0056] The second marking module is used to detect in real time whether there is at least one type of target event in the target detection frame of each frame image in each direction video. If there is, the image with the target event is marked as a valid frame of the corresponding target event type, wherein the target event includes: animals, non-maintenance personnel and watering water flow;

[0057] The second judgment module is used to save a preset number of valid frame time queues of the same target event in time sequence for each orientation video, and calculate whether the time difference between the first frame time and the last frame time of the valid frame time queue of the same target event is less than a second preset time. If so, there is a target event of the corresponding type in the current valid frame time queue of the corresponding orientation video, and the corresponding orientation photos and / or videos, event type, event occurrence time and location are saved for a certain length of time from the next moment after the last frame time of the current valid frame time queue, and uploaded to the service terminal.

[0058] Another aspect of the disclosed embodiment provides an electronic device, including: one or more processors;

[0059] The storage unit is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the method as described above.

[0060] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and the computer program can implement the method as described above when executed by a processor.

[0061] The beneficial effects of the embodiments of the present disclosure are:

[0062] The disclosed embodiment discloses a method for identifying the maintenance status of a botanical garden based on deep learning; based on AI recognition technology and information technology, efficient identification of grazing behavior, human destructive behavior, and watering and maintenance behavior is achieved, and process videos are saved at the same time; through the disclosed embodiment, the maintenance status of the botanical garden is monitored in real time, grazing and human destructive behavior in the botanical garden can be discovered and stopped in time, and the watering and maintenance of the botanical garden can be standardized, thereby protecting the environment of the botanical garden. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flowchart of a method for identifying the maintenance status of a botanical garden based on deep learning proposed in one aspect of an embodiment of the present disclosure;

[0064] Figure 2 2 is a schematic diagram of a structure of a botanical garden maintenance status recognition system based on deep learning proposed in another aspect of the embodiment of the present disclosure;

[0065] Figure 3 It is a flowchart of a method for identifying the maintenance status of a botanical garden based on deep learning proposed in another aspect of the embodiment of the present disclosure;

[0066] Figure 4 2 is a schematic diagram of a structure of a botanical garden maintenance status recognition system based on deep learning proposed in another aspect of the embodiment of the present disclosure;

[0067] Figure 5 It is a schematic flow chart of a method for identifying the maintenance status of a botanical garden in Embodiment 1. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiment of the present disclosure more clear, the embodiment of the present disclosure is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described here are only used to explain the embodiment of the present disclosure and are not used to limit the embodiment of the present disclosure.

[0069] In order to automatically identify grazing and human destructive behaviors, save corresponding videos as supervision evidence or issue alarm information in time when grazing and human destructive behaviors occur, the embodiments of the present disclosure propose a method and system for identifying the maintenance status of a botanical garden based on deep learning.

[0070] like Figure 1 As shown, the embodiment of the present disclosure provides a method S100 for identifying the maintenance status of a botanical garden based on deep learning, the method comprising:

[0071] Step S110, obtain multiple monitoring video streams at different locations of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix.

[0072] It should be noted that in order to fully monitor the botanical garden area, multiple monitoring devices will be set up around the botanical garden area to obtain a more comprehensive monitoring picture in the botanical garden area. Specifically, the number and location of the cameras are adjusted according to the actual situation such as the area of ​​the detection area and the cost that the owner can invest. The botanical garden maintenance status identification method of the embodiment of the present disclosure is applicable to any botanical garden, and the area monitored by the botanical garden monitoring equipment constitutes the area to be detected. The present disclosure uses the cat method of the pytorch framework to synthesize each frame image of each video stream at the same time into a four-dimensional matrix, and each picture is represented by a three-dimensional matrix. The three dimensions of the three-dimensional matrix are 3, the pixel value of the horizontal direction x, and the pixel value of the vertical direction y, where 3 represents the three RGB channels of the image. Stacking multiple pictures in the z-axis direction becomes a four-dimensional matrix, just like falling a stack of A4 paper, and the z-axis is perpendicular to the plane formed by the horizontal and vertical directions. One A4 paper represents one picture, and the stacked A4 paper pile represents a four-dimensional matrix. The Yolov5 deep learning network can directly detect the four-dimensional matrix and obtain the detection result of each picture. If the four-dimensional matrix method is not used to detect multiple pictures, then a loop detection method is required to detect each picture. According to experiments, the detection speed of the four-dimensional matrix detection method is more than 30% faster than the single picture loop detection method. Therefore, the four-dimensional matrix detection method is used to increase the detection efficiency. After obtaining the images of multiple cameras at the same time, these images are combined into a four-dimensional matrix, and then the four-dimensional matrix rectangle is used for reasoning. The output result contains the target detection frame of all camera images at that moment, and then these images with corresponding target detection frames are distributed to the corresponding camera monitoring screens. Multiple monitoring devices, that is, cameras are controlled by a computing device and a set of trigger conditions.

[0073] Step S120, input the four-dimensional matrix into the pre-trained botanical garden detection model, and obtain a corresponding number of target detection frames for each frame image; wherein the botanical garden detection model is obtained based on the YOLOv5 network training, and the targets include: animals, personnel and water splashes. The personnel include maintenance personnel and non-maintenance personnel, but maintenance personnel and non-maintenance personnel will be detected as personnel, the water splashes include natural water flow and water flow for watering, and both natural water flow and water flow for watering will be detected as water splashes, and the animals include sheep, cattle, horses and dogs, and sheep, cattle, horses and dogs will be detected as animals.

[0074] The disclosed embodiment synthesizes the corresponding frame images of each video stream into a four-dimensional matrix in real time and sequentially inputs it into a pre-trained botanical garden detection model, and each frame image obtains a corresponding number of target detection frames and detection labels of the corresponding targets.

[0075] Specifically, the botanical garden detection model is trained based on the YOLOv5s network and is used to detect people, animals, and water flowers.

[0076] As an example of a botanical garden detection model, the training process of the botanical garden detection model includes:

[0077] Step S121, extracting pictures of any one or more of the five categories of sheep, cows, horses, dogs and humans from the COCO dataset, and retaining the label information of the corresponding categories; wherein the category label of humans is person, and the category labels of sheep, cows, horses and dogs are all animal.

[0078] It should be noted that each picture includes at least one of the five categories: sheep, cows, horses, dogs and humans. All pictures extracted from the COCO dataset include pictures of the five categories: sheep, cows, horses, dogs and humans.

[0079] Step S122: collect water splash images as a water splash dataset, and mark water as a water splash label.

[0080] Here, water splash can also refer to water flow, including natural precipitation and artificial watering.

[0081] Step S123: respectively input the pictures with labels of sheep, cow, horse, dog, human and water splash as training sets to the YOLOv5 network to be trained to obtain a trained botanical garden detection model.

[0082] The botanical garden detection model described in the embodiment of the present disclosure can identify three types of targets: people as a type of detection target, the detection label is person; sheep, cattle, horses, and dogs as four types of detection targets, all of which are detected as animal; water splashes as a type of detection target, the detection label is water, and the YOLOv5 target detection network can detect and classify the above three types of targets. Specifically, when the botanical garden detection model is obtained based on YOLOv5s network training, pictures with sheep, cattle, horses, dogs, people, and water splashes are used as training sets to input into the YOLOv5s network to be trained.

[0083] Step S130, sequentially cache the video content of the first preset time for each directional video, sequentially extract each frame image in the video of the first preset time according to a preset frame extraction frequency, detect whether there is at least one type of target event in the target detection frame of each frame image in the current first preset time, if so, mark the image with at least one type of target event as a valid frame of the corresponding category target event, otherwise, do not mark it, wherein the target events include: animals, non-maintenance personnel and watering water flow.

[0084] In the disclosed embodiment, when the algorithm capability is insufficient and all detection cannot be achieved, each frame image in the video of the first preset time may be extracted in sequence according to the preset frame extraction frequency. For example, the first preset time is 20 seconds, there are 500 frames of pictures, and the preset frame extraction frequency is 5. About 100 frames may be selected for detection, and the 100 frames are detection frames, of which 60 frames detect splashes, and these 60 frames are splash valid frames. If multiple target detection frames in a frame image respectively detect at least two types of target events, such as animals and non-maintenance personnel target events, then the frame image will be simultaneously marked as a valid frame of the corresponding category target event, such as marked as an animal valid frame and a non-maintenance personnel valid frame. Similarly, if multiple target detection frames in a frame image respectively detect three types of target events, then the frame image will be simultaneously marked as three types of corresponding target event valid frames.

[0085] As an example of marking valid frames, the detecting whether a target detection frame of each frame image in the video of the current first preset time has at least one type of target event, and if so, marking the image having at least one type of target event as a valid frame of the corresponding category of target event at the same time, includes:

[0086] If an animal exists in each of the target detection frames, the image to which the animal belongs is marked as a valid animal frame;

[0087] If there is a person in each of the target detection frames, the image containing the person is input into a pre-trained image classification model to obtain a classification result of a maintenance person or a non-maintenance person, and the image belonging to the non-maintenance person is marked as a valid frame of the non-maintenance person;

[0088] If there is water splash in each of the target detection frames, the height and width of the water splash detection frame are obtained, and the classification result of natural water flow or watering water flow is obtained according to the height and width of the water splash detection frame, and the image belonging to the watering water flow is marked as a valid frame of the watering water flow; wherein, the image classification model is based on YOLOv5 network training.

[0089] In the implementation of the present disclosure, each target detection frame corresponds to a corresponding label, and the type of the detection target can be determined by the label. When the label shows that an animal exists in the detection target, it is judged as grazing behavior; when the label shows that a person exists in the detection target, a rectangular image containing the whole body of the human body and a very small amount of background is extracted, that is, the target detection frame, and the person is further classified by the person image classification model. When it is classified as a non-maintenance person, it is judged as human damage; when the label shows that there is a splash in the detection target, the natural water flow and the water flow are distinguished by the direction of the water flow trajectory. When the water flow is a watering flow, it is judged as watering maintenance. Grazing behavior and non-maintenance personnel are both artificial destructive behaviors, and watering behavior is maintenance behavior. If multiple target detection frames of a picture detect multiple types of target events at the same time, the picture is simultaneously marked as valid frames of the corresponding multiple types of target events. The above three target event situations, any one of which needs to be monitored in real time, and the following step S400 determines whether there is a target event. Water flow is water splash. When water is sprayed out from the water pipe in the botanical garden, it mostly appears in a scattered shape, so it is called water splash.

[0090] As an example of an image classification model training process, the image classification model training process includes:

[0091] Step S131, extract pictures containing human bodies from the COCO dataset and extract pictures of maintenance staff from any botanical garden monitoring scene video or photo, mark the pictures of human bodies and maintenance staff, mark the maintenance staff as staff, and mark the human bodies as non-maintenance staff; the basis for marking maintenance staff and non-maintenance staff is whether the clothing is maintenance clothing; maintenance staff wear maintenance clothing.

[0092] Step S132: input the marked maintenance personnel pictures and human body pictures as training sets into the YOLOv5 image classification network to be trained to obtain a trained image classification model.

[0093] In the disclosed embodiment, the image classification model is used to distinguish between maintenance personnel and non-maintenance personnel; randomly extract and intercept human cutouts in the COCO dataset or other datasets and pictures of maintenance personnel in work clothes at the botanical garden, classify and annotate the pictures of maintenance personnel and non-maintenance personnel, and store them separately, respectively construct data sets containing images of the two types of personnel, use them as training sets, and obtain an image classification model through training with the YOLOv5 network framework. The image classification model can classify the people in the picture by their clothing to determine whether they are maintenance personnel.

[0094] As an example of water splash detection, the classification results of natural water flow and watering water flow are obtained according to the height and width of the water splash detection frame, including:

[0095] Calculate the ratio of the width w and height h of the splash detection frame k=w / h;

[0096] It is determined whether the ratio k of the width w and height h of the water splash detection frame is less than a preset threshold k0. If so, the water splash is a natural water flow; if not, the water splash is a water flow for watering.

[0097] The disclosed embodiment obtains the width w and height h of the water flow area, calculates the aspect ratio k of the water flow, wherein the aspect ratio k=w / h, and distinguishes the water flow trajectory by judging the size of the aspect ratio k, and sets the aspect ratio threshold k0. When k<k0, it indicates that the water flow trajectory is a downward water flow trajectory, and when k≥k0, it indicates that the water flow trajectory is a parallel water flow trajectory. By specifying the downward water flow as a natural water flow and the parallel water flow as a watering water flow, natural precipitation and artificial watering can be distinguished.

[0098] Step S140, determine whether the ratio of the number of valid frames of the same target event detected within the current first preset time in each azimuth video to the number of detected frames is greater than or equal to the first preset threshold value. If so, there is a target event of the corresponding type in the current first preset time of the corresponding azimuth video, and save the corresponding azimuth photos and / or videos, event types and other data information within the current first preset time and upload them to the service terminal. Specifically, each target event in the corresponding azimuth photos and / or videos within the current first preset time can be saved separately and uploaded to the service terminal. Other data information includes: the time and place of occurrence of the event.

[0099] In the embodiment of the present disclosure, when any of the above three behaviors is successfully detected and the event recording mechanism is triggered, the first preset time photo and / or video is saved and other relevant data is uploaded to the cloud server. In the embodiment of the present disclosure, monitoring equipment is set in multiple positions of the botanical garden for real-time detection to obtain videos of various positions, and the videos of various positions are respectively saved after detection and analysis in steps S100 to S400. The so-called number of frames of valid frames detected in the first preset time period should be understood as a statistical analysis of all image frames of videos in various positions in the preset time period, and the statistical results are used to determine whether the target event has occurred in the first preset time period. The purpose of doing so is to increase the fault tolerance rate. If the result of each frame is used to determine whether the event occurs or not, errors may occur because the results of deep learning detection will be accidental. However, when the time scale is increased, the statistical results will eliminate the accidental and leave the inevitability. When the ratio of the number of frames of valid frames detected in the first preset time period to the total number of frames is greater than or equal to the first preset threshold, the video of the first preset time period is saved. All frames within the first preset time period will be pre-stored, that is, cached. If the ratio of the number of valid frames detected within the first preset time period to the total number of frames is greater than or equal to the preset threshold, these pre-stored frames will be merged into a video. Otherwise, these frames will be cleared and the next cycle will start. It should be noted that if multiple types of target event valid frames are detected within the first preset time, as mentioned above, one picture may be marked as corresponding multiple types of target event valid frames, and various types of target events are dynamically changing. Within the first preset time, the various types of target events detected may also change. Therefore, it is necessary to calculate the ratio of the number of valid frames of the same type of target events within the first preset time to the total number of frames detected within the first preset time. If, after detection, each frame image of the video at the current first preset time has a ratio of one target event valid frame to the number of detected frames greater than or equal to the first preset threshold, and another target event valid frame to the number of detected frames has a ratio less than the first preset threshold, then the corresponding type of target event exists in the corresponding orientation video within the current first preset time, and the latter target event does not exist; if, after detection, each frame image of the video at the current first preset time has two or three types of target event valid frames, and the ratio of each of the detected frames is greater than or equal to the first preset threshold, then the corresponding orientation video has two or three types of target events within the current first preset time. Then, multiple types of target events can be saved at the same time, or different types of target events can be saved separately.

[0100] As an example of a deduplication mechanism, the determining whether a ratio of the number of valid frames of the same target event detected within the current first preset time in each direction of the video to the number of detected frames is greater than or equal to a first preset threshold includes:

[0101] The ratio m=n1 / n0 of the number of frames n1 of valid frames of the same target event detected within the current first preset time t in each direction video and the number of detection frames n0 is determined to determine whether m is greater than or equal to the first preset threshold m0.

[0102] Exemplarily, the value range of t is 15s to 90s, and the value range of m0 is 0.45 to 0.55.

[0103] The event trigger mechanism described in the embodiment of the present disclosure includes: obtaining the number of frames in which the target event is detected within a period of time, and calculating the ratio of the number of frames in which the target event is detected within the period of time to the total number of frames detected. For the period of time t, the ratio of the number of frames in which the target is detected to the total number of frames is expressed as: m = n1 / n0, where n1 is the number of frames in which the target is detected within the period of time t, and n0 is the total number of frames detected within the period of time t. Set the event trigger threshold m0, if m ≥ m0, there is currently a target event; if m < m0, no target event currently occurs.

[0104] As an example of a deduplication mechanism, before saving the corresponding position photos and / or videos, event types, event times and event locations within the current first preset time and uploading them to the service terminal, the method further includes:

[0105] Calculate the time difference between the occurrence time of the target event within the current first preset time of the corresponding orientation video and the end time of the previous target event of the same type, and determine whether the time difference is greater than the reference time threshold. If so, the target event within the current first preset time is a new target event, and save the new target event related data and send it to the service terminal;

[0106] If not, then the end time of the target event within the current first preset time is recorded. The reference time threshold range is 40 minutes to 1.5 hours.

[0107] That is to say, after there is a target event of the corresponding type within the current first preset time of the corresponding position video, it is also necessary to calculate the time difference between the occurrence time of the target event within the current first preset time of the corresponding position video and the end time of the last target event of the same type, and determine whether the time difference is greater than the reference time threshold. If so, the target event within the current preset time is a new target event, and the corresponding position photo and / or video, event type, event occurrence time and location within the current first preset time are saved and uploaded to the service terminal;

[0108] If not, then the end time of the target event within the current first preset time is recorded.

[0109] In the disclosed embodiment, data uploading also includes an event deduplication mechanism: when the target event is detected for the first time, the data will be saved and uploaded. After the target event data is saved and uploaded, if the same type of target event is detected again within the reference time threshold t', the data will not be saved and uploaded repeatedly. Only this time will be recorded. Only when the time of the next detection of the target event is greater than the reference time threshold t' from the last time the same type of target event was recorded, will it be determined as a new target event. Assume t' = 1 hour. If there are multiple types of target events within the current first preset time, then when deduplicating, the last target event also needs to have multiple corresponding targets. The current target event and the last target event belong to the same type of target event, otherwise, they belong to different types of target events.

[0110] The disclosed embodiment obtains on-site image data of the area to be detected through monitoring equipment such as cameras or video cameras; performs target detection on the on-site image in the AI ​​device, and determines whether to continue personnel classification and water flow trajectory judgment. If no target (person and water flow) is detected, personnel classification and water flow trajectory judgment are not performed; through the event trigger mechanism and data deduplication mechanism, the event is recorded, saved and uploaded to the server. By analyzing the surveillance video of the botanical garden and its surroundings, a model can be established to identify and monitor grazing and human destruction events, protect the botanical garden ecology, liberate inefficient manual labor, and help coordinate the sustainable development of the hydropower station with ecological protection.

[0111] like Figure 2 As shown, another aspect of the embodiment of the present disclosure provides a system for identifying the maintenance status of a botanical garden based on deep learning, based on a server storing data and an AI device for target detection and image classification, the system includes:

[0112] The first acquisition module 110 is used to acquire multiple surveillance video streams at different locations of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix;

[0113] The first acquisition module 110 synthesizes the same frame images of the video streams acquired by monitoring devices such as cameras or video cameras in various directions into a four-dimensional matrix.

[0114] The first detection module 120 is used to input the four-dimensional matrix into a pre-trained botanical garden detection model, and each frame image obtains a corresponding number of target detection frames; wherein the botanical garden detection model is based on YOLOv5 network training, and the targets include: animals, people and water splashes, the people include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering; the first detection module 120 is used to detect animals, people and water splash targets in the image.

[0115] The first marking module 130 is used to sequentially cache the video content of the first preset time for each directional video, and detect whether there is at least one type of target event in the target detection frame of each frame image in the current first preset time. If so, the images with at least one type of target event are simultaneously marked as valid frames of the corresponding category of target events, wherein the target events include: animals, non-maintenance personnel, and watering water flows; the first marking module 130 is used to mark the valid frames of detected animals; perform maintenance personnel and non-maintenance personnel detection on the images of detected personnel, mark the images of non-maintenance personnel as valid frames, and perform natural water flow and watering water flow detection on the images of detected water flows; if no target (person and water flow) is detected, personnel classification and water flow trajectory judgment are not performed.

[0116] The first judgment module 140 is used to judge whether the ratio of the number of valid frames of the same target event detected in the current first preset time in each direction video to the number of detected frames is greater than or equal to the first preset threshold. If so, there is a target event of the corresponding type in the current first preset time of the corresponding direction video, and the corresponding direction photos and / or videos, event types, event occurrence time and location within the current first preset time are saved and uploaded to the service terminal. The first judgment module 140 is used to judge whether there is a target event based on the ratio of the valid frames marked in the video within the first preset time to the total number of detected frames. If so, the target events can be saved separately or together according to the actual situation.

[0117] The system further includes a first deduplication module, which is used to determine whether the time difference between the occurrence time of the target event within the current first preset time and the end time of the previous target event of the same type is greater than a reference time threshold, and if so, the target event within the current first preset time is a new target event, and the relevant data of the new target event is saved and sent;

[0118] If not, then the end time of the target event within the current first preset time is recorded.

[0119] The first deduplication module of the disclosed embodiment can remove redundant target events, facilitate viewing and timely discovery of destructive behaviors, and take corresponding emergency measures.

[0120] like Figure 3 As shown, another aspect of the present disclosure provides a method S200 for identifying the maintenance status of a botanical garden based on deep learning, the method comprising:

[0121] Step S210: obtain multiple monitoring video streams at different locations of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix.

[0122] On the other hand, step S210 of the method provided in the embodiment of the present disclosure adopts the same method as step S110 of the above embodiment to obtain multiple monitoring video streams in different directions of the area to be detected and synthesize the frame images of each video stream at the same time into a four-dimensional matrix.

[0123] Step S220, input the four-dimensional matrix into a pre-trained botanical garden detection model, and obtain a corresponding number of target detection frames for each frame image; wherein the botanical garden detection model is obtained based on YOLOv5 network training, and the targets include: animals, personnel, and water splashes. The personnel include maintenance personnel and non-maintenance personnel, but maintenance personnel and non-maintenance personnel will be detected as personnel, the water splashes include natural water flow and water flow for watering, and both natural water flow and water flow for watering will be detected as water splashes, and the animals include sheep, cattle, horses, and dogs, and sheep, cattle, horses, and dogs will be detected as animals.

[0124] On the other hand, the method provided in step S220 of the embodiment of the present disclosure adopts the same method as step S120 of the above embodiment to detect the target of each frame image in real time and train the botanical garden detection model.

[0125] Step S230, performing real-time detection on the target detection frame of each frame image in each orientation video to determine whether there is at least one type of target event. If so, the images with at least one type of target event are simultaneously marked as valid frames of the corresponding target event type, wherein the target events include: animals, non-maintenance personnel, and watering water flows.

[0126] On the other hand, the method provided in the embodiment of the present disclosure requires real-time detection of whether there is a target event in the target detection frame of each frame image. Similarly, the preset frame sampling frequency same as step 130 of the above embodiment can be used to detect images in videos of all directions to save computing power.

[0127] As an example of marking valid frames, the method for marking valid frames is the same as the marking method of step 130 in the above embodiment, wherein the target detection frame of each frame image in each orientation video is detected in real time to determine whether at least one type of target event exists, and if so, the image with at least one type of target event is marked as a valid frame of the corresponding target event type, including:

[0128] If an animal exists in each of the target detection frames, the image containing the animal is marked as a valid animal frame;

[0129] If there is a person in each of the target detection frames, the image containing the person is input into a pre-trained image classification model to obtain a classification result of a maintenance person or a non-maintenance person, and the image belonging to the non-maintenance person is marked as a valid frame of the non-maintenance person;

[0130] If there is water splash in each of the target detection frames, the height and width of the water splash detection frame are obtained, and the classification result of natural water flow or watering water flow is obtained according to the height and width of the water splash detection frame, and the image belonging to the watering water flow is marked as a valid frame of the watering water flow; wherein, the image classification model is obtained based on YOLOv5 network training.

[0131] In another aspect of the method provided by the present disclosure, step S230 is the same as step 130 of the above embodiment. If multiple target detection frames in a frame of image detect three types of target events respectively, then the frame of image will be marked as a valid frame of three corresponding target events at the same time. The training process of the image classification model is the same as step 130 of the above embodiment.

[0132] Step S240, save a preset number of valid frame time queues of the same target event for each orientation video in chronological order, and determine whether the time difference between the first frame time and the last frame time of the valid frame time queue of the same target event is less than a second preset time. If so, there is a target event of the corresponding type in the current valid frame time queue of the corresponding orientation video, and save the corresponding orientation photos and / or videos, event type, event occurrence time and location for a certain length of time from the next moment after the last frame time of the current valid frame time queue, and upload them to the service terminal.

[0133] On the other hand, the method provided by the embodiment of the present disclosure is that when any of the above three behaviors is successfully detected and the event recording mechanism is triggered, the scene is photographed and / or recorded and other relevant data is uploaded to the cloud server. On the other hand, the method provided by the embodiment of the present disclosure is to detect the target detection frame of each frame image in each direction video in real time, and pre-store a certain number of valid frame times of the same type of target event, and the valid frame image is not saved. Similarly, there may be multiple types of target events in one frame image. As long as the time difference between the first frame time and the last frame time of any type of target event valid frame time queue is less than the second preset time, the current detection has the target event. If the time difference between the first frame time and the last frame time of each valid frame time queue of multiple types of target events is less than the second preset time, the current detection has the corresponding multiple target events, then the next moment after the last frame time of the current target event valid frame time queue starts to save the corresponding orientation photos and / or videos, event types, event occurrence time and location for a certain length of time, and upload them to the service terminal. The second preset time, the certain duration and the preset number can be set according to actual conditions. The second preset time range can be 3 seconds to 7 seconds.

[0134] On the other hand, the method provided in the embodiment of the present disclosure starts to save the corresponding orientation photos and / or videos and related data for a certain period of time. During this period, the target event detection can be stopped, and the target event is detected again after the saving time ends. The target event can also be saved separately while being detected. In this case, a deduplication mechanism is required.

[0135] As an example of a deduplication mechanism, the method further includes: saving corresponding orientation photos and / or videos, event types, event occurrence time and occurrence location for a certain period of time starting from the next moment after the last frame time of the current valid frame time queue, and uploading to the service terminal;

[0136] Determine whether the time difference between the occurrence time of the current target event and the end time of the previous target event of the same type is greater than the reference time threshold. If so, the current target event valid frame time queue is the new target event, and data is saved and sent from the next moment of the current valid frame time queue;

[0137] If not, record the time of the current target event valid frame time queue.

[0138] On the other hand, the deduplication mechanism in the method provided in the embodiment of the present disclosure is the same as the reference time threshold of the above-mentioned embodiment, which will not be repeated here.

[0139] like Figure 4 As shown, another aspect of the embodiment of the present disclosure provides a system for identifying the maintenance status of a botanical garden based on deep learning, based on a server storing data and an AI device for target detection and image classification, the system includes:

[0140] The second acquisition module 210 is used to acquire multiple surveillance video streams at different locations of the area to be detected, and synthesize the frame images of each video stream at the same time into a four-dimensional matrix using the cat method of the pytorch framework;

[0141] The second detection module 220 is used to input the four-dimensional matrix into a pre-trained botanical garden detection model, and obtain a corresponding number of target detection frames for each frame image; wherein the botanical garden detection model is obtained based on YOLOv5 network training, and the targets include: animals, personnel and water splashes, the personnel include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering;

[0142] The second marking module 230 is used to perform real-time detection on the target detection frame of each frame image in each direction video to determine whether there is at least one type of target event. If so, the images with at least one type of target event are marked as valid frames of the corresponding target event type, wherein the target events include: animals, non-maintenance personnel, and watering water flow;

[0143] The second judgment module 240 is used to save a preset number of valid frame time queues of the same target event in time sequence for each orientation video, and calculate whether the time difference between the first frame time and the last frame time of the valid frame time queue of the same target event is less than a second preset time. If so, there is a target event of the corresponding type in the current valid frame time queue of the corresponding orientation video, and the corresponding orientation photos and / or videos, event types, event occurrence time and location are saved for a certain period of time from the next moment after the last frame time of the current valid frame time queue, and uploaded to the service terminal.

[0144] The system further includes a second deduplication module, which is used to determine whether the time difference between the occurrence time of the current target event and the end time of the previous target event of the same type is greater than a reference time threshold, and if so, the current target event valid frame time queue is a new target event, and data is saved and sent from the next moment of the current valid frame time queue;

[0145] If not, record the time of the current target event valid frame time queue.

[0146] The disclosed embodiment uses a target detection network to monitor three scenarios in a botanical garden, including grazing behavior, human destruction activities, and watering identification, and can save and upload monitoring data. The method uses a target detection network to detect grazing activities and identify several specified animals, people, and water splashes. In order to effectively identify grazing activities and human destruction activities around the reservoir, the method trains a detection model that can detect specified animals and people and a classification model that classifies different personnel; at the same time, in order to effectively distinguish between rainfall and watering, the water splash trajectory is judged, and when the water splash has an upward trajectory, it is judged that watering exists; in order to enhance the robustness of the algorithm, the disclosure calculates the number of frames in which target events, such as animals, people, or water splashes, are detected within a period of time; then calculates the time difference between the first frame time and the last frame time of a certain number of effective frame time queues in which targets are detected, and when the time difference is less than the threshold, it is judged that there is a target event at present; when it is successfully judged that there is a target in the current scene, the target event label is read to judge the target event type; and finally, the next moment of the tail time of the queue starts to save the current photos, videos, event types and other data information and upload them to the corresponding server. The beneficial effects of the disclosed embodiments are as follows: the maintenance status of a botanical garden can be robustly monitored and maintenance data can be saved.

[0147] Another aspect of the disclosed embodiment provides an electronic device, including: one or more processors;

[0148] The storage unit is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors can implement the method as described above.

[0149] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program can implement the method according to any one of claims 1 to 7 when executed by a processor.

[0150] Embodiment 1 of the present disclosure:

[0151] This embodiment 1 realizes the recognition function of grazing and human destruction behavior in botanical gardens around hydropower stations based on some hydropower stations in the Yalong River Basin. The recognition algorithm is deployed on the local server, and the corresponding image frames are obtained through the video stream addresses of various cameras on site.

[0152] like Figure 5 As shown, it mainly includes the following steps:

[0153] Step S1: Extract the images of five categories including sheep, cattle, horses, dogs, and humans from the COCO dataset and only retain the label information of these five objects. Assign the label of 'person' to humans as a target, and assign the label of 'animal' to sheep, cattle, horses, and dogs as another target. Collect photos of water splashes from the Internet or real scenes as a water splash dataset, and construct a target detection dataset containing three categories. Extract the cutouts of human images in the COCO dataset and the cutouts of human images taken by the on-site maintenance staff of the botanical garden, and divide the maintenance staff images and non-maintenance staff images into two categories. The classification is based on whether the clothes of the people in the pictures are designated by the maintenance staff. Construct the classified pictures into an image classification dataset.

[0154] Step S2: Use the YOLOv5 framework and the data enhancement method provided by it to obtain a new dataset for target detection and image classification. Select the YOLOv5s network, input the target detection dataset into two YOLOv5s target detection networks, one YOLOv5s network is used to train the botanical garden detection model, and one YOLOv5s network is used to train the image classification model; specifically, input the human, animal, and water splash target detection datasets into a YOLOv5s target detection network; the label of the human dataset is set to 'person', the label of the animal dataset is set to 'animal', the label of the water splash dataset is set to 'water', the number of categories (class) of each network is set to 3, the learning rate is 0.01, the optimizer uses SGD, and the training epoch is set to 200. After training, the weight file and the botanical garden detection model will be obtained. yolov5s is a target detection network belonging to the yolov5 framework. Yolov5 also includes different networks such as yolov5m and yolov5l. The difference between them lies in the difference in the number of network parameters. Yolov5s has fewer parameters but faster detection speed.

[0155] Step S3: Input the image classification dataset into another YOLOv5 image classification network, set the maintenance staff label to 'staff', set the non-maintenance staff label to 'non-staff', set the number of classification network output categories to 2, set the learning rate to 0.01, use Adam in the optimization area, and set the training epoch to 200. After training, a personnel classification weight file and image classification model are obtained.

[0156] Step S4: Use the opencv tool to read the video streams of different cameras, and use the cat method provided by the pytorch framework to synthesize the same frame read from different cameras into a four-dimensional matrix.

[0157] Step S5: Input this four-dimensional matrix into the botanical garden detection model respectively, and obtain the detection coordinate frame results of people, animals, and water splashes respectively; if there are people, input the person coordinate frame cutout into the image classification model to obtain the person classification result; if there are water splashes, obtain the height and width of the water splashes, calculate the aspect ratio, judge the water flow direction by the aspect ratio, and distinguish the type of water splashes; mark the frames in which animals, non-maintenance personnel, or water splashes are detected as 'valid frames'.

[0158] Step S6: Set a cycle time of 20s, and the frame rate of the video stream is 25fps. Then the total number of image frames of the 20s video stream is 20×25=500. The 500 frames of video are stored in the queue. The queue is first-in-first-out, last-in-last-out, and updated in real time. One frame is updated every 1 / 30 second. If the algorithm is not capable enough, all detections cannot be achieved. Set the frame extraction frequency to 5, and the actual number of image frames to be detected is 100=500÷5. Set the frame number threshold δ=50, and the preset threshold is 0.5. When a frame of a single camera is marked as a valid frame, let m increase by 1. Assuming that the number of valid frames corresponding to the event detected in 100 frames in the queue of 500 frames is 60, it can be seen that m is greater than δ, that is, the ratio of the number of valid frames to the number of detected frames is 0.6, and 0.6 is greater than 0.5, so it is determined that the corresponding event occurred in the area covered by the camera within the first 20s.

[0159] Step S7: Use the ImageDraw method in the Pi l low package to draw a frame for the target coordinates in the corresponding event, and use the VideoWriter function in opencv to synthesize all frames within the first 20 seconds of the camera where the target event occurs into a video and save it. Then perform the above steps S6 and S7 on the updated new queue, and repeat this cycle.

[0160] Step S8: To prevent repeated reporting of the same type of events, it is necessary to calculate the time difference between the current target event and the last recorded target event of the same type, and determine whether the time difference is greater than the preset threshold. When the time difference is greater than the preset threshold, save the current camera photos, videos and related data and upload them to the corresponding server.

[0161] Embodiment 2 of the present disclosure:

[0162] The second embodiment of the present invention is based on the rare plant garden near some hydropower stations in the Lancang River Basin, and realizes the maintenance monitoring function in the botanical garden. The main AI computing device is the Rockchip RK3588 AI box, and the AI ​​box and the botanical garden camera are in the same local area network. For compatibility, after using Pytorch to train the splash detection model, the Pytorch model is converted into an RKNN model using Rockchip's conversion code and deployed on the AI ​​box.

[0163] The main steps are as follows:

[0164] Step S1: Extract the images of five categories including sheep, cattle, horses, dogs, and humans from the COCO dataset and only retain the label information of these five objects. Assign the label of 'person' to humans as a target, and assign the label of 'animal' to sheep, cattle, horses, and dogs as another target. Collect photos of water splashes from the Internet or real scenes as a water splash dataset, and construct a target detection dataset containing three categories. Extract the cutouts of human images in the COCO dataset and the cutouts of human images taken by on-site maintenance personnel, and divide the pictures of maintenance personnel and non-maintenance personnel into two categories. The classification is based on whether the clothes of the people in the pictures are the designated clothes for maintenance personnel. Construct the classified pictures into an image classification dataset.

[0165] Step S2: Use the YOLOv5 framework and the data enhancement method provided by it to obtain a new dataset for target detection and image classification. Select the YOLOv5s network and input the target detection dataset into two YOLOv5s target detection networks. One YOLOv5s network is used to train the botanical garden detection model, and the other YOLOv5s network is used to train the image classification model. Specifically, input the human, animal, and water splash target detection datasets into a YOLOv5s target detection network. The label of the human dataset is set to 'person', the label of the animal dataset is set to 'animal', and the label of the water splash dataset is set to 'water'. The number of classes of each network is set to 3, the learning rate is 0.01, the optimizer uses SGD, and the training epoch is set to 200. After training, the weight file and the botanical garden detection model will be obtained.

[0166] Step S3: Input the image classification dataset into another YOLOv5 image classification network, set the maintenance staff label to 'staff', set the non-maintenance staff label to 'non-staff', set the number of classification network output categories to 2, set the learning rate to 0.01, use Adam in the optimization area, and set the training epoch to 200. After training, a personnel classification weight file and image classification model are obtained.

[0167] Step S4: Convert the Pytorch model to the RKNN model supported by Rockchip AI Box through Rockchip's RKNN deployment solution, and deploy the RKNN model and the corresponding inference code on the AI ​​Box.

[0168] Step S5: Obtain the video frame of the botanical garden scene camera and perform a cropping operation. The cropped frame is input into the deployed RKNN model code to obtain the detection coordinate frame results of people, animals, and water splashes respectively; if there are people, the person coordinate frame cutout is input into the image classification model to obtain the person classification result; if there are water splashes, the height and width of the water splashes are obtained, the aspect ratio is calculated, and the direction of the water flow is determined by the aspect ratio to distinguish between natural water flow and watering water flow; the time of the frame in which animals, non-maintenance personnel, or watering water splashes are detected is saved in the corresponding target event detection queue, and the queue length is set to 10 frames of valid frame time. The queue is first-in-first-out and last-in-last-out. Here, the queue only saves the time of valid frames.

[0169] Step S6: Determine whether a target event occurs according to the time in each queue. Specifically, first set the time threshold t to 4s, the video stream is 25 frames per second, then there are 100 frames of video in 4s, and the frame extraction strategy is 1 frame for 5 frames for detection, then a total of 20 frames of video are detected in 4s, and the time difference from the first row to the last row of the queue is calculated. If the time difference is less than 4s, it means that the time of 10 valid frames is within 4s, that is, the ratio of the number of valid frames to the number of detected frames is greater than 0.5, it can be inferred that there is a real target event at this time, and start recording several seconds of video.

[0170] Step S7: To prevent repeated reporting of the same event, it is necessary to determine the time difference between the current target event and the last recorded target event of the same type. When the time difference is greater than a preset threshold, the current camera photos, videos and related data are saved and uploaded to the corresponding server.

[0171] The above is only a preferred implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be considered as the protection scope of the embodiment of the present disclosure.

Claims

1. A method for identifying the maintenance status of a botanical garden based on deep learning, characterized in that: The method comprises: Obtain multiple surveillance video streams from different directions of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix; The four-dimensional matrix is ​​input into a pre-trained botanical garden detection model, and each frame image obtains a corresponding number of target detection frames; wherein the botanical garden detection model is obtained based on YOLOv5 network training, and the targets include: animals, personnel and water splashes, the personnel include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering; The video content of the first preset time is sequentially cached for each directional video, and whether the target detection frame of each frame image in the current first preset time has at least one type of target event, and if so, the image with the target event is simultaneously marked as a valid frame of the corresponding category of target event, wherein the target event includes: animals, non-maintenance personnel, and watering water flow; Determine whether the ratio of the number of valid frames of the same target event detected within the current first preset time in each orientation video to the number of detected frames is greater than or equal to the first preset threshold. If so, there is a target event of the corresponding type within the current first preset time in the corresponding orientation video, save the corresponding orientation photos and / or videos, event type, event occurrence time and location within the current first preset time and upload them to the service terminal.

2. The method according to claim 1, characterized in that The detecting whether there is at least one type of target event in the target detection frame of each frame image in the video of the current first preset time, and if there is, marking the image with the target event as a valid frame of the corresponding target event at the same time, includes: If an animal exists in each of the target detection frames, the image containing the animal is marked as a valid animal frame; If there is a person in each of the target detection frames, the image containing the person is input into a pre-trained image classification model to obtain a classification result of a maintenance person or a non-maintenance person, and the image belonging to the non-maintenance person is marked as a valid frame of the non-maintenance person; If there is water splash in each of the target detection frames, the height and width of the water splash detection frame are obtained, and the classification result of natural water flow or watering water flow is obtained according to the height and width of the water splash detection frame, and the image belonging to the watering water flow is marked as a valid frame of the watering water flow; wherein, the image classification model is obtained based on YOLOv5 network training.

3. The method according to claim 1 or 2, characterized in that: Before saving the corresponding position photo and / or video, event type, event time and event location within the current first preset time and uploading to the service terminal, the method further includes: Determine whether the time difference between the occurrence time of the target event within the current first preset time and the end time of the previous target event of the same type is greater than the reference time threshold. If so, the target event within the current first preset time is a new target event, and the relevant data of the new target event is saved and sent; If not, then the end time of the target event within the current first preset time is recorded.

4. A method for identifying the maintenance status of a botanical garden based on deep learning, characterized in that: The method comprises: Obtain multiple surveillance video streams from different directions of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix; The four-dimensional matrix is ​​input into a pre-trained botanical garden detection model, and each frame image obtains a corresponding number of target detection frames; wherein the botanical garden detection model is obtained based on YOLOv5 network training, and the targets include: animals, personnel and water splashes, the personnel include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering; Perform real-time detection on the target detection frame of each frame image in each direction video to see if there is at least one type of target event. If so, mark the image with the target event as a valid frame of the corresponding target event type, wherein the target events include: animals, non-maintenance personnel, and watering water flow; A preset number of valid frame time queues of the same target event are saved in chronological order for each orientation video, and it is determined whether the time difference between the first frame time and the last frame time of the valid frame time queue of the same target event is less than a second preset time. If so, there is a target event of the corresponding type in the current valid frame time queue of the corresponding orientation video, and the corresponding orientation photos and / or videos, event type, event occurrence time and location are saved for a certain length of time from the next moment after the last frame time of the current valid frame time queue, and uploaded to the service terminal.

5. The method according to claim 4, characterized in that The method of detecting in real time whether there is at least one type of target event in the target detection frame of each frame image in each orientation video, and if so, marking the image with the target event as a valid frame of the corresponding target event type, includes: If an animal exists in each of the target detection frames, the image containing the animal is marked as a valid animal frame; If there is a person in each of the target detection frames, the image containing the person is input into a pre-trained image classification model to obtain a classification result of a maintenance person or a non-maintenance person, and the image belonging to the non-maintenance person is marked as a valid frame of the non-maintenance person; If there is water splash in each of the target detection frames, the height and width of the water splash detection frame are obtained, and the classification result of natural water flow or watering water flow is obtained according to the height and width of the water splash detection frame, and the image belonging to the watering water flow is marked as a valid frame of the watering water flow; wherein, the image classification model is obtained based on YOLOv5 network training.

6. The method according to claim 4 or 5, characterized in that: Before the method starts to save the corresponding orientation photos and / or videos, event types, event occurrence time and occurrence location for a certain period of time from the moment after the last frame time of the current valid frame time queue, and uploads them to the service terminal, the method also includes: Determine whether the time difference between the occurrence time of the current target event and the end time of the previous target event of the same type is greater than the reference time threshold. If so, the current target event valid frame time queue is the new target event, and data is saved and sent from the next moment of the current valid frame time queue; If not, record the time of the current target event valid frame time queue.

7. A deep learning-based identification system for botanical garden maintenance status, characterized in that: The system comprises: The first acquisition module is used to acquire multiple surveillance video streams at different locations of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix; A first detection module is used to input the four-dimensional matrix into a pre-trained botanical garden detection model, and obtain a corresponding number of target detection frames for each frame image; wherein the botanical garden detection model is obtained based on YOLOv5 network training, the targets include: animals, people and water splashes, the people include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering; A first marking module is used to sequentially cache the video content of the first preset time for each directional video, detect whether there is at least one type of target event in the target detection frame of each frame image in the current first preset time, and if so, mark the image with the target event as a valid frame of the corresponding category of target event, wherein the target event includes: animals, non-maintenance personnel and watering water flow; The first judgment module is used to judge whether the ratio of the number of valid frames of the same target event detected within the current first preset time in each orientation video to the number of detected frames is greater than or equal to a first preset threshold. If so, there is a target event of the corresponding type within the current first preset time in the corresponding orientation video, and the corresponding orientation photos and / or videos, event type, event occurrence time and location within the current first preset time are saved and uploaded to the service terminal.

8. A deep learning-based identification system for botanical garden maintenance status, characterized in that: The system comprises: The second acquisition module is used to obtain multiple monitoring video streams at different locations of the area to be detected, and use the cat method of the pytorch framework to synthesize the frame images of each video stream at the same time into a four-dimensional matrix; The second detection module is used to input the four-dimensional matrix into a pre-trained botanical garden detection model, and each frame image obtains a corresponding number of target detection frames; wherein the botanical garden detection model is obtained based on YOLOv5 network training, the targets include: animals, people and water splashes, the people include maintenance personnel and non-maintenance personnel, and the water splashes include natural water flows and water flows for watering; The second marking module is used to detect in real time whether there is at least one type of target event in the target detection frame of each frame image in each direction video. If there is, the image with the target event is marked as a valid frame of the corresponding target event type, wherein the target event includes: animals, non-maintenance personnel and watering water flow; The second judgment module is used to save a preset number of valid frame time queues of the same target event in time sequence for each orientation video, and calculate whether the time difference between the first frame time and the last frame time of the valid frame time queue of the same target event is less than a second preset time. If so, there is a target event of the corresponding type in the current valid frame time queue of the corresponding orientation video, and the corresponding orientation photos and / or videos, event type, event occurrence time and location are saved for a certain length of time from the next moment after the last frame time of the current valid frame time queue, and uploaded to the service terminal.

9. An electronic device, characterized in that: include: one or more processors; A storage unit, used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 can be implemented.

Images