Iron tower bird nest monitoring method and system based on multimodal large model
By combining traditional image recognition technology and multimodal big model, using the filter prompt word template library to detect the tower bird nest, the false alarm problem of traditional visual model in complex environments is solved, and high-precision bird nest recognition and classification is achieved , reduces the workload of manual review and improves the degree of automation of the system.
Patent Information
- Application Number
- CN202411170410.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-26
AI Technical Summary
In the detection of tower bird nests, traditional visual models have serious false alarms due to environmental complexity and target diversity. The existing false alarm processing methods have a long cycle and limited effect, which is difficult to meet high-precision requirements, which increases the workload of manual review and system. Not real-time.
Combining traditional image recognition technology and multimodal big model, by initially identifying suspected bird nests and inputting them into fine-tuned multimodal big model, the filter prompt word template library is used for further review, and combining visual information and semantic understanding, it can achieve Accurate identification and classification of the bird's nest of the tower.
It significantly reduces the false alarm rate, improves the accuracy of event detection, reduces the workload of manual audits, enhances the generalization ability and automation of the system, and provides a more reliable and intelligent monitoring solution.
Smart Images

Figure CN118692028B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of iron tower safety monitoring, and in particular to an iron tower bird nest monitoring method and system based on a multi-modal large model. Background Art
[0002] In the bird nest detection scenario on towers (including power towers and communication towers), traditional visual models often find it difficult to accurately identify the location of bird nests due to the complex background and diverse environmental conditions around the towers, and they are prone to misidentifying other non-bird nest objects (such as branches, plastic bags, etc.) as bird nests.
[0003] With the continuous development of artificial intelligence technology, computer vision models such as the YOLO series and ResNet have been widely used in various monitoring and detection tasks, including traffic monitoring, security monitoring, environmental monitoring, and industrial detection. These models have made significant progress in automated detection, especially in identifying specific targets (such as bird nests on towers). However, in practical applications, the performance of visual models is still limited by factors such as environmental complexity and target diversity, resulting in a large number of false alarms. These false alarms not only increase the workload of manual review, but also affect the overall efficiency and accuracy of the system to a certain extent.
[0004] At present, the commonly used false alarm processing methods in the industry include updating the training data set by adding negative sample data in order to improve the generalization ability of the model. However, this method has problems such as long cycle and limited effect, and it is difficult to meet high-precision requirements. In addition, some systems try to combine large visual models (such as ViT large visual models) and the collection and update of false alarm data to improve detection accuracy, but the false alarm rate is still high, and still needs to rely on manual review to finally confirm the accuracy of the event. This method not only increases labor costs, but also limits the real-time and automation capabilities of the system. Summary of the invention
[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention aims to provide a tower bird nest monitoring method that combines traditional image recognition technology and emerging multimodal large model technology, and can combine visual information and semantic understanding, so as to achieve accurate detection and classification of tower bird nest events, improve the accuracy of event reporting, reduce false alarms, and significantly reduce the workload of manual review, thereby optimizing the overall performance of the system and providing related fields with more reliable and intelligent monitoring solutions.
[0006] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention specifically includes:
[0007] 1. A tower bird nest monitoring method based on a multi-modal large model, comprising the following steps:
[0008] Obtain monitoring video of the target tower, use the target detection algorithm to preliminarily identify suspected bird nests in the monitoring video, and report the suspected bird nest objects; the suspected bird nest objects include monitoring images, suspected bird nest locations, image timestamps, labels, and confidence levels;
[0009] The suspected bird's nest object is input into the fine-tuned multimodal large model, and the tower bird's nest recognition result is output according to the filtering prompt word template library; the filtering prompt word template library includes the event type and the standard event prompt word list constituting the key-value pair;
[0010] If the identification result includes a definite conclusion that a bird's nest exists on the tower, the suspected bird's nest object will be reported as an event object.
[0011] Preferably, the preliminary identification method of the suspected bird nest in the monitoring video includes:
[0012] Extracting monitoring images from the monitoring video frame by frame, extracting sub-images containing suspected bird nests from the monitoring images using regions of interest, and performing target recognition on the sub-images using a trained bird nest target detection model;
[0013] Obtain the position of the identified suspected bird nest in the sub-image, and restore the position to the suspected bird nest position obtained in the monitoring image;
[0014] Adding a bird nest tag to the suspected bird nest, and filtering the tag attribute to obtain a list of suspected bird nest objects;
[0015] Using a tracking algorithm to track the suspected bird nest object list, obtaining a suspected bird nest tracking object list; the suspected bird nest tracking object list includes the most reasonable suspected bird nest object, the current suspected bird nest object, the tracking state and the image timestamp;
[0016] When the tracking state changes to "disappeared", the most reasonable suspected bird nest object at present is taken as the suspected bird nest object.
[0017] Preferably, the sub-image extraction method includes:
[0018] Extracting an image of interest from the monitoring image based on the minimum circumscribed rectangle of the region of interest;
[0019] Taking the aspect ratio of the smallest circumscribed rectangle and the monitored image with the largest aspect ratio as the expansion ratio, taking the center of the smallest circumscribed rectangle as the expansion center, and expanding the image extraction frame outward according to the expansion ratio;
[0020] If the image extraction frame extends beyond the monitoring image boundary, the expansion center is translated in the opposite direction of the exceeding direction to ensure that the image extraction frame does not exceed the monitoring image boundary;
[0021] When any one of the width and height of the image extraction frame is equal to the width and height of the monitoring image, the monitoring image of the current image extraction frame is used as a sub-image.
[0022] Preferably, the method for updating the most reasonable suspected bird nest object in the suspected bird nest tracking object list includes:
[0023] When the identification frame size and the confidence of the current suspected bird's nest object are both greater than the suspected bird's nest position and the confidence of the current most reasonable suspected bird's nest object, the current suspected bird's nest object is used to update the current most reasonable suspected bird's nest object.
[0024] Preferably, the fine-tuning method of the multimodal large model includes:
[0025] Construct a correct event dataset including images of labeled bird nests and corresponding image description texts, and a false alarm event dataset including images of objects labeled as falsely reported as bird nests and description texts of the causes of the false alarms, and use them together as pre-training datasets to fine-tune the multimodal large model, thereby enhancing the multimodal large model's image and semantic understanding of the tower bird nests.
[0026] The present invention also provides a tower bird nest monitoring system based on a multi-modal large model, comprising:
[0027] A video acquisition module is used to acquire monitoring video of the target tower;
[0028] A preliminary identification module, connected to the video acquisition module, is used to perform preliminary identification on the monitoring video using a target detection algorithm to generate a suspected bird nest object; the suspected bird nest object includes a monitoring image, a suspected bird nest location, an image timestamp, a label, and a confidence level;
[0029] A multimodal large model module is connected to the preliminary recognition module and has a fine-tuned multimodal large model built in it, and is used to input the suspected bird nest object into the multimodal large model and output the tower bird nest recognition result according to the filtering prompt word template library; the filtering prompt word template library includes an event type and a standard event prompt word list that constitute a key-value pair;
[0030] The event reporting module is connected to the multimodal large model module and is used to report the suspected bird's nest object as an event object when the recognition result contains a definite conclusion that there is a bird's nest on the tower.
[0031] Preferably, the method by which the preliminary identification module performs preliminary identification on the monitoring video includes:
[0032] Extracting monitoring images from the monitoring video frame by frame, extracting sub-images containing suspected bird nests from the monitoring images using regions of interest, and performing target recognition on the sub-images using a trained bird nest target detection model;
[0033] Obtain the position of the identified suspected bird nest in the sub-image, and restore the position to the suspected bird nest position obtained in the monitoring image;
[0034] Adding a bird nest tag to the suspected bird nest, and filtering the tag attribute to obtain a list of suspected bird nest objects;
[0035] Using a tracking algorithm to track the suspected bird nest object list, obtaining a suspected bird nest tracking object list; the suspected bird nest tracking object list includes the most reasonable suspected bird nest object, the current suspected bird nest object, the tracking state and the image timestamp;
[0036] When the tracking state changes to "disappeared", the most reasonable suspected bird nest object at present is taken as the suspected bird nest object.
[0037] Preferably, the preliminary recognition module includes a sub-image extraction unit, and the method for extracting a sub-image by the sub-image extraction unit includes:
[0038] Extracting an image of interest from the monitoring image based on the minimum circumscribed rectangle of the region of interest;
[0039] Taking the aspect ratio of the smallest circumscribed rectangle and the monitored image with the largest aspect ratio as the expansion ratio, taking the center of the smallest circumscribed rectangle as the expansion center, and expanding the image extraction frame outward according to the expansion ratio;
[0040] If the image extraction frame extends beyond the monitoring image boundary, the expansion center is translated in the opposite direction of the exceeding direction to ensure that the image extraction frame does not exceed the monitoring image boundary;
[0041] When any one of the width and height of the image extraction frame is equal to the width and height of the monitoring image, the monitoring image of the current image extraction frame is used as a sub-image.
[0042] Preferably, the method for updating the most reasonable suspected bird nest object in the suspected bird nest tracking object list includes:
[0043] When the identification frame size and the confidence of the current suspected bird's nest object are both greater than the suspected bird's nest position and the confidence of the current most reasonable suspected bird's nest object, the current suspected bird's nest object is used to update the current most reasonable suspected bird's nest object.
[0044] Preferably, the fine-tuning method of the multimodal large model includes:
[0045] Construct a correct event dataset including images of labeled bird nests and corresponding image description texts, and a false alarm event dataset including images of objects labeled as falsely reported as bird nests and description texts of the causes of the false alarms, and use them together as pre-training datasets to fine-tune the multimodal large model, thereby enhancing the multimodal large model's image and semantic understanding of the tower bird nests.
[0046] Beneficial Effects
[0047] 1. Improve event detection accuracy: The present invention achieves accurate identification of bird nests on the tower by combining visual information and semantic understanding. Further review of the initially identified suspected bird nest objects using a fine-tuned multimodal large model can significantly reduce the false alarm rate and ensure that the reported bird nest events are more accurate. This method effectively overcomes the problem of traditional visual models being prone to false alarms in complex environments and greatly improves the detection accuracy of the system.
[0048] 2. Reduce the workload of manual review: The present invention can automatically screen out effective tower bird nest events, reduce the burden of manual review, reduce labor cost investment, and thus improve overall efficiency.
[0049] 3. Enhance the generalization ability of the system: The present invention can adapt to the detection of bird nests on towers in different scenarios and complex environments. By combining the processing of multimodal data, the system is not only suitable for the detection of bird nests on towers, but can also be extended to other similar target recognition tasks, and has broad application prospects.
[0050] 4. Optimize user experience: By improving detection accuracy and reducing false alarm rate, this system can provide users with more accurate and reliable alarm information, avoiding unnecessary interference caused by false alarms. At the same time, the system's automation level is improved, reducing the user's operating burden and improving the user's overall experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 A schematic flow chart of a method for monitoring bird nests on a tower based on a multi-modal large model provided in a preferred embodiment of the present invention;
[0052] Figure 2 A schematic flow chart of a method for preliminary identification of suspected bird nests provided in another preferred embodiment of the present invention;
[0053] Figure 3 A schematic flow chart of a sub-image extraction method provided in another preferred embodiment of the present invention;
[0054] Figure 4 The present invention is a schematic structural diagram of a tower bird nest monitoring system based on a multi-modal large model provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.
[0056] Example 1.
[0057] like Figure 1 As shown, this embodiment provides a tower bird nest monitoring method based on a multi-modal large model, comprising the steps of:
[0058] S1. Obtain monitoring video of the target tower, use the target detection algorithm to preliminarily identify suspected bird nests in the monitoring video, and report the suspected bird nest objects; the suspected bird nest objects include monitoring images, suspected bird nest locations, image timestamps, labels, and confidence levels.
[0059] It should be understood that the monitoring video acquisition method in the present invention has a variety of flexible implementation methods. Specifically, the acquisition of monitoring video can be achieved by timed automatic acquisition, that is, the system regularly acquires the monitoring video of the target area at preset time intervals to ensure the continuity and timeliness of continuous monitoring. In addition, the monitoring video can also be acquired by manual uploading by the user, and the user uploads the video file shot by the monitoring equipment to the system for analysis and processing according to actual needs. It should be noted that the specific method of video acquisition is not limited to the above two methods, and can also be adjusted according to the needs and technical conditions of the actual application scenario. The selection and implementation of these different video acquisition methods do not belong to the core content of the present invention, but can be determined and configured by those skilled in the art according to actual conditions to ensure that the monitoring system can efficiently and accurately acquire the required video data.
[0060] The target detection algorithm used in the present invention can be any one of the commonly used target detection algorithms, such as YOLO (YouOnly Look Once), Faster R-CNN (Region Convolutional Neural Network), SSD (SingleShot MultiBox Detector), etc. Since the algorithm is mainly used in the present invention to preliminarily identify suspected bird nests in the monitoring video, its role is limited to screening out areas where bird nests may exist, which is convenient for subsequent further analysis and processing. Therefore, for the selection of the target detection algorithm in this step, there is no need to pay too much attention to the accuracy or complexity of its recognition. In practical applications, as long as the selected target detection algorithm can effectively identify the suspected bird nest area within a reasonable computing resource and time range, the requirements of the present invention can be met. In other words, the algorithm only needs to have sufficient recognition ability to provide preliminary screening results, and the final bird nest identification and confirmation will be completed by the multimodal large model of the present invention and subsequent processing steps. Therefore, those skilled in the art can select and adjust the appropriate target detection algorithm according to the specific application scenario and hardware conditions to meet the overall performance requirements of the system.
[0061] In some preferred embodiments, for the video footage shot by the monitoring tower, considering that the intercepted monitoring image resolution is usually large, while the proportion of the bird's nest in the image is very small, it may make target detection more difficult. Especially when using conventional target monitoring algorithms, these algorithms usually require the size of the input image to be uniformly adjusted to a smaller pixel value to meet the input requirements of the model. However, this reduction in image size will further reduce the resolution of the bird's nest in the image, resulting in the loss of its detail information, which in turn affects the accuracy and efficiency of subsequent target recognition. Based on this, this embodiment provides the following method for preliminary identification of suspected bird nests, such as Figure 2 As shown, specifically including:
[0062] S11, extract the monitoring image in the monitoring video frame by frame, use the region of interest to extract the sub-image containing the suspected bird's nest in the monitoring image, and use the trained bird's nest target detection model to identify the sub-image. The region of interest (ROI) refers to a specific area in the image or video screen that contains the target object. In the present invention, the region of interest refers to the part of the image that may contain the bird's nest. Since the tower bird's nest usually occupies only a small part in the high-resolution monitoring image, extracting and focusing on this region of interest helps to improve the recognition efficiency and accuracy of the target detection algorithm. Those skilled in the art can know that the determination of the region of interest can be achieved by a variety of methods, such as pre-definition, automatic detection and other common methods, and the specific implementation of the present invention is not further limited. The bird's nest target detection model used in this embodiment is a common target monitoring model that is pre-trained using a data set containing a bird's nest. The sub-image refers to the original monitoring image, by extracting and enlarging the region of interest containing the suspected bird's nest target, so that the proportion of the bird's nest target in the sub-image is increased as much as possible. This operation is intended to improve the visibility of the bird's nest feature during the detection process, while retaining other relevant features in the original image as much as possible to ensure that subsequent target recognition can make full use of this information.
[0063] S12. Obtain the position of the suspected bird's nest identified in the sub-image, and restore the position to the suspected bird's nest position in the monitoring image. Since the sub-image is extracted from the area of interest of the original monitoring image, the identified bird's nest position needs to be mapped back to the original monitoring image. To this end, it is necessary to convert the coordinate system in the sub-image into the coordinate system in the original monitoring image. This step usually involves the calculation of scale conversion and position offset to obtain the actual coordinates of the suspected bird's nest, which is restored to the original monitoring image. This process ensures that the suspected bird's nest can be accurately located in the original monitoring image, thereby providing an accurate basis for subsequent processing and decision-making.
[0064] S13, adding a bird nest tag to the suspected bird nest, and filtering the tag attributes to obtain a suspected bird nest object list. The tag attribute filtering refers to screening the detection results according to preset tags and attribute conditions after detecting the suspected target (such as a bird nest) to filter out targets that do not meet specific standards or requirements, thereby obtaining a more accurate suspected target list. Those skilled in the art can select a suitable implementation method based on the prior art, and the present invention does not make further limitations. The filtered suspected bird nest object list includes several suspected bird nest objects in each frame.
[0065] S14. Target tracking is performed on the suspected bird nest object list using a tracking algorithm to obtain a suspected bird nest tracking object list; the suspected bird nest tracking object list includes the most reasonable suspected bird nest object at the moment, the current suspected bird nest object, the tracking status and the image timestamp. In this step, during the tracking process, when the confidence of the suspected bird nest gradually increases or the size of the identification box increases during the tracking process, the possibility that it is a real bird nest can be further confirmed. In some preferred embodiments, the method for updating the most reasonable suspected bird nest object at the moment in the suspected bird nest tracking object list includes: when the identification box size and the confidence of the current suspected bird nest object are both greater than the suspected bird nest position and confidence of the most reasonable suspected bird nest object at the moment, the most reasonable suspected bird nest object at the moment is updated with the current suspected bird nest object.
[0066] S15. When the tracking state changes to "disappeared", the most reasonable suspected bird's nest object is used as the suspected bird's nest object.
[0067] At this point, the preliminary identification of the suspected bird nest has been completed. However, as mentioned above, although the target recognition model can provide effective detection results in most cases, its performance is still affected by environmental complexity, target diversity and other external factors. These factors may lead to false positives in the recognition process, especially in complex scenarios, the recognition accuracy of the model may be significantly reduced. In order to improve the accuracy of target recognition, a common practice is to continuously improve the capabilities of the target recognition model. However, this approach has many challenges in practice. On the one hand, improving the model capabilities usually requires highly specialized technicians to optimize and debug the model, which requires high experience and skills of the technical team. On the other hand, enhancing the complexity of the model will also put higher requirements on the hardware resources, computing power and overall construction of the system, which may lead to a significant increase in the development and operation costs of the system. The prior art also provides some ways to review the preliminary recognition results through manual review to reduce false positives. This method relies on the experience and judgment of human experts. Through manual review and screening, invalid targets that are misidentified by the model can be effectively filtered out. However, although manual review can improve the accuracy of recognition, it also brings problems such as low work efficiency, high review cost, and more manual intervention, which limits the degree of automation of the system.
[0068] To this end, the present invention proposes to use a fine-tuned multimodal large model to verify the preliminary recognition results.
[0069] S2. Input the suspected bird's nest object into the fine-tuned multimodal large model, and output the tower bird's nest recognition result according to the filtering prompt word template library; the filtering prompt word template library includes event types and standard event prompt word lists that constitute key-value pairs.
[0070] A multimodal large model is a large-scale AI model that can process and understand multiple types of data (i.e., "modalities"). By integrating information from different modalities, multimodal large models can more comprehensively understand complex scenarios and tasks, thereby enabling more advanced AI applications. Commonly used models include the open source LLAVA class model and the commercial Claude and ChatGPT.
[0071] Since the generalization ability of the multimodal large model is strong, the effect of directly applying it in the present invention is often unsatisfactory. In order for the multimodal large model to have event-enhanced image semantic understanding, it needs to be fine-tuned. The fine-tuning of the multimodal large model is mainly achieved by constructing a pre-training data set that meets the requirements, processing the image data and text data in the data set, and merging the subsequent parts of the input target multimodal large model for training to generate a description or response instruction corresponding to the image. The specific training steps can be specifically designed by those skilled in the art according to actual needs and prior art. In some preferred embodiments, a method for constructing a pre-training data set for fine-tuning is provided, specifically including:
[0072] Construct a correct event dataset including images of bird nests with annotations and corresponding image description texts, and a false alarm event dataset including images of objects that are falsely reported as bird nests with annotations and description texts of the causes of false alarms, and use them together as pre-training datasets to fine-tune the multimodal large model and enhance the multimodal large model's image and semantic understanding of the tower bird nest. An example is given below to further illustrate the format and content of the dataset:
[0073] A dataset image file is bird_nest_01.jpg, and the text description file bird_nest_01.txt describes the image in the form of QA pairs. The content of bird_nest_01.txt is generally as follows:
[0074] 1. Picture description:
[0075] The picture was taken on XX / XX / XX. The picture shows a scene near a field railway. There is a railway track running through the middle of the image. There is a high tower on the right side of the railway. There is a bird's nest on the tower. The bird's nest on the tower will affect the safe operation of the power grid and needs to be reported to the relevant departments in a timely manner.
[0076] 2. Scene Reasoning:
[0077] Question: Is there a bird's nest in the picture?
[0078] Answer: There is a bird's nest in the picture.
[0079] Question: Where is the bird's nest in the picture?
[0080] Answer: The bird's nest is on the high tower on the right side of the railway.
[0081] The filtering prompt word template library includes preset templates for guiding the multimodal large model to perform screening and verification. The template library consists of a series of templates containing event types (iron tower bird's nest) and corresponding prompt word lists. It plays a key role in the process of the multimodal large model reviewing the preliminary recognition results, helping the model to more accurately determine whether the suspected target meets specific conditions, thereby reducing false alarms and improving recognition accuracy. In some preferred embodiments, the false alarm event data set can be manually analyzed to construct a set of false alarm descriptions of different types under the same event type, and generate an event filtering prompt word template library. Its format is map<event type, list<event prompt word>>. In the iron tower bird's nest event of the present invention, its prompt word list can be:
[0082] The type of the tower bird's nest event is: bird_nest. Is there a tower bird's nest event in this picture?
[0083] Is there an iron tower in this picture?
[0084] Is there a bird's nest on the tower in the picture?
[0085] The multimodal large model obtains the prompt word list of the event from the filtering prompt word template library through the provided suspected event type. The first data in the list is used as the text question, and the subsequent data is normalized into the output of the large model in the form of regular expressions.
[0086] S3. If the recognition result contains a definite conclusion that a bird's nest exists on the tower, the suspected bird's nest object is reported as an event object. Taking the above-mentioned filter prompt word template library format of map<event type, list<event prompt word>> as an example, if the recognition result output by the multimodal large model contains the following content, the suspected bird's nest object is reported as an event object:
[0087] {
[0088] "Does the tower exist?": "Yes",
[0089] "Are there bird nests on the tower?": "Yes"
[0090] }.
[0091] Example 2.
[0092] This embodiment is developed on the basis of the above-mentioned embodiment 1. Figure 3 As shown, this embodiment provides an example of a specific method for extracting a sub-image, including:
[0093] S111, extracting an image of interest from the monitoring image based on the minimum bounding rectangle of the region of interest, wherein the region of interest is usually a polygonal region.
[0094] S112, taking the aspect ratio of the smallest circumscribed rectangle and the monitored image with the largest aspect ratio as the expansion ratio, taking the center of the smallest circumscribed rectangle as the expansion center, and expanding the image extraction frame outward according to the expansion ratio. By expanding the image extraction frame outward according to the expansion ratio, the extraction frame can cover a larger area. This expansion process helps to retain more background and related information without losing target information, thereby improving the accuracy and robustness of target recognition.
[0095] S113. If the image extraction frame extends beyond the boundary of the monitoring image, the center of the expansion is translated in the opposite direction of the direction of the expansion to ensure that the image extraction frame does not exceed the boundary of the monitoring image. In this step, in order to prevent the image extraction frame from exceeding the boundary of the monitoring image during the expansion process, the position of the extraction frame needs to be adjusted. The purpose of this translation operation is to ensure that the entire image extraction frame is still within the boundary of the monitoring image, thereby preventing the extraction frame from overflowing the monitoring image range and causing partial image information to be lost.
[0096] S114. When any one of the width and height of the image extraction frame is equal to the width and height of the monitoring image, the monitoring image of the current image extraction frame is used as a sub-image. During the expansion process of the image extraction frame, it may encounter a situation where one of its width or height is equal to the width or height of the entire monitoring image. This means that the extraction frame has been expanded to the maximum possible size of the monitoring image and cannot be expanded further. Once this happens, that is, the width or height of the extraction frame is equal to the corresponding size of the monitoring image, the system will directly regard the monitoring image area covered by the extraction frame as the final sub-image. This sub-image will contain all the key information in the area of interest, and because the extraction frame has reached the maximum expandable size of the monitoring image, it is ensured that as much background information as possible is retained in the sub-image.
[0097] Through the above steps, the final generated sub-image not only contains all the target features of the region of interest, but also retains the information of the surrounding environment as much as possible, providing richer contextual data for subsequent target recognition and analysis, thereby helping to improve the accuracy of recognition.
[0098] Example 3.
[0099] like Figure 4 As shown, this embodiment provides a tower bird nest monitoring system based on a multi-modal large model, including:
[0100] A video acquisition module is used to acquire monitoring video of the target tower;
[0101] A preliminary identification module, connected to the video acquisition module, is used to perform preliminary identification on the monitoring video using a target detection algorithm to generate a suspected bird nest object; the suspected bird nest object includes a monitoring image, a suspected bird nest location, an image timestamp, a label, and a confidence level;
[0102] A multimodal large model module is connected to the preliminary recognition module and has a fine-tuned multimodal large model built in it, and is used to input the suspected bird nest object into the multimodal large model and output the tower bird nest recognition result according to the filtering prompt word template library; the filtering prompt word template library includes an event type and a standard event prompt word list that constitute a key-value pair;
[0103] The event reporting module is connected to the multimodal large model module and is used to report the suspected bird's nest object as an event object when the recognition result contains a definite conclusion that there is a bird's nest on the tower.
[0104] In some preferred embodiments, the method by which the preliminary recognition module performs preliminary recognition on the monitoring video includes:
[0105] Extracting monitoring images from the monitoring video frame by frame, extracting sub-images containing suspected bird nests from the monitoring images using regions of interest, and performing target recognition on the sub-images using a trained bird nest target detection model;
[0106] Obtain the position of the identified suspected bird nest in the sub-image, and restore the position to the suspected bird nest position obtained in the monitoring image;
[0107] Adding a bird nest tag to the suspected bird nest, and filtering the tag attribute to obtain a list of suspected bird nest objects;
[0108] Using a tracking algorithm to track the suspected bird nest object list, obtaining a suspected bird nest tracking object list; the suspected bird nest tracking object list includes the most reasonable suspected bird nest object, the current suspected bird nest object, the tracking state and the image timestamp;
[0109] When the tracking state changes to "disappeared", the most reasonable suspected bird nest object at present is taken as the suspected bird nest object.
[0110] In some preferred embodiments, the preliminary recognition module includes a sub-image extraction unit, and the method for extracting a sub-image by the sub-image extraction unit includes:
[0111] Extracting an image of interest from the monitoring image based on the minimum circumscribed rectangle of the region of interest;
[0112] Taking the aspect ratio of the smallest circumscribed rectangle and the monitored image with the largest aspect ratio as the expansion ratio, taking the center of the smallest circumscribed rectangle as the expansion center, and expanding the image extraction frame outward according to the expansion ratio;
[0113] If the image extraction frame extends beyond the monitoring image boundary, the expansion center is translated in the opposite direction of the exceeding direction to ensure that the image extraction frame does not exceed the monitoring image boundary;
[0114] When any one of the width and height of the image extraction frame is equal to the width and height of the monitoring image, the monitoring image of the current image extraction frame is used as a sub-image.
[0115] In some preferred embodiments, the method for updating the most reasonable suspected bird nest object in the suspected bird nest tracking object list includes:
[0116] When the identification frame size and the confidence of the current suspected bird's nest object are both greater than the suspected bird's nest position and the confidence of the current most reasonable suspected bird's nest object, the current suspected bird's nest object is used to update the current most reasonable suspected bird's nest object.
[0117] In some preferred embodiments, the fine-tuning method of the multimodal large model includes:
[0118] Construct a correct event dataset including images of labeled bird nests and corresponding image description texts, and a false alarm event dataset including images of objects labeled as falsely reported as bird nests and description texts of the causes of the false alarms, and use them together as pre-training datasets to fine-tune the multimodal large model, thereby enhancing the multimodal large model's image and semantic understanding of the tower bird nests.
[0119] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A tower bird nest monitoring method based on a multimodal large model, characterized in that: Includes steps: Obtain monitoring video of the target tower, use the target detection algorithm to preliminarily identify suspected bird nests in the monitoring video, and report the suspected bird nest objects; the suspected bird nest objects include monitoring images, suspected bird nest locations, image timestamps, labels, and confidence levels; Input the suspected bird's nest object into the fine-tuned multimodal large model, and output the tower bird's nest recognition result according to the filtered prompt word template library; The filtering prompt word template library includes event types and standard event prompt word lists that constitute key-value pairs; If the identification result contains a definite conclusion that a bird's nest exists on the tower, the suspected bird's nest object is reported as an event object; The fine-tuning method of the multimodal large model includes: Construct a correct event dataset including images of labeled bird nests and corresponding image description texts, and a false alarm event dataset including images of objects labeled as falsely reported as bird nests and description texts of the causes of the false alarms, and use them together as pre-training datasets to fine-tune the multimodal large model, thereby enhancing the multimodal large model's image and semantic understanding of the tower bird nests.
2. The tower bird nest monitoring method based on a multi-modal large model as claimed in claim 1, characterized in that: The preliminary identification method of the suspected bird nest in the monitoring video includes: Extracting monitoring images from the monitoring video frame by frame, extracting sub-images containing suspected bird nests from the monitoring images using regions of interest, and performing target recognition on the sub-images using a trained bird nest target detection model; Obtain the position of the identified suspected bird nest in the sub-image, and restore the position to the suspected bird nest position obtained in the monitoring image; Adding a bird nest tag to the suspected bird nest, and filtering the tag attribute to obtain a list of suspected bird nest objects; Using a tracking algorithm to track the suspected bird nest object list, obtaining a suspected bird nest tracking object list; the suspected bird nest tracking object list includes the most reasonable suspected bird nest object, the current suspected bird nest object, the tracking state and the image timestamp; When the tracking state changes to "disappeared", the most reasonable suspected bird's nest object at present is taken as the suspected bird's nest object.
3. The tower bird nest monitoring method based on a multi-modal large model as claimed in claim 2 is characterized in that: The sub-image extraction method comprises: Extracting an image of interest from the monitoring image based on the minimum circumscribed rectangle of the region of interest; Taking the aspect ratio of the smallest circumscribed rectangle and the monitored image with the largest aspect ratio as the expansion ratio, taking the center of the smallest circumscribed rectangle as the expansion center, and expanding the image extraction frame outward according to the expansion ratio; If the image extraction frame extends beyond the monitoring image boundary, the expansion center is translated in the opposite direction of the exceeding direction to ensure that the image extraction frame does not exceed the monitoring image boundary; When any one of the width and height of the image extraction frame is equal to the width and height of the monitoring image, the monitoring image of the current image extraction frame is used as a sub-image.
4. The tower bird nest monitoring method based on a multimodal large model as claimed in claim 2 is characterized in that: The method for updating the most reasonable suspected bird nest object in the suspected bird nest tracking object list includes: When the identification frame size and the confidence of the current suspected bird's nest object are both greater than the suspected bird's nest position and the confidence of the current most reasonable suspected bird's nest object, the current suspected bird's nest object is used to update the current most reasonable suspected bird's nest object.
5. The tower bird nest monitoring system based on multi-modal large model is characterized by: include: A video acquisition module is used to acquire monitoring video of the target tower; A preliminary identification module, connected to the video acquisition module, is used to perform preliminary identification on the monitoring video using a target detection algorithm to generate a suspected bird nest object; the suspected bird nest object includes a monitoring image, a suspected bird nest location, an image timestamp, a label, and a confidence level; A multimodal large model module is connected to the preliminary recognition module and has a fine-tuned multimodal large model built in it, and is used to input the suspected bird nest object into the multimodal large model and output the tower bird nest recognition result according to the filtering prompt word template library; the filtering prompt word template library includes an event type and a standard event prompt word list that constitute a key-value pair; An event reporting module, connected to the multimodal large model module, is used to report the suspected bird's nest object as an event object when the identification result contains a confirmed conclusion that a bird's nest exists on the tower; The fine-tuning method of the multimodal large model includes: Construct a correct event dataset including images of labeled bird nests and corresponding image description texts, and a false alarm event dataset including images of objects labeled as falsely reported as bird nests and description texts of the causes of the false alarms, and use them together as pre-training datasets to fine-tune the multimodal large model, thereby enhancing the multimodal large model's image and semantic understanding of the tower bird nests.
6. The iron tower bird nest monitoring system based on multi-modal large model as claimed in claim 5, characterized in that: The method for the preliminary recognition module to perform preliminary recognition on the monitoring video includes: Extracting monitoring images from the monitoring video frame by frame, extracting sub-images containing suspected bird nests from the monitoring images using regions of interest, and performing target recognition on the sub-images using a trained bird nest target detection model; Obtain the position of the identified suspected bird nest in the sub-image, and restore the position to the suspected bird nest position obtained in the monitoring image; Adding a bird nest tag to the suspected bird nest, and filtering the tag attribute to obtain a list of suspected bird nest objects; Using a tracking algorithm to track the suspected bird nest object list, obtaining a suspected bird nest tracking object list; the suspected bird nest tracking object list includes the most reasonable suspected bird nest object, the current suspected bird nest object, the tracking state and the image timestamp; When the tracking state changes to "disappeared", the most reasonable suspected bird's nest object at present is taken as the suspected bird's nest object.
7. The iron tower bird nest monitoring system based on multi-modal large model as claimed in claim 6 is characterized in that: The preliminary recognition module includes a sub-image extraction unit, and the method for extracting a sub-image by the sub-image extraction unit includes: Extracting an image of interest from the monitoring image based on the minimum circumscribed rectangle of the region of interest; Taking the aspect ratio of the smallest circumscribed rectangle and the monitored image with the largest aspect ratio as the expansion ratio, taking the center of the smallest circumscribed rectangle as the expansion center, and expanding the image extraction frame outward according to the expansion ratio; If the image extraction frame extends beyond the monitoring image boundary, the expansion center is translated in the opposite direction of the exceeding direction to ensure that the image extraction frame does not exceed the monitoring image boundary; When any one of the width and height of the image extraction frame is equal to the width and height of the monitoring image, the monitoring image of the current image extraction frame is used as a sub-image.
8. The iron tower bird nest monitoring system based on multi-modal large model as claimed in claim 6, characterized in that: The method for updating the most reasonable suspected bird nest object in the suspected bird nest tracking object list includes: When the identification frame size and the confidence of the current suspected bird's nest object are both greater than the suspected bird's nest position and the confidence of the current most reasonable suspected bird's nest object, the current suspected bird's nest object is used to update the current most reasonable suspected bird's nest object.
