Method for monitoring a rot infect and computing device

Abstract

The invention provides a method for monitoring a rat disease and computing equipment. The method comprises the following steps: acquiring an infrared video frame image; transmitting the video frame image into a first neural network model and a second neural network model for target identification; judging whether a first target identified by the first neural network model and a second target identified by the second neural network model are the same target or not; if the first target and the second target are the same target, determining that the same target is a mouse target, and obtaining a first recognition result; if the first target and the second target are different, respectively transmitting the first target and the second target into a classifier for secondary identification to obtain a second identification result; and according to the first identification result and the second identification result, whether early warning is carried out is determined. According to the technical scheme of the invention, high-precision target identification can be realized, the error of target identification is reduced, the false detection rate is reduced, the interference of environmental factors is reduced, and the target detection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and computing device for monitoring rodent infestation. Background Art

[0002] In the catering industry, the problem of rat infestation has always been a problem that has troubled many catering units. The activities of rats not only cause food contamination, but also may cause the spread of diseases, seriously threatening consumers' food safety and health.

[0003] In recent years, with the development of technology, most methods have adopted deep learning technology and used convolutional neural network (CNN) target detection algorithms to solve this type of problem. For example, the common real-time target detection algorithm (YOLO) series of target detection algorithms are widely used. Although the recognition accuracy has been improved compared with the previous traditional image recognition methods, compared with the complex kitchen scenes, the recognition accuracy is far from meeting the accuracy standards of actual application scenarios, and the false detection rate is high.

[0004] To this end, a technical solution is needed that can achieve high-precision target recognition, reduce target recognition errors, reduce false detection rates, reduce interference from environmental factors, and improve target detection efficiency. Summary of the invention

[0005] The present invention aims to provide a method and computing equipment for monitoring rodent infestation, which can achieve high-precision target recognition, reduce target recognition errors, reduce false detection rates, reduce interference from environmental factors, and improve target detection efficiency.

[0006] According to one aspect of the present invention, there is provided a method for monitoring rodent infestation, the method comprising:

[0007] Acquire infrared video frame images;

[0008] The video frame image is transmitted to the first neural network model and the second neural network model for target recognition respectively;

[0009] Determining whether a first target recognized by the first neural network model and a second target recognized by the second neural network model are the same target;

[0010] If the first target and the second target are the same target, determining that the same target is a mouse target, and obtaining a first recognition result;

[0011] If the first target and the second target are different, the first target and the second target are respectively transmitted to the classifier for secondary recognition to obtain a second recognition result;

[0012] Determine whether to issue a warning based on the first recognition result and the second recognition result.

[0013] According to some embodiments, the first neural network model is a detector based on a neural network target detection algorithm, and the second neural network model is a segmenter based on a neural network image semantic segmentation algorithm.

[0014] According to some embodiments, the detector is based on an improved yolov7 structure, wherein a receptive field module is attached to the yolov7 backbone network, and the original SiLU activation layer is replaced by a Relu activation layer to accelerate training, and the receptive field module expands the receptive field by using convolution kernels of different sizes at different scales.

[0015] According to some embodiments, the receptive field module includes a plurality of convolutional layers, concatenation layers, and ReLU activation layers of different scales, wherein the video frame image is passed into the first neural network model for target recognition, including:

[0016] The input feature map is subjected to feature extraction through multiple convolutional layers of different scales and convolutional layers with adjusted expansion coefficients to obtain feature maps of different scales, thereby enhancing the detection of small targets such as mice, which are smaller than 30*30 pixels.

[0017] The feature maps of different scales are spliced ​​and fused through 1x1 convolutional layers to improve the detection speed and accuracy of mouse targets while the increased computational effort is less than the threshold.

[0018] The fused features are then residually fused with the input feature map and sent to the activation layer.

[0019] According to some embodiments, after the video frame image is passed into the detector and the segmenter for object recognition, the method further includes:

[0020] The detector filters target frames whose detector scores are less than a first predetermined threshold;

[0021] The detector filters the target frames whose width and / or height are smaller than a second predetermined threshold;

[0022] The detector filters the target frames whose width and / or height are greater than a third predetermined threshold.

[0023] According to some embodiments, after the video frame image is passed into the detector and the segmenter for object recognition, the method further includes:

[0024] The segmenter obtains one or more mouse target contour point sets;

[0025] The target contour area enclosed by each mouse target contour was calculated;

[0026] Compare each calculated target contour area with a minimum area threshold and a maximum area threshold, and retain a set of target contours that are larger than the minimum area threshold and smaller than the maximum area threshold;

[0027] By finding the circumscribed rectangle of the contour point set, the target frame data of the retained target contour set is obtained.

[0028] According to some embodiments, determining whether the first target identified by the detector and the second target identified by the segmenter are the same includes:

[0029] Calculate the overlap intersection-over-union ratio of the target frame of the first target and the target frame of the second target;

[0030] When the overlap intersection-and-union ratio value is greater than a preset overlap intersection-and-union ratio threshold, it indicates that the detector and the segmenter have detected the same target;

[0031] When the overlap intersection-and-union ratio value is less than a preset overlap intersection-and-union ratio threshold, a secondary identification is required.

[0032] According to some embodiments, if the first target and the second target are different, the first target and the second target are input into a classifier for secondary recognition to obtain a final recognition result, including:

[0033] Acquire a secondary recognition target image through the target frame data of the first target and the target frame information of the second target;

[0034] The secondary recognition target image is sent to the classifier to output the corresponding non-mouse target category score and mouse target category score;

[0035] When the mouse target category score is greater than the preset category score threshold, there is a mouse target in the current secondary recognition target image.

[0036] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method as described above is implemented.

[0037] According to another aspect of the present invention, there is provided a computing device, comprising:

[0038] Processor; and

[0039] A memory stores a computer program, and when the computer program is executed by the processor, the method described in any one of the above items is implemented.

[0040] According to an embodiment of the present invention, by acquiring a video frame image, the video frame image is passed into a detector and a segmenter for target recognition, and it is determined whether the first target recognized by the detector and the second target recognized by the segmenter are the same. If the first target and the second target are different, they are passed into a classifier for secondary recognition to obtain a final recognition result, and it is determined whether to issue an early warning. The present invention combines target detection and semantic segmentation, performs a primary recognition through a detector and a segmenter, performs a secondary recognition through a classifier, and completes target recognition by combining the two recognition results. It can achieve high-precision target recognition, reduce target recognition errors, reduce false detection rates, reduce interference from environmental factors, and improve target detection efficiency.

[0041] According to some embodiments, the accuracy of recognition can be significantly improved by performing target recognition through a detector and a segmenter respectively, and then performing secondary verification through a classifier. The detector and the segmenter extract features from different angles and levels, which helps to reduce false detections and missed detections. The detector focuses on the location and category of the target, while the segmenter focuses on the fine contours and boundaries of the target. The combination of the two can provide more comprehensive information and improve the reliability of recognition.

[0042] According to some embodiments, through a multi-stage recognition process, the influence of environmental factors such as complex background, occlusion, and illumination changes can be effectively dealt with. Even if a false detection occurs in a certain stage, it can be corrected in the subsequent stage. Through secondary recognition, false alarms caused by misjudgment of a single model can be reduced, and the robustness of the system can be improved.

[0043] According to some embodiments, the detector and the segmenter can be processed in parallel to speed up the recognition speed and meet the real-time requirements. Through multi-stage processing, most non-target objects can be quickly filtered out in the early stage, reducing the computational burden of the subsequent stage and improving the overall efficiency.

[0044] According to some embodiments, by integrating the results of detectors, segmenters, and classifiers, multi-dimensional decisions can be made to improve the accuracy of early warnings. Thresholds and parameters are dynamically adjusted according to actual conditions to adapt to different application scenarios and environmental changes. This multi-stage recognition method is applicable to a variety of scenarios, such as wildlife monitoring, pet monitoring, security prevention, etc., and has high versatility and adaptability.

[0045] According to some embodiments, the entire system is composed of multiple modules, and different detectors, segmenters and classifiers can be selected according to specific needs, which is highly flexible. Each module is relatively independent and can be optimized and upgraded separately without affecting the operation of other parts, which is convenient for system maintenance and function expansion.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0048] Figure 1 A flow chart of a method for monitoring rodent infestation is shown according to an example embodiment.

[0049] Figure 2 A schematic diagram illustrating a rodent infestation identification process according to an example embodiment.

[0050] Figure 3a A schematic diagram illustrating a receptive field module according to an example embodiment is shown.

[0051] Figure 3b A schematic diagram illustrating a detector network structure according to an example embodiment.

[0052] Figure 3c Schematic diagram showing a detailed explanation of a detector network structure according to an example embodiment.

[0053] Figure 4 A schematic diagram illustrating a splitter network structure according to an example embodiment.

[0054] Figure 5 A schematic diagram illustrating overlap intersection over union according to example embodiments.

[0055] Figure 6 A block diagram of a computing device is shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0057] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0058] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0059] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0060] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the present inventive concept. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0061] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0062] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present invention, and therefore cannot be used to limit the protection scope of the present invention.

[0063] Rats, as common rodents, have been a problem for human living environments and agricultural production in many areas due to their rapid reproduction, strong reproductive capacity, and hidden activities. Globally, the spread of rat infestation has become a serious public health problem, especially in some economically backward areas or areas prone to natural disasters, where the solution to the problem is particularly urgent. The threat of rats to the human living environment cannot be ignored. Rats in cities may carry viruses and bacteria, such as plague bacteria, which pose health risks to humans by spreading diseases. In addition, the gnawing behavior of rats may cause damage to items such as wires and furniture, and even cause safety accidents such as fires. Identification of rat infestation is particularly important, as it can detect the presence of rats in a timely manner and provide a basis for formulating timely and effective prevention and control measures.

[0064] To this end, the present invention proposes a method for identifying rodent infestation, which can achieve high-precision target identification, reduce target identification errors, reduce false detection rates, reduce interference from environmental factors, and improve target detection efficiency.

[0065] Exemplary embodiments of the present invention are described below with reference to the accompanying drawings.

[0066] Figure 1 A flow chart of a method for monitoring rodent infestation is shown according to an example embodiment.

[0067] See also Figure 1 , in S101, an infrared video frame image is acquired.

[0068] According to some embodiments, a single frame is extracted from a surveillance video file for target recognition.

[0069] In S103, the video frame image is transmitted to the first neural network model and the second neural network model for target recognition respectively.

[0070] According to some embodiments, the method according to claim 1 is characterized in that the first neural network model is a detector based on a neural network target detection algorithm, and the second neural network model is a segmenter based on a neural network image semantic segmentation algorithm. The detector is pre-trained according to a constructed training data set, and the training data set includes mouse data of visible light during the day and mouse data of infrared scenes at night; the mouse target in the mouse data is annotated with a target box through an open source annotation tool to obtain a first training data label.

[0071] The coordinates of the upper left corner point of the mouse target frame and the width and height of the target frame included in all training samples are obtained to generate the first training data label; after obtaining the first training data label, all data in the first training data set are scaled.

[0072] According to some embodiments, the detector is based on an improved yolov7 structure, wherein a receptive field module is attached to the yolov7 backbone network, and the original SiLU activation layer is replaced by a Relu activation layer to accelerate training, and the receptive field module expands the receptive field by using convolution kernels of different sizes at different scales.

[0073] The receptive field module includes a plurality of convolutional layers, splicing layers and Relu activation layers of different scales. The video frame image is passed into the first neural network model for target recognition. For example, the input feature map is extracted through a plurality of convolutional layers of different scales and a convolutional layer after the expansion coefficient is adjusted to obtain feature maps of different scales, thereby enhancing the detection of small targets such as mice. Here, small targets refer to those smaller than 30*30 pixels. The feature maps of different scales are spliced ​​through channels and fused through 1x1 convolutional layers to improve the detection speed and accuracy of mouse targets when the increased amount of calculation is less than the threshold. The fused features are then residually fused with the input feature map and sent to the activation layer. The yolov7 with the addition of the RBF module brings a corresponding amount of calculation. After 1*1 convolution, the accuracy of mouse target detection can be improved through multi-scale features without much increase in the amount of calculation.

[0074] The detector is based on an improved yolov7 structure, wherein a receptive field module is attached to the main network of yolov7, wherein the receptive field module expands the receptive field by using convolution kernels of different sizes at different scales, and replaces the original SiLU activation layer with a Relu activation layer. The receptive field module includes a plurality of convolution layers of different scales, a splicing layer and a Relu activation layer, wherein the convolution layer after the expansion coefficient is adjusted is spliced ​​in the splicing layer and is residually connected with the first convolution layer before being fed into the Relu activation layer.

[0075] To train the detector, we first need to build a training data set. Taking into account the fact that mice may appear in the kitchen both during the day and at night, the training data set contains mouse data in visible light during the day and infrared scenes at night. The foreground mouse target image is randomly generated on different background images through text control to generate mouse target data in different positions, directions, and lighting. When the data is ready, the mouse target is annotated with the target box through an open source annotation tool to generate the coordinates of the upper left corner of the mouse target box x, y and the width and height (w, h) of the target box contained in each training sample. After the training data labels are ready, all data are preprocessed, scaled to 640*640 input size, and sent to the improved yolov7 network structure in batches for training and learning to obtain the detector with final learning convergence.

[0076] According to some embodiments, the mouse target contour in the training data set of the detector is annotated; the pixels of the mouse target contour are marked as 1, and the pixels of the non-mouse target contour are marked as 0 to obtain a binary image as the second training data label of the segmenter; all data in the second training data set are scaled.

[0077] The detector filters the target frames whose detector scores are less than the first predetermined threshold; the detector filters the target frames whose width and / or height are less than the second predetermined threshold; the detector filters the target frames whose width and / or height are greater than the third predetermined threshold. The segmenter obtains one or more mouse target contour point sets; calculates the target contour area surrounded by each mouse target contour; compares each calculated target contour area with the minimum area threshold and the maximum area threshold, and retains the target contour set that is greater than the minimum area threshold and less than the maximum area threshold; obtains the target frame data of the retained target contour set by finding the circumscribed rectangle of the contour point set.

[0078] The main function of the segmenter is to segment the foreground target of the mouse, and the other non-mouse targets are the background. Different from the training labels of the detector, the training labels of the segmenter are binary images with pixel contents of 0 and 1. Pixel content of 1 represents the mouse target, and pixel content of 0 represents the background except for the mouse target. The label data of the segmenter still uses the data used by the detector to generate segmentation labels. The mouse target outline is obtained by automatic annotation using a general segmentation tool (SAM, Segment Anything tool), and then a binary image with pixel contents of 0 and 1 is obtained. After the training labels are prepared, the training data and training labels are preprocessed, scaled to 640*640 input size, and sent to the double dilated residual network (DDRNet network) for training until the training loss function converges (dice_loss converges), and the segmenter training is completed.

[0079] In S105, it is determined whether the first target recognized by the first neural network model and the second target recognized by the second neural network model are the same target.

[0080] According to some embodiments, the target frame of the first target and the target frame of the second target are calculated to obtain an overlap intersection-and-union ratio, and the value of the overlap intersection-and-union ratio is compared with the size of a preset overlap intersection-and-union ratio threshold to determine whether the first target recognized by the first neural network model and the second target recognized by the second neural network model are the same target.

[0081] In S107, if the first target and the second target are the same target, the same target is determined to be a mouse target, and a first recognition result is obtained.

[0082] When the overlap IoU value is greater than the preset overlap IoU threshold, it means that the detector and the segmenter detect the same target. When the overlap IoU value is less than the preset overlap IoU threshold, it is considered not to be a mouse target and secondary identification is required.

[0083] In S109, if the first target and the second target are different, the first target and the second target are respectively transmitted to a classifier for secondary recognition to obtain a second recognition result.

[0084] The detector outputs the target frame and the corresponding detection score. Filter all the mouse target result frames to exclude those that are not mouse targets. Discard the target frames with a detector score less than 0.5, discard the target frames with a width less than 10 pixels and a height less than 10 pixels, and discard the target frames with a width greater than 500 pixels and a height greater than 500 pixels. The detector obtains the final valid target frame information through filtering, that is, the detection score of each target frame, the coordinates of the upper left corner point X, Y, and the width W and height H of the target frame.

[0085] After processing and analysis, the segmenter obtains a set of contour points of one or more targets. For example, each target contour point set contains multiple points such as (X1, Y1), (X2, Y2), (X3, Y3), (X4, Y4), (X5, Y5), etc. Each predicted target contour point set is first subjected to a non-target filter. First, the area S1, S2...Sn enclosed by each target contour is calculated, and then the target contour area obtained by each calculation method is compared with the minimum area threshold and the maximum area threshold, and only the target contour set greater than the minimum area threshold and less than the maximum area threshold is retained, where the minimum area threshold is 50 pixels and the maximum area threshold is 800. Finally, the target contour set that is retained is used to obtain the target frame information of each contour set by finding the circumscribed rectangle of the contour point set, that is, the coordinates X, Y of the upper left corner point and the width W and height H of the target frame.

[0086] The target frames finally retained by the detector and segmenter are integrated, and the intersection over union (IOU) of each two target frames in the two sets of results is calculated in turn, such as Figure 5 The diagonal area shows that the IOU threshold is set to 0.95. When the IOU value calculated between every two target boxes is greater than the set IOU threshold, it means that the target detected by the detector and the segmenter is the same target, that is, the mouse target is recognized, and the two target boxes are deduplicated, and only one of the target result boxes is retained.

[0087] In S107, if the first target and the second target are the same target, it is determined that the same target is a mouse target.

[0088] According to some embodiments, a secondary recognition target image is obtained through the target frame data of the first target and the target frame information of the second target; the secondary recognition target image is sent to the classifier to output the corresponding non-mouse target category score and the mouse target category score; when the mouse target category score is greater than the preset category score threshold, there is a mouse target in the current secondary recognition target image.

[0089] The classifier mainly adopts the resnet50 network structure, and the classification data is obtained by cross entropy loss training convergence. The classification training set has two labels, namely, the mouse target attribute label is 1, and the mouse target image is mainly obtained by cropping on the basis of the detector training data target frame, and the other type of non-mouse target image is obtained by randomly deducting images of random target frame sizes from the background image as non-mouse targets, which may be targets misdetected by the detector and segmenter. This type of data label attribute is 0. After the training labels are prepared, the data is preprocessed, the training data is scaled to a uniform size of 96*96, and sent to the Resnet50 network for training and learning to obtain a learned classifier model.

[0090] In step S105, when the IOU value is less than the set IOU threshold, it is considered not a mouse target and needs to be recognized again. At this time, the remaining target frames of the detector will be processed. The detector and segmenter may segment multiple targets, including those that are not real targets. If the segmenter and detector segment the same target at the same time, it is a real target. After excluding the same target detected by both detectors, the remaining target frames need to be reviewed again to avoid missed detection.

[0091] The target image is obtained through the target frame information. This target image is a small target frame image. The small target frame images are sent to the classifier for secondary judgment. Each target image is sent to the classifier to output the corresponding non-mouse target and mouse target category scores score1 and score2. The preset category score threshold is set to 0.9. When the score2 category score is greater than the set category score threshold, the target image corresponding to the target frame retained by the current detector is a true mouse target, otherwise the current target frame is discarded. Similarly, the target image cut out by the remaining target frame of the segmenter is subjected to the same operation, and only the target frame when the mouse target category score is greater than the set category score threshold is retained, and the remaining non-target frames are discarded.

[0092] In S111, whether to issue a warning is determined based on the first recognition result and the second recognition result.

[0093] The recognition results obtained in S105, S107 and S109 are summarized, and the final mouse target result box information is output as an alarm and sent to the corresponding result platform for display.

[0094] In practical applications, the accuracy of the method of the present invention is more than 90% in actual project applications, the false detection rate is less than 5%, the mouse target detection accuracy is high, and the accuracy is higher than that of single target detection algorithms or segmentation algorithms. It greatly reduces false identification and greatly reduces the false detection rate.

[0095] Figure 2 A schematic diagram illustrating a rodent infestation identification process according to an example embodiment.

[0096] The technical difficulties in identifying rat infestation mainly lie in the following aspects: first, rats often appear at night, and the images presented by surveillance cameras are infrared images. Compared with RGB image features, the features are single, and it is difficult for the feature extraction network to extract the features of rats, resulting in great difficulty in identification; second, rat targets are relatively small, and the YOLO series target detection algorithms have certain limitations in identifying small targets; third, in kitchen scenes, there are many non-target interference backgrounds, and target detection algorithms are prone to false detection; fourth, there are often relatively few rat training samples, which makes it difficult to improve the accuracy of the trained target detection model.

[0097] Based on the existing target detection methods, the method of the present invention proposes a method of dual recognition combining target detection and image semantic segmentation, which can not only improve the detection rate in actual projects, but also greatly reduce the false detection rate in the actual application process. This method adopts the improved yolov7 target detection method and the image semantic segmentation algorithm DDRNet semantic segmentation algorithm to respectively recognize the images in the video frame images, and then performs secondary processing on the results obtained by the two algorithms to obtain the final recognition result. The classifier in the figure uses the commonly used ResNet50 classification network to perform secondary recognition on the target in this method, and can review the detection target results and the segmentation target results to ensure that the recognized target is a real mouse target and reduce false detection. The purpose of using the detector and the segmenter at the same time is to improve the detection rate of the real target, detect all suspected targets separately, and play a role in improving the recognition accuracy.

[0098] See also Figure 2 , Figure 2 Shows the process of rodent infestation identification.

[0099] The video frame images are sent to the detector and the segmenter respectively for image analysis and result processing. The detector filters all the mouse target result frames detected to exclude those target frames that are not mouse targets. There are three filtering processes in the present invention. First, the target frames with a detector score less than 0.5 are discarded. If the score detection score is too small, it may be a false detection. Second, the target frame with a width of less than 10 pixels and a height of less than 10 pixels is discarded. The probability of false detection of the target frame width and height less than 10 pixels is extremely high. Third, the target frame with a width greater than 500 pixels and a height greater than 500 pixels is also discarded. Under normal circumstances, the mouse target frame detected will not be greater than 500 pixels. Greater than 500 pixels is also likely to be a false detection caused by the detector itself. The detector obtains the final effective target frame information through filtering, i.e., the detection score of each target frame, the coordinates X, Y of the upper left corner point, and the width W and height H of the target frame.

[0100] After processing and analysis, the segmenter obtains a set of contour points of one or more targets. Each predicted target contour point set is first subjected to a non-target filter. First, the area enclosed by each target contour is calculated, and then the target contour area obtained by each calculation method is compared with the minimum area threshold and the maximum area threshold. Only the target contour sets that are larger than the minimum area threshold and smaller than the maximum area threshold are retained. Those that are not in this range are considered to be mis-segmented by the segmenter. Finally, the target contour sets that are retained are used to obtain the target frame information of each contour set by finding the circumscribed rectangle of the contour point set, that is, the coordinates X, Y of the upper left corner point and the width W and height H of the target frame.

[0101] Finally, the target frames retained by the detector and segmenter are integrated. First, the intersection over union (IOU) of each two target frames in the two sets of results is calculated in turn. The diagonal area in the figure below shows that when the IOU value calculated for each two target frames is greater than the set IOU threshold, it means that the target detected by the detector and segmenter is the same target. In this case, the mouse target is considered to be identified and the two target frames are deduplicated, and only one of the target result frames is retained. When the IOU value is less than the set IOU threshold, it is considered not to be a mouse target and needs to be recognized twice. At this time, the remaining target frames of the detector will be processed, and the target image will be obtained through the target frame information. They will be sent to the classifier for secondary judgment. Each target image is sent to the classifier to output the corresponding non-mouse target and mouse target category scores score1 and score2. When the score2 category score is greater than the set category score threshold, the target image corresponding to the target frame retained by the current detector is a true mouse target, otherwise the current target frame is discarded. Similarly, the target image cut out by the segmenter is subjected to the same operation, and only the target frame whose mouse target category score is greater than the set category score threshold is retained, and the remaining non-target frames are discarded. Finally, the final mouse target result frame information of a frame of video image is output as an alarm and sent to the corresponding result platform for display.

[0102] Figure 3a A schematic diagram illustrating a receptive field module according to an example embodiment is shown.

[0103] Figure 3b A schematic diagram showing a detector network structure according to an example embodiment.

[0104] Figure 3c Schematic diagram showing a detailed explanation of a detector network structure according to an example embodiment.

[0105] According to some embodiments, the detector module is improved on the structure of the yolov7 algorithm, which is improved on the basis of the original network structure.

[0106] For the yolov7 feature extraction network, a receptive field block (RBF) is added, such as Figure 3a As shown in the figure, by using convolution kernels of different sizes at different scales, the receptive field is expanded, which is more conducive to the detection of small targets and makes up for the defect of the original YOLOv7 target detection algorithm that is insufficient for small target detection. After multiple convolution layers and convolution layers with adjusted expansion coefficients extract features from the feature map of the previous layer input, the feature maps of different scales are obtained for channel splicing, and then feature fusion is performed through a 1x1 convolution layer, and finally residual fusion is performed with the feature map of the previous layer input and sent to the activation layer.

[0107] For the activation layer, remove the original SiLU activation layer and use the relatively less time-consuming ReLu to accelerate training. The SiLu activation layer is more time-consuming on hardware, while the Relu layer is not so time-consuming. Sppcspc is a module that combines spatial pyramid pooling (SPP) and cross-stage partial network (CSPNet). SPPCSPC effectively extracts and fuses multi-scale features by adding multiple parallel maximum pooling operations to a series of convolutions and combining the structure of the cross-stage partial network (CSPN). The final detector network structure is as follows: Figure 3b shown.

[0108] See also Figure 3c , Figure 3c The detailed composition of the detector network structure is shown. CBS is specifically composed of convolutional layers, BN (Batch Normalization) batch normalization layers, and Relu (activation layers). The ELAN module is composed of multiple CBS modules connected together. Cat (Concatention) is a concatenation layer, and the ELAN-H module is composed of multiple CBS modules connected in different ways. The UP module is composed of a CBS module and an unsample (up-sampling layer).

[0109] After the detector is designed, it is trained. To train the detector, a training data set must be constructed first. The training data set takes into account that in actual applications, mice may appear in the kitchen both during the day when no one is there and at night when no one is there. Therefore, the training data set contains mouse data in visible light during the day and infrared scenes at night, so that the detector scene generalization ability learned by training is stronger. Considering that there are relatively few mouse data samples, especially infrared data at night is even less and difficult to obtain, the present invention uses the Inpait technology in the Stable Diffusion (SD) image generation technology to generate offline data. The main method uses the foreground target image to randomly generate mouse target data in different positions, directions, and lighting on different kitchen background images through text control, which greatly increases the data samples and greatly reduces the difficulty of obtaining data samples.

[0110] Figure 4 A schematic diagram illustrating a splitter network structure according to an example embodiment.

[0111] See also Figure 4 ,The segmenter module adopts the DDRNet network structure for training, and the input also adopts a relatively large ,640*640 input size to ensure the segmentation accuracy of the segmenter.,Among them, RB is the residual basic module, and RBB is the residual module with bottelneck.

[0112] RB is a residual structure module composed of two 3x3 convolutional layers. RBB is a residual structure composed of convolution with a bottleneck layer, that is, three convolutional layers, 1x1, 3x3 and 1x1 convolutional layers. Seg.Head segmentation head is composed of convolution. Loss function in training. This picture shows the network structure of DAPPM module.

[0113] The training label of the segmenter is a binary image with pixel contents of 0 and 1. Pixel content 1 indicates a mouse target, and pixel content 0 indicates the background except for the mouse target. The label data of the segmenter is still generated by using the data used by the detector. The mouse target outline is obtained by automatic annotation using the SAM (Segment Anything) tool, and then a binary image with pixel contents of 0 and 1 is obtained. The purpose of this is to reduce unnecessary manual annotation costs. After the training labels are prepared, the training data and training labels are preprocessed, scaled to 640*640 input size, and sent to the DDRNet network for training until the training dice_loss converges and the segmenter training is completed.

[0114] The loss function in the image segmentation task is:

[0115]

[0116] Among them, the X value is the result of the model's forward inference, which is the predicted pixel value, and Y is the true label, the true pixel value. It is based on the Dice coefficient (also known as -Dice coefficient or F1 score), which is used to measure the similarity between two sets. In the context of deep learning, Dice Loss is mainly used to evaluate the overlap between the predicted segmentation map and the true label.

[0117] In order to convert the Dice coefficient into a loss function, we usually take its opposite or 1 minus the Dice coefficient, so that when the prediction result is exactly the same as the true label, the loss value is 0; conversely, the greater the difference, the higher the loss value.

[0118] Figure 6 A block diagram of a computing device is shown according to an exemplary embodiment.

[0119] like Figure 6 As shown, computing device 30 includes processor 12 and memory 14. Computing device 30 may also include bus 22, network interface 16, and I / O interface 18. Processor 12, memory 14, network interface 16, and I / O interface 18 may communicate with each other via bus 22.

[0120] The processor 12 may include one or more general-purpose CPUs (Central Processing Units, processors), microprocessors, or application-specific integrated circuits, etc., for executing relevant program instructions. According to some embodiments, the computing device 30 may also include a high-performance graphics card (GPU) 20 for accelerating the processor 12.

[0121] The memory 14 may include a machine system readable medium in the form of a volatile memory, such as a random access memory (RAM), a read-only memory (ROM) and / or a cache memory. The memory 14 is used to store one or more programs including instructions and data. The processor 12 can read the instructions stored in the memory 14 to execute the above-mentioned method according to the embodiment of the present invention.

[0122] The computing device 30 may also communicate with one or more networks via the network interface 16. The network interface 16 may be a wireless network interface.

[0123] The bus 22 may include an address bus, a data bus, a control bus, etc. The bus 22 provides a path for exchanging information between components.

[0124] It should be noted that, in the specific implementation process, the computing device 30 may also include other components necessary for normal operation. In addition, those skilled in the art may understand that the above device may only include components necessary for implementing the embodiments of this specification, and need not include all components shown in the figure.

[0125] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a microdrive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), a network storage device, a cloud storage device, or any type of medium or device suitable for storing instructions and / or data.

[0126] An embodiment of the present invention further provides a computer program product, which includes a computer program. The computer program is operable to enable a computer to execute part or all of the steps of any one of the methods described in the above method embodiments.

[0127] Those skilled in the art can clearly understand that the technical solution of the present invention can be implemented with the help of software and / or hardware. "Unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array, an integrated circuit, etc.

[0128] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0129] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided by the present invention, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0134] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0135] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended clauses.

Claims

1. A method for monitoring rodent infestation, the method comprising: Acquire infrared video frame images; The video frame image is transmitted to the first neural network model and the second neural network model for target recognition respectively; Determining whether a first target recognized by the first neural network model and a second target recognized by the second neural network model are the same target; If the first target and the second target are the same target, determining that the same target is a mouse target, and obtaining a first recognition result; If the first target and the second target are different, the first target and the second target are respectively transmitted to the classifier for secondary recognition to obtain a second recognition result; Determine whether to issue an early warning based on the first recognition result and the second recognition result.

2. The method according to claim 1, characterized in that The first neural network model is a detector based on a neural network target detection algorithm, and the second neural network model is a segmenter based on a neural network image semantic segmentation algorithm.

3. The method according to claim 2, characterized in that The detector is based on an improved yolov7 structure, in which a receptive field module is attached to the yolov7 backbone network, and the original SiLU activation layer is replaced by a Relu activation layer to accelerate training. The receptive field module expands the receptive field by using convolution kernels of different sizes at different scales.

4. The method according to claim 3, characterized in that The receptive field module includes a plurality of convolutional layers, concatenation layers and ReLU activation layers of different scales, wherein the video frame image is passed into the first neural network model for target recognition, including: The input feature map is subjected to feature extraction through multiple convolutional layers of different scales and convolutional layers with adjusted expansion coefficients to obtain feature maps of different scales, thereby enhancing the detection of small targets such as mice, which are smaller than 30*30 pixels. The feature maps of different scales are spliced ​​and fused through 1x1 convolutional layers to improve the detection speed and accuracy of mouse targets while the increased computational effort is less than the threshold. The fused features are then residually fused with the input feature map and sent to the activation layer.

5. The method according to claim 1, characterized in that After the video frame image is passed into the detector and the segmenter for target recognition, the following steps are also included: The detector filters target frames whose detector scores are less than a first predetermined threshold; The detector filters the target frames whose width and / or height are smaller than a second predetermined threshold; The detector filters the target frames whose width and / or height are greater than a third predetermined threshold.

6. The method according to claim 1, characterized in that After the video frame image is passed into the detector and the segmenter for target recognition, the following steps are also included: The segmenter obtains one or more mouse target contour point sets; The target contour area enclosed by each mouse target contour was calculated; Compare each calculated target contour area with a minimum area threshold and a maximum area threshold, and retain a set of target contours that are larger than the minimum area threshold and smaller than the maximum area threshold; By finding the circumscribed rectangle of the contour point set, the target frame data of the retained target contour set is obtained.

7. The method according to claim 1, characterized in that Determining whether the first target identified by the detector and the second target identified by the segmenter are the same includes: Calculate the overlap intersection-over-union ratio of the target frame of the first target and the target frame of the second target; When the overlap intersection-and-union ratio value is greater than a preset overlap intersection-and-union ratio threshold, it indicates that the detector and the segmenter have detected the same target; When the overlap intersection-and-union ratio value is less than a preset overlap intersection-and-union ratio threshold, a secondary identification is required.

8. The method according to claim 1, characterized in that If the first target and the second target are different, the first target and the second target are passed into the classifier for secondary recognition to obtain a final recognition result, including: Acquire a secondary recognition target image through the target frame data of the first target and the target frame information of the second target; The secondary recognition target image is sent to the classifier to output the corresponding non-mouse target category score and mouse target category score; When the mouse target category score is greater than the preset category score threshold, there is a mouse target in the current secondary recognition target image.

9. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

10. A computing device, characterized in that: include: processor; as well as A memory storing a computer program, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

Images