Automatic inspection beehive

Abstract

The present disclosure proposes an automatic inspection beehive, including a housing, a comb frame, an image capturing device, and a computing module. The comb frame is arranged within the housing. The image capturing device is disposed on the comb frame and is configured to capture a digital image of the interior of the housing. The computing module executes a machine learning model based on the digital image to determine whether the digital image belong to one of multiple categories.

Description

Technical Field

[0001] This disclosure relates to a peak box that utilizes artificial intelligence technology to achieve automated inspection. Prior Technology

[0002] Bees play a vital role in precision and eco-friendly agriculture. Before large-scale human intervention in agricultural production, bees were key pollinators in the natural food chain, having an indispensable impact on crop growth and reproduction. Statistics show that approximately one-third of the world's crops rely on bee pollination, and in Taiwan, more than forty crop varieties are pollinated by bees. According to data from the Agriculture and Food Agency and the Taiwan Beekeeping Association, the number of beehives in Taiwan increased steadily from approximately 98,000 in 2010 to approximately 159,000 in 2019, and further reached approximately 180,000 in 2020, with an annual honey production of 8,387 metric tons. This demonstrates the stable development of the beekeeping industry in Taiwan and its significant contribution to agricultural production.

[0003] However, current beekeeping management techniques still primarily rely on manual inspection of beehives to ensure the health of bee colonies and honey production. Traditional inspection methods require beekeepers to open each hive individually to check the honeycomb cells and assess the health of the bees and the storage condition of the honey. Therefore, existing technologies still face many challenges in bee colony monitoring and management, necessitating more efficient solutions to enhance the automation and intelligent management capabilities of the beekeeping industry. Summary of the Invention

[0004] To address the aforementioned problems, this disclosure proposes an automated inspection beehive, comprising a hive body, honeycomb sheets, a first image capturing device, and a computing module. The honeycomb sheets are disposed within the hive body. The first image capturing device is disposed on the honeycomb sheets and is used to capture digital images of the interior of the hive body. The computing module is communicatively connected to the first image capturing device and is used to acquire the digital images and execute a machine learning model based on the digital images to determine whether the digital images belong to one of several categories.

[0005] In one embodiment of this disclosure, the above categories include queen bees, bee mites, and incompletely capped bees.

[0006] In one embodiment of this disclosure, the aforementioned automatic inspection beehive further includes a thermal imaging camera mounted on the hive panel to acquire thermal images. When the machine learning model outputs a category belonging to bee mites, the calculation module obtains the location of the bee mites and reads the corresponding temperature value in the thermal image based on this location. The calculation module also determines whether the temperature value is less than a temperature threshold; if so, it confirms the presence of bee mites; otherwise, it determines that the digital image does not contain bee mites.

[0007] In one embodiment of this disclosure, the aforementioned machine learning model is used to segment a digital image to extract the queen bee region from the image. A computational module is used to fit an ellipse to the queen bee region to determine multiple parameters of the ellipse.

[0008] In one embodiment of this disclosure, the above-mentioned calculation module is used to calculate the image length and image width based on the parameters of the ellipse, and to calculate the corresponding real length and real width based on the conversion ratio.

[0009] In one embodiment of this disclosure, the aforementioned computational module is further configured to: binarize the queen bee region to obtain a binarized image; perform a high-pass filter on the binarized image to obtain an edge image; obtain multiple segments of the edge image; for each segment, approximate the segment into multiple endpoints; calculate the cross product vector between two line segments formed by the endpoints to adjust the direction of the endpoints; determine whether the endpoints between two line segments are convex or concave based on the sign of one component in the cross product vector; if the angle between two line segments is greater than a critical angle value and the endpoints between two line segments are convex, then delete the endpoints between these two line segments; and perform an optimization algorithm on the remaining endpoints to determine the parameters of the ellipse.

[0010] In one embodiment of this disclosure, the aforementioned automatic inspection beehive further includes a top cover for covering the beehive body. Additionally, a solar panel is disposed on the upper surface of the top cover.

[0011] In one embodiment of this disclosure, the aforementioned automatic inspection beehive further includes a second image capturing device, which is disposed on one side of the top cover and faces the entrance and exit of the beehive.

[0012] In one embodiment of this disclosure, the first image capturing device described above is an infrared camera.

[0013] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings for detailed explanation. Simple Explanation of the Diagram

[0014] Figure 1 is a schematic diagram illustrating an automated inspection beehive according to one embodiment. Figure 2 is a schematic diagram of an automatic inspection peak box system according to one embodiment. Figure 3 is a schematic diagram illustrating digital and thermal images according to one embodiment. Figure 4 is a flowchart illustrating the calculation of the queen bee's length and width according to one embodiment. Figure 5 is a schematic diagram illustrating edge smoothing processing according to one embodiment. Figure 6 is a flowchart illustrating the operation of an automated inspection beehive according to one embodiment. Implementation

[0015] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered identical or similar when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the present invention's patent application.

[0016] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0017] Figure 1 is a schematic diagram illustrating an automatic inspection hive according to an embodiment. Referring to Figure 1, the automatic inspection hive includes a hive body 110, a top cover 120, a computing module 121, an image capturing device 130, honeycomb panels 140, image capturing devices 141 and 142, and a thermal imaging camera 143.

[0018] The hive 110 has a receiving space, and honeycomb sheets 140 are disposed within the receiving space of the hive 110. In this embodiment, image capturing devices 141 and 142 are disposed on both sides of the honeycomb sheet 140, while a thermal imaging camera 143 is disposed on one side. However, the present invention does not limit the number or placement of the image capturing devices 141 and 142 and the thermal imaging camera 143. Other honeycomb sheets may also be disposed within the receiving space of the hive 110; these honeycomb sheets contain honeycomb but do not have components such as the image capturing device 141. The honeycomb sheet 140 may also be referred to as a "pseudo-honeycomb sheet," and its purpose is to monitor the condition inside the hive.

[0019] The top cover 120 is used to cover the housing 110. In some embodiments, a solar panel 122 is provided on the upper surface of the top cover 120. The thickness of the top cover 120 is greater than that of a typical beehive top cover to accommodate components such as batteries and computing modules 121. In addition, an image capturing device 130 is provided on one side of the top cover 120. This image capturing device 130 captures images facing the entrance 112 of the housing 110, through which bees 131 will pass.

[0020] Figure 2 is a schematic diagram of an automatic inspection peak box system according to one embodiment. Referring to Figure 2, the computing module 121 is communicatively connected to image capturing devices 141, 142, and 130, a thermal imaging camera 143, a communication module 210, and an infrared light source 220. Here, the communication connection can refer to any wired or wireless communication method.

[0021] The computing module 121 may include a central processing unit, a graphics processing unit, a microprocessor, a microcontroller, an image processing chip, a deep-learning processing unit (DPU), a neural network processing unit (NPU), a tensor processing unit (TPU), application-specific integrated circuits (ASICs), and a programmable logic device (PLD). In this embodiment, the computing module 121 is disposed within the upper cover 120; therefore, the computing module 121 may also be referred to as an edge computing unit. In other embodiments, the computing module 121 may also be a server in the cloud.

[0022] Image capturing devices 141, 142, and 130 may include a charge-coupled device (CCD) sensor, a complementary metal-oxide-semiconductor (CMOS) sensor, or other suitable photosensitive elements. In some embodiments, image capturing devices 141, 142, and 130 may also include dual cameras, a structured light sensing device, a laser, or any element capable of sensing scene depth. In this embodiment, image capturing devices 141 and 142 are infrared cameras, while image capturing device 130 is a visible light camera. In other embodiments, image capturing devices 141 and 142 may also be visible light cameras, and this invention is not limited thereto. In some embodiments, image capturing devices 141 and 142 may employ replaceable miniature cameras, thereby facilitating their placement on chip 140. The infrared light source 220, for example, is a light-emitting diode (LED) used to provide infrared light, thereby increasing the brightness of the captured image.

[0023] The communication module 210 may include circuitry supporting communication functions such as cellular networking (or mobile network), near-field communication, infrared communication, Bluetooth, and Wi-Fi. In some embodiments, the image capturing devices 141, 142, and 130, and the thermal imaging camera 143 use memory cards with Wi-Fi functionality, and the communication module 210 may also be the communication circuitry on these memory cards. When the computing module 121 is implemented as a server in the cloud, the images captured by the image capturing devices 141, 142, and 130, and the thermal imaging camera 143 can be transmitted to the computing module 121 via the communication module 210.

[0024] Figure 3 is a schematic diagram illustrating a digital image and a thermal image according to one embodiment. Referring to Figures 2 and 3, taking the digital image 310 captured by the image capturing device 141 as an example, this digital image 310 is about the interior of the housing 110, and shows, for example, the condition of another chip. The thermal imaging camera 143 is used to capture a thermal image 320 about the interior of the housing 110, the field of view of which at least partially overlaps with the field of view of the digital image 310.

[0025] The computing module 121 executes a machine learning model based on the digital image 310 to determine whether the digital image 310 belongs to one of several categories. This machine learning model may include, for example, a convolutional neural network, the architecture of which may employ LeNet, AlexNet, VGG, GoogLeNet, ResNet, DenseNet, or YOLO (You Only Look Once), etc., and this invention is not limited thereto. On the other hand, the aforementioned categories may include queen bee, bee mites, incompletely capped bees, bee with incomplete capping, complete capping, worker bees, drones, etc. In some embodiments, the machine learning model is used to perform segmentation; therefore, the output of the machine learning model is an image, where each pixel indicates the aforementioned category, for example, "1" represents queen bee, and "2" represents bee mites. In other embodiments, the machine learning model may also perform classification or detection, and thus the machine learning model may output one or more numerical values ​​and one or more bounding box coordinates, the numerical values ​​representing the probability (or confidence value) of the corresponding category occurring, and the bounding boxes indicating the location where the corresponding category occurs.

[0026] When bees are infected with the bee mites, their body temperature is lower. Therefore, when the machine learning model outputs a category belonging to bee mites, thermal image 320 can be used for secondary confirmation. The bee mites are relatively small compared to bees. During the training phase, bees infected with bee mites are marked; therefore, during inference, the machine learning model outputs the segmentation result or bounding box of the bee infected with bee mites. The calculation module 121 first obtains the location marked as a bee mite in the digital image 310, and then reads the temperature value at the corresponding location in the thermal image 320. The calculation module 121 determines whether this temperature value is less than a temperature threshold; if so, it confirms the presence of bee mites; otherwise, it determines that the bee does not belong to this category. In some embodiments, since it is not easy to determine the shape of the bee from the thermal image 320, inputting the thermal image 320 into the machine learning model will not yield better results; therefore, the thermal image 320 is used to secondary confirm the output of the machine learning model.

[0027] In some embodiments, thermal images 320 can also assist in generating training labels. For example, a model with low accuracy can be built first to detect bee mites, and then thermal images can be used to filter out false positives before manual labeling, thus reducing the burden of manual labeling.

[0028] In some embodiments, after the machine learning model segments the digital image, one of the classifications is the queen bee (also known as the queen bee). The segmentation result output by the machine learning model indicates the location of the queen bee, called the queen bee region. Next, the calculation module 121 fits an ellipse to the queen bee region to determine the parameters of the ellipse, based on which the length and width of the queen bee can be calculated. Figure 4 is a flowchart illustrating the calculation of the queen bee's length and width according to one embodiment. Referring to Figure 4, this process includes steps 401 to 407. Images 412 to 416 are also shown in Figure 4, representing the processing results of steps 402 to 406, respectively.

[0029] In step 401, the digital image is input into the machine learning model. In step 402, it is determined whether the queen bee is detected; for example, the cropping results may indicate whether any pixels are classified as queen bees.

[0030] If the queen bee is detected, the queen bee region is obtained in step 403 (as shown in image 413). For example, the cutting result can be regarded as a mask, and image 413 can be obtained by performing an element-wise AND operation on this mask and the digital image.

[0031] In step 404, the queen bee region is binarized to obtain a binarized image 414. For example, in the binarized image 414, pixels belonging to the queen bee region can be set to "1", and the rest can be set to "0".

[0032] In step 405, a high-pass filter is applied to the binarized image 415 to obtain the edge image 415. In this embodiment, the high-pass filter is a Sobel filter, but the present invention is not limited thereto.

[0033] In step 406, an ellipse is fitted to the edge image 415, thereby determining several parameters of the ellipse. The parameters of the ellipse include the X coordinate, Y coordinate, angle, semi-major axis, and semi-minor axis. In some embodiments, since the segments in the edge image 415 are not smooth, which may cause errors, some smoothing processing may be performed first.

[0034] Figure 5 is a schematic diagram illustrating edge image smoothing according to one embodiment. First, multiple segments 501-503 are obtained. In some embodiments, a pixel value in the edge image 415 is considered an edge if it is greater than or less than 0, and not if it is equal to 0. Here, each pixel belonging to an edge can be connected to one of its eight surrounding pixels that also belong to edges to form a segment. In other words, different segments do not have adjacent pixels that are also edges.

[0035] Next, for each segment, the segment is approximated as multiple endpoints. Taking segment 503 as an example, segment 503 can be approximated as endpoints P1 to P6. In some embodiments, two points can be taken starting from one end of segment 503, and the equation of the line between these two points can be calculated. Next, the distance from the third point to the equation of the line is calculated. If this distance is less than a distance threshold, it is determined that they belong to the same line segment; otherwise, the third point is regarded as an endpoint, and starting from the fourth point, they belong to the next line segment, and so on until all points in segment 503 have been processed. In other words, an additional endpoint will be added where the curvature of segment 503 is large. In some embodiments, the endpoint can also be determined based on the curvature. In the example of Figure 5, endpoints P1 and P2 form a line segment S1; endpoints P2 and P3 form a line segment S2; endpoints P3 and P4 form a line segment S3; endpoints P4 and P5 form a line segment S4; and endpoints P5 and P6 form a line segment S5.

[0036] Next, the cross product between two line segments is calculated to adjust the orientation of these endpoints. Assume the coordinates of endpoint P1 are (x1, y1), endpoint P2 are (x2, y2), and endpoint P3 are (x3, y3). Then, setting the Z-coordinate to 0, line segment S1 can be represented as the vector (x2-x1, y2-y1, 0), and line segment S2 can be represented as the vector (x3-x2, y3-y2, 0). The cross product between line segments S1 and S2 is the vector (0, 0, (x2-x1)(y3-y2)-(x3-x2)(y2-y1)), called the cross product vector. Furthermore, the sign of the third component in the cross product vector indicates the direction of the Z-axis, representing whether endpoints P1 to P3 are ordered clockwise or counterclockwise. For every two line segments, the order can be determined, and then the order of segment 503 is decided by voting. This assumes all segments should be ordered counter-clockwise. If a segment is clockwise, the order of all endpoints in that segment can be reversed. For example, reversing the order of endpoints P1 to P6 results in endpoints P6 to P1.

[0037] After adjusting the direction of the endpoints, the sign of the third component in the outer product vector determines whether an endpoint is convex or concave. In this example, if the third component is less than 0, it is concave; if it is greater than 0, it is convex. For example, endpoint P2 is concave, and endpoint P4 is convex. In this case, the concave endpoint needs to be retained, and the convex endpoint needs to be deleted. Since segment 503 may have slight distortion, an angle threshold can be set to filter these convex and concave points. If the angle between two line segments is greater than this angle threshold and the endpoint between these two line segments is convex, then this endpoint is deleted. For example, the angle between line segments S3 and S4 is greater than the angle threshold, and endpoint P4 is convex, so endpoint P4 can be deleted. After deleting endpoint P4, segment 503 is left with line segments S1, S2, and S5. In some embodiments, the shorter line segment S5 can be further deleted, while the longer and continuous line segments S1 and S2 are retained. In this embodiment, line segment S5 is retained, so segment 503 will have endpoints P1~P3, P5, and P6 remaining.

[0038] After performing the above processing on all segments 501-503, an optimization algorithm is executed on the remaining endpoints to determine the parameters of the ellipse. For example, the parameters of the ellipse include the center coordinates (h, k) and angles. The semi-major axis a and the semi-minor axis b are given. The optimization algorithm is shown in the following mathematical formula 1. [Mathematical Expression 1]

[0039] Where d(x,y) represents the distance from a given endpoint (x,y) to the ellipse, such as the Euclidean distance. Here, all endpoints are substituted into equation 1 to calculate the distances, which are then summed to find a set of parameters (h,k). Let a, b, and make the accumulated distance have a minimum value.

[0040] The above approach first divides the ellipse into multiple segments, approximating them as multiple endpoints, and then removes the convex points. As a result, the number of remaining endpoints is smaller, which can reduce the time required for the optimization algorithm. In addition, the removal of noise (convex points) also results in more accurate results.

[0041] Referring back to Figure 4, in step 407, the true length and true width are calculated based on the parameters of the ellipse. Specifically, the major axis of the ellipse is 2a (called the image length), and the minor axis is 2b (called the image width). The units for this image length and image width are the number of pixels. Next, the true length corresponding to the image length and the true width corresponding to the image width can be calculated based on a conversion ratio. For example, a calibration procedure can be performed on the image capturing device 141 beforehand to calculate the ratio between a unit length (e.g., centimeters) and the number of pixels in the real world. This calibration procedure can be obtained by placing a ruler and a frame in the scene and then having the image capturing device 141 capture the image of the ruler. In other embodiments, parameters such as the focal length of the image capturing device 141 can also be calculated using computer vision or other means to convert the real-world coordinates to image coordinates. The unit of the aforementioned conversion ratio is, for example, centimeters / pixels; therefore, 2a can be multiplied by the conversion ratio to obtain the true length, and 2b can be multiplied by the conversion ratio to obtain the true width. The length and width of the queen bee can be calculated using the process shown in Figure 4.

[0042] Figure 6 is a flowchart illustrating the operation of an automated beehive inspection system according to one embodiment. Referring to Figure 6, in step 601, a digital image is acquired from the image capturing device. In step 602, a machine learning model is executed based on the digital image. If the output of the machine learning model indicates that the digital image belongs to the bee-crab-man category, a secondary confirmation is performed using thermal imaging in step 603. If the output of the machine learning model indicates that the digital image belongs to the queen bee category, the queen bee's size is calculated using elliptic fitting in step 604. In some embodiments, the digital image captured by the image capturing device 130 located at the entrance / exit can also be input into the machine learning model to determine the number of bees passing through the entrance / exit per hour. After steps 603 and 604, or after determining that the bee belongs to another category, in step 605, the relevant judgment results and data can be uploaded to a server or a user's device (e.g., a mobile phone). The user can view this information through an application or webpage to know the condition inside the beehive. If the beehive contains a queen bee, it means that the bee colony is still laying eggs and is a healthy colony. If a bee colony is found inside the hive, it indicates that the colony is infected, and the system will alert the user to check. An incompletely capped hive refers to a hole in a cell, likely caused by a bee colony passing through. Additionally, a warning will be issued if too few bees pass through the entrance / exit, prompting the user to check. This method eliminates the need for the user to manually open the hive, reducing beekeeping costs and providing an objective way to assess the colony's condition.

[0043] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0044] 110: Box 112: Entrance / Exit 120: Top Cover 121: Computing Module 122: Solar panels 130, 141, 142: Image capturing device 131: Bee 140: Nestlings 143: Thermal imaging camera 210: Communication Module 220: Infrared light source 310: Digital Image 320: Thermal Image 401~407, 601~605: Steps 412~416: Images 501~503: Excerpts P1~P6: Endpoints S1~S5: Line segments

Claims

1. An automatic inspection beehive, comprising: a hive body; a honeycomb plate for placement within the hive body; a first image capturing device disposed on the honeycomb plate for capturing a digital image of the interior of the hive body; and a computing module communicatively connected to the first image capturing device for acquiring the digital image and executing a machine learning model based on the digital image to determine whether the digital image belongs to one of multiple categories, wherein the machine learning model is further configured to segment the digital image to extract a queen region from the digital image, wherein the computing module is configured to fit an ellipse to the queen region to determine multiple parameters of the ellipse, comprising: performing binarization on the queen region to obtain a binarized image; performing a high-pass filter on the binarized image to obtain an edge image; acquiring multiple segments of the edge image; and approximating each of the segments as multiple endpoints; Calculate the cross product vector between two line segments formed by at least three of the endpoints, and adjust the order of the endpoints by clockwise or counterclockwise rotation based on the sign of one component of the cross product vector; determine whether the endpoint between the two line segments is a convex or concave point based on the sign of the component in the cross product vector; delete the endpoint between the two line segments if the angle between the two line segments is greater than a critical angle value and the endpoint between the two line segments is a convex point; and perform an optimization algorithm on the remaining endpoints to determine the parameters of the ellipse.

2. The automated inspection hive as described in claim 1, wherein the categories include queen bees, bee mites, and incompletely capped hives.

3. The automatic inspection beehive as described in claim 2 further includes: A thermal imaging camera is installed on the nest to acquire a thermal image. When the category output by the machine learning model belongs to the bee crab mite, the computing module is used to obtain a location of the bee crab mite and read a corresponding temperature value in the thermal image based on the location. The computing module is used to determine whether the temperature value is less than a temperature threshold. If it is, the bee crab mite is confirmed; otherwise, the digital image is determined not to contain the bee crab mite.

4. The automatic inspection beehive as claimed in claim 1, wherein the calculation module is used to calculate an image length and an image width based on the parameters, and to calculate an actual length corresponding to the image length and an actual width corresponding to the image width based on a conversion ratio.

5. The automatic inspection beehive as claimed in claim 1 further comprises: a top cover for covering the beehive body; and a solar panel disposed on the upper surface of the top cover.

6. The automatic inspection beehive as described in claim 5 further includes: a second image capturing device disposed on one side of the top cover and facing an entrance / exit of the beehive body.

7. The automatic inspection beehive as described in claim 1, wherein the first image capturing device is an infrared camera.

Images