Leaf similarity measurement method and device based on semantic segmentation and key point detection

Abstract

The application discloses a kind of based on semantic segmentation and key point detection's leaf similarity measurement method and device, comprising: the leaf image collected is labeled, obtains the leaf image labeled well;The leaf image labeled well is input into the fusion network model established in advance and is trained, obtains the fusion network model trained, and obtains the contour information and both end point information of leaf image;The fusion network model is using ResNet as the feature information of backbone network extraction input image, including Mask R-CNN branch and Simple Baselines branch;According to the contour information and both end point information of the leaf image, scale normalization is carried out to leaf image, and the normalized leaf image is obtained;The leaf image of scale normalization is registered and aligned, and similarity is calculated.The application obtains leaf contour information and petiole point and leaf tip point information, and is registered and aligned respectively, to improve the accuracy of similarity.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target recognition in computer vision, and particularly relates to a leaf similarity measurement method and device based on semantic segmentation and key point detection. BACKGROUND

[0002] Leaf similarity refers to comparing the similarity of two leaves in terms of shape, size, texture and other characteristics in biology or botany. Usually, a number is used to represent the similarity of two leaves, and the higher the value is, the more similar the two leaves are. In botany, leaf similarity is an important research content, which can be used to help distinguish different plant varieties and identify whether plants belong to the same species. The calculation of leaf similarity usually needs the help of computer technology, which is called "leaf image recognition".

[0003] Currently, this field mainly includes: 1) feature extraction, studying how to extract more accurate and rich leaf features from leaf images to improve the accuracy of leaf similarity calculation; 2) similarity calculation, studying how to design more effective and accurate leaf similarity calculation methods to improve the accuracy of leaf recognition; 3) dataset construction, studying how to construct more complete and rich leaf image datasets to support more extensive leaf recognition applications; 4) application scenarios, studying how to apply leaf recognition technology to more extensive botanical fields, such as plant germplasm resource identification, plant growth monitoring, plant disease diagnosis, etc.

[0004] The disadvantage of such devices is that the difference between the two ends of the leaf and the distance during the shooting process are not fully considered during the leaf similarity comparison process, which leads to a large difference in similarity.

[0005] Therefore, it is necessary to provide a leaf similarity measurement system and method based on semantic segmentation and key point detection. SUMMARY

[0006] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a leaf similarity measurement method and device based on semantic segmentation and key point detection, which improves the accuracy of similarity by determining the leaf petiole point and the leaf tip point and aligning them respectively.

[0007] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0008] The present application provides a leaf similarity measurement method based on semantic segmentation and key point detection, which includes the following steps:

[0009] The collected leaf images are labeled to obtain labeled leaf images;

[0010] The annotated leaf image is input into a pre-established fusion network model for training, a trained fusion network model is obtained, and contour information and two-end point information of the leaf image are obtained; the fusion network model uses ResNet as a backbone network to extract feature information of the leaf image, includes a Mask R-CNN branch and a SimpleBaselines branch, the Mask R-CNN branch is used to obtain the contour information of the leaf, and the Simple Baselines branch is used to obtain the two-end point information of the leaf;

[0011] According to the contour information and the two-end point information of the leaf image, the leaf image is subjected to scale normalization to obtain a normalized leaf image.

[0012] The scale-normalized leaf image is registered and aligned, and a similarity is calculated.

[0013] As a preferred technical solution, the annotated content includes leaf contour information and leaf two-end points; the leaf two-end points are petiole points and leaf tip points.

[0014] As a preferred technical solution, the annotated leaf image is input into a pre-established fusion network model for training, and the specific training process is as follows:

[0015] An original image and annotated information are input, and a set of feature maps is obtained through a feature extraction network;

[0016] For the Mask R-CNN branch: an RPN network is used to generate a candidate region on the feature map, for each candidate region, an RoIAlign layer is used to align it to a fixed-size feature block, and is sent to two parallel branches; in the classification and regression branch, a fully connected layer is used to output the class and boundary box correction value of each candidate region; in the mask branch, a full convolution layer is used to output the pixel-level mask of each candidate region, and a 1x1 convolution layer and a pixel-level Softmax function are used to obtain the probability of each pixel belonging to different categories;

[0017] For the Simple Baselines branch: a convolution layer is used to output a heat map corresponding to the key point category on each feature map, and is normalized through a softmax function; the maximum value position on each heat map is found as the predicted key point coordinate, and is refined through Gaussian peak adjustment;

[0018] The predicted result is compared with the real label, and a loss function is calculated;

[0019] Using the calculated loss function, the weights and biases of the existing model are updated using a backpropagation algorithm;

[0020] The model is trained iteratively until the preset stopping condition is met; the trained fusion network model is obtained, and the contour information and end point information of the leaf image are obtained accordingly.

[0021] As a preferred technical solution, the scale normalization of the leaf images aims to make the pixel distance between the petiole point and the leaf tip point consistent across multiple leaf images. Specifically:

[0022] First, let the two leaf images be A and B, with widths w respectively. A and w b The heights are h A and h B Let the coordinates of the two endpoints on image A be (x1, y1) and (x2, y2), and the coordinates of the two endpoints on image B be (x3, y3) and (x4, y4). Then the pixel distance between the two endpoints on the two leaf images is as follows:

[0023]

[0024] Secondly, A or B needs to be scaled so that d A and d B To be equal, assuming A is scaled, we need to find a scaling factor s such that: s*d A =d B That is, s = d B / d A That is, scaling A by a scaling factor of s.

[0025] As a preferred technical solution, the registration and alignment of the scale-normalized leaf image specifically involves:

[0026] Let there be two leaf images, A and B. Suppose that image A is processed so that the petiole point and leaf tip point of the leaf in image A are aligned with the petiole point and leaf tip point of the leaf in image B, respectively.

[0027] Petiole point registration and alignment: Let the pixel coordinates of the petiole point of the leaf blade in image A be (x,y) and the pixel coordinates of the petiole point of the leaf blade in image B be (x′,y′). To make the pixel coordinates of the two petiole points the same, the leaf blade in image A needs to be translated along the x-axis by Δx = x′ - x and along the y-axis by Δy = y′ - y, so that the petiole points of the leaf blades in the two images are at the same pixel coordinate position.

[0028] Leaf tip registration and alignment: Let the pixel coordinates of the leaf tip in image A be (m, n), and the pixel coordinates of the registered petiole be (m0, n0). To align the leaf tips in the two images, image A needs to be rotated around (m0, n0) as the rotation center as follows, where (m′, n′) are the pixel coordinates of the leaf tip after the rotation:

[0029]

[0030] This equation represents the coordinate transformation in a two-dimensional space after rotating counterclockwise by an angle θ around the point (m0, n0), where θ is calculated using the law of cosines. Let the coordinates of the two leaf tips be (m1, n1) and (m2, n2), then...

[0031]

[0032] in

[0033]

[0034] Where a, b, and c are the lengths of the three sides of the cosine theorem;

[0035] After the above matrix transformation, the image is rotated to the corresponding position, so that the petiole point and the leaf tip point of the two leaves are aligned respectively.

[0036] As a preferred technical solution, the calculation of similarity specifically includes:

[0037] The pixel values ​​of the registered and aligned leaf images are treated as binary variables, where 0 represents the background, 1 represents the interior of the leaf, two 1s represent overlapping pixels between two leaves, and non-zero pixels represent all pixels between two leaves. The ratio of the total number of pixels with two 1s to the total number of non-zero pixels is the leaf similarity value, i.e.:

[0038]

[0039] The leaf similarity ranges from 0 to 1. When the two sets completely overlap, the leaf similarity is 1. The closer the leaf similarity is to 1, the more similar the two sets are.

[0040] In another aspect, the present invention provides a leaf similarity measurement system based on semantic segmentation and key point detection, which is applied to the aforementioned leaf similarity measurement method based on semantic segmentation and key point detection, including an image annotation module, a contour and key point acquisition module, a normalization module, and a similarity calculation module.

[0041] The image annotation module is used to annotate the acquired leaf images to obtain an annotated leaf images;

[0042] The contour and keypoint acquisition module is used to input the labeled leaf image into a pre-established fusion network model for training, to obtain the trained fusion network model, and to obtain the contour information and endpoint information of the leaf image. The fusion network model uses ResNet as the backbone network to extract feature information of the leaf image, including a Mask R-CNN branch and a Simple Baselines branch. The Mask R-CNN branch is used to obtain the contour information of the leaf, and the Simple Baselines branch is used to obtain the endpoint information of the leaf.

[0043] The normalization module is used to normalize the scale of the leaf image based on the contour information and end point information of the leaf image to obtain a normalized leaf image.

[0044] The similarity calculation module is used to register and align the scale-normalized leaf images and calculate the similarity.

[0045] In another aspect, the present invention provides an electronic device, characterized in that the electronic device comprises:

[0046] At least one processor; and,

[0047] A memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the leaf similarity measurement method based on semantic segmentation and key point detection.

[0049] In another aspect, the present invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the blade similarity measurement method based on semantic segmentation and key point detection.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] 1. A fusion network based on semantic segmentation algorithm and key point detection algorithm can quickly obtain semantic information and key point information of blades;

[0052] 2. Because the images have been scaled, the calculation of leaf similarity is not affected by factors such as image shooting distance and leaf age;

[0053] 3. By identifying and aligning the petiole point and leaf tip point of the leaf respectively, the accuracy of similarity can be improved. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of a leaf similarity measurement method based on semantic segmentation and key point detection according to an embodiment of the present invention;

[0056] Figure 2 This is a flowchart illustrating the fusion network model processing in an embodiment of the present invention;

[0057] Figure 3 (a) in the figure is the original blade diagram of the embodiment of the present invention;

[0058] Figure 3 (b) in the figure represents the blade outline information and the two ends of the blade in an embodiment of the present invention;

[0059] Figure 4 This is a normalized diagram of the blade dimensions according to an embodiment of the present invention;

[0060] Figure 5 This is a registration and alignment diagram of the petiole point according to an embodiment of the present invention;

[0061] Figure 6 This is a registration and alignment diagram of the two ends of the blade in an embodiment of the present invention;

[0062] Figure 7 This is a visualization of the intersection and union of two blades in an embodiment of the present invention.

[0063] Figure 8 This is a block diagram of a leaf similarity measurement system based on semantic segmentation and key point detection according to an embodiment of the present invention.

[0064] Figure 9 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0066] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0067] Please see Figure 1 In one embodiment of this application, a leaf similarity measurement method based on semantic segmentation and key point detection is provided, including the following steps:

[0068] S1. Label the collected leaf images to obtain labeled leaf images;

[0069] Furthermore, the annotation includes leaf outline information and the two ends of the leaf; the two ends of the leaf are the petiole point and the leaf tip point.

[0070] S2. Input the labeled leaf image into a pre-established fusion network model for training to obtain the trained fusion network model and acquire the contour information and endpoint information of the leaf image. The fusion network model uses ResNet as the backbone network to extract feature information from the leaf image, including a Mask R-CNN branch and a SimpleBaselines branch. The Mask R-CNN branch is used to obtain the contour information of the leaf; the SimpleBaselines branch is used to obtain the endpoint information of the leaf. Please refer to [link to relevant documentation]. Figure 2 , Figure 3 (a) Figure 3 (b) in the middle;

[0071] Furthermore, the labeled leaf images are input into a pre-established fusion network model for training. The specific training process is as follows:

[0072] S21. Input the original image and the labeled information, and obtain a set of feature maps through the feature extraction network;

[0073] S22. For the Mask R-CNN branch: The RPN network is used on the feature map to generate candidate regions. For each candidate region, the RoIAlign layer is used to align it to a fixed-size feature block, and the result is fed into two parallel branches. In the classification and regression branches, fully connected layers are used to output the class and bounding box correction values ​​for each candidate region. In the masking branch, fully convolutional layers are used to output pixel-level masks for each candidate region, and 1×1 convolutional layers and a pixel-level Softmax function are used to obtain the probability of each pixel belonging to a different class.

[0074] S23. For the Simple Baselines branch: Use a convolutional layer on each feature map to output a heatmap corresponding to the key point category, and normalize it using the softmax function; find the location of the maximum value on each heatmap as the predicted key point coordinates, and refine it using Gaussian Peak Adjustment (GPA);

[0075] S24. Compare the predicted results with the true labels and calculate the loss function;

[0076] S25. Using the calculated loss function, update the weights and biases of the existing model using the backpropagation algorithm;

[0077] S26. Iterate the training of the model until the preset stopping condition is reached; obtain the trained fusion network model, and obtain the contour information and end point information of the leaf image accordingly.

[0078] S3. Based on the contour information and endpoint information of the blade image, scale normalize the blade image to obtain a normalized blade image. Please refer to [link to relevant documentation]. Figure 4 ;

[0079] Furthermore, the scale normalization of the leaf images ensures that the pixel distance between the petiole point and the leaf tip point is consistent in two leaf images. Specifically:

[0080] First, let the two leaf images be A and B, with widths w respectively. A and w b The heights are h A and h B Let the coordinates of the two endpoints on image A be (x1, y1) and (x2, y2), and the coordinates of the two endpoints on image B be (x3, y3) and (x4, y4). Then the pixel distance between the two endpoints on the two leaf images is as follows:

[0081]

[0082] Secondly, A or B needs to be scaled so that d A and d B To be equal, assuming A is scaled, we need to find a scaling factor s such that: s*d A =d B That is, s = d B / d A When scaling A, the scaling factor is s.

[0083] S4. Register and align the scale-normalized leaf images and calculate the similarity.

[0084] Furthermore, let the two leaf images be A and B. Suppose that image A is processed so that the petiole point and leaf tip point of the leaf in image A are aligned with the petiole point and leaf tip point of the leaf in image B, respectively.

[0085] Please see Figure 5 Petiole point registration and alignment: Let the pixel coordinates of the petiole point in image A be (x,y) and the pixel coordinates of the petiole point in image B be (x′,y′). To make the pixel coordinates of the two petiole points the same, the blade in image A needs to be translated along the x-axis by Δx = x′ - x and along the y-axis by Δy = y′ - y, so that the petiole points of the blades in the two images are at the same pixel coordinate position.

[0086] To align the leaf tips in two images, the pixel positions of the leaf petiole and leaf tip are made consistent. This invention uses matrix transformations to describe translation, scaling, and rotation operations of objects in two-dimensional or three-dimensional space, thereby allowing the image to be rotated to a specified position. Let the pixel coordinates of the leaf tip in image A be (m, n), and the pixel coordinates of the registered petiole be (m0, n0). To align the leaf tips in the two images, image A needs to be rotated around (m0, n0) as the rotation center, where (m′, n′) are the pixel coordinates of the leaf tip after rotation in image A:

[0087]

[0088] This equation represents the coordinate transformation in a two-dimensional space after rotating counterclockwise by an angle θ around the point (m0, n0), where θ is calculated using the law of cosines. Let the coordinates of the two leaf tips be (m1, n1) and (m2, n2), then...

[0089]

[0090] in

[0091]

[0092] Where a, b, and c are the lengths of the three sides according to the Law of Cosines;

[0093] For further details, please refer to Figure 6 After the above matrix transformation, the image is rotated to the corresponding position, so that the petiole point and the leaf tip point of the two leaves are aligned respectively.

[0094] Specifically, the similarity calculation is as follows:

[0095] Similarity is the degree of overlap between two leaf blades. The pixel values ​​of the registered and aligned leaf images are treated as binary variables, where 0 represents the background, 1 represents the interior of the leaf, two 1s represent overlapping pixels between the two leaf blades, and non-zero values ​​represent all pixels between the two leaf blades. The ratio of the total number of pixels with two 1s to the total number of non-zero pixels is the leaf similarity value, which is between 0 and 1.

[0096]

[0097] Specifically, when the two sets completely overlap, the leaf similarity is 1. The closer the leaf similarity is to 1, the more similar the two sets are. Please refer to [link / reference needed]. Figure 7 .

[0098] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0099] Based on the same ideas as the blade similarity measurement method based on semantic segmentation and keypoint detection in the above embodiments, the present invention also provides a blade similarity measurement system based on semantic segmentation and keypoint detection. This system can be used to execute the aforementioned blade similarity measurement method based on semantic segmentation and keypoint detection. For ease of explanation, the structural schematic diagram of the embodiment of the blade similarity measurement system based on semantic segmentation and keypoint detection only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0100] Please see Figure 8 In another embodiment of this application, a leaf similarity measurement system 100 based on semantic segmentation and key point detection is provided. The system includes an image annotation module 101, a contour and key point acquisition module 102, a normalization module 103, and a similarity calculation module 104.

[0101] The image annotation module 101 is used to annotate the acquired leaf images to obtain an annotated leaf images;

[0102] The contour and key point acquisition module 102 is used to input the labeled leaf image into a pre-established fusion network model for training, to obtain the trained fusion network model, and to obtain the contour information and end point information of the leaf image; the fusion network model uses ResNet as the backbone network to extract feature information of the input image, including a Mask R-CNN branch and a Simple Baselines branch. The Mask R-CNN branch is used to obtain the contour information of two leaves; the Simple Baselines branch is used to obtain the end point information of the leaf.

[0103] The normalization module 103 is used to perform scale normalization on the leaf image based on the contour information and end point information of the leaf image to obtain a normalized leaf image.

[0104] The similarity calculation module 104 is used to register and align the scale-normalized leaf image and calculate the similarity.

[0105] It should be noted that the leaf similarity measurement system based on semantic segmentation and key point detection of the present invention corresponds one-to-one with the leaf similarity measurement method based on semantic segmentation and key point detection of the present invention. The technical features and beneficial effects described in the embodiments of the leaf similarity measurement method based on semantic segmentation and key point detection are applicable to the embodiments of the leaf similarity measurement system based on semantic segmentation and key point detection. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0106] Furthermore, in the embodiments of the blade similarity measurement system based on semantic segmentation and key point detection described above, the logical division of each program module is merely an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the blade similarity measurement system based on semantic segmentation and key point detection can be divided into different program modules to complete all or part of the functions described above.

[0107] Please see Figure 9 In one embodiment, an electronic device is provided for implementing a blade similarity measurement method based on semantic segmentation and key point detection. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a blade similarity measurement program 203 based on semantic segmentation and key point detection.

[0108] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the leaf similarity measurement program 203 for semantic segmentation and key point detection, but also to temporarily store data that has been output or will be output.

[0109] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0110] Figure 9 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 9 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0111] The leaf similarity measurement program 203 based on semantic segmentation and key point detection stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0112] The collected leaf images are labeled to obtain labeled leaf images;

[0113] The labeled leaf images are input into a pre-established fusion network model for training, resulting in a trained fusion network model that obtains the contour information and endpoint information of the leaf images. The fusion network model uses ResNet as the backbone network to extract feature information from the input images, including a Mask R-CNN branch and a SimpleBaselines branch. The Mask R-CNN branch is used to obtain the contour information of the two leaves, and the SimpleBaselines branch is used to obtain the endpoint information of the leaves.

[0114] Based on the contour information and end point information of the leaf image, the leaf image is scaled and normalized to obtain a normalized leaf image.

[0115] The scale-normalized leaf images are registered and aligned, and their similarity is calculated.

[0116] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0117] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A leaf similarity measurement method based on semantic segmentation and key point detection, characterized in that, Includes the following steps: The collected leaf images are labeled to obtain labeled leaf images; The labeled leaf images are input into a pre-established fusion network model for training to obtain the trained fusion network model, which then obtains the contour information and end point information of the leaf images. The fusion network model uses ResNet as the backbone network to extract feature information from the leaf images, including a Mask R-CNN branch and a Simple Baselines branch. The Mask R-CNN branch is used to obtain the contour information of the leaf, and the Simple Baselines branch is used to obtain the end point information of the leaf. Based on the contour information and end point information of the leaf image, the leaf image is scaled and normalized to obtain a normalized leaf image. The scale-normalized leaf images are registered and aligned, and their similarity is calculated. The registration and alignment of the scale-normalized leaf image specifically involves: Let there be two leaf images, A and B. Suppose that image A is processed so that the petiole point and leaf tip point of the leaf in image A are aligned with the petiole point and leaf tip point of the leaf in image B, respectively. Petiole point registration and alignment: Let the pixel coordinates of the petiole point of the leaf blade in image A be (x,y) and the pixel coordinates of the petiole point of the leaf blade in image B be (x′,y′). To make the pixel coordinates of the two petiole points the same, the leaf blade in image A needs to be translated along the x-axis by Δx = x′ - x and along the y-axis by Δy = y′ - y, so that the petiole points of the leaf blades in the two images are at the same pixel coordinate position. Leaf tip registration and alignment: Let the pixel coordinates of the leaf tip in image A be (m, n), and the pixel coordinates of the registered petiole be (m0, n0). To align the leaf tips in the two images, image A needs to be rotated around (m0, n0) as the rotation center as follows, where (m′, n′) are the pixel coordinates of the leaf tip after the rotation: This equation represents the coordinate transformation in a two-dimensional space after rotating counterclockwise by an angle θ around the point (m0, n0), where θ is calculated using the law of cosines. Let the coordinates of the two leaf tips be (m1, n1) and (m2, n2), then: in Where a, b, and c are the lengths of the three sides of the cosine theorem; After the coordinate transformation described above, the image is rotated to the corresponding position, so that the petiole point and the leaf tip point of the two leaves are aligned respectively.

2. The leaf similarity measurement method based on semantic segmentation and key point detection according to claim 1, characterized in that, The annotation includes leaf outline information and the two ends of the leaf; the two ends of the leaf are the petiole point and the leaf tip point.

3. The leaf similarity measurement method based on semantic segmentation and key point detection according to claim 1, characterized in that, The labeled leaf images are input into a pre-established fusion network model for training. The specific training process is as follows: The original image and labeled information are input, and a set of feature maps are obtained through a feature extraction network. For the Mask R-CNN branch: candidate regions are generated using an RPN network on the feature map. For each candidate region, the RoIAlign layer is used to align it to a fixed-size feature block, and the result is fed into two parallel branches. In the classification and regression branches, fully connected layers are used to output the class and bounding box correction values ​​for each candidate region. In the masking branch, fully convolutional layers are used to output a pixel-level mask for each candidate region, and a 1×1 convolutional layer and a pixel-level Softmax function are used to obtain the probability of each pixel belonging to a different class. For the Simple Baselines branch: a convolutional layer is used on each feature map to output a heatmap corresponding to the keypoint category, and normalized by the softmax function; the location of the maximum value on each heatmap is found as the predicted keypoint coordinates, and refined by Gaussian peak adjustment; The predicted results are compared with the true labels, and the loss function is calculated. Using the calculated loss function, the weights and biases of the existing model are updated using the backpropagation algorithm. The model is trained iteratively until a preset stopping condition is reached, resulting in a well-trained fusion network model. This model is then used to obtain the contour information and end-point information of the leaf image.

4. The leaf similarity measurement method based on semantic segmentation and key point detection according to claim 1, characterized in that, The scale normalization of the leaf images involves making the pixel distance between the petiole and leaf tip points consistent across multiple leaf images. Specifically: First, let the two leaf images be A and B, with widths w respectively. A and w b The heights are h A and h B Let the coordinates of the two endpoints on image A be (x1, y1) and (x2, y2), and the coordinates of the two endpoints on image B be (x3, y3) and (x4, y4). Then the pixel distance between the two endpoints on the two leaf images is as follows: Secondly, A or B needs to be scaled so that d A and d B To be equal, assuming A is scaled, we need to find a scaling factor s such that: s*d A =d B That is, s = d B / d A That is, scaling A by a scaling factor of s.

5. The leaf similarity measurement method based on semantic segmentation and key point detection according to claim 1, characterized in that, The calculation of similarity specifically includes: The pixel values ​​of the registered and aligned leaf images are treated as binary variables, where 0 represents the background, 1 represents the interior of the leaf, two 1s represent overlapping pixels between two leaves, and non-zero pixels represent all pixels between two leaves. The ratio of the total number of pixels with two 1s to the total number of non-zero pixels is the leaf similarity value, i.e.: The leaf similarity ranges from 0 to 1. When the two sets completely overlap, the leaf similarity is 1. The closer the leaf similarity is to 1, the more similar the two sets are.

6. A blade similarity measurement system based on semantic segmentation and key point detection, characterized in that, The leaf similarity measurement method based on semantic segmentation and key point detection, applied to any one of claims 1-5, includes an image annotation module, a contour and key point acquisition module, a normalization module, and a similarity calculation module; The image annotation module is used to annotate the acquired leaf images to obtain an annotated leaf images; The contour and keypoint acquisition module is used to input the labeled leaf image into a pre-established fusion network model for training, to obtain the trained fusion network model, and to obtain the contour information and endpoint information of the leaf image. The fusion network model uses ResNet as the backbone network to extract feature information of the leaf image, including a Mask R-CNN branch and a Simple Baselines branch. The Mask R-CNN branch is used to obtain the contour information of the leaf, and the Simple Baselines branch is used to obtain the endpoint information of the leaf. The normalization module is used to normalize the scale of the leaf image based on the contour information and end point information of the leaf image to obtain a normalized leaf image. The similarity calculation module is used to register and align the scale-normalized leaf images and calculate the similarity.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the leaf similarity measurement method based on semantic segmentation and key point detection as described in any one of claims 1-5.

8. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the blade similarity measurement method based on semantic segmentation and key point detection as described in any one of claims 1-5.

Images