Method, system and storage medium for evaluating silkworm pupa status for making silkworm pupa

Abstract

The present invention discloses a method, system and storage medium for evaluating the state of silkworm pupae for making pupae, which belongs to the field of image processing technology, and includes: obtaining two-dimensional images and depth maps of target silkworm pupae at multiple viewpoints, and obtaining a three-dimensional shape model of the target silkworm pupae based on the depth map; selecting a priority viewpoint from all viewpoints, photographing new silkworm pupae at the priority viewpoint, obtaining target point cloud data of the new silkworm pupae, matching the target point cloud data with each three-dimensional shape model, if there is a three-dimensional shape model with a matching degree greater than a first threshold and it is the first shape model, the new silkworm pupae type is a normal silkworm pupae, if it is a second shape model, then the type of the new silkworm pupae is evaluated based on the first shape model corresponding to the second shape model; if there is no three-dimensional shape model with a matching degree greater than the first threshold, then the type of the new silkworm pupae is obtained based on a neural network. The present invention improves the recognition efficiency of abnormal silkworm pupae in the process of making pupae.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method, system and storage medium for evaluating the state of silkworm pupa used in making pupae. Background Art

[0002] In the field of pupae production, the quality of silkworm pupae directly affects the quality of the final product. In particular, incomplete silkworm pupae are not suitable for pupae production because they cannot provide all the structures and nutrients required for pupae production, and incomplete silkworm pupae may be damaged during breeding, collection or processing, which may affect the quality and appearance of the final product when making pupae.

[0003] The existing classification methods for silkworm pupae, such as the Chinese patent application with publication number CN115115633A, propose an intelligent picking method and system for spinning cocoons; the method obtains a cocoon image to obtain a cocoon connected domain of a single cocoon image; a template set is constructed according to the cocoon connected domain, the template set includes n templates, and the cocoon connected domain is opened using the n templates to obtain n internal connected domains of the cocoon; by performing an expansion operation on the n internal connected domains of the cocoons, all judgment connected domains of the n internal connected domains of the cocoons are obtained, and then a judgment image of each internal connected domain of the cocoons is obtained, and a cocoon edge curve is established by analyzing and calculating each cocoon judgment image, and then n cocoon edge curves are obtained; a single cocoon is evaluated according to the characteristics of the n cocoon edge curves to determine the quality of the cocoon, and then the quality of each cocoon is determined and the cocoons are classified.

[0004] However, in the above-mentioned prior art, the analysis based on a single cocoon image may not be able to fully capture the three-dimensional morphological characteristics of the silkworm pupa.

[0005] For another example, Chinese patent application with publication number CN107993244A proposes an automatic corn detection method, which includes the following steps: undersize detection; corn kernel overall color detection; corn kernel damage detection; wormhole detection; mildew detection; and surface crack detection.

[0006] In the above prior art, if the image quality is not high (for example, low resolution, high noise), the detection accuracy may be affected, especially in edge detection and crack recognition. Therefore, a fast, accurate and automated silkworm pupa status assessment method is needed to assess whether silkworm pupae can be used to make dead pupae. Summary of the invention

[0007] To solve the above problems, the present invention provides a method, system and storage medium for evaluating the state of silkworm pupa used in the production of silkworm pupae, so as to solve the problems in the prior art.

[0008] In order to achieve the above-mentioned purpose of the invention, the present invention provides a method for evaluating the state of silkworm pupae used in the production of dead pupae, comprising:

[0009] S1: Acquire two-dimensional images of target silkworm pupae of various shapes at multiple viewpoints, wherein the target silkworm pupae include silkworm pupae in normal state and abnormal state, acquire a depth map corresponding to the two-dimensional image at each viewpoint, and generate standard point cloud data representing the shape of the target silkworm pupae based on the depth map;

[0010] S2: obtaining a three-dimensional shape model of the target silkworm pupa based on the standard point cloud data, wherein the three-dimensional shape model includes a first shape model of the target silkworm pupa in a normal state and a second shape model in an abnormal state, and associating the first shape model and the second shape model of each target silkworm pupa into a database;

[0011] S3: Filter out a priority viewpoint from all viewpoints, photograph the new silkworm pupa at the priority viewpoint, obtain standard point cloud data of the new silkworm pupa, define it as target point cloud data, match the target point cloud data with each three-dimensional shape model in the database, and determine whether there is a three-dimensional shape model with a matching degree greater than a first threshold. If so, if the three-dimensional shape model is the first shape model, determine that the type of the new silkworm pupa is a normal silkworm pupa; if the three-dimensional shape model is the second shape model, evaluate the type of the new silkworm pupa based on the first shape model corresponding to the second shape model;

[0012] S4: If there is no three-dimensional shape model whose matching degree is greater than the first threshold, a neural network model is established, and the two-dimensional image is input into the neural network model as a training set, and the neural network model extracts first feature information therefrom, wherein the first feature information includes normal features and abnormal features, and a two-dimensional image of a new silkworm pupa under a preferred perspective is obtained, which is defined as a target image, and the target image is input into the neural network model, and second feature information of the new silkworm pupa is output, and the type of the new silkworm pupa is obtained based on the second feature information.

[0013] Further, generating standard point cloud data representing the shape of the target silkworm pupa based on the depth map comprises the following steps:

[0014] S31: Divide the area where the standard point cloud data is located into a plurality of voxel units, obtain the centroid of the voxel based on all the point cloud data in the same voxel unit, define it as a first target point, and remove other point cloud data in the standard point cloud data except the first target point;

[0015] S32: setting a first area range with each first target point as the center, calculating the number of all first target points within the first area range, defining the first target point as a first value, and if the first value is less than a second threshold, defining the first target point as a first abnormal point, and removing the first abnormal point;

[0016] S33: Repeat step S32 until all first target points are processed, and define the remaining combination of the first target points as standard point cloud data.

[0017] Further, selecting a priority viewpoint from multiple viewpoints includes the following steps:

[0018] Obtain the normal vector of each three-dimensional point in the standard point cloud data, obtain the coordinate information and direction information of each viewpoint, and calculate the first score of the two-dimensional image taken at each viewpoint based on the first formula , the first formula is: ,in, is the number of 3D points corresponding to each viewpoint, is the normal vector of the optical axis at the jth viewpoint, is the viewpoint vector from the jth viewpoint to the i-th 3D point, is the normal vector of the i-th 3D point, represents the distance between the pixel corresponding to the i-th 3D point and the center of the 2D image corresponding to the j-th viewpoint, is the dot product between the normal vector of the i-th 3D point and the view vector, =1 or 0, indicating whether the i-th standard point cloud data is visible from the j-th viewpoint, and defining the viewpoint whose first score is greater than the third threshold as the priority viewpoint.

[0019] Furthermore, obtaining the three-dimensional shape model of the target silkworm pupa comprises the following steps:

[0020] The standard point cloud data under different priority viewpoints are aligned based on the phase-limited correlation method, the optimal translation displacement between target areas at the same actual position under different viewpoints is obtained based on the phase-limited correlation function, and the standard point cloud data under different viewpoints are translated and rotated based on the optimal translation displacement to align the standard point cloud data to the same three-dimensional shape model.

[0021] Further, evaluating the type of silkworm pupa based on the first shape model corresponding to the second shape model comprises the following steps:

[0022] A three-dimensional point in the target point cloud data is defined as a second target point, and a three-dimensional point in the first shape model is defined as a third target point. The third target point closest to the second target point is obtained based on an iterative nearest point algorithm, and the position difference between the two points is calculated. If the position difference is greater than a fourth threshold, the second target point is defined as an abnormal point cloud, and adjacent abnormal point clouds are connected to form an abnormal area. If multiple abnormal point clouds are arranged along a straight line or a curve, the formed abnormal area is defined as a first type of abnormal area. A circumscribed rectangle including the first type of abnormal area is determined, and the area of ​​the first type of abnormal area is obtained based on the size parameters of the circumscribed rectangle. If the area is greater than a fifth threshold, the silkworm pupa is defined as a first type of abnormal silkworm pupa, otherwise it is a normal silkworm pupa.

[0023] Furthermore, evaluating the type of silkworm pupa based on the first shape model corresponding to the second shape model further includes the following steps:

[0024] If the abnormal point cloud forms a closed polygon, the formed abnormal area is defined as a second-class abnormal area, and a circumscribed sphere including the second-class abnormal area is determined. The volume of the second-class abnormal area is obtained based on the size parameters of the circumscribed sphere. If the volume is greater than a sixth threshold, the silkworm pupa is defined as a second-class abnormal silkworm pupa, otherwise it is a normal silkworm pupa.

[0025] Further, obtaining a new type of silkworm pupa based on the second characteristic information comprises the following steps:

[0026] The neural network model is a convolutional neural network model. The convolution layer of the convolutional neural network model extracts second feature information from the target image. The second feature information is flattened into a one-dimensional vector and input into the fully connected layer. The fully connected layer maps the second feature information to the category space to generate a first score for each category of the new silkworm pupae. The categories of the new silkworm pupae include normal categories and abnormal categories. Based on the Softmax layer, the first scores corresponding to the two categories are respectively converted into first probabilities and second probabilities, and the sizes of the first probability and the second probability are judged. The neural network model outputs the silkworm pupae category corresponding to the larger probability.

[0027] Furthermore, the calculation of the matching degree includes the following steps:

[0028] For each point in the target point cloud data, the nearest neighbor point on the surface of the three-dimensional shape model is obtained, the Euclidean distance from the point to the nearest neighbor point is calculated, and the average value of the Euclidean distances of all points is taken as the matching degree.

[0029] The present invention also provides a silkworm pupa state evaluation system for making pupae, which is used to implement the above method. The system mainly includes:

[0030] An acquisition module is used to acquire two-dimensional images of target silkworm pupae of various shapes at multiple viewpoints, wherein the target silkworm pupae include silkworm pupae in a normal state and an abnormal state, acquire a depth map corresponding to the two-dimensional image at each viewpoint, and generate standard point cloud data representing the shape of the target silkworm pupae based on the depth map;

[0031] A model generation module is used to obtain a three-dimensional shape model of the target silkworm pupa based on the standard point cloud data, wherein the three-dimensional shape model includes a first shape model of the target silkworm pupa in a normal state and a second shape model in an abnormal state, and the first shape model and the second shape model of each target silkworm pupa are associated and stored in a database;

[0032] A matching module is provided, which selects a priority viewpoint from all viewpoints, shoots a new silkworm pupa at the priority viewpoint, obtains standard point cloud data of the new silkworm pupa, defines the data as target point cloud data, matches the target point cloud data with each three-dimensional shape model in a database, and determines whether there is a three-dimensional shape model with a matching degree greater than a first threshold value. If so, if the three-dimensional shape model is a first shape model, the type of the new silkworm pupa is determined to be a normal silkworm pupa; if the three-dimensional shape model is a second shape model, the type of the new silkworm pupa is evaluated based on the first shape model corresponding to the second shape model;

[0033] A neural network evaluation module is provided, if there is no three-dimensional shape model whose matching degree is greater than the first threshold value, then a neural network model is established, the two-dimensional image is input into the neural network model as a training set, the neural network model extracts first feature information therefrom, the first feature information includes normal features and abnormal features, a two-dimensional image of a new silkworm pupa under the priority perspective is obtained, defined as a target image, the target image is input into the neural network model, second feature information of the new silkworm pupa is output, and the type of the new silkworm pupa is obtained based on the second feature information.

[0034] The present invention also provides a computer storage medium, which stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the above-mentioned method.

[0035] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0036] The present invention constructs a three-dimensional shape model of silkworm pupa by acquiring two-dimensional images and depth maps under multiple viewpoints. The generated three-dimensional shape model can provide a comprehensive view of the morphology of silkworm pupa, which is helpful to capture the subtle morphological features of silkworm pupa; the priority viewpoints are screened and their data are aligned to a unified three-dimensional model, which can ensure the accuracy and consistency of the model, while reducing the amount of data processing and improving efficiency; when the present invention judges the type of silkworm pupa, first, the target point cloud data of the new silkworm pupa is matched with the three-dimensional shape model in the database, which can quickly identify whether the new silkworm pupa is normal or abnormal. This method reduces the dependence on manual inspection and improves the speed and accuracy of classification. When the three-dimensional model matching is unsuccessful, the neural network model can be used as a supplementary method to identify the type of silkworm pupa by learning the features in the two-dimensional image. This method can capture those subtle anomalies that are difficult to identify through three-dimensional model matching. The whole process can be highly automated, reducing manual intervention and improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of the steps of a method for evaluating the state of silkworm pupa used for preparing silkworm pupa according to the present invention;

[0038] Figure 2 The present invention is a structural diagram of a silkworm pupa state assessment system for making silkworm pupae. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0041] like Figure 1 As shown, a method for evaluating the state of silkworm pupae used for making pupae comprises:

[0042] S1: Obtain two-dimensional images of target silkworm pupae of various shapes under multiple viewpoints, where the target silkworm pupae include silkworm pupae in normal and abnormal states, obtain a depth map corresponding to the two-dimensional image under each viewpoint, and generate standard point cloud data representing the shape of the target silkworm pupae based on the depth map.

[0043] Specifically, pupae are specially treated silkworm pupae, which are usually used in traditional Chinese medicine or for certain specific industrial purposes. The production of pupae requires that the silkworm pupae maintain a certain integrity and quality standard. Only normal silkworm pupae are considered suitable for making pupae, further ensuring the effectiveness and safety of the final product.

[0044] Assuming that all silkworm pupae in the present invention are disease-free and meet the maturity required for making pupae, a camera is used to shoot target silkworm pupae of different shapes from different angles to obtain two-dimensional images of the target silkworm pupae in different viewpoints. The target silkworm pupae in a normal state (which can be used to make pupae) have a complete appearance without obvious damage or cracks, and the target silkworm pupae in an abnormal state (which cannot be used to make pupae) have an incomplete, damaged or cracked appearance. The depth information of the silkworm pupae is extracted from the two-dimensional image based on the stereo matching method, and the depth map of the target silkworm pupae in the two states is obtained. Based on the camera parameters used to shoot the two-dimensional image (including external parameters, internal parameters, and coordinate system conversion), the physical distance from each pixel point in the depth map (generally a pixel point area is selected as a point cloud) to the shooting viewpoint is obtained, the coordinates of each pixel point in the three-dimensional space are calculated, and all the calculated three-dimensional coordinate points are combined to form standard point cloud data. The standard point cloud data is the point cloud data after excluding abnormal point clouds. The specific implementation method will be explained later.

[0045] S2: Obtain a three-dimensional shape model of the target silkworm pupa based on the standard point cloud data, wherein the three-dimensional shape model includes a first shape model of the target silkworm pupa in a normal state and a second shape model in an abnormal state, and the first shape model and the second shape model of each target silkworm pupa are associated and stored in a database.

[0046] Specifically, the point cloud data from different viewpoints are merged into a unified three-dimensional model using three-dimensional reconstruction technology. Target silkworm pupae of different shapes may generate multiple different three-dimensional shape models, including a first shape model in a normal state, and a second shape model including a shape model containing cracks and a shape model containing different notch volumes. The first shape model and the second shape model of silkworm pupae of the same shape are associated and stored.

[0047] S3: Filter out a priority viewpoint from all viewpoints, photograph the new silkworm pupa under the priority viewpoint, obtain standard point cloud data of the new silkworm pupa, define it as target point cloud data, match the target point cloud data with each 3D shape model in the database, and determine whether there is a 3D shape model with a matching degree greater than a first threshold. If so, if the 3D shape model is the first shape model, determine that the type of the new silkworm pupa is a normal silkworm pupa; if the 3D shape model is the second shape model, evaluate the type of the new silkworm pupa based on the first shape model corresponding to the second shape model.

[0048] Specifically, a new silkworm pupa is obtained and photographed at multiple priority perspectives. The target point cloud data of the new silkworm pupa is obtained in the same way. The newly collected target point cloud data is compared with the three-dimensional shape model stored in the database. The matching degree ranges from [0, 1]. The first threshold is assumed to be 0.9. If there is a shape model in the database with a matching degree greater than 0.9 with the new target point cloud data, it means that the new silkworm pupa has found a model in the database that is very close to its shape feature. If the three-dimensional shape model is a model of a normal silkworm pupa, it is directly determined that the new silkworm pupa is a normal silkworm pupa. If the matched three-dimensional shape model is a model of an abnormal silkworm pupa, a first shape model related to the second shape model is used here to match the target point cloud data of the new silkworm pupa. The first shape model is a model of a normal silkworm pupa under the same shape as the second shape model. By comparing it with the model of a normal silkworm pupa, the abnormal position and area size of the silkworm pupa can be more accurately identified. Moreover, if the silkworm pupa matched to the abnormal model is directly determined as an abnormal silkworm pupa, some special cases may be ignored, such as small measurement errors or normal variations.

[0049] S4: If there is no three-dimensional shape model with a matching degree greater than the first threshold, a neural network model is established, and the two-dimensional image is input into the neural network model as a training set. The neural network model extracts normal features and abnormal features therefrom, obtains a new two-dimensional image of the silkworm pupa under a preferred perspective, and defines it as a target image. The target image is input into the neural network model, and the feature information of the new silkworm pupa is output, and the type of the new silkworm pupa is obtained based on the feature information.

[0050] Specifically, when there is no high-matching three-dimensional shape model in the database, a neural network model is used to deal with this problem. The CNN convolutional neural network should be used to extract image feature information. The two-dimensional images of the target silkworm pupae are used as training sets to input into the neural network model. These images contain samples of normal silkworm pupae and abnormal silkworm pupae. The neural network model extracts normal features and abnormal features from the training set, including the shape, size, color, and surface (whether there are scratches, dents, or notches) of the target silkworm pupae in normal and abnormal states. The two-dimensional image of the new silkworm pupa under the preferred perspective is input into the neural network model, and the neural network outputs the feature information of the new silkworm pupa. How to obtain the type of the new silkworm pupa based on the feature information will be explained later.

[0051] The present invention first uses the three-dimensional shape model generated in the database to match the target point cloud data of new silkworm pupae. Once the target point cloud data is successfully matched with the model in the database, the type of silkworm pupa can be quickly obtained without a complicated learning process or a long calculation. Compared with directly using the trained neural network model to judge, less computing resources and time are required. Moreover, the three-dimensional shape models in the database of the present invention are all models of silkworm pupae obtained through three-dimensional reconstruction, including a variety of shapes and a variety of states, which can cover most normal and abnormal types of silkworm pupae.

[0052] If the target point cloud data does not successfully match the model in the database, the neural network can provide a second line of defense, integrating multiple features for analysis, including those that are difficult to capture with traditional model matching methods, such as texture and color changes. If it is detected that the new silkworm pupae are significantly different in color from normal silkworm pupae, the type of silkworm pupae can be judged immediately, while also reducing the occurrence of misjudgments.

[0053] As a preferred technical solution of the present invention, generating standard point cloud data representing the shape of the target silkworm pupa based on the depth map includes the following steps:

[0054] S31: Divide the area where the standard point cloud data is located into multiple voxel units, obtain the center of gravity of the voxel unit based on all the point cloud data in the same voxel unit, define it as the first target point, and eliminate other point cloud data in the standard point cloud data except the first target point.

[0055] S32: setting a first area range with each first target point as the center, calculating the number of all first target points within the first area range, defining it as a first value, and if the first value is less than a second threshold, defining the first target point as a first outlier point, and removing the first outlier point.

[0056] S33: Repeat step S32 until all first target points are processed, and define the remaining combination of first target points as standard point cloud data.

[0057] Specifically, the area where the standard point cloud data is located is divided into multiple voxel units. The voxel unit is the smallest unit in the three-dimensional space, similar to the pixel in the two-dimensional image. Each voxel unit represents a cube of a fixed size in the three-dimensional space. For all the point cloud data in each voxel unit, their center of gravity (i.e., the average position) is calculated. This center of gravity is defined as the first target point. By dividing the voxel units and calculating the center of gravity, multiple three-dimensional points in the same area of ​​the multi-view image are converted into one three-dimensional point, the number of three-dimensional points in the three-dimensional point group is reduced, redundant data is removed, and key information is retained. A region range (first region range) is set with each first target point as the center. This first region range is larger than the voxel unit. The number of all first target points in the region is calculated to obtain a first value. If the first value is less than the preset second threshold, it means that there are fewer point cloud data in the region. At this time, the first target point is a point for processing isolated abnormalities. These points may be caused by noise or erroneous measurement. Eliminating these abnormal points helps to improve the accuracy and reliability of the point cloud data. Step S32 is repeated until all first target points are processed and all abnormal points are cleared.

[0058] Selecting the preferred viewpoint from multiple viewpoints includes the following steps:

[0059] Obtain the normal vector of each 3D point in the standard point cloud data, obtain the coordinate information and direction information of each viewpoint, and calculate the first score of the 2D image taken at each viewpoint based on the first formula , the first formula is: ,in, is the number of 3D points corresponding to each viewpoint, is the normal vector of the optical axis at the jth viewpoint, is the viewpoint vector from the jth viewpoint to the i-th 3D point, is the normal vector of the i-th 3D point, represents the distance between the pixel corresponding to the i-th 3D point and the center of the 2D image corresponding to the j-th viewpoint, is the dot product between the normal vector of the i-th 3D point and the view vector, =1 or 0, indicating whether the i-th standard point cloud data is visible from the j-th viewpoint, and the viewpoint whose first score is greater than the third threshold is defined as the priority viewpoint.

[0060] Specifically, for each 3D point in the target point cloud data, its normal vector is calculated. The normal vector provides information about the surface direction of each point in the point cloud. The 2D image taken at each viewpoint is scored according to the first formula to quantify the quality of the 2D image taken at each viewpoint. In the first formula, This dot product can be used to measure the degree of alignment between the viewpoint direction and the 3D point position. The larger the value of, the better the alignment between the viewpoint direction and the 3D point position, which helps to improve the clarity of the image; Used to evaluate whether the surface of a 3D point is facing the viewpoint, The larger the value of, the more the surface of the 3D point is oriented toward the viewpoint, which helps capture more surface details; Is a binary value, indicating that the i-th standard point cloud data is from the Is the viewpoint visible? If so, =1; if it is not visible, = 0. The first formula quantifies the image quality of each viewpoint by comprehensively considering the geometric relationship between the viewpoint and the target point cloud data. The higher the score, the better the image taken from the viewpoint and the more suitable it is for subsequent processing and analysis.

[0061] The priority viewpoint is the viewpoint that can capture the key features of the target object to the greatest extent. A comprehensive and accurate shape model of the target silkworm pupa can be generated based on the two-dimensional graphics under multiple priority viewpoints.

[0062] Obtaining the three-dimensional shape model of the target silkworm pupa includes the following steps:

[0063] The standard point cloud data under different priority viewpoints are aligned based on the phase-limited correlation method, and the optimal translation displacement between target areas at the same actual position under different viewpoints is obtained based on the phase-limited correlation function. The standard point cloud data under different viewpoints are translated and rotated based on the optimal translation displacement, so that the standard point cloud data are aligned to the same three-dimensional shape model.

[0064] Specifically, the phase-restricted correlation method is used to measure the similarity between two images. In point cloud data alignment, it can be used to identify and align the same surface area in point cloud data from different viewpoints. Phase correlation analysis is performed on the two-dimensional projections of the two sets of point cloud data to obtain a correlation graph. The peak is found in the correlation graph. This peak represents the optimal translation displacement between the two images. The translation displacement vector indicates how to translate one image to the position of another image. The point cloud data is translated using the calculated displacement vector so that the target area at the same actual position is aligned. The phase correlation function can provide sub-pixel displacement estimation and build a high-precision three-dimensional model. The key feature points representing the target silkworm pupa are extracted from the standard point cloud data. The extracted feature points are used to build a three-dimensional model of the silkworm pupa through three-dimensional modeling software or programming libraries (such as PCLP, VTK, etc.). The three-dimensional model is optimized using Poisson surface or meshing technology to ensure the smoothness and accuracy of the model.

[0065] Evaluating the type of silkworm pupa based on the first shape model corresponding to the second shape model comprises the following steps:

[0066] A three-dimensional point in the target point cloud data is defined as a second target point, and a three-dimensional point in the first shape model is defined as a third target point. The third target point closest to the second target point is obtained based on an iterative nearest point algorithm, and the position difference between the two points is calculated. If the position difference is greater than a fourth threshold, the second target point is defined as an abnormal point cloud, and adjacent abnormal point clouds are connected to form an abnormal area. If multiple abnormal point clouds are arranged along a straight line or a curve, the formed abnormal area is defined as a first-class abnormal area. A circumscribed rectangle including the first-class abnormal area is determined, and the area of ​​the first-class abnormal area is obtained based on the size parameters of the circumscribed rectangle. If the area is greater than a fifth threshold, the silkworm pupa is defined as a first-class abnormal silkworm pupa, otherwise it is a normal silkworm pupa.

[0067] Specifically, based on the iterative nearest point algorithm, the nearest neighbor points of the three-dimensional points in the new silkworm pupa and the three-dimensional points in the first shape model are found, and the point clouds are aligned by iteratively updating the correspondence between the point clouds to obtain the position difference between the two point clouds. If the position difference is greater than the preset fourth threshold, it means that the position of the new silkworm pupa is deviated from that of the normal silkworm pupa at this position, and these points are defined as abnormal point clouds. If these abnormal point clouds are arranged along a straight line or a curve, this arrangement may indicate a specific type of abnormality, such as a crack or deformation. By analyzing the arrangement of the point clouds, the starting and ending points of the crack can be accurately located, and the area of ​​the crack can be estimated using a circumscribed rectangle. If the area is greater than the fifth threshold, the silkworm pupa is defined as a first-class abnormal silkworm pupa, that is, a silkworm pupa with a crack or deformation. The method of using a circumscribed rectangle to estimate the area of ​​the crack is much faster than trying to accurately calculate the area along the crack contour. In the quality control process, rapid evaluation of the crack area is crucial to determining whether the silkworm pupa is suitable for further processing.

[0068] Evaluating the type of silkworm pupa based on the first shape model corresponding to the second shape model also includes the following steps:

[0069] If the abnormal point cloud forms a closed polygon, the formed abnormal area is defined as a second-class abnormal area, and the circumscribed sphere including the second-class abnormal area is determined. The volume of the second-class abnormal area is obtained based on the size parameters of the circumscribed sphere. If the volume is greater than the sixth threshold, the silkworm pupa is defined as a second-class abnormal silkworm pupa, otherwise it is a normal silkworm pupa.

[0070] Specifically, the geometric shape of the abnormal point cloud is analyzed to determine whether they form a closed polygonal area. The closed polygonal abnormal area may indicate that the surface of the silkworm pupa is damaged or incomplete. A circumscribed sphere is determined for the second type of abnormal area. If the volume of the abnormal area exceeds the sixth threshold, the silkworm pupa is defined as the second type of abnormal silkworm pupa, that is, a silkworm pupa with damage or incompleteness.

[0071] Obtaining a new type of silkworm pupa based on the second characteristic information comprises the following steps:

[0072] The neural network model is a convolutional neural network model. The convolution layer of the convolutional neural network model extracts the second feature information from the target image. The second feature information is flattened into a one-dimensional vector and input into the fully connected layer. The fully connected layer maps the second feature information to the category space to generate the first score of each category of new silkworm pupae. The new silkworm pupae categories include normal categories and abnormal categories. Based on the Softmax layer, the first scores corresponding to the two categories are respectively converted into the first probability and the second probability, and the size of the first probability and the second probability is judged. The neural network model outputs the silkworm pupa category corresponding to the larger probability.

[0073] Specifically, the convolutional layer of CNN extracts the second feature information from the input silkworm pupa image, including the color, size and shape of the silkworm pupa, maps the extracted high-level features to the category space, generates category scores, and converts the output of the fully connected layer into a probability distribution based on the Softmax function in the Softmax layer, so that each category has a probability value. These probability values ​​represent the possibility that the input image belongs to each category, compare the probabilities of normal and abnormal categories, and output the type of silkworm pupa with a larger probability.

[0074] like Figure 2 As shown, the present invention also provides a silkworm pupa state evaluation system for making pupae, which is used to implement the above method, and the system mainly includes:

[0075] The acquisition module is used to obtain two-dimensional images of target silkworm pupae of various shapes under multiple viewpoints, the target silkworm pupae include silkworm pupae in normal state and abnormal state, obtain the depth map corresponding to the two-dimensional image under each viewpoint, and generate standard point cloud data representing the shape of the target silkworm pupae based on the depth map.

[0076] The model generation module obtains the three-dimensional shape model of the target silkworm pupa based on the standard point cloud data. The three-dimensional shape model includes a first shape model of the target silkworm pupa in a normal state and a second shape model in an abnormal state. The first shape model and the second shape model of each target silkworm pupa are associated and stored in a database.

[0077] A matching module selects a priority viewpoint from all viewpoints, shoots the new silkworm pupa under the priority viewpoint, obtains standard point cloud data of the new silkworm pupa, defines it as target point cloud data, matches the target point cloud data with each 3D shape model in the database, and determines whether there is a 3D shape model with a matching degree greater than a first threshold. If so, if the 3D shape model is the first shape model, the type of the new silkworm pupa is determined to be a normal silkworm pupa; if the 3D shape model is the second shape model, the type of the new silkworm pupa is evaluated based on the first shape model corresponding to the second shape model.

[0078] A neural network evaluation module establishes a neural network model if there is no three-dimensional shape model with a matching degree greater than a first threshold value, and inputs the two-dimensional image as a training set into the neural network model. The neural network model extracts first feature information therefrom, and the first feature information includes normal features and abnormal features. A two-dimensional image of a new silkworm pupa under a preferred perspective is obtained and defined as a target image. The target image is input into the neural network model, and the second feature information of the new silkworm pupa is output. Based on the second feature information, the type of the new silkworm pupa is obtained.

[0079] The present invention also provides a computer storage medium, which stores program instructions. When the program instructions are executed, the device where the computer storage medium is located is controlled to execute the above method.

[0080] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0081] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0082] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The above embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0084] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for evaluating the state of silkworm pupae used in the production of pupae, characterized in that: The method comprises the following steps: S1: Acquire two-dimensional images of target silkworm pupae of various shapes at multiple viewpoints, wherein the target silkworm pupae include silkworm pupae in normal state and abnormal state, acquire a depth map corresponding to the two-dimensional image at each viewpoint, and generate standard point cloud data representing the shape of the target silkworm pupae based on the depth map; S2: obtaining a three-dimensional shape model of the target silkworm pupa based on the standard point cloud data, wherein the three-dimensional shape model includes a first shape model of the target silkworm pupa in a normal state and a second shape model in an abnormal state, and associating the first shape model and the second shape model of each target silkworm pupa into a database; S3: Filter out a priority viewpoint from all viewpoints, photograph the new silkworm pupa at the priority viewpoint, obtain standard point cloud data of the new silkworm pupa, define it as target point cloud data, match the target point cloud data with each three-dimensional shape model in the database, and determine whether there is a three-dimensional shape model with a matching degree greater than a first threshold. If so, if the three-dimensional shape model is the first shape model, determine that the type of the new silkworm pupa is a normal silkworm pupa; if the three-dimensional shape model is the second shape model, evaluate the type of the new silkworm pupa based on the first shape model corresponding to the second shape model; S4: If there is no three-dimensional shape model whose matching degree is greater than the first threshold, a neural network model is established, and the two-dimensional image is input into the neural network model as a training set, and the neural network model extracts first feature information therefrom, wherein the first feature information includes normal features and abnormal features, and a two-dimensional image of a new silkworm pupa under a preferred perspective is obtained, which is defined as a target image, and the target image is input into the neural network model, and second feature information of the new silkworm pupa is output, and the type of the new silkworm pupa is obtained based on the second feature information.

2. The method according to claim 1, characterized in that Generating standard point cloud data representing the shape of the target silkworm pupa based on the depth map comprises the following steps: S31: Divide the area where the standard point cloud data is located into a plurality of voxel units, obtain the centroid of the voxel based on all the point cloud data in the same voxel unit, define it as a first target point, and remove other point cloud data in the standard point cloud data except the first target point; S32: setting a first area range with each first target point as the center, calculating the number of all first target points within the first area range, defining the first target point as a first value, and if the first value is less than a second threshold, defining the first target point as a first abnormal point, and removing the first abnormal point; S33: Repeat step S32 until all first target points are processed, and define the remaining combination of the first target points as standard point cloud data.

3. The method according to claim 2, characterized in that Selecting the preferred viewpoint from multiple viewpoints includes the following steps: Obtain the normal vector of each three-dimensional point in the standard point cloud data, obtain the coordinate information and direction information of each viewpoint, and calculate the first score of the two-dimensional image taken at each viewpoint based on the first formula , the first formula is: ,in, is the number of 3D points corresponding to each viewpoint, is the normal vector of the optical axis at the jth viewpoint, is the viewpoint vector from the jth viewpoint to the i-th 3D point, is the normal vector of the i-th 3D point, represents the distance between the pixel corresponding to the i-th 3D point and the center of the 2D image corresponding to the j-th viewpoint, is the dot product between the normal vector of the i-th 3D point and the view vector, =1 or 0, indicating whether the i-th standard point cloud data is visible from the j-th viewpoint, and defining the viewpoint whose first score is greater than the third threshold as the priority viewpoint.

4. The method according to claim 3, characterized in that Acquiring the three-dimensional shape model of the target silkworm pupa comprises the following steps: The standard point cloud data under different priority viewpoints are aligned based on the phase-limited correlation method, the optimal translation displacement between target areas at the same actual position under different viewpoints is obtained based on the phase-limited correlation function, and the standard point cloud data under different viewpoints are translated and rotated based on the optimal translation displacement to align the standard point cloud data to the same three-dimensional shape model.

5. The method according to claim 1, characterized in that Evaluating the type of silkworm pupa based on the first shape model corresponding to the second shape model comprises the following steps: A three-dimensional point in the target point cloud data is defined as a second target point, and a three-dimensional point in the first shape model is defined as a third target point. The third target point closest to the second target point is obtained based on an iterative nearest point algorithm, and the position difference between the two points is calculated. If the position difference is greater than a fourth threshold, the second target point is defined as an abnormal point cloud, and adjacent abnormal point clouds are connected to form an abnormal area. If multiple abnormal point clouds are arranged along a straight line or a curve, the formed abnormal area is defined as a first type of abnormal area. A circumscribed rectangle including the first type of abnormal area is determined, and the area of ​​the first type of abnormal area is obtained based on the size parameters of the circumscribed rectangle. If the area is greater than a fifth threshold, the silkworm pupa is defined as a first type of abnormal silkworm pupa, otherwise it is a normal silkworm pupa.

6. The method according to claim 5, characterized in that Evaluating the type of silkworm pupa based on the first shape model corresponding to the second shape model further comprises the following steps: If the abnormal point cloud forms a closed polygon, the formed abnormal area is defined as a second-class abnormal area, and a circumscribed sphere including the second-class abnormal area is determined. The volume of the second-class abnormal area is obtained based on the size parameters of the circumscribed sphere. If the volume is greater than a sixth threshold, the silkworm pupa is defined as a second-class abnormal silkworm pupa, otherwise it is a normal silkworm pupa.

7. The method according to claim 1, characterized in that Obtaining a new type of silkworm pupa based on the second characteristic information comprises the following steps: The neural network model is a convolutional neural network model. The convolution layer of the convolutional neural network model extracts second feature information from the target image. The second feature information is flattened into a one-dimensional vector and input into the fully connected layer. The fully connected layer maps the second feature information to the category space to generate a first score for each category of the new silkworm pupae. The categories of the new silkworm pupae include normal categories and abnormal categories. Based on the Softmax layer, the first scores corresponding to the two categories are respectively converted into first probabilities and second probabilities, and the sizes of the first probability and the second probability are judged. The neural network model outputs the silkworm pupae category corresponding to the larger probability.

8. The method according to claim 1, characterized in that: The calculation of the matching degree includes the following steps: For each point in the target point cloud data, the nearest neighbor point on the surface of the three-dimensional shape model is obtained, the Euclidean distance from the point to the nearest neighbor point is calculated, and the average value of the Euclidean distances of all points is taken as the matching degree.

9. A silkworm pupa status assessment system for pupa production, used to implement the method according to any one of claims 1 to 8, characterized in that: The system includes the following modules: An acquisition module is used to acquire two-dimensional images of target silkworm pupae of various shapes at multiple viewpoints, wherein the target silkworm pupae include silkworm pupae in a normal state and an abnormal state, acquire a depth map corresponding to the two-dimensional image at each viewpoint, and generate standard point cloud data representing the shape of the target silkworm pupae based on the depth map; A model generation module is used to obtain a three-dimensional shape model of the target silkworm pupa based on the standard point cloud data, wherein the three-dimensional shape model includes a first shape model of the target silkworm pupa in a normal state and a second shape model in an abnormal state, and the first shape model and the second shape model of each target silkworm pupa are associated and stored in a database; A matching module is provided, which selects a priority viewpoint from all viewpoints, shoots a new silkworm pupa at the priority viewpoint, obtains standard point cloud data of the new silkworm pupa, defines the data as target point cloud data, matches the target point cloud data with each three-dimensional shape model in a database, and determines whether there is a three-dimensional shape model with a matching degree greater than a first threshold value. If so, if the three-dimensional shape model is a first shape model, the type of the new silkworm pupa is determined to be a normal silkworm pupa; if the three-dimensional shape model is a second shape model, the type of the new silkworm pupa is evaluated based on the first shape model corresponding to the second shape model; A neural network evaluation module is provided, if there is no three-dimensional shape model whose matching degree is greater than the first threshold value, then a neural network model is established, the two-dimensional image is input into the neural network model as a training set, the neural network model extracts first feature information therefrom, the first feature information includes normal features and abnormal features, a two-dimensional image of a new silkworm pupa under the priority perspective is obtained, defined as a target image, the target image is input into the neural network model, second feature information of the new silkworm pupa is output, and the type of the new silkworm pupa is obtained based on the second feature information.

10. A computer storage medium, characterized in that: The computer storage medium stores program instructions, wherein when the program instructions are executed, the device where the computer storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images