Family tree identification method, device, equipment and computer readable storage medium

Abstract

The application provides a family tree identification method, device and equipment and a computer readable storage medium. The method comprises: acquiring a to-be-identified image; inputting the to-be-identified image into a first convolutional neural network model trained by transfer learning to output an image type; when the image type is a family tree type, inputting the to-be-identified image into a second convolutional neural network model and a third convolutional neural network model trained by transfer learning respectively to output individual features in the family tree and a kinship line of generational individuals; and according to the individual features and the kinship line, a final family tree structure is parsed in combination with a drawing rule of the family tree. The method labels the collected family tree, trains the first, second and third convolutional neural network models respectively according to the principle of transfer learning, and realizes a high identification rate of the family tree. Meanwhile, the method has high training efficiency and can adapt to the requirements of incremental learning in actual applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition and processing, and particularly relates to a pedigree recognition method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] A pedigree is a common tool used to analyze the family background of patients in medical genetics. It can reflect various aspects of information related to the health problems of probands in their families, mainly including four aspects: family structure, demographic data, family life events and health problems. In addition, the genetic relationship of patients in the pedigree can be used to determine the inheritance mode of monogenic genetic diseases, such as autosomal dominant inheritance, autosomal recessive inheritance, sex-linked dominant inheritance and sex-linked recessive inheritance. With the widespread development of disease gene positioning, researchers often need to use pedigrees as a tool for data collection and preservation. Therefore, it is particularly important to recognize pedigrees.

[0003] At present, in the pedigree recognition, it mainly depends on the experience of professionals to make judgments. This method is time-consuming and laborious, and is usually subjective. For those who lack experience, it is likely to make a wrong judgment. Even well-trained medical workers may get different results. SUMMARY

[0004] Therefore, in the embodiments of the present application, a pedigree recognition method, device, equipment and computer readable storage medium are provided.

[0005] In a first aspect, the embodiments of the present application provide a pedigree recognition method, which comprises:

[0006] obtaining a to-be-recognized image;

[0007] inputting the to-be-recognized image into a first convolutional neural network model trained by transfer learning to output an image type;

[0008] when the image type is a pedigree type, inputting the to-be-recognized image into a second convolutional neural network model and a third convolutional neural network model trained by transfer learning respectively to output individual features and a kinship line;

[0009] establishing a pedigree tree according to the individual features and the kinship line.

[0010] In a second aspect, the embodiments of the present application provide a pedigree recognition device, which comprises:

[0011] a to-be-recognized image obtaining module, configured to obtain a to-be-recognized image;

[0012] an image type output module, configured to input the image to be recognized into a first convolutional neural network model trained by transfer learning, and output an image type;

[0013] a feature and relationship line output module, configured to, when the image type is a family tree type, input the image to be recognized into a second convolutional neural network model and a third convolutional neural network model trained by transfer learning respectively, to output individual features and a kinship relationship line;

[0014] a family tree establishment module, configured to establish a family tree according to the individual features and the kinship relationship line.

[0015] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory; one or more processors coupled with the memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to execute the family tree identification method provided in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores program codes, and the program codes can be called by a processor to execute the family tree identification method provided in the first aspect. The family tree identification method, device, equipment and computer readable storage medium provided in the embodiments of the present application first acquire an image to be recognized; then input the image to be recognized into a first convolutional neural network model trained by transfer learning, and output an image type; when the image type is a family tree type, input the image to be recognized into a second convolutional neural network model and a third convolutional neural network model trained by transfer learning respectively, to output individual features and a kinship relationship line; and finally establish a family tree according to the individual features and the kinship relationship line. The method labels the collected family tree, trains the first, second and third convolutional neural network models according to the principle of transfer learning, and realizes high recognition rate of the family tree. Meanwhile, the method has high training efficiency and can adapt to the requirement of incremental learning in actual application. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0018] Figure 1 Application scenario diagram of the family tree identification method provided in the embodiments of the present application;

[0019] Figure 2A flowchart of a family tree recognition method provided in an embodiment of the present application is shown in FIG. 1.

[0020] Figure 3 An individual feature label provided in an embodiment of the present application is shown in FIG. 2.

[0021] Figure 4 Position information of an individual feature label and a kinship line label provided in an embodiment of the present application is shown in FIG. 3.

[0022] Figure 5 A change graph of mAP and Loss values in a training process of a second convolutional neural network model provided in an embodiment of the present application is shown in FIG. 4.

[0023] Figure 6 A kinship line label provided in an embodiment of the present application is shown in FIG. 5.

[0024] Figure 7 A structure of a kinship line label required by a third convolutional neural network model provided in an embodiment of the present application is shown in FIG. 6.

[0025] Figure 8 A change graph of mAP and Loss values in a training process of a third convolutional neural network model provided in an embodiment of the present application is shown in FIG. 7.

[0026] Figure 9 A family tree provided in an embodiment of the present application is shown in FIG. 8.

[0027] Figure 10 A structure of a family tree recognition device provided in an embodiment of the present application is shown in FIG. 9.

[0028] Figure 11 A structure of a terminal device provided in an embodiment of the present application is shown in FIG. 10.

[0029] Figure 12 A structure of a computer readable storage medium provided in an embodiment of the present application is shown in FIG. 11. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described in detail below. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0031] In order to describe the present application in more detail, a family tree recognition method, device, terminal device and computer storage medium provided by the present application are described in detail below with reference to the accompanying drawings.

[0032] Please refer toFigure 1 , Figure 1 A schematic diagram of an application scenario of the family tree identification method provided in the embodiments of the present application is shown, which includes the terminal device 100 provided in the embodiments of the present application. The terminal device 100 can be various electronic devices (such as the structural diagrams of 102, 104, 106 and 108) with a display screen, including but not limited to smart phones and computer devices. The computer device can be at least one of a desktop computer, a portable computer, a laptop computer, a tablet computer and the like. A user operates the terminal device 100 to issue an identification operation instruction of a family tree, and the terminal device 100 executes the family tree identification method of the present application. For details, refer to the family tree identification method embodiments.

[0033] Secondly, the terminal device 100 can be used to represent one of a plurality of terminal devices. The embodiments of the present application only take the terminal device 100 as an example. Those skilled in the art can know that the number of the terminal devices can be more or less. For example, the terminal devices can be only a few, or the terminal devices can be dozens or hundreds, or more. The number and type of the terminal devices are not limited in the embodiments of the present application. The terminal device 100 can be used to execute the family tree identification method provided in the embodiments of the present application.

[0034] In an optional implementation, the application scenario includes the terminal device 100 provided in the embodiments of the present application, and further includes a server. The server is connected with the terminal device through a network. The network is used to provide a communication link medium between the terminal device and the server. The network can include various connection types, such as wired, wireless communication links or optical fiber cables and the like.

[0035] It should be understood that the number of terminal devices, networks and servers is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers. For example, the server can be a server cluster composed of multiple servers. The terminal device interacts with the server through the network to receive or send messages and the like. The server can be a server providing various services. The server can be used to execute the steps of the family tree identification method provided in the embodiments of the present application. In addition, the terminal device can execute part of the steps in the terminal device and part of the steps in the server when executing the family tree identification method provided in the embodiments of the present application. Here, it is not limited.

[0036] Based on this, the embodiments of the present application provide a family tree identification method. Please refer to Figure 2 , Figure 2 A flowchart of a family tree identification method provided in the embodiments of the present application is shown. Taking the terminal device in the application scenario as an example, the method includes the following steps: Figure 1 ​

[0037] In step S110, an image to be recognized is acquired.

[0038] The image to be recognized can be any image to be recognized, and can be a family tree or a non-family tree. The family tree, also referred to as a genealogy, is a visual view presenting various relationships in a family and various information about the family, such as blood relationships and marriage relationships from great-grandparents to three generations of oneself. As a clinical tool, the family tree can facilitate, quickly and effectively, the collection of information about the family in a horizontal dimension and the analysis of family structure and family relationship patterns.

[0039] In step S120, the image to be recognized is input to a first convolutional neural network model trained by transfer learning, and an image type is output.

[0040] In step S130, when the image type is a family tree type, the image to be recognized is input to a second convolutional neural network model and a third convolutional neural network model trained by transfer learning, respectively, to output individual features and a relationship line.

[0041] A convolutional neural network (CNN) is a representative classification of a deep neural network, and is a kind of feedforward neural network. Artificial neurons of the CNN can respond to surrounding units in a partial coverage range, and have excellent performance in processing large images. The CNN is composed of one or more convolution layers and fully connected layers (corresponding to a classical neural network), and also includes associated weights and a pooling layer. This structure enables the CNN to utilize the two-dimensional structure of input data. Compared with other deep learning structures, the CNN can give better results in image and speech recognition.

[0042] Transfer learning is a kind of machine learning method, which simply uses a model developed for task A as an initial point to develop a model for task B. For example, a model for recognizing animals can also be retrained for recognizing cars.

[0043] Since the input image to be recognized can be a family tree or can not be a family tree, the individual features and the relationship line cannot be recognized if it is not a family tree. Therefore, the first convolutional neural network model trained by transfer learning is used to classify the category of the image to be recognized, that is, the first convolutional neural network model is used to determine the type of the image to be recognized, and to judge whether the image to be recognized is a family tree or a non-family tree.

[0044] When the image to be recognized is determined to be a family tree, the second convolutional neural network model and the third convolutional neural network model are used to extract individual features and a relationship line, respectively.

[0045] The individual characteristics refer to characteristics related to each individual in the family tree, including but not limited to one or more of gender, whether or not suffering from a disease, whether or not alive, and whether or not a proband. The line of kinship refers to lines connecting individuals in the family tree, representing different kinship, including but not limited to a line of spouse relationship, a line of sibling relationship, and a line of parent / child relationship.

[0046] In addition, the first, second and third convolutional neural network models can be the same model or different models, which can be selected according to actual needs.

[0047] Step S140: establishing a family tree according to the individual characteristics and the line of kinship.

[0048] Specifically, the family tree of the entire family can be established according to the individual characteristics and the line of kinship. After obtaining the family tree, the family tree can be stored in a specific data structure (for example, in Json format) for later use, for example, for genetic disease research.

[0049] The family tree recognition method provided by the embodiments of the present application labels the collected family tree, respectively trains the first, second and third convolutional neural network models according to the principle of transfer learning, and realizes high recognition rate of the family tree. At the same time, the method has high training efficiency and can adapt to the requirements of incremental learning in actual application.

[0050] Further, an embodiment of training the first convolutional neural network model is provided, which is described in detail as follows.

[0051] In one embodiment, training the first convolutional neural network model comprises:

[0052] S1: obtaining an image sample; wherein the image sample comprises a family tree and a non-family tree.

[0053] Specifically, the image samples include multiple images of family trees and non-family trees. The image samples are obtained from network platforms (e.g., Bing, Google, and CNKI scientific picture library). The family trees are searched and downloaded from the three platforms using web queries according to the two keywords of “family tree” and “pedigree”, and then the downloaded images are processed to remove the family trees that do not meet the requirements. The family trees that do not meet the requirements include but are not limited to the following cases: (1) a single picture has multiple family trees. When multiple family trees are in one picture, it is difficult to automatically divide different family trees by the model due to the indefinite distance between the family trees; (2) the picture is too blurred. If the blurred family tree is used for model training, it will affect the correct identification of the model on the clear picture, so the picture that is blurred to the extent that the human eye cannot easily recognize is removed; (3) the family tree is not drawn according to the standard. The correct identification of individual characteristics and kinship lines depends on the standard drawing rules of the family tree. Irregular family trees will affect the judgment of the model, such as the same generation not being in the same vertical coordinate and the kinship lines overlapping each other.

[0054] It should be noted that the more the number of family trees and non-family trees in the image samples, the more accurate the model will be trained. However, if the number is too large, the model training time will be longer, so in actual model training, a sufficient number of image samples can be selected. In this embodiment, 786 family trees and 649 non-family trees are selected to form the image samples.

[0055] S2: The family trees are labeled with a first picture type label, and the non-family trees are labeled with a second picture type label.

[0056] After obtaining the non-family trees and the family trees that meet the requirements, the picture classification labels (i.e., the first picture type label and the second picture type label) can be used to label the family trees and the non-family trees. The first picture type label is a family tree type label, and the second picture type label is a non-family tree type label.

[0057] In addition, after labeling the family trees and the non-family trees, the labeled family trees and non-family trees can be divided into a training set and a test set (in this example, the training set: test set = 4:1) according to a certain proportion, and then the training set is used to train the first convolutional neural network model. After the model training is completed, the test set is used to test the trained first convolutional neural network model to determine the prediction effect of the model.

[0058] S3: The labeled family trees and non-family trees are input into the first convolutional neural network model, the first loss function is calculated, the parameters are updated until convergence, and the trained first convolutional neural network model is obtained.

[0059] The model training is to give an input vector and a target output value, then input the input vector into one or more network structures or functions to obtain an actual output value, calculate the bias according to the target output value and the actual output value, and determine whether the bias is within the allowable range; if it is within the allowable range, the training is ended and the related parameters are fixed; if it is not within the allowable range, some parameters in the neural network structure or function are constantly adjusted until the bias is within the allowable range or a certain ending condition is reached, the training is ended and the related parameters are fixed, and finally the trained model can be obtained according to the fixed related parameters. In the embodiment, the family tree and the non-family tree in the training set are input into the first convolutional neural network to calculate the first loss function, update the parameters until convergence, and obtain the first convolutional neural network model trained by transfer learning.

[0060] In a preferred embodiment, the first convolutional neural network is an improved VGG16 model; wherein the VGG16 model includes multiple convolutional layers, pooling layers and fully connected layers; the last fully connected layer after improvement includes two neurons.

[0061] Specifically, the VGG16 model performs excellently in image classification task processing, so the improved VGG16 model is used to identify the type of the image to be identified in the embodiment. The improved VGG16 model includes 13 convolutional layers (Convolutional Layer), 5 pooling layers (Pool layer), 3 fully connected layers (Fully connected Layer), and the last fully connected layer has two neurons for image two-classification. The expression of the fully connected layer is: In the formula, wherein yl represents the output of the nth neuron in the lth layer, Nl represents the number of output layer neurons, yl-1,m represents the output of the mth neuron in the l-1th layer, wl-1,m,n is the weight of the connection between the mth neuron in the l-1th layer and the nth neuron in the lth layer, and σl is the relu activation function, bl,n is the bias of the nth neuron.

[0062] In addition, the training process of the improved VGG16 model is as follows: the family tree and the non-family tree marked with image classification labels are used to train the improved VGG16 weight, and finally an image classification model capable of distinguishing the family tree and the non-family tree is obtained. This method can quickly and accurately train the model, so that the image type can be quickly identified by using the model, which is convenient to operate and efficient.

[0063] Next, an embodiment of training a second convolutional neural network model is provided, which is described in detail as follows.

[0064] In one embodiment, training the second convolutional neural network model comprises:

[0065] S4 obtains a pedigree sample.

[0066] S5: The pedigree sample is labeled with individual feature labels.

[0067] S6: The labeled pedigree sample is input into the second convolutional neural network model, a second loss function is calculated, and the parameters are updated until convergence to obtain the trained second convolutional neural network model.

[0068] Specifically, the pedigree sample is determined to be a picture of a pedigree by the trained first convolutional neural network model.

[0069] In the pedigree, individuals are usually represented by graphs, and lines represent kinship. For example, each "□" or "○" represents an individual, the vertical direction represents different generations; "□" represents a male, and "○" represents a female; "□", "○" are connected by a horizontal line, indicating that they are in a "spouse relationship"; a vertical line is drawn from the middle point of the marriage line to the lower end, and the individual connected at the lower end is the child of the previous generation of husband and wife. If there are more than two children, they can be arranged from left to right according to birth order, and the individuals enclosed by "left and right corners" are siblings. The pedigree Figure 1 is usually composed of three generations. The elders are on the top, and the descendants are on the bottom. In the same generation, the elders are on the left, and the young are on the right. Solid indicates that the individual exhibits a phenotype of a related genetic variation disease (i.e., is ill), and hollow indicates that the individual is not ill. The diagonal line indicates that the individual has died. Since the judgment of kinship requires the integration of information such as individual gender, relationship line direction, etc., using a single model to recognize multiple types of information can easily cause incorrect recognition and thus affect accuracy. Therefore, in this embodiment, a second convolutional neural network and a third convolutional neural network are used to recognize individual features and kinship lines, respectively.

[0070] The individual feature label can be used to determine the individual feature type in the pedigree. Since the individual feature specifically includes one or more of gender, whether or not ill, whether or not alive, and whether or not a proband, the types of individual feature labels are multiple. Each individual feature label corresponds to an individual graph, and the specific types of individual feature labels are shown in Table 1. Figure 3

[0071] In addition, since the pedigree includes individuals of different generations, although the individuals of different generations are represented by the same set of graphs, their positions in the pedigree are different. Therefore, when labeling the individuals of the pedigree with individual feature labels, label position information is required. The specific types of individual feature labels are shown in Table 2. Figure 4

[0072] ​​In an optional implementation, the labelImg software can be used to label the individual feature labels of the individuals in the pedigree chart.

[0073] In addition, after the individuals in the pedigree chart are labeled with the individual feature labels, the labeled pedigree chart can be divided into a training set and a test set according to a certain proportion, and then the training set is used to train the second convolutional neural network model. After the model training is completed, the test set is used to test the trained second network convolutional neural model to determine the prediction effect of the model.

[0074] In this embodiment, the pedigree chart in the training set is input into the second convolutional neural network, the second loss function is calculated, the parameters are updated until convergence, and the trained second convolutional neural network model is obtained.

[0075] In one embodiment, the second convolutional neural network is a YoloV5 model, and the loss function used when training the YoloV5 model is CIOU_Loss.

[0076] The YoloV5 model includes four parts of structures of an input end, a Backbone, a Neck and a Prediction; and the YoloV5 uses the CIOU_Loss as the loss function at the output end. During the model training, the training set of the individual feature labels is used to train the YoloV5 model to obtain the YoloV5 model for identifying the individual features in the pedigree chart. Optionally, during the model training, the neural network parameters can be: iteration number = 26000, Batch_size = 64, Learning_rate = 0.001. Among them, Batch refers to updating the model weight by using a small part of samples in the training set for one time of back propagation, and the number of samples in this batch is Batch size.

[0077] After obtaining the trained YoloV5 model, the test set of the individual feature labels can be used to test the performance of the YoloV5 model, and the test result is as shown in Figure 5 Figure 5 The mAP value and the loss value (Loss) of the test set are represented by accuracy, wherein accuracy = (the number of correctly predicted samples) / (the total number of samples).

[0078] This method can quickly and accurately train a model so as to quickly identify the individual features in the pedigree chart by using the model, and the operation is convenient and efficient.

[0079] Further, an embodiment of training a third convolutional neural network model is provided, which is described in detail as follows.

[0080] In one embodiment, training the third convolutional neural network model includes:​

[0081] S7: Obtain a pedigree sample.

[0082] S8: Label the pedigree sample with the kinship label.

[0083] S9: Input the labeled pedigree sample into the third convolutional neural network model, calculate the third loss function, and update the neural network parameters until convergence to obtain the trained third convolutional neural network model.

[0084] The pedigree sample is predicted by the trained first convolutional neural network model, and is determined to be a picture of a pedigree.

[0085] The kinship line label refers to a label used to represent the kinship between individuals. According to the label, the kinship between individuals can be determined. The kinship includes "spouse relationship (i.e. spouse relationship)", "parent / child relationship" and "sibling (i.e. sibling relationship)". These kinship relationships can be represented by the "left and right" and "up and down" directional relationship lines. In addition, since the "gender" and "disease" features are crucial for pedigree analysis, and in order to verify the multiple effects of subsequent kinship analysis, the "gender" and "disease" information of the individual feature label is also retained in this step. Therefore, the kinship line label has four types of information: "gender", "disease", "up and down directional relationship line" and "left and right directional relationship line", a total of 25 labels, including label number, label meaning and label number. Each kinship line label corresponds to an individual graph, as shown in Figure 6 For the convenience of subsequent program recognition, in addition to the left and right corner lines for identifying sibling relationships (i.e. sibling relationships) in this step, the other four types of information are designed as four sections, which represent different information in turn, as shown in Figure 7 ; Figure 7 The first two fields are used to verify each other with the individual feature label, which describes the "gender" and "disease" of the individual graph, respectively. The last two fields mainly describe the position of the line in the individual graph (i.e. reflecting the kinship), of which the third field describes the left and right directional line, and the fourth field describes the up and down directional line. For example, the label with the number "0" has a line representing the spouse relationship on the right side, and no line in the up and down direction, so the third field is "right side" and the fourth field is "none".

[0086] In addition, when labeling the individuals of the pedigree with the kinship label, the label position information is needed, as shown in Figure 4

[0087] ​In an optional implementation, the labelImg software can be used to label the individuals in the pedigree diagram with the relationship lines.

[0088] In addition, after the pedigree diagram is labeled with the relationship lines, the labeled pedigree diagram can be divided into a training set and a test set according to a certain proportion, and then the training set is used to train the third convolutional neural network model. After the model training is completed, the test set is used to test the trained third convolutional neural network model to determine the prediction effect of the model.

[0089] In this embodiment, the pedigree diagram in the training set is input into the third convolutional neural network to calculate the third loss function, and the neural network parameters are updated until convergence to obtain the trained third convolutional neural network model.

[0090] In one embodiment, the third convolutional neural network is a YoloV5 model, and the loss function used when training the YoloV5 model is CIOU_Loss.

[0091] In this embodiment, YoloV5 uses CIOU_Loss as the loss function at the output end. During model training, the training set with the relationship lines is used to train the YoloV5 model to obtain the YoloV5 model for identifying the relationship lines between individuals in the pedigree diagram. Optionally, during model training, the neural network parameters can be: iteration number (Epoch) = 50000, Batch_size = 64, Learning_rate = 0.001.

[0092] After obtaining the trained YoloV5 model, the test set can be used to test the performance of the YoloV5 model, and the test results are as shown in Figure 8 Figure 8 The mAP value and the loss value (Loss) of the test set are represented.

[0093] This method can quickly and accurately train a model so as to quickly identify the relationship lines in the pedigree diagram using the model, which is convenient to operate and efficient.

[0094] Finally, a construction method of a pedigree tree is provided, which is described in detail as follows.

[0095] In one embodiment, in step S140, the pedigree tree is established according to the individual characteristics and the relationship lines, including: identifying the generation individuals and the distribution of the generation individuals according to the individual characteristics; identifying the spouse relationship, sibling relationship and parent / child relationship of the generation individuals according to the relationship lines; and constructing the pedigree tree according to the distribution of the generation individuals, the spouse relationship, the sibling relationship and the parent / child relationship of the individuals. ​

[0096] Specifically, the construction process of the family tree is as follows: (1) identification of generations, sorting all individual y-coordinates identified by the second convolutional neural network model from small to large, and distributing them to different generations according to whether the y-coordinates of individuals overlap, which can obtain the individual distribution of each generation for subsequent judgment. (2) Identification of couple relationship: if the individuals in the same generation meet the condition of 1 male + 1 female, and the relative relationship line in the left and right directions, it is judged that the two are in a couple relationship. This step can obtain the husband and wife relationship of each generation. (3) Identification of sibling relationship: if the individuals in the same generation contain upward relationship lines and are wrapped by a pair of left and right corner sibling relationship lines, it is considered that these individuals are siblings. This step can obtain the brother and sister relationship of each generation. (4) Identification of parent / child relationship: between the adjacent two generations, if the individuals in the upper generation are husband and wife and the individuals in the lower generation are siblings, there is an overlap in the x-coordinate direction, then it is considered that these individuals constitute a small family. This step can obtain the relationship of parents / children. (5) According to the generation distribution, husband and wife relationship, brother and sister relationship, and parent / child relationship obtained in the previous four steps, a family tree that can be understood semantically is constructed, and is stored in a specific data structure. The structure of the family tree is as follows: Figure 9 Wherein G1, G2, G3 in the family tree represent the first, second, and third generations; Member represents the individual number in each generation; yline represents the y-coordinate center value of the generation; Couple represents the husband and wife list in the generation; C1, C2, C3 represent the first, second, and third pairs of husband and wife in the generation; cp_member represents the individual number of the couple relationship; childer_member represents the individual number of the children of the couple relationship; Sibling represents the sibling relationship list in the generation; S1, S2, S3 represent the first, second, and third pairs of sibling relationships in the generation; Cap represents the sibling relationship line number in the sibling relationship; Sib_member represents the individual number in the sibling relationship; Kids represents the only child list in the generation; K1, K2, K3 represent the first, second, and third only children in the generation.

[0097] The family tree is convenient for storage and reuse, and also convenient for computer program language to understand the family tree.

[0098] It should be understood that, although Figure 2 The steps in the flowchart of the method are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover Figure 2At least one of the steps in the above method can include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the execution of the sub-steps or stages is not necessarily sequential, but can be performed alternately or alternately with at least one of the other steps or the sub-steps or stages of the other steps.

[0099] The method for identifying a family tree is described in detail in the embodiments of the present application. The method can be implemented in various forms of devices, and therefore the present application also discloses a device for identifying a family tree corresponding to the method. The following embodiments are given to explain the device in detail.

[0100] Please refer to Figure 10 The device for identifying a family tree disclosed in the embodiments of the present application mainly includes:

[0101] The image to be identified is obtained by the image to be identified acquisition module 1010.

[0102] The image type output module 1020 is configured to input the image to be identified into the first convolutional neural network model trained by transfer learning, and output the image type.

[0103] The feature and relationship line output module 1030 is configured to input the image to be identified into the second convolutional neural network model and the third convolutional neural network model trained by transfer learning when the image type is the family tree type, so as to output the individual features and the kinship relationship line.

[0104] The family tree establishment module 1040 is configured to establish a family tree according to the individual features and the kinship relationship line.

[0105] In one embodiment, the device further includes:

[0106] The first sample acquisition module is configured to acquire image samples, wherein the image samples include family trees and non-family trees.

[0107] The first labeling module is configured to label the family trees with a first picture type label.

[0108] The second labeling module is configured to label the non-family trees with a second picture type label.

[0109] The first model training module is configured to input the labeled family trees and non-family trees into the first convolutional neural network model, calculate a first loss function, update parameters until convergence, and obtain a trained first convolutional neural network model.

[0110] The second sample acquisition module is configured to acquire family tree samples.

[0111] The third marking module is configured to mark the pedigree chart sample by using the individual feature label.

[0112] The second model training module is configured to input the marked pedigree chart sample into a second convolutional neural network model, calculate a second loss function, update parameters until convergence, and obtain a trained second convolutional neural network model.

[0113] The third sample obtaining module is configured to obtain the pedigree chart sample.

[0114] The fourth marking module is configured to mark the pedigree chart sample by using the kinship line label.

[0115] The third training module is configured to input the marked pedigree chart sample into a third convolutional neural network model, calculate a third loss function, update parameters until convergence, and obtain a trained third convolutional neural network model.

[0116] In an embodiment, the family tree establishing module 1040 is configured to identify generation individuals and distribution of the generation individuals according to the individual features, identify spouse relationship, sibling relationship and parent / child relationship of the individuals according to the kinship lines, and construct a family tree according to the distribution of the generation individuals, the spouse relationship, the sibling relationship and the parent / child relationship of the individuals.

[0117] In an embodiment, the first convolutional neural network is an improved VGG16 model; the improved VGG16 model includes multiple convolutional layers, pooling layers and fully connected layers; and the last fully connected layer includes two neurons.

[0118] In an embodiment, the second convolutional neural network and / or the third convolutional neural network is a YoloV5 model, and a loss function used when training the YoloV5 model is CIOU_Loss.

[0119] The specific limitations of the pedigree chart recognition device can refer to the limitations of the method described above, which will not be repeated here. Each module in the above device can be realized by software, hardware and their combination in whole or in part. Each module described above can be embedded in or independent of the processor in the terminal device in hardware form, or can be stored in the memory in the terminal device in the form of software, so as to call and execute the operation corresponding to each module by the processor.

[0120] Please refer to Figure 11 , Figure 11It shows a structural block diagram of a terminal device provided by an embodiment of the present application. The terminal device 110 can be a computer device. The terminal device 110 in the present application can include one or more of the following components: a processor 112, a memory 114, and one or more application programs, wherein the one or more application programs can be stored in the memory 114 and configured to be executed by the one or more processors 112, and the one or more application programs are configured to execute the method described in the above family tree identification method embodiment.

[0121] The processor 112 can include one or more processing cores. The processor 112 connects various parts within the entire terminal device 110 through various interfaces and lines, executes various functions of the terminal device 110 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 114, and calling data stored in the memory 114. Optionally, the processor 112 can be implemented in at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 112 can integrate a combination of one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU) for reporting and verifying embedded point data, and modem. Among them, the CPU mainly processes the operating system, user interface and application programs, etc.; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 112, but can be realized by a separate communication chip.

[0122] The memory 114 can include Random Access Memory (RAM) and can also include Read-Only Memory (ROM). The memory 114 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 114 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as touch function, sound playing function, image playing function, etc.), instructions for implementing each of the following method embodiments, etc. The data storage area can also store data created by the terminal device 110 in use, etc.

[0123] Those skilled in the art can understand that, Figure 11The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] To sum up, the terminal device provided by the embodiments of the present application is used to implement the corresponding family tree identification method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here.

[0125] Please refer to Figure 12 which shows a structural block diagram of a computer-readable storage medium provided by the embodiments of the present application. The computer-readable storage medium 120 stores program codes, which can be called and executed by a processor to implement the method described in the foregoing family tree identification method embodiments.

[0126] The computer-readable storage medium 120 can be an electronic storage such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM. Alternatively, the computer-readable storage medium 120 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 120 has a storage space for program codes 122 for executing any method steps in the foregoing methods. These program codes can be read out from or written into one or more computer program products. The program codes 122 can be compressed in an appropriate form, for example.

[0127] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0128] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of identifying a family tree, characterized by, The method comprises: acquiring an image to be identified; the image to be identified comprises a family tree and a non-family tree; inputting the image to be identified into a first convolutional neural network model trained by transfer learning, and outputting an image type; when the image type is a family tree type, inputting the image to be identified into a second convolutional neural network model and a third convolutional neural network model trained by transfer learning respectively, to output individual characteristics and a kinship line; the individual characteristics comprise one or more of gender, whether suffering from a disease, whether alive, and whether a proband; establishing a family tree according to the individual characteristics and the kinship line; wherein the establishing of the family tree according to the individual characteristics and the kinship line comprises: identifying generation individuals and generation individual distribution according to the individual characteristics; identifying spouse relationships, sibling relationships and parent / child relationships of the generation individuals according to the kinship line; constructing a family tree according to the generation individual distribution, the spouse relationships, the sibling relationships and the parent / child relationships of the generation individuals.

2. The method of claim 1, wherein, The training of the first convolutional neural network model comprises: acquiring image samples; wherein the image samples comprise family trees and non-family trees; labeling the family trees with first picture type labels and labeling the non-family trees with second picture type labels; inputting the labeled family trees and non-family trees into the first convolutional neural network model, calculating a first loss function, updating neural network parameters until convergence, to obtain a trained first convolutional neural network model.

3. The method of claim 1, wherein, The training of the second convolutional neural network model comprises: acquiring family tree samples; labeling the family tree samples with individual characteristic labels; inputting the labeled family tree samples into the second convolutional neural network model, calculating a second loss function, updating parameters until convergence, to obtain a trained second convolutional neural network model.

4. The method of claim 1, wherein, The training of the third convolutional neural network model comprises: acquiring family tree samples; labeling the family tree samples with kinship line labels; inputting the labeled family tree samples into the third convolutional neural network model, calculating a third loss function, updating parameters until convergence, to obtain a trained third convolutional neural network model.

5. The method according to any one of claims 1 to 4, characterized in that, The first convolutional neural network is an improved VGG16 model; wherein the improved VGG16 model comprises a plurality of convolutional layers, pooling layers and fully connected layers; and the last fully connected layer comprises two neurons.

6. The method according to any one of claims 1 to 4, characterized in that, The second convolutional neural network and / or the third convolutional neural network is a YoloV5 model, wherein a loss function used when training the YoloV5 model is CIOU_Loss.

7. An identification device of a family tree, characterized by The device comprises: an image to be identified acquisition module, configured to acquire an image to be identified; the image to be identified comprises a family tree and a non-family tree; an image type output module, configured to input the image to be identified into a first convolutional neural network model trained by transfer learning, and output an image type; The feature and relationship line output module is configured to input the to-be-identified image into a second convolutional neural network model and a third convolutional neural network model trained by transfer learning when the image type is a family tree type, so as to output individual features and a kinship relationship line; the individual features include one or more of gender, whether suffering from a disease, whether alive, and whether a proband; The family tree establishment module is configured to establish a family tree according to the individual features and the kinship relationship line; The family tree establishment module is configured to: identify generation individuals and generation individual distribution according to the individual features; identify a spouse relationship, a sibling relationship, and a parent / child relationship of an individual according to the kinship relationship line; construct a family tree according to the generation individual distribution, the spouse relationship, the sibling relationship, and the parent / child relationship of the individual.

8. A terminal device, comprising: The family tree establishment module includes: a memory; one or more processors coupled to the memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, and the program code can be called and executed by the processor to perform the method according to any one of claims 1-7.

Images