Method and device for evaluating nasal septum deviation based on image recognition

Abstract

The present application relates to the field of image recognition, and discloses a method for evaluating nasal septum deviation based on image recognition, the method comprising: obtaining a nasal septum deviation image; wherein the nasal septum deviation image comprises a cross-sectional CT image of the nasal cavity; in the nasal septum deviation image, determining multiple key points of the nasal septum cartilage according to a nasal septum deviation detection model; wherein the multiple key points of the nasal septum cartilage comprise the front end point, the deflection point and the rear end point of the nasal septum cartilage; according to the multiple key points of the nasal septum cartilage, determining the angle of the connection line of the multiple key points of the nasal septum cartilage, and taking the angle of the connection line of the multiple key points as the nasal septum deviation angle; wherein the angle of the connection line of the multiple key points comprises the angle of a first connection line connecting the front end point and the deflection point, and a second connection line connecting the front end point and the rear end point. The present application automatically identifies the degree of nasal septum deviation, and improves the treatment effect of nasal septum deviation of patients.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and more specifically, to a method and device for evaluating nasal septum deviation based on image recognition. Background Art

[0002] Nasal septum deviation is a common anatomical variation worldwide. Patients will experience symptoms such as nasal obstruction, sinusitis, vocal cord abnormalities or dyspnea, which seriously affect the patient's physical health and quality of life. According to statistics, analysis data of nasal CT scans of adults show that the incidence of nasal septum deviation is 86.6%. Studies have found that the incidence of nasal septum deviation increases with age, and the risk of nasal septum deviation increases by 3.2% for every 10 years of age. Therefore, early screening for nasal septum deviation is crucial. Imaging examination is an important means of screening for nasal septum deviation. Through imaging examinations such as X-ray examinations, computed tomography (CT) and magnetic resonance imaging (MRI), the structure and deviation of the nasal septum can be observed more accurately. Figure 1 The following is a schematic diagram of CT images of a normal nasal septum and a deviated nasal septum. Figure 1 The figure a in the middle shows the appearance of a normal nasal septum in CT images. Figure 1 Figure b shows the appearance of nasal septum deviation in CT images.

[0003] Figure 2 This is a schematic diagram of the medical community's classification of nasal septum deviation. Figure 2 As shown in the figure, according to the different deviation shapes of the nasal septum, the medical community divides nasal septum deviation into 6 basic types from type I to type VI, of which type III and type I are more common. In addition, there is type VII nasal septum deviation, which is Figure 2 There are six basic types of nasal septum deviation, from type I to type VI. Since the conditions of different patients vary greatly, doctors in clinical practice need to accurately quantify the patient's nasal septum deviation in order to make an accurate diagnosis of the patient's condition. Figure 3 A schematic diagram of the nasal septum deviation angle is shown in Figure 1. Figure 3 As shown in the figure, the nasal septum deviation angle α is often used in clinic to describe the degree of deviation of the patient. The larger the nasal septum deviation angle α, the higher the degree of nasal septum deviation of the patient. At present, the measurement of the nasal septum deviation angle relies entirely on manual measurement. Doctors need to analyze and measure the patient's nasal anatomical structure on CT or other types of images. Manual measurement of nasal septum deviation is inefficient and has large measurement errors, making it difficult to achieve an accurate quantitative description of the patient's condition, which affects the treatment of the patient's nasal septum deviation. Summary of the invention

[0004] The purpose of this application is to provide a method and device for evaluating nasal septum deviation based on image recognition, which solves the technical problems of low efficiency and large measurement error in manual measurement of nasal septum deviation, and achieves the technical effect of automatically identifying the degree of nasal septum deviation.

[0005] An embodiment of the present application provides a method and device for evaluating nasal septum deviation based on image recognition, the method comprising: acquiring a nasal septum deviation image; wherein the nasal septum deviation image comprises a cross-sectional CT image of the nasal cavity; in the nasal septum deviation image, determining multiple key points of the nasal septum cartilage according to a nasal septum deviation detection model; wherein the multiple key points of the nasal septum cartilage comprise a front end point, a deviation point and a rear end point of the nasal septum cartilage; according to the multiple key points of the nasal septum cartilage, determining an angle of a line connecting the multiple key points of the nasal septum cartilage, and taking the angle of the line connecting the multiple key points as the nasal septum deviation angle; wherein the angle of the line connecting the multiple key points comprises an angle of a first line connecting the front end point and the deviation point, and an angle of a second line connecting the front end point and the rear end point.

[0006] In a possible implementation, in a nasal septum deviation image, multiple key points of the nasal septum cartilage are determined according to a nasal septum deviation detection model, including: downsampling through a general feature extraction module to determine the general features of the nasal septum image; wherein the general feature extraction module includes a Resnet18 model pre-trained on ImageNet; through a nasal septum semantic feature extraction module, the semantic features of multiple key points are determined according to the general features of the nasal septum image; wherein the nasal septum semantic feature extraction module includes a Unet model; through a key point output module, the intermediate coordinates and position deviation values ​​of multiple key points of the nasal septum cartilage are determined according to the semantic features of the multiple key points of the nasal septum cartilage, and the final coordinates of the multiple key points of the nasal septum cartilage are determined according to the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage.

[0007] In another possible implementation, the final coordinates of multiple key points of the nasal septum cartilage are determined based on the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage, including: adding and integerizing the intermediate coordinates and position deviation values ​​corresponding to the multiple key points of the nasal septum cartilage to obtain the final coordinates of the multiple key points of the nasal septum cartilage.

[0008] In another possible implementation, the intermediate coordinates and position deviation values ​​of multiple key points of the nasal septum cartilage are determined according to the semantic features of the multiple key points of the nasal septum cartilage through a key point output module, and the method also includes: determining the heat map of the multiple key points of the nasal septum cartilage according to the semantic features of the multiple key points of the nasal septum cartilage through a key point output module, and upsampling the heat map of the multiple key points of the nasal septum cartilage to determine the intermediate coordinates of the multiple key points of the nasal septum cartilage.

[0009] In another possible implementation, in the nasal septum deviation image, multiple key points of the nasal septum cartilage are determined, which also includes: in the nasal septum deviation image, the front end point of the nasal septum cartilage is marked in red, the deviation point of the nasal septum cartilage is marked in green, and the posterior end point of the nasal septum cartilage is marked in blue.

[0010] In another possible implementation, when training a nasal septum deviation detection model, the sample image is annotated by the following method: retaining the pixel value of each pixel of the sample image between -1024hu and 1024hu, and normalizing to obtain a normalized sample image; wherein the sample image includes multiple annotated points of the nasal septum cartilage corresponding to multiple key points of the nasal septum cartilage; determining an annotation grid for annotating the normalized sample image according to the parameters of the downsampling of the general feature extraction module, and determining multiple annotation grid points and annotation grid point correction values ​​corresponding to the multiple annotated points of the nasal septum cartilage in the annotation grid, and using the multiple annotation grid points and annotation grid point correction values ​​corresponding to each sample image as training labels; wherein the annotation grid points correspond to the heat map of the key points, the annotation grid point correction values ​​correspond to the position deviation values ​​of the key points, and the annotation grid point correction values ​​are used to correct the coordinates of the key points of the downsampled nasal septum cartilage.

[0011] An embodiment of the present application also provides a nasal septum deviation assessment device based on image recognition, including a unit for executing the above method.

[0012] An embodiment of the present application also provides a nasal septum deviation assessment device based on image recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method described above is implemented when the processor executes the computer program.

[0013] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0014] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the method described above when executed by a processor.

[0015] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0016] The embodiment of the present application provides a method for evaluating nasal septum deviation based on image recognition, the method comprising: obtaining a nasal septum deviation image; wherein the nasal septum deviation image comprises a cross-sectional CT image of the nasal cavity; in the nasal septum deviation image, determining multiple key points of the nasal septum cartilage according to a nasal septum deviation detection model; wherein the multiple key points of the nasal septum cartilage comprise the front end point, the deflection point and the rear end point of the nasal septum detection model; according to the multiple key points of the nasal septum cartilage, determining the angle of the multiple key points of the nasal septum cartilage, and taking the angle of the multiple key points as the nasal septum deviation angle; wherein the angle of the multiple key points comprises the angle of the first line connecting the front end point and the deflection point and the second line connecting the front end point and the rear end point. The embodiment of the present application can quickly and accurately identify the nasal septum deviation angle through image recognition, improve the efficiency and accuracy of identifying the nasal septum deviation angle, improve the efficiency of screening the nasal septum deviation angle, and also improve the treatment effect of the patient's nasal septum deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 The following are schematic diagrams of CT images of a normal nasal septum and a deviated nasal septum;

[0019] Figure 2 This is a schematic diagram of the medical community's classification of nasal septum deviation.

[0020] Figure 3 This is a schematic diagram of a case of nasal septum deviation angle;

[0021] Figure 4 A schematic diagram of a flow chart of a method for evaluating nasal septum deviation based on image recognition provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a process for identifying a deviated nasal septum provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a process for labeling a sample image provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a nasal septum deviation assessment device based on image recognition provided in an embodiment of the present application;

[0025] Figure 8A schematic diagram of another nasal septum deviation assessment device based on image recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0027] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0029] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0031] In the prior art, the measurement of the deviation angle of the nasal septum is completely dependent on manual measurement, and doctors need to analyze and measure the patient's nasal anatomical structure on CT or other types of images. Manual measurement of nasal septum deviation is inefficient and has large measurement errors, making it difficult to achieve an accurate quantitative description of the patient's condition, which affects the treatment of the patient's nasal septum deviation.

[0032] Based on the above reasons, the embodiment of the present application provides a method for evaluating nasal septum deviation based on image recognition, the method comprising: obtaining a nasal septum deviation image; wherein the nasal septum deviation image includes a cross-sectional CT image of the nasal cavity; in the nasal septum deviation image, determining multiple key points of the nasal septum cartilage according to the nasal septum deviation detection model; wherein the multiple key points of the nasal septum cartilage include the front end point, the deflection point and the rear end point of the nasal septum detection model; according to the multiple key points of the nasal septum cartilage, determining the angle of the multiple key points of the nasal septum cartilage, and taking the angle of the multiple key points as the nasal septum deviation angle; wherein the angle of the multiple key points includes the angle of the first line connecting the front end point and the deflection point and the second line connecting the front end point and the rear end point. The embodiment of the present application can quickly and accurately identify the nasal septum deviation angle through image recognition, improve the efficiency and accuracy of identifying the nasal septum deviation angle, improve the efficiency of screening the nasal septum deviation angle, and also improve the treatment effect of the patient's nasal septum deviation.

[0033] In some scenarios, a nasal septum deviation assessment method based on image recognition in an embodiment of the present application can be applied to nasal septum deviation screening. When a doctor needs to analyze and measure the patient's nasal anatomical structure on CT or other types of images, this method can accurately and automatically measure the nasal septum deviation.

[0034] In other scenarios, a nasal septum deviation assessment method based on image recognition in an embodiment of the present application can be applied to the treatment of nasal septum deviation. This method can accurately measure the nasal septum deviation, improve the accuracy of the assessment of the patient's nasal septum deviation, and improve the treatment effect of the patient's nasal septum deviation.

[0035] The following is a detailed description of a nasal septum deviation assessment method based on image recognition provided in an embodiment of the present application with reference to specific examples.

[0036] Figure 4 A schematic diagram of a flow chart of a method for evaluating nasal septum deviation based on image recognition provided in an embodiment of the present application, such as Figure 4 As shown, the method includes S110 to S130, and S110 to S130 are described in detail below.

[0037] S110, obtaining a nasal septum deviation image, wherein the nasal septum deviation image includes a nasal cross-sectional CT image.

[0038] When automatically identifying the deviation of the nasal septum, a cross-sectional CT image of the patient's nasal cavity can be taken as a nasal septum deviation image, and then image recognition is performed on the nasal septum deviation image to further detect the degree of deviation of the nasal septum.

[0039] S120: In the nasal septum deviation image, determine multiple key points of the nasal septum cartilage according to the nasal septum deviation detection model, wherein the multiple key points of the nasal septum cartilage include the front end point, the deviation point and the rear end point of the nasal septum detection model.

[0040] In nasal septum image recognition, the neural network model can be used to automatically identify multiple key points of the nasal septum cartilage.

[0041] like Figure 2 As shown, when the embodiment of the present application is used, type I nasal septum deviation and type III nasal septum deviation among the six types of nasal septum deviation are automatically identified. Among type I nasal septum deviation and type III nasal septum deviation, Figure 2 As shown, the multiple key points of the nasal septum cartilage include the front end point, the deflection point and the rear end point of the nasal septum cartilage. The front end point of the nasal septum cartilage is Figure 2 The top end of the nasal septum cartilage in the middle, the deflection point of the nasal septum cartilage is Figure 2 The lateral deviation point of the middle position of the nasal septum cartilage, the posterior end point of the nasal septum cartilage is Figure 2 The lowest point of the nasal septum cartilage.

[0042] S130, according to the multiple key points of the nasal septum cartilage, determine the angle of the connection lines of the multiple key points of the nasal septum cartilage, and use the angle of the connection lines of the multiple key points as the nasal septum deflection angle. The angle of the connection lines of the multiple key points includes the angle of a first connection line connecting the front end point and the deflection point and the angle of a second connection line connecting the front end point and the rear end point.

[0043] After identifying multiple key points of the nasal septum cartilage, the angle of the lines connecting the multiple key points of the nasal septum cartilage can be determined based on the multiple key points of the nasal septum cartilage, and the angle of the lines connecting the multiple key points can be used as the nasal septum deviation angle, and then the severity of the patient's nasal septum deviation can be evaluated based on the nasal septum deviation angle.

[0044] like Figure 2 and Figure 3 As shown, when automatically identifying type I nasal septum deviation and type III nasal septum deviation among the six types of nasal septum deviation, when determining the nasal septum deviation angle according to the angle between the lines connecting multiple key points, the first line connecting the front end point and the deviation point and the second line connecting the front end point and the rear end point can be first determined, and the angle between the first line connecting the front end point and the deviation point and the second line connecting the front end point and the rear end point is used as the nasal septum deviation angle.

[0045] For example, after obtaining the nasal septum deviation angle, the angle value of the nasal septum deviation angle can be further automatically detected through an image recognition algorithm to achieve quantitative detection of the nasal septum deviation angle and improve the accuracy and efficiency of detecting and evaluating the nasal septum deviation angle.

[0046] The beneficial effect brought about by the above-mentioned implementation method is that it can automatically detect the nasal septum deviation angle, realize high-precision automatic detection and evaluation of the nasal septum deviation angle, and assist in improving the doctor's screening efficiency and treatment effect of nasal septum deviation.

[0047] In some implementations, in the above S120, in the nasal septum deviation image, multiple key points of the nasal septum cartilage are determined according to the nasal septum deviation detection model, including S121 to S122, and S121 to S122 are specifically described below.

[0048] S121, downsampling is performed through a universal feature extraction module to determine universal features of the nasal septum image, wherein the universal feature extraction module includes a Resnet18 model pre-trained on ImageNet.

[0049] Figure 5 A schematic diagram of a process for identifying a deviated nasal septum is provided in an embodiment of the present application, such as Figure 5 As shown, when identifying the nasal septum deviation image, the general feature extraction module ( Figure 5 The universal feature extractor (in the figure) is used for down-sampling to determine the universal features of the nasal septum image, so as to facilitate the subsequent processing and recognition of the universal features of the nasal septum image.

[0050] Exemplarily, the general feature extraction module can be a Resnet18 model pre-trained on ImageNet.

[0051] S122, determining semantic features of multiple key points according to the common features of the nasal septum image through a nasal septum semantic feature extraction module, wherein the nasal septum semantic feature extraction module includes a Unet model.

[0052] like Figure 5 As shown, the nasal septum semantic feature extraction module can be used to determine the semantic features of multiple key points based on the common features of the nasal septum image, so as to facilitate the subsequent identification of multiple key points of the nasal septum cartilage based on the semantic features of the multiple key points.

[0053] Exemplarily, the nasal septum semantic feature extraction module can be a Unet model.

[0054] S123. Through the key point output module, the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage are determined according to the semantic features of the multiple key points of the nasal septum cartilage, and the final coordinates of the multiple key points of the nasal septum cartilage are determined according to the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage.

[0055] like Figure 5As shown, the intermediate coordinates and position deviation values ​​of multiple key points of the nasal septum cartilage can be determined according to the semantic features of the multiple key points of the nasal septum cartilage through the key point output module, and the intermediate coordinates and position deviation values ​​are used to determine the final coordinates of the multiple key points of the nasal septum cartilage. Subsequently, the final coordinates of the multiple key points of the nasal septum cartilage can be determined according to the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage. After obtaining the final coordinates of the multiple key points of the nasal septum cartilage, the angle of the line connecting the multiple key points of the nasal septum cartilage can be determined through S130, and the angle of the line connecting the multiple key points is used as the nasal septum deviation angle.

[0056] The beneficial effect of the above implementation method is that by downsampling, the resolution of the image is reduced, the number of parameters and the amount of calculation are reduced, thereby improving the running speed and efficiency of the model. At the same time, downsampling helps to improve the model's ability to understand the overall structure and content of the image, and improves the efficiency and accuracy of identifying nasal septum deviation.

[0057] In some implementations, the final coordinates of multiple key points of the nasal septum cartilage are determined based on the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage, including: adding and integerizing the intermediate coordinates and position deviation values ​​corresponding to the multiple key points of the nasal septum cartilage, to obtain the final coordinates of the multiple key points of the nasal septum cartilage.

[0058] like Figure 5 As shown, the intermediate coordinates (X', Y') corresponding to the multiple key points obtained by S123 can be (240, 245), (266, 104) and (265, 332), respectively, and the position deviation values ​​(σX, σY) corresponding to the multiple key points can be (1.1, 0.5), (0.2, 2.1) and (5.2, 1.6), respectively. Further, (240, 245) and (1.1, 0.5) can be added and rounded, (266, 104) and (0.2, 2.1) can be added and rounded, and (265, 332) and (5.2, 1.6) can be added and rounded to obtain (242, 246), (266, 106) and (270, 334) as the final coordinates of the multiple key points of the nasal septum cartilage.

[0059] The beneficial effect brought about by the above-mentioned implementation method is that by adding and integerizing the intermediate coordinates and position deviation values ​​corresponding to multiple key points of the nasal septum cartilage, the intermediate coordinates are corrected, thereby improving the accuracy of detection and identification of the final coordinates of multiple key points of the nasal septum cartilage.

[0060] In some implementations, in the above S123, the intermediate coordinates and position deviation values ​​of multiple key points of the nasal septum cartilage are determined according to the semantic features of the multiple key points of the nasal septum cartilage through the key point output module, and it also includes: determining the heat map of the multiple key points of the nasal septum cartilage according to the semantic features of the multiple key points of the nasal septum cartilage through the key point output module, and upsampling the heat map of the multiple key points of the nasal septum cartilage to determine the intermediate coordinates of the multiple key points of the nasal septum cartilage.

[0061] When using, Figure 5 As shown, through the key point output module, the heat map of multiple key points of the nasal septum cartilage is determined according to the semantic features of the multiple key points of the nasal septum cartilage. The heat map of the multiple key points represents the probability of the multiple key points appearing in the nasal septum deviation image. Subsequently, the heat map of the multiple key points of the nasal septum cartilage is upsampled to determine the intermediate coordinates of the multiple key points of the nasal septum cartilage.

[0062] When used, upsampling can increase the resolution of the heat map, thereby improving the positioning accuracy of the key point position. At the same time, upsampling can reduce the blur in the image of nasal septum cartilage recognition, making the position recognition of the key point more accurate.

[0063] The beneficial effect brought about by the above implementation method is that the blurriness of the image of nasal septum cartilage recognition can be reduced through upsampling, so that the position recognition of the key points of the nasal septum deviation is more accurate.

[0064] In some implementations, in the above-mentioned S120, multiple key points of the nasal septum cartilage are determined in the nasal septum deviation image, which also includes: in the nasal septum deviation image, the front end point of the nasal septum cartilage is marked in red, the deviation point of the nasal septum cartilage is marked in green, and the posterior end point of the nasal septum cartilage is marked in blue.

[0065] When in use, by marking the front end point of the nasal septum cartilage in red, the deviation point of the nasal septum cartilage in green, and the posterior end point of the nasal septum cartilage in blue in the nasal septum deviation image, these three key points can better describe the anatomical structure of the nasal septum deviation and have obvious recognition in different lesion images, making it easy for doctors to evaluate the state of the nasal septum deviation based on the nasal septum key points of different colors.

[0066] The beneficial effect brought about by the above implementation method is that it can improve the recognition of nasal septum deviation in different pathological images, making it easier for doctors to evaluate the state of nasal septum deviation based on nasal septum key points of different colors.

[0067] In some implementations, when training a nasal septum deviation detection model, sample images are labeled through S210 to S220, and S210 to S220 are described in detail below.

[0068] S210, retaining the pixel value of each pixel of the sample image between -1024hu and 1024hu, and normalizing to obtain a normalized sample image. The sample image includes a plurality of annotated points of the nasal septum cartilage corresponding to a plurality of key points of the nasal septum cartilage.

[0069] When in use, the pixel value of each pixel of the sample image can be retained between -1024hu and 1024hu by formula (1), and normalized to obtain a normalized sample image. By normalizing the image, the pixel value can be mapped to a floating point number between 0 and 1, which is convenient for subsequent feature extraction.

[0070]

[0071] When selecting a sample image, the selected sample image includes a plurality of annotated points of the nasal septum cartilage corresponding to a plurality of key points of the nasal septum cartilage, so as to facilitate identification of the plurality of key points of the nasal septum cartilage during use.

[0072] S220, according to the parameters of the general feature extraction module for downsampling, determine the annotation grid for annotating the normalized sample image, and determine the multiple annotation grid points and annotation grid point correction values ​​corresponding to the multiple annotation points of the nasal septum cartilage in the annotation grid, and use the multiple annotation grid points and annotation grid point correction values ​​corresponding to each sample image as training labels. Among them, the annotation grid points correspond to the heat map of the key points, the annotation grid point correction values ​​correspond to the position deviation values ​​of the key points, and the annotation grid point correction values ​​are used to correct the intermediate coordinates of the key points of the nasal septum cartilage after upsampling.

[0073] During model training, sample annotation may be performed on the output result image obtained after downsampling of the general feature extraction module, and annotation may be performed in the output result image obtained after downsampling through an annotation grid.

[0074] During model training, in order to ensure the annotation accuracy of the output result image obtained after downsampling, multiple annotation grid points and annotation grid point correction values ​​corresponding to multiple annotation points of the nasal septum cartilage can be determined in the annotation grid. The annotation grid points are position parameters for preliminary annotation of the positions of the key points, and the annotation grid point correction values ​​are parameters for correcting the positions of the key points.

[0075] After obtaining the labeled grid points and the labeled grid point correction values, the multiple labeled grid points and the labeled grid point correction values ​​corresponding to each sample image are used as training labels, and a key point, a labeled grid point and a labeled grid point correction value correspond to each other.

[0076] Among them, the labeled grid points correspond to the heat map of the key points, and the heat map of the key points is used for subsequent upsampling to obtain the intermediate coordinates of multiple key points of the nasal septum cartilage.

[0077] When in use, the grid point correction value corresponds to the position deviation value of the key point, and the grid point correction value is used to correct the intermediate coordinates of the key points of the nasal septum cartilage after upsampling. The grid point correction value can be calculated by taking the modulus of the grid width from the original coordinates of the key points of the nasal septum cartilage.

[0078] Figure 6 A schematic diagram of a process of labeling a sample image provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the downsampling parameters of the general feature extractor can be designed as 5-layer downsampling, and the size of the feature layer obtained is 1 / 4 of the input image. Therefore, the coordinates of the downsampled annotation grid can be obtained based on the coordinates of multiple annotation points of the nasal septum cartilage in the input image.

[0079] like Figure 6 As shown, taking the sampled size of 16*16 as an example, the coordinates of multiple labeled grid points of the nasal septum cartilage correspond to the coordinates of multiple key points in the normalized sample image (x i ,y i ), the size of the annotation grid obtained after the normalized sample image (resolution is 512*512) is downsampled by 5 layers is 16*16, and each annotation grid point in the annotation grid corresponds to a 32*32 sub-region in the normalized sample image. The coordinates of the annotation grid points of the normalized sample image after downsampling can be calculated by taking the annotation coordinates modulo 32 to calculate mod((x i ,y i ), 32) to obtain the coordinates of the marked grid points.

[0080] like Figure 6 As shown, the grid correction value is recorded as The shape of the annotation grid correction value is 3*2, corresponding to the offset of the three key points in the x and y directions respectively. The annotation grid correction value is calculated by taking the modulus of the annotation coordinates to the grid width, which is used to correct the middle coordinates of the key points of the nasal septum cartilage after upsampling.

[0081] like Figure 6 As shown, the multiple annotation grid point labels generated according to the grid coordinates can be recorded as The size of multiple label grid points is 3*32*32. Each 32*32 label grid point represents a key point of the normalized sample image. In each label grid point, only the pixel value consistent with the label grid point is 1, and the rest are 0.

[0082] During model training, the input is CT cross-sectional image data, which is then processed by the general feature extractor, nasal septum semantic feature extractor, and key point output module to output the corresponding heat map (denoted as ) and the offset (denoted as ). The model is optimized by batch gradient descent algorithm, the training batch size is 32, and the model training rounds are 200 times.

[0083] As shown in formula (2), during model training, the loss function of the model training consists of two parts, where L is the total loss and L hm is the heat map loss, L delta is the coordinate deviation loss, and α is the weight coefficient. As shown in formula (3), L hm Heatmap used to calculate predictions and label heatmap The difference is expressed using cross entropy. As shown in Formula 4, L delta Used to calculate key point prediction deviation and true deviation The error between them is expressed as the mean absolute error. Where B represents the training batch.

[0084] L=L hm +αL delta (2)

[0085]

[0086]

[0087] The beneficial effect brought about by the above implementation method is that the results are labeled by grid coordinates, which reduces the amount of calculation in the training process, and the obtained grid coordinates are corrected by marking grid point correction values, which ensures the labeling accuracy and the training effect of the model.

[0088] An embodiment of the present application also provides a nasal septum deviation assessment device based on image recognition, including a unit for executing the above method.

[0089] Figure 7 A schematic diagram of a nasal septum deviation assessment device based on image recognition provided in an embodiment of the present application, such as Figure 7 As shown, the device 1 of this embodiment includes a processing unit 11, a storage unit 12 and a transceiver unit 13. The processing unit 11 is used to process data, the storage unit 12 is used to store data, and the transceiver unit 13 is used to send and receive data. The processing unit 11, the storage unit 12 and the transceiver unit 13 cooperate with each other to implement the above method. The beneficial effects brought by the device have been described in the above method and will not be repeated here.

[0090] An embodiment of the present application also provides a nasal septum deviation assessment device based on image recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method described above is implemented when the processor executes the computer program.

[0091] Figure 8 A schematic diagram of another nasal septum deviation assessment device based on image recognition provided in an embodiment of the present application, such as Figure 8 As shown, the device 2 of this embodiment includes: at least one processor 20 ( Figure 2 Only one processor 20 is shown in the figure), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, and the processor 20 implements the steps in any of the above-mentioned method embodiments when executing the computer program 22. The beneficial effects brought by the device have been described in the above-mentioned method and will not be repeated here.

[0092] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0093] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0094] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0095] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0097] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for evaluating nasal septum deviation based on image recognition, characterized in that: The method comprises: Obtaining a nasal septum deviation image; wherein the nasal septum deviation image includes a cross-sectional CT image of the nasal cavity; In the nasal septum deviation image, multiple key points of the nasal septum cartilage are determined according to the nasal septum deviation detection model; wherein the multiple key points of the nasal septum cartilage include the front end point, the deviation point and the rear end point of the nasal septum cartilage; According to the multiple key points of the nasal septum cartilage, the angle of the connection lines of the multiple key points of the nasal septum cartilage is determined, and the angle of the connection lines of the multiple key points is used as the nasal septum deviation angle; wherein the angle of the connection lines of the multiple key points includes the angle of a first connection line connecting the front end point and the deflection point and the angle of a second connection line connecting the front end point and the rear end point; When training the nasal septum deviation detection model, the sample images are annotated using the following method: Retaining the pixel value of each pixel of the sample image between -1024hu and 1024hu, and normalizing it to obtain a normalized sample image; wherein the sample image includes a plurality of annotated points of the nasal septum cartilage corresponding to a plurality of key points of the nasal septum cartilage; According to the parameters of downsampling in the general feature extraction module, a labeling grid for labeling the normalized sample image is determined, and a plurality of labeling grid points and labeling grid point correction values ​​corresponding to a plurality of labeling points of the nasal septum cartilage are determined in the labeling grid, and the plurality of labeling grid points and labeling grid point correction values ​​corresponding to each sample image are used as training labels; wherein the labeling grid points correspond to the heat map of the key points, the labeling grid point correction values ​​correspond to the position deviation values ​​of the key points, and the labeling grid point correction values ​​are used to correct the coordinates of the key points of the downsampled nasal septum cartilage.

2. The method for evaluating nasal septum deviation based on image recognition according to claim 1, characterized in that: In the nasal septum deviation image, multiple key points of the nasal septum cartilage are determined according to the nasal septum deviation detection model, including: Downsampling is performed through a universal feature extraction module to determine universal features of the nasal septum image; wherein the universal feature extraction module includes a Resnet18 model pre-trained on ImageNet; Determine semantic features of multiple key points according to common features of the nasal septum image through a nasal septum semantic feature extraction module; wherein the nasal septum semantic feature extraction module includes a Unet model; Through the key point output module, the intermediate coordinates and position deviation values ​​of multiple key points of the nasal septum cartilage are determined according to the semantic features of the multiple key points of the nasal septum cartilage, and the final coordinates of the multiple key points of the nasal septum cartilage are determined according to the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage.

3. The method for evaluating nasal septum deviation based on image recognition according to claim 2, characterized in that: The final coordinates of the multiple key points of the nasal septum cartilage are determined according to the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage, including: The intermediate coordinates and position deviation values ​​corresponding to the multiple key points of the nasal septum cartilage are added and integerized to obtain the final coordinates of the multiple key points of the nasal septum cartilage.

4. The method for evaluating nasal septum deviation based on image recognition according to claim 3, characterized in that: Through the key point output module, the intermediate coordinates and position deviation values ​​of the multiple key points of the nasal septum cartilage are determined according to the semantic features of the multiple key points of the nasal septum cartilage, and also include: Through the key point output module, the heat map of multiple key points of the nasal septum cartilage is determined according to the semantic features of the multiple key points of the nasal septum cartilage, and the heat map of the multiple key points of the nasal septum cartilage is upsampled to determine the intermediate coordinates of the multiple key points of the nasal septum cartilage.

5. The method for evaluating nasal septum deviation based on image recognition according to claim 4, characterized in that: In the nasal septum deviation image, multiple key points of the nasal septum cartilage are identified, including: In the nasal septum deviation image, the front end point of the nasal septum cartilage is marked in red, the deviation point of the nasal septum cartilage is marked in green, and the rear end point of the nasal septum cartilage is marked in blue.

6. A nasal septum deviation assessment device based on image recognition, characterized in that: The method comprises means for performing the method according to any one of claims 1 to 5.

7. A nasal septum deviation assessment device based on image recognition, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images