Obstructive sleep apnea risk prediction method and device

Abstract

The embodiment of the invention relates to an obstructive sleep apnea risk prediction method and device. The method comprises the steps that images of the front face, the left side face, the right side face and the interior of the oral cavity of a testee are shot, and depth feature adding is conducted on the shot images; constructing a face and neck feature map by using the face and neck semantic segmentation model and the face and neck key point recognition model; the face width / eye width ratio, the chin process and the chin neck angle are measured and calculated based on the feature map; measuring and calculating the soft palate length and the uvula size by using a throat semantic segmentation model; inputting a feature vector formed by measurement and calculation data and a face and neck feature map into an apnea hypopnea index prediction model for prediction; risk level identification is carried out based on the prediction index; and forming a testee prediction report by the feature vector, the prediction index and the risk level. When the method is applied to obstructive sleep apnea risk assessment, the assessment convenience can be improved, the assessment time can be shortened, the assessment efficiency can be improved, and the assessment cost can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a method and device for predicting the risk of obstructive sleep apnea. BACKGROUND

[0002] Obstructive sleep apnea (OSA) is a common sleep disorder. At present, the risk assessment of OSA for a subject needs to use professional equipment to monitor the subject overnight in a hospital or a specific sleep monitoring agency, and finally the risk assessment is performed based on the analysis result of long-term monitoring data. Obviously, this conventional risk assessment method is not only cumbersome and time-consuming, but also has a high assessment cost. SUMMARY

[0003] The present application is aimed at the defects of the prior art, and provides a method and device for predicting the risk of obstructive sleep apnea, an electronic device and a computer readable storage medium. Four tool models are pre-trained, including a face-neck semantic segmentation model, a throat semantic segmentation model, a face-neck key point recognition model and an apnea hypopnea index prediction model. The risk of OSA is predicted based on the four tool models. Specifically, the front, left and right side and oral cavity images of the subject are first captured, and depth features are added to the four captured images. Then, the face-neck semantic segmentation model and the face-neck key point recognition model are used to add face-neck semantic and key point features to the front / left / right images, and the three images with added features are subjected to image rotation, point cloud conversion and registration. The three-dimensional face-neck feature map is constructed based on the registered three-dimensional point cloud. The face width / eye width ratio, chin protrusion and chin-neck angle are calculated based on the three-dimensional face-neck feature map. The three-dimensional dimensions of the soft palate length and the uvula in the oral cavity image are calculated using the throat semantic segmentation model. Then, the feature vector composed of the chin features (face width / eye width ratio, chin protrusion, chin-neck angle) and the upper airway features (soft palate length and uvula size) and the three-dimensional face-neck feature Figure One The apnea hypopnea index prediction model is inputted with the feature vector, the predicted index is used to identify the risk level of obstructive sleep apnea based on the index-risk level correspondence, and finally the prediction report of the current subject is composed of the feature vector, the predicted index and the risk level. The present application can improve the convenience of assessment, shorten the assessment time, improve the assessment efficiency and reduce the assessment cost when applied to the risk assessment of OSA.

[0004] To achieve the above object, a method for predicting the risk of obstructive sleep apnea is provided in the first aspect of the present application, which comprises the following steps.

[0005] Pre-set four artificial intelligence models; and respectively perform model training on the four artificial intelligence models; the four artificial intelligence models include a face-neck semantic segmentation model, a throat semantic segmentation model, a face-neck key point recognition model, and an apnea hypopnea index prediction model;

[0006] After the model training of the four artificial intelligence models is completed, a front view, left side view, right side view and oral cavity view of the subject are captured using the same camera according to a preset image capturing requirement to obtain corresponding first front view, first left side view, first right side view and first oral cavity view; and pixel-level deep feature addition processing is performed on the first front view, the first left side view, the first right side view and the first oral cavity view to obtain corresponding second front view, second left side view, second right side view and second oral cavity view;

[0007] Face-neck semantic and key point feature addition is performed on the second front view, the second left side view and the second right side view using the face-neck semantic segmentation model and the face-neck key point recognition model, image rotation is performed on the three images after the feature addition is completed, point cloud conversion and registration are performed on the three images after the rotation is completed, three-dimensional face-neck feature map construction is performed based on the registered three-dimensional point cloud to obtain a first face-neck feature map; and face width / eye width ratio, chin protrusion degree and chin-neck angle are calculated based on the first face-neck feature map to obtain corresponding first face width / eye width ratio, first chin protrusion degree and first chin-neck angle;

[0008] The three-dimensional dimensions of the soft palate length and the uvula in the second oral cavity view are calculated using the throat semantic segmentation model to obtain corresponding first soft palate length and first uvula size; the first uvula size includes uvula length, uvula width and uvula thickness;

[0009] The first feature vector is composed of the first face width / eye width ratio, the first chin protrusion degree, the first chin-neck angle, the first soft palate length and the first uvula size; the first feature vector and the first face-neck feature map are input into the apnea hypopnea index prediction model to predict the apnea hypopnea index to obtain a corresponding first prediction index; and based on a preset index-risk level correspondence relationship, the first prediction index is used to identify the risk level of obstructive sleep apnea to obtain a corresponding first risk level; the index-risk level correspondence relationship includes a plurality of correspondence records; each correspondence record is composed of an index value range and a corresponding risk level;

[0010] The first feature vector, the first prediction index and the first risk level constitute a first prediction report of the current subject.

[0011] Preferably, the face-neck semantic segmentation model is implemented based on a type of image segmentation model; the face-neck semantic segmentation model is used for image feature extraction on a first feature map input by the model, and performs pixel-level face, neck and background semantic segmentation based on the extracted features and outputs the segmentation results as a corresponding first semantic map; the first feature map is a front, left or right image with depth features and containing human face and neck parts; pixel point features of the first feature map include first RGB three primary colors and first depth; the image size of the first semantic map remains consistent with the first feature map; pixel point features of the first semantic map include first semantic types; the first semantic types include face, neck and background;

[0012] The throat semantic segmentation model is implemented based on a type of image segmentation model; the throat semantic segmentation model is used for image feature extraction on a second feature map input by the model, and performs pixel-level soft palate, uvula and background semantic segmentation based on the extracted features and outputs the segmentation results as a corresponding second semantic map; the first feature map is an intraoral image with depth features and containing human throat parts; pixel point features of the second feature map include second RGB three primary colors and second depth; the image size of the second semantic map remains consistent with the second feature map; pixel point features of the second semantic map include second semantic types; the second semantic types include soft palate, uvula and background;

[0013] The face-neck key point recognition model is implemented based on a type of face key point recognition model; the face-neck key point recognition model is used for image feature extraction on a third feature map input by the model, and performs pixel-level key point classification and prediction processing based on the extracted features and a preset total set of face-neck key points, and outputs the prediction results as a corresponding fourth feature map; the third feature map is a front, left or right image containing human face and neck parts and having depth and face-neck semantic features; pixel point features of the third feature map include third RGB three primary colors, third depth and third semantic types; the third semantic types include face, neck and background; the image size of the fourth feature map remains consistent with the third feature map; pixel point features of the fourth feature map include first key point types; the type range of the first key point types consists of the total set of face-neck key points and non-key points; the total set of face-neck key points includes multiple types of face key points and multiple types of neck key points;

[0014] The apnea hypopnea index prediction model is used for feature extraction processing on the first feature vector and the first face-neck feature map input by the model respectively, feature fusion processing on the two-way features, regression prediction processing on the apnea hypopnea index based on the fused features, and output of the corresponding first prediction index;

[0015] The first feature vector comprises the first face width / eye width ratio, the first gony degree, the first gony-neck angle, the first soft palate length, and the first uvula size; the first uvula size comprises the uvula length, the uvula width, and the uvula thickness;

[0016] The first face-neck feature map is a 3D image containing a human face-neck part and having face-neck part semantic features and key point features; the voxel point features of the first face-neck feature map comprise fourth RGB three primary colors, fourth semantic types, and second key point types; the fourth semantic types comprise a face, a neck, and a background; the type range of the second key point types is composed of the face-neck key point total set and non-key points;

[0017] The first model input end of the apnea hypopnea index prediction model is configured to receive the first feature vector, the second model input end is configured to receive the first face-neck feature map, and the model output end is configured to output the corresponding first prediction index;

[0018] The apnea hypopnea index prediction model comprises a first feature extraction network, a second feature extraction network, a feature fusion module, and a regression prediction network; the input end of the first feature extraction network is connected with the first model input end, and the output end is connected with the first input end of the feature fusion module; the input end of the second feature extraction network is connected with the second model input end, and the output end is connected with the second input end of the feature fusion module; the output end of the feature fusion module is connected with the input end of the regression prediction network; and the output end of the regression prediction network is connected with the model output end;

[0019] The first feature extraction network is sequentially connected by a first encoder and a first feature mapping network; the first encoder and the first feature mapping network are each implemented based on an MLP model;

[0020] The first feature extraction network is configured to perform feature coding processing on the first feature vector, map an encoded feature tensor to a preset public feature vector space to obtain a corresponding first mapping feature vector H1, and send the first mapping feature vector H1 to the feature fusion module; specifically, the first encoder is configured to perform feature coding processing on the first feature vector to obtain a corresponding first feature tensor, and send the first feature tensor to the first feature mapping network; and the first feature mapping network is configured to perform feature vector mapping of the public feature vector space on the first feature tensor to obtain the corresponding first mapping feature vector H1, and send the first mapping feature vector H1 to the feature fusion module;

[0021] The second path feature extraction network is sequentially connected by a second encoder and a second feature mapping network; the second encoder is implemented based on a 3D convolutional neural network, a 3D residual neural network or a 3D graph neural network; and the second feature mapping network is implemented based on an MLP model;

[0022] The second path feature extraction network is used for feature encoding processing on the first face-neck feature map and mapping of an encoded feature tensor to the public feature vector space to obtain a corresponding second path mapping feature vector H2, which is sent to the feature fusion module. Specifically, the second encoder is used for feature encoding processing on the first face-neck feature map to obtain a corresponding second feature tensor, which is sent to the second feature mapping network; and the second feature mapping network is used for feature vector mapping of the second feature tensor to the public feature vector space to obtain a corresponding second path mapping feature vector H2, which is sent to the feature fusion module.

[0023] The feature fusion module is used for weighted summation calculation on the first path mapping feature vector H1 and the second path mapping feature vector H2, and the calculation result is sent to the regression prediction network as a corresponding first fusion feature vector H3; wherein H3=w1H1+w2H1, w1 and w2 are first and second learnable weighting parameters.

[0024] The regression prediction network is sequentially connected by one or more hidden layers and an output layer; each hidden layer is sequentially connected by a fully connected layer and a nonlinear activation function, and the nonlinear activation function defaults to a ReLU activation function; the output layer is implemented based on a fully connected layer; and the regression prediction network is used for regression prediction processing on an apnea hypopnea index according to the first fusion feature vector H3 to obtain a corresponding first prediction index and output.

[0025] Preferably, the four artificial intelligence models are trained respectively, specifically including:

[0026] The face-neck semantic segmentation model is trained based on a preset first data set; the throat semantic segmentation model is trained based on a preset second data set; the face-neck key point recognition model is trained based on a preset third data set; the apnea hypopnea index prediction model is trained based on a preset fourth data set; and after the first, second, third and fourth model training are all completed, it is confirmed that the model training of the four artificial intelligence models is completed.

[0027] The first data set includes a plurality of first data records; the first data record includes a first training feature map and a first label semantic map; the first training feature map is a front image, a left image or a right image with depth features and containing human face and neck parts, or an arbitrary image with depth features but not containing human face and neck parts; the feature composition of the pixel point features of the first training feature map is consistent with the first feature map; the image size of the first label semantic map is consistent with the first training feature map; the feature composition of the pixel point features of the first label semantic map is consistent with the first semantic map.

[0028] The second data set includes a plurality of second data records; the second data record includes a second training feature map and a second label semantic map; the second training feature map is an oral cavity internal image with depth features and containing human throat parts, or an arbitrary image with depth features but not containing human throat parts; the feature composition of the pixel point features of the second training feature map is consistent with the second feature map; the image size of the second label semantic map is consistent with the second training feature map; the feature composition of the pixel point features of the second label semantic map is consistent with the second semantic map.

[0029] The third data set includes a plurality of third data records; the third data record includes a third training feature map and a first label feature map; the third training feature map is a front image, a left image or a right image containing human face and neck parts and having depth and face and neck semantic features; the feature composition of the pixel point features of the third training feature map is consistent with the third feature map; the image size of the first label feature map is consistent with the third training feature map; the feature composition of the pixel point features of the first label feature map is consistent with the fourth feature map.

[0030] The fourth data set includes a plurality of fourth data records; the fourth data record includes a first training feature vector, a first training face and neck feature map and a first label index; the vector data composition of the first training feature vector is consistent with the first feature vector, which is composed of corresponding face width / eye width ratio, chin protrusion, chin and neck angle, soft palate length and uvula size, wherein the uvula size is composed of corresponding length, width and thickness; the first training face and neck feature map is a 3D image containing human face and neck parts and having face and neck semantic features and key point features; the feature composition of the voxel point features of the first training face and neck feature map is consistent with the first face and neck feature map; the first label index is a corresponding apnea hypopnea index.

[0031] Preferably, the image shooting requirements include: shooting images of the front face, left side face and right side face of the human face respectively to generate the front face image, the left side image and the right side image, under the premise that the front face, left side face and right side face of the human face are parallel to the imaging plane of the camera lens respectively; guiding the tip of the tongue of the subject to rest on the lower teeth and open the mouth to the maximum extent to make an "ah" sound, and shooting an image of the inside of the oral cavity of the subject to generate the oral cavity image at this time; the front face image requires that the image contains the face and neck positions, and the face needs to expose both ears, and the ratio of the imaging area of the face and neck to the image area is greater than a preset first proportion threshold; the left side face or the right side face requires that the image contains the corresponding left or right side face and neck positions, and the face needs to expose the corresponding left or right ear, and the ratio of the imaging area of the face and neck to the image area is greater than the first proportion threshold; the tonsils, soft palate and uvula edges of the throat part in the oral cavity image are clear, and the ratio of the imaging area of the throat part to the image area is greater than a preset second proportion threshold;

[0032] The pixel point features of the first front face image, the first left side image, the first right side image and the first oral cavity image include RGB three primary colors;

[0033] The pixel point features of the second front face image, the second left side image, the second right side image and the second oral cavity image include RGB three primary colors and depth;

[0034] The pixel-level depth feature adding processing of the first front face image, the first left side image, the first right side image and the first oral cavity image obtains the corresponding second front face image, second left side image, second right side image and second oral cavity image, specifically: using a preset depth image generation tool to generate the depth image of the first front face image, the first left side image, the first right side image and the first oral cavity image to obtain the corresponding front face depth image, left side depth image, right side depth image and oral cavity depth image; and performing pixel feature fusion processing on the first front face image and the front face depth image to obtain the corresponding second front face image; and performing pixel feature fusion processing on the first left side image and the left side depth image to obtain the corresponding second left side image; and performing pixel feature fusion processing on the first right side image and the right side depth image to obtain the corresponding second right side image; and performing pixel feature fusion processing on the first oral cavity image and the oral cavity depth image to obtain the corresponding second oral cavity image; the depth image generation tool includes MiDaS tool, DPT model tool and OpenCV tool.

[0035] Preferably, the face-neck semantic segmentation model and the face-neck key point recognition model are used to add face-neck semantic and key point features to the second front view, the second left view and the second right view, the three-view images after feature addition are rotated, the three-view images after rotation are converted into point clouds and registered, and a first face-neck feature map is constructed based on the registered three-view point clouds, specifically including:

[0036] The second front view, the second left view and the second right view are respectively input into the face-neck semantic segmentation model as a corresponding first feature map to obtain a corresponding first semantic map; pixel feature fusion is performed on each first feature map and its corresponding first semantic map to obtain a corresponding first fusion map; and the first fusion map corresponding to the second front view, the second left view and the second right view is taken as a corresponding third front view, a third left view and a third right view; the pixel point features of the third front view, the third left view and the third right view include the first RGB three primary colors, the first depth and the first semantic type;

[0037] The third front view, the third left view and the third right view are respectively input into the face-neck key point recognition model as a corresponding third feature map to obtain a corresponding fourth feature map; pixel feature fusion is performed on each third feature map and its corresponding fourth feature map to obtain a corresponding second fusion map; and the second fusion map corresponding to the third front view, the third left view and the third right view is taken as a corresponding fourth front view, a fourth left view and a fourth right view; the pixel point features of the fourth front view, the fourth left view and the fourth right view include the third RGB three primary colors, the third depth, the third semantic type and the first key point type;

[0038] and the pixel points of the third semantic type being other than background points on the fourth front view after rotation form a corresponding front pixel point set; wherein the third depth of each pixel point is adaptively adjusted in the process of rotating or translating the fourth front view;

[0039] and the fourth left view or the fourth right view is taken as a corresponding current side view; and the pixel points of the first key point type of inter-brow point, nose tip point and chin point on the current side view are recorded as a corresponding second inter-brow point, first nose tip point and first chin point; and a first plane is made based on the second inter-brow point, the first nose tip point and the first chin point; and the current side view is rotated with the goal of keeping the depth of all pixel points on the first plane consistent; and the pixel points of the third semantic type being other than background points on the current side view after rotation form a corresponding left pixel point set or right pixel point set; wherein the third depth of each pixel point is adaptively adjusted in the process of rotating the fourth left view and the fourth right view;

[0040] and the fourth left view or the fourth right view is taken as a corresponding current side view; and the pixel points of the first key point type of inter-brow point, nose tip point and chin point on the current side view are recorded as a corresponding second inter-brow point, first nose tip point and first chin point; and a first plane is made based on the second inter-brow point, the first nose tip point and the first chin point; and the current side view is rotated with the goal of keeping the depth of all pixel points on the first plane consistent; and the pixel points of the third semantic type being other than background points on the current side view after rotation form a corresponding left pixel point set or right pixel point set; wherein the third depth of each pixel point is adaptively adjusted in the process of rotating the fourth left view and the fourth right view;

[0041] And the point in the front point cloud corresponding to the glabella point, the rhinion point, the left infraorbital point, and the right infraorbital point is recorded as the corresponding third glabella point, the second rhinion point, the second left infraorbital point, and the second right infraorbital point; and the point in the left point cloud corresponding to the glabella point, the rhinion point, and the left infraorbital point is recorded as the fourth glabella point, the third rhinion point, and the third left infraorbital point; and the point in the right point cloud corresponding to the glabella point, the rhinion point, and the right infraorbital point is recorded as the fifth glabella point, the fourth rhinion point, and the third right infraorbital point; and the third and fourth glabella points form a first matching point pair, the second and third rhinion points form a second matching point pair, the second and third left infraorbital points form a third matching point pair, the third and fifth glabella points form a fourth matching point pair, the second and fourth rhinion points form a fifth matching point pair, and the second and third right infraorbital points form a sixth matching point pair; and the front point cloud is taken as the reference point cloud, and the sum of the point distances of the first, second, and third matching point pairs is minimized as the registration target, the left point cloud is subjected to point cloud registration processing; and the front point cloud is taken as the reference point cloud, and the sum of the point distances of the fourth, fifth, and sixth matching point pairs is minimized as the registration target, the right point cloud is subjected to point cloud registration processing; wherein the three-dimensional coordinates of each point of the current point cloud are adaptively adjusted during the registration of the left point cloud and the right point cloud.

[0042] And the first fusion point cloud is obtained through point cloud fusion processing based on the front point cloud, the left point cloud, and the right point cloud after the point cloud registration is completed; and a three-dimensional voxel space is constructed based on the width, height, and depth of the first fusion point cloud; and a three-dimensional feature map is constructed based on the first fusion point cloud and the three-dimensional voxel space to obtain the first face-neck feature map; the voxel point features of the first face-neck feature map include the fourth RGB three primary colors, the fourth semantic type, and the second key point type; the fourth semantic type includes the face, the neck, and the background; and the type range of the second key point type is composed of the face-neck key point total set and the non-key point.

[0043] Preferably, the first face width / eye width ratio, the first chin prominence, and the first chin-neck angle are obtained through the calculation of the face width / eye width ratio, the chin prominence, and the chin-neck angle based on the first face-neck feature map, and specifically include:

[0044] The straight-line distance between the two voxel points of the first face-neck feature map, in which the second key point type is the left and right tragus point, is calculated and the calculation result is taken as the first face width; the straight-line distance between the two voxel points of the first face-neck feature map, in which the second key point type is the left and right infraorbital point, is calculated and the calculation result is taken as the first eye width; and the first face width / eye width ratio is calculated based on the first face width and the first eye width, and the first face width / eye width ratio = first face width: first eye width.

[0045] and the five voxel points of the second key point type on the first face-neck feature map are recorded as the corresponding glabella, rhinion, menton, left suborbicularis ovals, and right suborbicularis ovals voxel points, respectively; a plane on which the glabella, rhinion, and menton voxel points are located is taken as a corresponding first sagittal section; a section that passes through the left and right suborbicularis ovals voxel points and is perpendicular to the first sagittal section is taken as a corresponding first transverse section; a section that passes through the menton voxel point and is perpendicular to the first sagittal section and parallel to the first transverse section is taken as a corresponding second transverse section; a perpendicular line of the second transverse section that passes through the rhinion voxel point is recorded as a corresponding first perpendicular line; directions on the first sagittal section that are towards the inside and outside of the face are taken as the rear and front of the first perpendicular line; a perpendicular line of the second transverse section that passes through the menton voxel point is recorded as a corresponding second perpendicular line; the line distance between the first and second perpendicular lines is calculated to obtain a corresponding first distance d, and the first mentum protrusion is set as d when the second perpendicular line is in front of the first perpendicular line, and the first mentum protrusion is set as -d when the second perpendicular line is behind the first perpendicular line;

[0046] and the two voxel points of the second key point type on the first face-neck feature map are recorded as the corresponding mentum and neck voxel points; a spatial vector from the rhinion voxel point to the menton voxel point is recorded as a corresponding vector A, and a spatial vector from the mentum voxel point to the neck voxel point is recorded as a corresponding vector B; the vector angle θ is calculated based on the vectors A and B; and the first mentum-neck angle is calculated based on the vector angle θ; α = 180° - θ + δ, where α is the first mentum-neck angle and δ is a preset compensation angle;

[0047] The first face width / eye width ratio, the first mentum protrusion, and the first mentum-neck angle obtained this time are taken as the measurement results this time.

[0048] Preferably, the three-dimensional dimensions of the soft palate length and the uvula in the second oral cavity graph are measured using the pharynx-throat semantic segmentation model to obtain a corresponding first soft palate length and a first uvula size, specifically including:

[0049] The second oral cavity graph is taken as a corresponding second feature graph, which is input into the pharynx-throat semantic segmentation model to obtain a corresponding second semantic graph; and the second semantic graph obtained this time is subjected to point cloud conversion processing to obtain a corresponding oral cavity internal point cloud; the point features of the oral cavity internal point cloud include a second semantic type;

[0050] and record the points of the second semantic type as soft palate points or uvula points in the obtained point cloud of the inside of the oral cavity; calculate the coordinates of the center points of all the soft palate points to obtain a corresponding first center point, and calculate the coordinates of the center points of all the uvula points to obtain a corresponding second center point, and draw a straight line based on the first center point and the second center point to obtain a corresponding first center line; take the soft palate point with the highest height among all the soft palate points on the first center line as a soft palate vertex; take the two soft palate points with the highest and lowest heights among all the uvula points on the first center line as a corresponding uvula base center point and a uvula tip point; calculate the straight line distance from the soft palate vertex to the uvula tip point and take the calculation result as a corresponding first soft palate length; and calculate the straight line distance from the uvula base center point to the uvula tip point and take the calculation result as a corresponding uvula length;

[0051] take the two uvula points farthest left and farthest right among all the uvula points as a corresponding left side end point and a right side end point; calculate the vertical distances from the left and right side end points to the first center line to obtain a corresponding left side vertical distance and a right side vertical distance; and calculate the sum of the left and right side vertical distances and take the calculation result as a corresponding uvula width;

[0052] take the two uvula points closest and farthest among all the uvula points as a corresponding near end point and a far end point; calculate the vertical distances from the near and far end points to the first center line to obtain a corresponding near end vertical distance and a far end vertical distance; and calculate the sum of the near and far end vertical distances and take the calculation result as a corresponding uvula thickness;

[0053] and the uvula length, the uvula width and the uvula thickness obtained are used to form a corresponding first uvula size;

[0054] and the first soft palate length and the first uvula size obtained are taken as the calculation result of this time.

[0055] The second aspect of the embodiment of the present application provides a device for implementing the risk prediction method of obstructive sleep apnea described in the first aspect, and the device comprises a tool model preparation module, a subject image acquisition module, a face-neck image processing module, an oral cavity image processing module, an index and risk grade prediction module, and a prediction report generation module.

[0056] The tool model preparation module is used for pre-setting four artificial intelligence models, and model training is performed on the four artificial intelligence models respectively; the four artificial intelligence models comprise a face-neck semantic segmentation model, a throat semantic segmentation model, a face-neck key point recognition model, and an apnea hypopnea index prediction model.

[0057] The subject image acquisition module is configured to, after the model training of the four artificial intelligence models is completed, capture the front, left and right side, and oral cavity images of the subject to obtain corresponding first front, left, right, and oral cavity images according to preset image capturing requirements and using the same camera; and perform pixel-level deep feature addition processing on the first front, left, right, and oral cavity images to obtain corresponding second front, left, right, and oral cavity images;

[0058] The face-neck image processing module is configured to add face-neck semantic and key point features to the second front, left, and right images by using the face-neck semantic segmentation model and the face-neck key point recognition model, rotate the three images after the feature addition, perform point cloud conversion and registration on the three images after the rotation, and construct a first face-neck feature map based on the registered three point clouds to obtain a first face-neck feature map; and calculate the face width / eye width ratio, chin protrusion, and chin-neck angle based on the first face-neck feature map to obtain corresponding first face width / eye width ratio, first chin protrusion, and first chin-neck angle.

[0059] The oral cavity image processing module is configured to calculate the three-dimensional size of the soft palate length and the uvula in the second oral cavity image by using the throat semantic segmentation model to obtain corresponding first soft palate length and first uvula size; the first uvula size includes the length, width, and thickness of the uvula.

[0060] The index and risk level prediction module is configured to form a corresponding first feature vector from the first face width / eye width ratio, the first chin protrusion, the first chin-neck angle, the first soft palate length, and the first uvula size; input the first feature vector and the first face-neck feature map into the apnea hypopnea index prediction model to predict the apnea hypopnea index to obtain a corresponding first predicted index; and identify the risk level of obstructive sleep apnea based on a preset index-risk level correspondence according to the first predicted index to obtain a corresponding first risk level; the index-risk level correspondence includes a plurality of correspondence records; each correspondence record is composed of an index value range and a corresponding risk level.

[0061] The prediction report generation module is configured to generate a first prediction report of the current subject from the first feature vector, the first predicted index, and the first risk level.

[0062] The third aspect of the embodiment of the present application provides an electronic device, including a memory, a processor, and a transceiver.

[0063] The processor is used for coupling with the memory, reading and executing instructions in the memory to realize the method steps of the first aspect;

[0064] The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transmission and reception.

[0065] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores computer instructions, when the computer instructions are executed by a computer, the computer instructions make the computer execute the instructions of the method of the first aspect.

[0066] The embodiment of the present application provides a method and device for predicting the risk of obstructive sleep apnea, an electronic device and a computer readable storage medium. From the above content, it can be known that four tool models are pre-trained in the embodiment of the present application: a face-neck semantic segmentation model, a throat semantic segmentation model, a face-neck key point recognition model and an apnea hypopnea index prediction model. The OSA risk prediction is realized based on the four tool models, specifically: first, the front, left and right side and oral cavity internal images of the measured person are photographed, and depth features are added to the four photographed images; then, the face-neck semantic segmentation model and the face-neck key point recognition model are used to add face-neck semantic and key point features to the front / left / right images, and the three images with added features are subjected to image rotation, point cloud conversion and registration, and a three-dimensional face-neck feature map is constructed based on the registered three-dimensional point cloud; the face width / eye width ratio, the chin prominence and the chin-neck angle are calculated based on the three-dimensional face-neck feature map; the three-dimensional dimensions of the soft palate length and the uvula in the oral cavity image are calculated by using the throat semantic segmentation model; then, the feature vector composed of the chin features (face width / eye width ratio, chin prominence, chin-neck angle) and the upper airway features (soft palate length and uvula size) and the three-dimensional face-neck feature Figure One The apnea hypopnea index prediction model is input to predict the apnea hypopnea index; the risk level of obstructive sleep apnea is identified according to the predicted index based on the index-risk level correspondence; finally, the feature vector, the predicted index and the risk level constitute the prediction report of the current measured person. The embodiment of the present application is applied to OSA risk assessment, which not only improves the convenience of assessment, but also shortens the assessment time, improves the assessment efficiency and reduces the assessment cost. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 A method for predicting the risk of obstructive sleep apnea is provided for the first embodiment of the present application;

[0068] Figure 2 A module diagram of the apnea hypopnea index prediction model is provided for the first embodiment of the present application;

[0069] Figure 3 A first distance diagram provided for the first embodiment of the present application;

[0070] Figure 4 A vector A, B diagram provided for the first embodiment of the present application;

[0071] Figure 5 A module structure diagram of a obstructive sleep apnea risk prediction device provided for the second embodiment of the present application;

[0072] Figure 6 A structure diagram of an electronic device provided for the third embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0074] The first embodiment of the present application provides a obstructive sleep apnea risk prediction method, which comprises the following steps: Figure 1 A obstructive sleep apnea risk prediction method diagram provided for the first embodiment of the present application, the method mainly comprises the following steps:

[0075] Step 1, four artificial intelligence models are set in advance; and the four artificial intelligence models are trained respectively.

[0076] Here, the four artificial intelligence models of the embodiment of the present application include a face-neck semantic segmentation model, a throat semantic segmentation model, a face-neck key point recognition model, and an apnea hypopnea index prediction model. Among them:

[0077] 1) Face-neck semantic segmentation model:

[0078] The face-neck semantic segmentation model of the embodiment of the present application is realized based on an image segmentation model. The image segmentation model is, for example, Unet series model, SAM series model, SEGFormer series model, etc.

[0079] The face-neck semantic segmentation model is configured to perform image feature extraction on a first feature map input to the model and perform pixel-level face, neck and background semantic segmentation based on the extracted features and output the segmentation results as a corresponding first semantic map. The first feature map is a front, left or right image of a human face and neck with depth features. The pixel point features of the first feature map include first RGB three primary colors and first depth. The image size of the first semantic map is consistent with that of the first feature map. The pixel point features of the first semantic map include first semantic types. The first semantic types include face, neck and background.

[0080] 2) The throat semantic segmentation model:

[0081] The throat semantic segmentation model of the embodiment of the present application is implemented based on a type of image segmentation model. The image segmentation model is, for example, a Unet series model, a SAM series model, a SEGFormer series model, etc.

[0082] The throat semantic segmentation model is configured to perform image feature extraction on a second feature map input to the model and perform pixel-level soft palate, uvula and background semantic segmentation based on the extracted features and output the segmentation results as a corresponding second semantic map. The first feature map is an intraoral image of a human throat with depth features. The pixel point features of the second feature map include second RGB three primary colors and second depth. The image size of the second semantic map is consistent with that of the second feature map. The pixel point features of the second semantic map include second semantic types. The second semantic types include soft palate, uvula and background.

[0083] 3) The face-neck key point recognition model:

[0084] The face-neck key point recognition model of the embodiment of the present application is implemented based on a type of face key point recognition model. The face key point recognition model is, for example, a Dlib model, a MTCNN (Multi-task Cascaded Convolutional Networks) model, a HRNet (High-Resolution Net) model, a FAN (Face Alignment Network) model, a 3DDFA (3D Dense Face Alignment) model, etc.

[0085] The face-neck key point recognition model is configured to perform image feature extraction on a third feature map input to the model, and perform pixel-level key point classification and prediction processing based on the extracted features and a preset total set of face-neck key points, and output the prediction results as a corresponding fourth feature map. The third feature map is a front image, a left image or a right image containing a human face-neck part and having depth and face-neck semantic features. The pixel point features of the third feature map include third RGB three primary colors, third depth and third semantic type. The third semantic type includes face, neck and background. The image size of the fourth feature map remains consistent with that of the third feature map. The pixel point features of the fourth feature map include first key point type. The type range of the first key point type is composed of the total set of face-neck key points and non-key points. The total set of face-neck key points includes multiple types of face key points and multiple types of neck key points. The face key points and neck key points mentioned herein are common key points in anthropometry and cephalometry, such as glabella (G), nasion (N), pronasale (Prn), left / right orbitale (Or), left / right tragion (Tr), pogonion (Pog), menton (Me), and cervical (C).

[0086] 4) Apnea Hypopnea Index (AH I) prediction model:

[0087] The apnea hypopnea index prediction model is configured to perform feature extraction processing on a first feature vector and a first face-neck feature map input to the model respectively, perform feature fusion processing on the two-way features, perform regression prediction processing on the apnea hypopnea index based on the fused features, and output a corresponding first prediction index.

[0088] The first feature vector includes a first face width / eye width ratio, a first pogonion, a first pogonion, a first soft palate length, and a first uvula size. The first uvula size includes a uvula length, a uvula width, and a uvula thickness. The first face-neck feature map is a 3D image containing a human face-neck part and having face-neck semantic features and key point features. The voxel point features of the first face-neck feature map include fourth RGB three primary colors, fourth semantic type and second key point type. The fourth semantic type includes face, neck and background. The type range of the second key point type is composed of the total set of face-neck key points and non-key points. The first prediction index is an AH I index.

[0089] As Figure 2As shown in the block diagram of the apnea hypopnea index prediction model provided for the first embodiment of the present application, the first model input end of the apnea hypopnea index prediction model is configured to receive the first feature vector, the second model input end is configured to receive the first face-neck feature map, and the model output end is configured to output the corresponding first predicted index.

[0090] As shown in the block diagram of the apnea hypopnea index prediction model provided for the first embodiment of the present application, the first model input end of the apnea hypopnea index prediction model is configured to receive the first feature vector, the second model input end is configured to receive the first face-neck feature map, and the model output end is configured to output the corresponding first predicted index. Figure 2 As shown in the block diagram of the apnea hypopnea index prediction model provided for the first embodiment of the present application, the first model input end of the apnea hypopnea index prediction model is configured to receive the first feature vector, the second model input end is configured to receive the first face-neck feature map, and the model output end is configured to output the corresponding first predicted index.

[0091] The functions of the model components of the apnea hypopnea index prediction model are as follows.

[0092] a) As shown in the block diagram of the apnea hypopnea index prediction model provided for the first embodiment of the present application, the first model input end of the apnea hypopnea index prediction model is configured to receive the first feature vector, the second model input end is configured to receive the first face-neck feature map, and the model output end is configured to output the corresponding first predicted index. Figure 2 The first feature extraction network is configured to perform feature encoding processing on the first feature vector and map the encoded feature tensor to a preset public feature vector space to obtain a corresponding first mapping feature vector H1, which is sent to the feature fusion module.

[0093] The first feature vector is encoded by the first encoder to obtain a corresponding first feature tensor, which is sent to the first feature mapping network. The first feature tensor is mapped to a feature vector in the public feature vector space by the first feature mapping network to obtain a corresponding first mapping feature vector H1, which is sent to the feature fusion module.

[0094] b) As shown in the block diagram of the apnea hypopnea index prediction model provided for the first embodiment of the present application, the first model input end of the apnea hypopnea index prediction model is configured to receive the first feature vector, the second model input end is configured to receive the first face-neck feature map, and the model output end is configured to output the corresponding first predicted index.

[0095] Figure 2 The second feature extraction network is configured to perform feature encoding processing on the first face-neck feature map and map the encoded feature tensor to a public feature vector space to obtain a corresponding second mapping feature vector H2, which is sent to the feature fusion module.

[0096] The second feature extraction network is configured to perform feature encoding processing on the first face-neck feature map and map the encoded feature tensor to a public feature vector space to obtain a corresponding second mapping feature vector H2, which is sent to the feature fusion module.

[0097] ​The first face-neck feature map is subjected to feature encoding processing by the second encoder to obtain a corresponding second feature tensor, which is sent to the second feature mapping network; the second feature tensor is subjected to feature vector mapping of a public feature vector space by the second feature mapping network to obtain a corresponding second mapping feature vector H2, which is sent to the feature fusion module.

[0098] c) The feature fusion module is configured to perform weighted summation calculation on the first mapping feature vector H1 and the second mapping feature vector H2 and send the calculation result as a corresponding first fusion feature vector H3 to the regression prediction network.

[0099] H3=w1H1+w2H1, w1 and w2 are the first and second weighting parameters; the two weighting parameters can be learned through model training.

[0100] d) The regression prediction network is sequentially connected by one or more hidden layers and an output layer; each hidden layer is sequentially connected by a fully connected layer and a nonlinear activation function, and the nonlinear activation function defaults to a ReLU activation function; the output layer is implemented based on a fully connected layer.

[0101] The regression prediction network is configured to perform regression prediction processing on the apnea hypopnea index according to the first fusion feature vector H3 to obtain a corresponding first predicted index and output the first predicted index.

[0102] In step 1, four artificial intelligence models are trained, specifically including:

[0103] In step 11, the face-neck semantic segmentation model is subjected to first model training based on a preset first data set.

[0104] The first data set includes a plurality of first data records; each first data record includes a first training feature map and a first label semantic map; the first training feature map is a front image, a left image or a right image with depth features and containing a human face and neck, or an arbitrary image with depth features but not containing a human face and neck; the feature composition of the pixel features of the first training feature map is consistent with that of the first feature map; the image size of the first label semantic map is consistent with that of the first training feature map; the feature composition of the pixel features of the first label semantic map is consistent with that of the first semantic map.

[0105] Specifically, in step 111, the first data set is randomly divided into two sub-data sets, denoted as a corresponding first training set and a first evaluation set, based on a preset first segmentation ratio.

[0106] The first segmentation ratio is a pre-set ratio parameter; the first training set and the first evaluation set each include a plurality of first data records; the total number of records of the first training set and the first evaluation set satisfies the first segmentation ratio.

[0107] Step 112, each first data record of the first training set is taken as a corresponding current training record; and the first training feature map of the current training record is taken as a current first feature map, which is input into the face-neck semantic segmentation model for processing, and the first semantic map obtained by this processing is taken as a corresponding first predicted semantic map; and a corresponding first prediction-label pair is formed by the first predicted semantic map and the first label semantic map of the current training record;

[0108] Step 113, all the first prediction-label pairs obtained are taken into the preset first model loss function to obtain a corresponding first loss value;

[0109] The first model loss function is realized based on a cross-entropy loss function;

[0110] Step 114, whether the first loss value meets a preset first loss value range is identified; if the first loss value meets the first loss value range, step 115 is turned to; if the first loss value does not meet the first loss value range, a round of modulation is performed on the model parameters of the face-neck semantic segmentation model in the direction of making the first model loss function reach a minimum value based on a preset first model optimizer, and step 112 is returned after the end of this round of modulation;

[0111] The first loss value range is a pre-set numerical range; and the first model optimizer at least includes an Adam optimizer and an SGD optimizer;

[0112] Step 115, each first data record of the first evaluation set is taken as a corresponding current evaluation record; and the first training feature map of the current evaluation record is taken as a current first feature map, which is input into the face-neck semantic segmentation model for processing, and the first semantic map obtained by this processing is taken as a corresponding second predicted semantic map; and a corresponding second prediction-label pair is formed by the second predicted semantic map and the first label semantic map of the current evaluation record; and the first accuracy, the first precision, the first recall and the first F1 score are calculated based on all the second prediction-label pairs obtained;

[0113] Step 116, whether the first accuracy, the first precision, the first recall and the first F1 score all meet corresponding first accuracy range, first precision range, first recall range and first F1 score range respectively is identified; if not, step 111 is returned for continuous training; if yes, it is confirmed that the first model training is ended;

[0114] Here, the first accuracy range, the first precision range, the first recall range and the first F1 score range are four pre-set numerical ranges;

[0115] Step 12, and based on the preset second data set, the throat semantic segmentation model is subjected to second model training;

[0116] The second data set includes a plurality of second data records; the second data record includes a second training feature map and a second label semantic map; the second training feature map is an intraoral image with depth features and containing a human throat part, or an arbitrary image with depth features but not containing a human throat part; the feature composition components of the pixel point features of the second training feature map are consistent with the second feature map; the image size of the second label semantic map is consistent with the second training feature map; the feature composition components of the pixel point features of the second label semantic map are consistent with the second semantic map;

[0117] Specifically, step 121, based on the preset second segmentation ratio, the second data set is randomly segmented into two sub-data sets, denoted as a corresponding second training set and a second evaluation set;

[0118] The second segmentation ratio is a pre-set ratio parameter; the second training set and the second evaluation set are both composed of a plurality of second data records; the total number of records of the second training set and the second evaluation set satisfies the second segmentation ratio;

[0119] Step 122, each second data record of the second training set is taken as a corresponding current training record; and the second training feature map of the current training record is taken as a current second feature map to input the throat semantic segmentation model for processing, and the second semantic map obtained by this processing is taken as a corresponding third predicted semantic map; and a corresponding third prediction-label pair is composed of the third predicted semantic map and the second label semantic map of the current training record;

[0120] Step 123, all the third prediction-label pairs obtained are brought into the preset second model loss function to calculate a corresponding second loss value;

[0121] The second model loss function is realized based on a cross-entropy loss function;

[0122] Step 124, whether the second loss value satisfies a preset second loss value range is identified; if the second loss value satisfies the second loss value range, step 125 is turned to; if the second loss value does not satisfy the second loss value range, based on a preset second model optimizer, the model parameters of the throat semantic segmentation model are modulated in a direction of making the second model loss function reach a minimum value for one round, and after the end of this round of modulation, step 122 is returned;

[0123] The second loss value range is a pre-set numerical range; the second model optimizer at least includes an Adam optimizer and an SGD optimizer;

[0124] Step 125, each second data record of the second evaluation set is taken as a corresponding current evaluation record; the second training feature map of the current evaluation record is taken as a current second feature map, which is input into the throat semantic segmentation model for processing, and the second semantic map obtained by this processing is taken as a corresponding fourth prediction semantic map; a corresponding fourth prediction-label pair is formed by the fourth prediction semantic map and the second label semantic map of the current evaluation record; and the second accuracy, the second precision, the second recall and the second F1 score are calculated based on all the fourth prediction-labels obtained.

[0125] Step 126, whether the second accuracy, the second precision, the second recall and the second F1 score all meet the corresponding second accuracy range, the second precision range, the second recall range and the second F1 score range respectively is identified; if not, return to step 121 to continue training; if so, it is confirmed that the second model training is completed.

[0126] Here, the second accuracy range, the second precision range, the second recall range and the second F1 score range are four pre-set numerical ranges.

[0127] Step 13, and based on the third data set, the face-neck key point recognition model is subjected to third model training.

[0128] Among them, the third data set includes a plurality of third data records; the third data record includes a third training feature map and a first label feature map; the third training feature map is a front image, a left image or a right image containing a human face-neck part and having depth and face-neck semantic features; the feature composition components of the pixel point features of the third training feature map are consistent with the third feature map; the image size of the first label feature map is consistent with the third training feature map; the feature composition components of the pixel point features of the first label feature map are consistent with the fourth feature map.

[0129] Specifically, it includes: step 131, based on the pre-set third segmentation ratio, the third data set is randomly divided into two sub-data sets, which are denoted as a corresponding third training set and a third evaluation set.

[0130] Among them, the third segmentation ratio is a pre-set ratio parameter; the third training set and the third evaluation set are both composed of a plurality of third data records; the total number of records of the third training set and the third evaluation set meets the third segmentation ratio;

[0131] Step 132, each third data record of the third training set is taken as a corresponding current training record; and the third training feature map of the current training record is taken as a current third feature map, which is input into the face-neck key point recognition model for processing, and a fourth feature map obtained by this processing is taken as a corresponding first predicted feature map; and a corresponding fifth prediction-label pair is formed by the first predicted feature map and the first label feature map of the current training record;

[0132] Step 133, all the obtained fifth prediction-label pairs are brought into a preset third model loss function to obtain a corresponding third loss value;

[0133] The third model loss function is realized based on a cross-entropy loss function;

[0134] Step 134, whether the third loss value meets a preset third loss value range is identified; if the third loss value meets the third loss value range, step 135 is turned to; if the third loss value does not meet the third loss value range, a model parameter of the face-neck key point recognition model is modulated in a direction of making the third model loss function reach a minimum value based on a preset third model optimizer, and step 132 is returned after the modulation is ended;

[0135] The third loss value range is a preset numerical range; and the third model optimizer at least includes an Adam optimizer and an SGD optimizer;

[0136] Step 135, each third data record of the third evaluation set is taken as a corresponding current evaluation record; and the third training feature map of the current evaluation record is taken as a current third feature map, which is input into the face-neck key point recognition model for processing, and a fourth feature map obtained by this processing is taken as a corresponding second predicted feature map; and a corresponding sixth prediction-label pair is formed by the second predicted feature map and the first label feature map of the current evaluation record; and the third accuracy, the third precision, the third recall and the third F1 score are obtained by calculating the accuracy, the precision, the recall and the F1 score based on all the obtained sixth prediction-label pairs;

[0137] Step 136, whether the third accuracy, the third precision, the third recall and the third F1 score all meet corresponding third accuracy range, third precision range, third recall range and third F1 score range respectively is identified; if not, step 131 is returned to continue training; if yes, it is confirmed that the third model training is ended;

[0138] Here, the third accuracy range, the third precision range, the third recall range and the third F1 score range are four preset numerical ranges;

[0139] Step 14, and based on the fourth preset data set, the fourth model training is performed on the apnea hypopnea index prediction model;

[0140] The fourth data set includes a plurality of fourth data records; the fourth data record includes a first training feature vector, a first training face-neck feature map, and a first label index; the vector data components of the first training feature vector are consistent with the first feature vector, and are composed of the corresponding face width / eye width ratio, chin prominence, chin-neck angle, soft palate length, and uvula size, wherein the uvula size is composed of the corresponding length, width, and thickness; the first training face-neck feature map is a 3D image containing the face and neck parts of the human body and having semantic features and key point features of the face and neck parts; the feature components of the voxel point features of the first training face-neck feature map are consistent with the first face-neck feature map; and the first label index is a corresponding apnea hypopnea index.

[0141] Specifically, step 141 includes: based on a preset fourth segmentation ratio, the fourth data set is randomly segmented into two sub-data sets, denoted as a corresponding fourth training set and a fourth evaluation set;

[0142] The fourth segmentation ratio is a pre-set ratio parameter; the fourth training set and the fourth evaluation set are both composed of a plurality of fourth data records; and the total number of records of the fourth training set and the fourth evaluation set satisfies the fourth segmentation ratio.

[0143] Step 142, each fourth data record of the fourth training set is taken as a corresponding current training record; and the first training feature vector and the first training face-neck feature map of the current training record are taken as the current first feature vector and the first face-neck feature map to input the apnea hypopnea index prediction model for processing, and the first prediction index obtained by this processing and the first label index of the current training record are taken to form a corresponding seventh prediction-label pair.

[0144] Step 143, all the obtained seventh prediction-label pairs are taken into a preset fourth model loss function to obtain a corresponding fourth loss value;

[0145] The fourth model loss function is implemented based on an L1 loss function or an L2 loss function;

[0146] Step 144, whether the fourth loss value satisfies a preset fourth loss value range is identified; if the fourth loss value satisfies the fourth loss value range, step 145 is performed; if the fourth loss value does not satisfy the fourth loss value range, a model parameter of the apnea hypopnea index prediction model is modulated in a direction of minimizing the fourth model loss function based on a preset fourth model optimizer, and after the modulation is completed, step 142 is returned;

[0147] The fourth loss value range is a pre-set numerical range; and the fourth model optimizer includes at least an Adam optimizer and an SGD optimizer.

[0148] In step 145, each fourth data record of the fourth evaluation set is taken as a corresponding current evaluation record; the third training feature map of the current evaluation record is taken as a current third feature map and input into the face-neck key point recognition model for processing; the first training feature vector and the first training face-neck feature map of the current evaluation record are taken as a current first feature vector and a first face-neck feature map and input into the apnea-hypoventilation index prediction model for processing; the first predicted index obtained by the processing and the first label set of the current evaluation record are taken as a corresponding eighth prediction-label pair; and all the obtained eighth prediction-label pairs are taken into the pre-set first model evaluation function to obtain a corresponding first evaluation value.

[0149] The first model evaluation function is implemented based on an MAE function, an MSE function or an RMSE function.

[0150] In step 146, it is identified whether the first evaluation value meets a pre-set first evaluation value range; if not, the step 141 is returned to continue the training; and if yes, it is confirmed that the fourth model training is ended.

[0151] The first evaluation value range is a pre-set numerical range.

[0152] In step 15, after the training of the first, second, third and fourth models is ended, it is confirmed that the model training of the four artificial intelligence models is ended.

[0153] In step 2, after the model training of the four artificial intelligence models is ended, the front view, left and right side views and the inside view of the oral cavity of the subject are photographed by using the same camera according to a pre-set image shooting requirement to obtain a corresponding first front view, a first left view, a first right view and a first oral cavity view; and pixel-level deep feature addition processing is performed on the first front view, the first left view, the first right view and the first oral cavity view to obtain a corresponding second front view, a second left view, a second right view and a second oral cavity view.

[0154] Specifically, in step 21, after the model training of the four artificial intelligence models is ended, the front view, left and right side views and the inside view of the oral cavity of the subject are photographed by using the same camera according to a pre-set image shooting requirement to obtain a corresponding first front view, a first left view, a first right view and a first oral cavity view.

[0155] Here, the image shooting requirements of the embodiment of the present application include: shooting images of the front face, left side face and right side face of the human face respectively to generate a front face image, a left side image and a right side image, with the premise that the front face, left side face and right side face are respectively parallel to the imaging plane of the camera lens; guiding the tongue tip of the subject to rest on the lower teeth and open the mouth to the maximum extent to make an "ah" sound, and shooting an image of the inside of the oral cavity of the subject at this time to generate an oral cavity image; the front face image requires that the image contains the face and neck positions, and the face needs to expose both ears, and the imaging area of the face and neck and the image area ratio is greater than a preset first proportion threshold, and the left side face or right side face requires that the image contains the corresponding left or right face and neck positions, and the face needs to expose the corresponding left or right ear, and the imaging area of the face and neck and the image area ratio is greater than the first proportion threshold; the tonsils, soft palate and uvula edges of the throat position in the oral cavity image are clear, and the imaging area of the throat position and the image area ratio is greater than a preset second proportion threshold; the first and second proportion thresholds are two preset threshold parameters;

[0156] The first front face image, the first left side image, the first right side image and the first oral cavity image of the embodiment of the present application are all RGB images, and the pixel point features of the first front face image, the first left side image, the first right side image and the first oral cavity image include RGB three primary colors;

[0157] Step 22, and the first front face image, the first left side image, the first right side image and the first oral cavity image are subjected to pixel-level depth feature addition processing to obtain corresponding second front face image, second left side image, second right side image and second oral cavity image;

[0158] Specifically, it includes: using a preset depth image generation tool to generate a depth image of the first front face image, the first left side image, the first right side image and the first oral cavity image to obtain a corresponding front face depth image, a left side depth image, a right side depth image and an oral cavity depth image; and the first front face image and the front face depth image are subjected to pixel feature fusion processing to obtain a corresponding second front face image; and the first left side image and the left side depth image are subjected to pixel feature fusion processing to obtain a corresponding second left side image; and the first right side image and the right side depth image are subjected to pixel feature fusion processing to obtain a corresponding second right side image; and the first oral cavity image and the oral cavity depth image are subjected to pixel feature fusion processing to obtain a corresponding second oral cavity image;

[0159] Here, the depth image generation tool of the embodiment of the present application includes MiDaS tool, DPT model tool, OpenCV tool, etc.; and the pixel point features of the second front face image, the second left side image, the second right side image and the second oral cavity image of the embodiment of the present application include RGB three primary colors and depth.

[0160] Step 3, using the face-neck semantic segmentation model and the face-neck key point recognition model to add face-neck semantic and key point features to the second front view, the second left view and the second right view, rotating the three views after the feature addition, converting and registering the point clouds of the three views after the rotation, and constructing a first face-neck feature map based on the registered three-dimensional point clouds to obtain the first face-neck feature map; and calculating the face width / eye width ratio, the chin protrusion and the chin-neck angle based on the first face-neck feature map to obtain the corresponding first face width / eye width ratio, the first chin protrusion and the first chin-neck angle;

[0161] Specifically includes: step 31, using the face-neck semantic segmentation model and the face-neck key point recognition model to add face-neck semantic and key point features to the second front view, the second left view and the second right view, rotating the three views after the feature addition, converting and registering the point clouds of the three views after the rotation, and constructing a first face-neck feature map based on the registered three-dimensional point clouds to obtain the first face-neck feature map;

[0162] Specifically includes: step 311, taking the second front view, the second left view and the second right view as a corresponding first feature map respectively, inputting the face-neck semantic segmentation model to obtain a corresponding first semantic map; and performing pixel feature fusion on each first feature map and its corresponding first semantic map to obtain a corresponding first fusion map; and taking the first fusion map corresponding to the second front view, the second left view and the second right view as the corresponding third front view, the third left view and the third right view;

[0163] Wherein, the pixel point features of the third front view, the third left view and the third right view include the first RGB three primary colors, the first depth and the first semantic type;

[0164] Step 312, and taking the third front view, the third left view and the third right view as a corresponding third feature map respectively, inputting the face-neck key point recognition model to obtain a corresponding fourth feature map; and performing pixel feature fusion on each third feature map and its corresponding fourth feature map to obtain a corresponding second fusion map; and taking the second fusion map corresponding to the third front view, the third left view and the third right view as the corresponding fourth front view, the fourth left view and the fourth right view;

[0165] Wherein, the pixel point features of the fourth front view, the fourth left view and the fourth right view include the third RGB three primary colors, the third depth, the third semantic type and the first key point type;

[0166] Step 313, and mark the pixel points of the first key point type of the inter-brow point, the rhinion point, the left infraorbital point and the right infraorbital point on the fourth front face image as the corresponding first inter-brow point, the first rhinion point, the first left infraorbital point and the first right infraorbital point; draw a straight line based on the first inter-brow point and the first rhinion point to obtain a first inter-brow-rhinion connecting line; draw a straight line based on the first left infraorbital point and the first right infraorbital point to obtain a first infraorbital connecting line; rotate and translate the fourth front face image with the first inter-brow-rhinion connecting line as the image center line as the target, and then rotate the fourth front face image with the first left infraorbital point and the first right infraorbital point having the same depth as the target and the first inter-brow-rhinion connecting line as the rotation axis; and form the corresponding front face pixel point set from the pixel points of the third semantic type other than the background point on the rotated fourth front face image.

[0167] In the process of rotating or translating the fourth front face image, the third depth of each pixel point is adaptively adjusted.

[0168] Step 314, and take the fourth left side image or the fourth right side image as the corresponding current side face image; mark the pixel points of the first key point type of the inter-brow point, the nose tip point and the chin point on the current side face image as the corresponding second inter-brow point, the first nose tip point and the first chin point; draw a plane based on the second inter-brow point, the first nose tip point and the first chin point to obtain a first plane; rotate the current side face image with the depth of all pixel points on the first plane remaining the same as the target; and form the corresponding left side pixel point set or the right side pixel point set from the pixel points of the third semantic type other than the background point on the rotated current side face image.

[0169] In the process of rotating the fourth left side image and the fourth right side image, the third depth of each pixel point is adaptively adjusted.

[0170] Step 315, and perform point cloud conversion processing on the front face pixel point set, the left side pixel point set and the right side pixel point set to obtain the corresponding front face point cloud, the left side point cloud and the right side point cloud.

[0171] The point features of the front face point cloud, the left side point cloud and the right side point cloud include the third RGB three primary colors, the third depth, the third semantic type and the first key point type.

[0172] Step 316, and record the positions corresponding to the glabella point, the nasion point, the left infraorbital point and the right infraorbital point in the front point cloud as the corresponding third glabella point, the second nasion point, the second left infraorbital point and the second right infraorbital point respectively; record the positions corresponding to the glabella point, the nasion point and the left infraorbital point in the left point cloud as the fourth glabella point, the third nasion point and the third left infraorbital point respectively; record the positions corresponding to the glabella point, the nasion point and the right infraorbital point in the right point cloud as the fifth glabella point, the fourth nasion point and the third right infraorbital point respectively; form the first matching point pair by the third and fourth glabella points, form the second matching point pair by the second and third nasion points, form the third matching point pair by the second and third left infraorbital points, form the fourth matching point pair by the third and fifth glabella points, form the fifth matching point pair by the second and fourth nasion points, and form the sixth matching point pair by the second and third right infraorbital points; perform point cloud registration on the left point cloud with the front point cloud as the reference point cloud and the minimum sum of the distances between the points of the first, second and third matching point pairs as the registration target; perform point cloud registration on the right point cloud with the front point cloud as the reference point cloud and the minimum sum of the distances between the points of the fourth, fifth and sixth matching point pairs as the registration target;

[0173] In the process of registering the left point cloud and the right point cloud, the three-dimensional coordinates of each point of the current point cloud are adaptively adjusted.

[0174] Step 317, perform point cloud fusion processing based on the front point cloud, the left point cloud and the right point cloud after the point cloud registration to obtain the corresponding first fused point cloud; perform three-dimensional voxel space construction based on the width, height and depth of the first fused point cloud; and perform three-dimensional feature map construction based on the first fused point cloud and the three-dimensional voxel space to obtain the corresponding first face-neck feature map.

[0175] Here, the voxel point features of the first face-neck feature map of the embodiment of the present application include the fourth RGB three primary colors, the fourth semantic type and the second key point type; the fourth semantic type includes the face, the neck and the background; the type range of the second key point type is composed of the total set of face-neck key points and non-key points.

[0176] Step 32, calculate the face width / eye width ratio, the chin protrusion and the chin-neck angle based on the first face-neck feature map to obtain the corresponding first face width / eye width ratio, the first chin protrusion and the first chin-neck angle.

[0177] Specifically, it includes: step 321, calculate the straight-line distance between the two voxel points of the first face-neck feature map whose second key point type is the left and right tragus points respectively and take the calculation result as the corresponding first face width; calculate the straight-line distance between the two voxel points of the first face-neck feature map whose second key point type is the left and right infraorbital points respectively and take the calculation result as the corresponding first eye width; and calculate the first face width / eye width ratio based on the first face width and the first eye width.

[0178] Here, the first face width / eye width ratio = first face width: first eye width;

[0179] Step 322, and record the five voxel points of the second key point type of the first face-neck feature map as the corresponding glabella, nasi, mentum, left suborbicularis, and right suborbicularis voxel points, respectively; take the plane where the glabella, nasi, and mentum voxel points are located as the corresponding first sagittal section; take the section that passes through the left suborbicularis and right suborbicularis voxel points and is perpendicular to the first sagittal section as the corresponding first transverse section; take the section that passes through the mentum voxel point and is perpendicular to the first sagittal section and parallel to the first transverse section as the corresponding second transverse section; take the perpendicular line of the second transverse section that passes through the nasi voxel point as the corresponding first perpendicular line; take the direction towards the inside and outside of the face on the first sagittal section as the rear and front of the first perpendicular line; take the perpendicular line of the second transverse section that passes through the mentum voxel point as the corresponding second perpendicular line; and calculate the line distance of the first and second perpendicular lines to obtain the corresponding first distance d, and set the first mentum protrusion = d when the second perpendicular line is in front of the first perpendicular line, and set the first mentum protrusion = -d when the second perpendicular line is behind the first perpendicular line;

[0180] Here, the first distance d is as shown in Figure 3 The first distance provided by the embodiment one of the present application is shown in the schematic diagram;

[0181] Step 323, and record the two voxel points of the second key point type of the first face-neck feature map as the corresponding submentum and neck voxel points; take the spatial vector from the nasi voxel point to the mentum voxel point as the corresponding vector A, and take the spatial vector from the submentum voxel point to the neck voxel point as the corresponding vector B; calculate the vector included angle θ based on the vectors A and B; and calculate the corresponding first mentum-neck angle based on the vector included angle θ;

[0182] Here, the vectors A and B are as shown in Figure 4 The vectors A and B provided by the embodiment one of the present application are shown in the schematic diagram;

[0183] The calculation process of the vector included angle θ and the first mentum-neck angle α is as follows:

[0184]

[0185] The calculation formula of the first mentum-neck angle α is an empirical formula, in which δ is a preset compensation angle;

[0186] Step 324, take the first face width / eye width ratio, the first mentum protrusion, and the first mentum-neck angle obtained this time as the measurement result output this time.

[0187] Step 4, the length of the soft palate and the three-dimensional size of the uvula in the second oral cavity image are calculated by using the throat semantic segmentation model to obtain the corresponding first soft palate length and first uvula size;

[0188] The first uvula size includes the length of the uvula, the width of the uvula, and the thickness of the uvula.

[0189] Specifically, step 41, input the second oral cavity image as a corresponding second feature map into the throat semantic segmentation model for processing to obtain a corresponding second semantic map; and perform point cloud conversion processing on the obtained second semantic map to obtain a corresponding oral cavity internal point cloud;

[0190] The point features of the oral cavity internal point cloud include the second semantic type.

[0191] Step 42, record the points with the second semantic type of soft palate or uvula in the oral cavity internal point cloud as corresponding soft palate points or uvula points; calculate the center point coordinates of all soft palate points to obtain a corresponding first center point, and calculate the center point coordinates of all uvula points to obtain a corresponding second center point, and draw a straight line based on the first center point and the second center point to obtain a corresponding first center line; take the highest soft palate point on the first center line as the soft palate vertex; take the two soft palate points with the highest and lowest heights on the first center line as the corresponding uvula base center point and uvula tip point; calculate the straight line distance from the soft palate vertex to the uvula tip point and take the calculation result as the corresponding first soft palate length; calculate the straight line distance from the uvula base center point to the uvula tip point and take the calculation result as the corresponding uvula length.

[0192] Step 43, take the leftmost and rightmost two uvula points among all uvula points as the corresponding left and right end points; calculate the vertical distances from the left and right end points to the first center line to obtain the corresponding left and right vertical distances; and sum the left and right vertical distances to obtain the corresponding uvula width.

[0193] Step 44, take the nearest and farthest two uvula points among all uvula points as the corresponding near and far end points; calculate the vertical distances from the near and far end points to the first center line to obtain the corresponding near and far vertical distances; and sum the near and far vertical distances to obtain the corresponding uvula thickness.

[0194] Step 45, the first uvula size is composed of the uvula length, the uvula width, and the uvula thickness.

[0195] Step 46, output the first soft palate length and the first uvula size obtained as the calculation result.

[0196] Step 5, a corresponding first feature vector is composed of the first face width / eye width ratio, the first chin prominence, the first chin-neck angle, the first soft palate length, and the first uvula size; the first feature vector and the first face-neck feature map are input into an apnea hypopnea index prediction model to predict the apnea hypopnea index to obtain a corresponding first predicted index; and based on a preset index-risk level correspondence relationship, the first predicted index is used to identify the risk level of obstructive sleep apnea to obtain a corresponding first risk level.

[0197] Here, the index-risk level correspondence relationship of the embodiment of the application includes a plurality of correspondence records; each correspondence record is composed of an index value range and a corresponding risk level.

[0198] Step 6, a first prediction report of the current subject is composed of the first feature vector, the first predicted index, and the first risk level.

[0199] Figure 5 A module structure diagram of an obstructive sleep apnea risk prediction device provided by the second embodiment of the application is provided, and the device is a terminal device or a server for realizing the method embodiment, or a device capable of enabling the terminal device or the server to realize the method embodiment, such as a device or a chip system of the terminal device or the server. As shown in the figure, the device includes a tool model preparation module 201, a subject image acquisition module 202, a face-neck map processing module 203, an oral cavity map processing module 204, an index and risk level prediction module 205, and a prediction report generation module 206. Figure 5 The tool model preparation module 201 is used to pre-set four artificial intelligence models, and the four artificial intelligence models are respectively subjected to model training; the four artificial intelligence models include a face-neck semantic segmentation model, a throat semantic segmentation model, a face-neck key point recognition model, and an apnea hypopnea index prediction model.

[0200] The tool model preparation module 201 is used to pre-set four artificial intelligence models, and the four artificial intelligence models are respectively subjected to model training; the four artificial intelligence models include a face-neck semantic segmentation model, a throat semantic segmentation model, a face-neck key point recognition model, and an apnea hypopnea index prediction model.

[0201] The subject image acquisition module 202 is used to, after the model training of the four artificial intelligence models is completed, capture the front, left and right side, and oral cavity internal images of the subject to obtain a corresponding first front image, a first left image, a first right image, and a first oral cavity image according to a preset image shooting requirement and using the same camera; and the first front image, the first left image, the first right image, and the first oral cavity image are subjected to pixel-level deep feature addition processing to obtain a corresponding second front image, a second left image, a second right image, and a second oral cavity image.

[0202] The face-neck image processing module 203 is configured to add face-neck semantic and key point features to the second front image, the second left image and the second right image by using a face-neck semantic segmentation model and a face-neck key point recognition model, rotate the three images after the feature addition, convert and register point clouds of the three images after the rotation, and construct a first face-neck feature map based on the registered three-dimensional point clouds.

[0203] The oral cavity image processing module 204 is configured to measure the three-dimensional size of the soft palate length and the uvula in the second oral cavity image by using a throat semantic segmentation model to obtain a first soft palate length and a first uvula size; the first uvula size includes a uvula length, a uvula width and a uvula thickness.

[0204] The index and risk level prediction module 205 is configured to form a first feature vector corresponding to the first face-neck feature map and the first face-neck feature map, and input the first feature vector and the first face-neck feature map into an apnea hypopnea index prediction model to predict the apnea hypopnea index to obtain a first predicted index; and based on a preset index-risk level correspondence relationship, the first predicted index is used to identify the risk level of obstructive sleep apnea to obtain a first risk level; the index-risk level correspondence relationship includes a plurality of correspondence records; each correspondence record is composed of an index value range and a corresponding risk level.

[0205] The prediction report generation module 206 is configured to form a first prediction report of the current subject by the first feature vector, the first predicted index and the first risk level.

[0206] The obstructive sleep apnea risk prediction device provided by the embodiment of the present application can execute the method steps in the above-mentioned method embodiment, and has similar implementation principles and technical effects, which will not be repeated here.

[0207] It should be noted that the division of the various modules of the above apparatus is only a logical functional division, and in actual implementation, all or part of them can be integrated into one physical entity, or can be physically separated. These modules can all be implemented in the form of software invoked by a processing element; all can be implemented in the form of hardware; or some modules can be implemented in the form of software invoked by a processing element, and some modules can be implemented in the form of hardware. For example, the tool model preparation module can be a separately established processing element, or can be integrated in a chip of the above apparatus, in addition, it can also be stored in the form of program code in the memory of the above apparatus, and the function of the above determination module is invoked and executed by a processing element of the above apparatus. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or can be independently implemented. The processing element described herein can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of the hardware in the processor element or the instructions in the form of software.

[0208] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of program code invoked by a processing element, the processing element can be a general purpose processor, such as a central processing unit (CPU) or other processor that can invoke program code. For another example, these modules can be integrated together to implement in the form of system on a chip (SOC).

[0209] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0210] Figure 6 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 6 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0211] exist Figure 6The system bus 305 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as a client, a read-write library and a read-only library). The memory can include a Random Access Memory (RAM), and can also include a Non-Volatile Memory, such as at least one disk memory.

[0212] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), a Graphics Processing Unit (GPU), etc.; can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0213] It should be noted that the embodiments of the present application also provide a computer readable storage medium, which stores instructions, and when the instructions run on a computer, the computer executes the method and process provided in the above embodiments.

[0214] The embodiment of the present application provides a method and device for predicting the risk of obstructive sleep apnea, an electronic device and a computer readable storage medium. According to the above content, the embodiment of the present application pre-trains four tool models: a face-neck semantic segmentation model, a throat semantic segmentation model, a face-neck key point recognition model and an apnea hypopnea index prediction model. The OSA risk prediction is realized based on the four tool models, specifically: first, the front, left and right side and oral cavity images of the measured person are photographed, and depth features are added to the four photographed images; then, the face-neck semantic segmentation model and the face-neck key point recognition model are used to add face-neck semantic and key point features to the front, left and right side images, and the three images with added features are subjected to image rotation, point cloud conversion and registration, and a three-dimensional face-neck feature map is constructed based on the registered three-dimensional point cloud; the face width / eye width ratio, chin protrusion and chin-neck angle are calculated based on the three-dimensional face-neck feature map; the three-dimensional dimensions of the soft palate length and the uvula in the oral cavity image are calculated using the throat semantic segmentation model; then, the feature vector composed of the chin features (face width / eye width ratio, chin protrusion, chin-neck angle) and the upper airway features (soft palate length and uvula size) and the three-dimensional face-neck feature Figure One are input into the apnea hypopnea index prediction model to predict the apnea hypopnea index; the risk level of obstructive sleep apnea is identified according to the predicted index based on the index-risk level correspondence; finally, the prediction report of the current measured person is composed of the feature vector, the predicted index and the risk level. The embodiment of the present application applied to OSA risk assessment not only improves the convenience of assessment, but also shortens the assessment time, improves the assessment efficiency and reduces the assessment cost.

[0215] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0216] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for predicting the risk of obstructive sleep apnea, characterized in that, The method includes: Four artificial intelligence models are pre-set; and the four artificial intelligence models are trained respectively; the four artificial intelligence models include a face and neck semantic segmentation model, a throat semantic segmentation model, a face and neck key point recognition model, and an apnea-hypopnea index prediction model. After the training of the four artificial intelligence models is completed, the front, left and right sides, and inside the oral cavity of the test subject are captured using the same camera according to the preset image capture requirements to obtain the corresponding first front view, first left side view, first right side view, and first oral cavity view; and pixel-level depth feature addition processing is performed on the first front view, first left side view, first right side view, and first oral cavity view to obtain the corresponding second front view, second left side view, second right side view, and second oral cavity view. The face and neck semantic segmentation model and the face and neck key point recognition model are used to add face and neck semantic and key point features to the second frontal image, the second left image, and the second right image. The three images with added features are rotated, and point cloud transformation and registration are performed on the rotated three images. Based on the registered three point clouds, a three-dimensional face and neck feature map is constructed to obtain a first face and neck feature map. Based on the first face and neck feature map, the face width / eye width ratio, chin protrusion, and chin-neck angle are calculated to obtain the corresponding first face width / eye width ratio, first chin protrusion, and first chin-neck angle. The length of the soft palate and the three-dimensional dimensions of the uvula in the second oral cavity image are calculated using the pharyngeal semantic segmentation model to obtain the corresponding first soft palate length and first uvula dimensions; the first uvula dimensions include uvula length, uvula width and uvula thickness. A first feature vector is formed by the first face width / eye width ratio, the first chin projection, the first chin-neck angle, the first soft palate length, and the first uvula size. The first feature vector and the first face-neck feature map are then input into the apnea-hypopnea index prediction model to predict the apnea-hypopnea index and obtain the corresponding first predicted index. Based on a preset index-risk level correspondence, the first predicted index is used to identify the obstructive sleep apnea risk level and obtain the corresponding first risk level. The index-risk level correspondence includes multiple correspondence records; each correspondence record consists of an index value range and a corresponding risk level. The first prediction report for the subject is composed of the first feature vector, the first prediction index, and the first risk level.

2. The method for predicting the risk of obstructive sleep apnea according to claim 1, characterized in that, The face and neck semantic segmentation model is implemented based on a type of image segmentation model. The model extracts image features from the first feature map input to the model and performs pixel-level semantic segmentation of the face, neck, and background based on the extracted features, outputting the segmentation result as the corresponding first semantic map. The first feature map is a frontal, left-side, or right-side image containing depth features and including the human face and neck area. The pixel features of the first feature map include the first RGB primary colors and the first depth. The image size of the first semantic map is consistent with that of the first feature map. The pixel features of the first semantic map include a first semantic type; the first semantic type includes face, neck, and background. The pharyngeal semantic segmentation model is implemented based on a type of image segmentation model. This model extracts image features from the second feature map input to the model and performs pixel-level semantic segmentation of the soft palate, uvula, and background based on the extracted features, outputting the segmentation result as the corresponding second semantic map. The first feature map is an image of the inside of the oral cavity containing depth features and representing the human pharyngeal region. The pixel features of the second feature map include the second RGB primary colors and the second depth. The image size of the second semantic map is consistent with that of the second feature map. The pixel features of the second semantic map include a second semantic type, which includes the soft palate, uvula, and background. The face and neck key point recognition model is implemented based on a type of face key point recognition model. This model extracts image features from the third feature map input to the model and performs pixel-level key point classification and prediction based on the extracted features and a preset set of face and neck key points. The prediction result is then output as the corresponding fourth feature map. The third feature map is a frontal, left-side, or right-side image containing the human face and neck area and possessing depth and semantic features related to the face and neck. The pixel features of the third feature map include the third RGB primary colors, the third depth, and the third semantic type. The third semantic type includes face, neck, and background. The image size of the fourth feature map is consistent with that of the third feature map. The pixel features of the fourth feature map include a first key point type. The type range of the first key point type consists of the set of face and neck key points and non-key points. The set of face and neck key points includes multiple types of face key points and multiple types of neck key points. The apnea-hypopnea index prediction model is used to perform feature extraction processing on the first feature vector and the first face and neck feature map input to the model, perform feature fusion processing on the dual features, perform regression prediction processing on the apnea-hypopnea index based on the fused features, and output the corresponding first prediction index. The first feature vector includes the first face width / eye width ratio, the first chin projection, the first chin-neck angle, the first soft palate length, and the first uvula size; the first uvula size includes the uvula length, the uvula width, and the uvula thickness; The first face and neck feature map is a 3D image that includes the human face and neck area and has face and neck semantic features and key point features; the voxel features of the first face and neck feature map include the fourth RGB three primary colors, the fourth semantic type and the second key point type; the fourth semantic type includes face, neck and background; the type range of the second key point type is composed of the set of face and neck key points and non-key points; The first model input terminal of the apnea-hypopnea index prediction model is used to receive the first feature vector, the second mode input terminal is used to receive the first face and neck feature map, and the model input terminal is used to output the corresponding first prediction index. The apnea-hypopnea index prediction model includes a first feature extraction network, a second feature extraction network, a feature fusion module, and a regression prediction network. The input of the first feature extraction network is connected to the input of the first model, and its output is connected to the first input of the feature fusion module. The input of the second feature extraction network is connected to the input of the second model, and its output is connected to the second input of the feature fusion module. The output of the feature fusion module is connected to the input of the regression prediction network. The output of the regression prediction network is connected to the output of the model. The first feature extraction network is composed of a first encoder and a first feature mapping network connected sequentially; the first encoder and the first feature mapping network are each implemented based on an MLP model. The first feature extraction network is used to perform feature encoding processing on the first feature vector and map the encoded feature tensor to a preset common feature vector space to obtain the corresponding first-path mapped feature vector H1, which is then sent to the feature fusion module. Specifically, the first encoder performs feature encoding processing on the first feature vector to obtain the corresponding first feature tensor, which is then sent to the first feature mapping network; the first feature mapping network maps the first feature tensor to the feature vectors in the common feature vector space to obtain the corresponding first-path mapped feature vector H1, which is then sent to the feature fusion module. The second feature extraction network is composed of a second encoder and a second feature mapping network connected sequentially; the second encoder is implemented based on a type of 3D convolutional neural network, 3D residual neural network, or 3D graph neural network; the second feature mapping network is implemented based on an MLP model. The second feature extraction network is used to perform feature encoding processing on the first face and neck feature map and map the encoded feature tensor to the common feature vector space to obtain the corresponding second-path mapped feature vector H2, which is then sent to the feature fusion module. Specifically, the second encoder performs feature encoding processing on the first face and neck feature map to obtain the corresponding second feature tensor, which is then sent to the second feature mapping network; the second feature mapping network maps the second feature tensor to the feature vectors in the common feature vector space to obtain the corresponding second-path mapped feature vector H2, which is then sent to the feature fusion module. The feature fusion module is used to perform a weighted summation calculation on the first path mapping feature vector H1 and the second path mapping feature vector H2, and send the calculation result as the corresponding first fused feature vector H3 to the regression prediction network; wherein, H3 = w1H1 + w2H1, w1 and w2 are learnable first weighting parameters and second weighting parameters; The regression prediction network consists of one or more hidden layers and one output layer connected sequentially; each hidden layer consists of a fully connected layer and a nonlinear activation function connected sequentially, with the nonlinear activation function being the ReLU activation function by default; the output layer is implemented based on a fully connected layer; the regression prediction network is used to perform regression prediction processing on the apnea-hypopnea index based on the first fusion feature vector H3 to obtain the corresponding first prediction index and output it.

3. The method for predicting the risk of obstructive sleep apnea according to claim 2, characterized in that, The training of the four artificial intelligence models specifically includes: The face and neck semantic segmentation model is trained using a first preset dataset; the throat semantic segmentation model is trained using a second preset dataset; the face and neck key point recognition model is trained using a third preset dataset; and the apnea-hypopnea index prediction model is trained using a fourth preset dataset. After the training of the first, second, third, and fourth models is completed, the training of the four artificial intelligence models is confirmed to be complete. The first dataset includes multiple first data records; each first data record includes a first training feature map and a first label semantic map; the first training feature map is a frontal image, left-side image, or right-side image with depth features and containing the human face and neck area, or any image with depth features but not containing the human face and neck area; the feature components of the pixel features of the first training feature map are consistent with those of the first feature map; the image size of the first label semantic map is consistent with that of the first training feature map; and the feature components of the pixel features of the first label semantic map are consistent with those of the first semantic map. The second dataset includes multiple second data records; each second data record includes a second training feature map and a second label semantic map; the second training feature map is either an image of the inside of the mouth with depth features and containing the human throat, or any image with depth features but not containing the human throat; the feature composition of the pixel features of the second training feature map is consistent with that of the second feature map; the image size of the second label semantic map is consistent with that of the second training feature map; the feature composition of the pixel features of the second label semantic map is consistent with that of the second semantic map. The third dataset includes multiple third data records; each third data record includes a third training feature map and a first label feature map; the third training feature map is a frontal, left-side, or right-side image containing the human face and neck area and possessing depth and semantic features of the face and neck; the feature components of the pixel features of the third training feature map are consistent with those of the third feature map; the image size of the first label feature map is consistent with that of the third training feature map; the feature components of the pixel features of the first label feature map are consistent with those of the fourth feature map. The fourth dataset includes multiple fourth data records; each fourth data record includes a first training feature vector, a first training face-neck feature map, and a first label index; the vector data components of the first training feature vector are consistent with those of the first feature vector, consisting of the corresponding face width / eye width ratio, chin protrusion, chin-neck angle, soft palate length, and uvula size, wherein the uvula size is composed of the corresponding length, width, and thickness; the first training face-neck feature map is a 3D image containing the human face and neck area and carrying face and neck semantic features and key point features; the feature components of the voxel point features of the first training face-neck feature map are consistent with those of the first face-neck feature map; the first label index is a corresponding apnea-hypopnea index.

4. The method for predicting the risk of obstructive sleep apnea according to claim 1, characterized in that, The image capture requirements include: taking images of the front, left, and right sides of the face, respectively, with the front, left, and right sides parallel to the imaging plane of the camera lens, to generate the frontal image, the left-side image, and the right-side image; guiding the subject to touch their lower teeth with their tongue and open their mouth to the maximum extent to make an "ah" sound, and at this time taking an image of the inside of the subject's mouth to generate the oral cavity image; the frontal image requires that the image include the face and neck, and both ears must be visible on the face, and the ratio of the face and neck imaging area to the image area is greater than a preset first ratio threshold; the left-side or right-side image requires that the image include the corresponding left or right side of the face and neck, and the corresponding left or right ear must be visible on the face, and the ratio of the face and neck imaging area to the image area is greater than the first ratio threshold; the oral cavity image shows that the edges of the tonsils, soft palate, and uvula in the pharynx are clear, and the ratio of the pharyngeal imaging area to the image area is greater than a preset second ratio threshold; The pixel features of the first frontal image, the first left side image, the first right side image, and the first oral cavity image include the three primary colors of RGB; The pixel features of the second frontal image, the second left side image, the second right side image, and the second oral cavity image include RGB primary colors and depth; The step of adding pixel-level depth features to the first frontal image, first left side image, first right side image, and first oral cavity image to obtain the corresponding second frontal image, second left side image, second right side image, and second oral cavity image specifically involves: using a preset depth map generation tool to generate depth maps for the first frontal image, first left side image, first right side image, and first oral cavity image to obtain the corresponding frontal depth map, left side depth map, right side depth map, and oral cavity depth map; performing pixel feature fusion processing on the first frontal image and the frontal depth map to obtain the corresponding second frontal image; performing pixel feature fusion processing on the first left side image and the left side depth map to obtain the corresponding second left side image; performing pixel feature fusion processing on the first right side image and the right side depth map to obtain the corresponding second right side image; and performing pixel feature fusion processing on the first oral cavity image and the oral cavity depth map to obtain the corresponding second oral cavity image. The depth map generation tool includes MiDaS tools, DPT model tools, and OpenCV tools.

5. The method for predicting the risk of obstructive sleep apnea according to claim 2, characterized in that, The process involves adding semantic and key point features to the second frontal image, the second left image, and the second right image using the face and neck semantic segmentation model and the face and neck key point recognition model. Then, the three images with added features are rotated, and point cloud transformation and registration are performed on the rotated three images. Finally, a first face and neck feature map is constructed based on the registered three-dimensional point cloud. Specifically, this includes: The second frontal image, the second left-side image, and the second right-side image are each used as a corresponding first feature map and input into the face and neck semantic segmentation model for processing to obtain the corresponding first semantic map. Then, pixel features of each of the obtained first feature maps are fused with their corresponding first semantic maps to obtain a corresponding first fused image. The first fused image corresponding to the second frontal image, the second left-side image, and the second right-side image is used as the corresponding third frontal image, the third left-side image, and the third right-side image. The pixel features of the third frontal image, the third left-side image, and the third right-side image include the first RGB primary colors, the first depth, and the first semantic type. The third frontal image, the third left-side image, and the third right-side image are each used as a corresponding third feature image and input into the face and neck key point recognition model for processing to obtain a corresponding fourth feature image. Then, the pixel features of each of the obtained third feature images are fused with their corresponding fourth feature images to obtain a corresponding second fusion image. The second fusion image corresponding to the third frontal image, the third left-side image, and the third right-side image is used as the corresponding fourth frontal image, the fourth left-side image, and the fourth right-side image. The pixel features of the fourth frontal image, the fourth left-side image, and the fourth right-side image include the third RGB primary colors, the third depth, the third semantic type, and the first key point type. The pixels in the fourth frontal image whose first key point type is eyebrow point, nasal root point, left infraorbital point, and right infraorbital point are recorded as the corresponding first eyebrow point, first nasal root point, first left infraorbital point, and first right infraorbital point. A first eyebrow-nasal root line is obtained by drawing a straight line between the first eyebrow point and the first nasal root point. A first infraorbital point line is obtained by drawing a straight line between the first left infraorbital point and the first right infraorbital point. The fourth frontal image is rotated and translated with the first eyebrow-nasal root line as the image center line as the target. Then, the fourth frontal image is rotated with the first eyebrow-nasal root line as the rotation axis, with the first left infraorbital point and the first right infraorbital point having the same depth as the target. The corresponding frontal pixel set is formed by the pixels in the fourth frontal image whose third semantic type is not background point. During the rotation or translation of the fourth frontal image, the third depth of each pixel is adaptively adjusted. The fourth left-side image or the fourth right-side image is taken as the corresponding current side image; the pixels on the current side image whose first key point type is eyebrow point, nose tip point, and chin point are recorded as the corresponding second eyebrow point, first nose tip point, and first chin point; a first plane is obtained by constructing a plane based on the second eyebrow point, the first nose tip point, and the first chin point; the current side image is rotated with the goal of keeping the depth of all pixels on the first plane consistent; and the corresponding left-side pixel set or right-side pixel set is formed by the pixels on the rotated current side image whose third semantic type is not background point; wherein, during the rotation of the fourth left-side image and the fourth right-side image, the third depth of each pixel is adaptively adjusted; The front pixel set, the left pixel set, and the right pixel set are respectively processed by point cloud transformation to obtain the corresponding front point cloud, left point cloud, and right point cloud; the point features of the front point cloud, the left point cloud, and the right point cloud include the third RGB three primary colors, the third depth, the third semantic type, and the first key point type; The points in the frontal point cloud corresponding to the glabella, nasal root, left infraorbital point, and right infraorbital point are designated as the corresponding third glabella, second nasal root, second left infraorbital point, and second right infraorbital point. The points in the left-side point cloud corresponding to the glabella, nasal root, and left infraorbital point are designated as the corresponding fourth glabella, third nasal root, and third left infraorbital point. The points in the right-side point cloud corresponding to the glabella, nasal root, and right infraorbital point are designated as the corresponding fifth glabella, fourth nasal root, and third right infraorbital point. The third and fourth glabella points form the first matching point pair, the second and third nasal root points form the second matching point pair, and the second and third left infraorbital points form the third matching point pair. The system first sets up matching point pairs; then, it forms a fourth matching point pair with the third and fifth glabella points, a fifth matching point pair with the second and fourth nasal root points, and a sixth matching point pair with the second and third right infraorbital points. Using the frontal point cloud as the reference point cloud and minimizing the sum of the point spacing of the first, second, and third matching point pairs as the registration target, it performs point cloud registration processing on the left-side point cloud. Similarly, using the frontal point cloud as the reference point cloud and minimizing the sum of the point spacing of the fourth, fifth, and sixth matching point pairs as the registration target, it performs point cloud registration processing on the right-side point cloud. During the registration process for the left-side and right-side point clouds, the three-dimensional coordinates of each point in the current point cloud are adaptively adjusted. The first fused point cloud is obtained by performing point cloud fusion processing on the front point cloud, the left point cloud, and the right point cloud after point cloud registration. A three-dimensional voxel space is constructed based on the point cloud width, height, and depth of the first fused point cloud. A three-dimensional feature map is constructed based on the first fused point cloud and the three-dimensional voxel space to obtain the first face and neck feature map. The voxel point features of the first face and neck feature map include the fourth RGB three primary colors, the fourth semantic type, and the second key point type. The fourth semantic type includes face, neck, and background. The type range of the second key point type consists of the set of face and neck key points and non-key points.

6. The method for predicting the risk of obstructive sleep apnea according to claim 2, characterized in that, The calculation of the face width / eye width ratio, chin protrusion, and chin-neck angle based on the first face and neck feature map to obtain the corresponding first face width / eye width ratio, first chin protrusion, and first chin-neck angle specifically includes: The straight-line distance between two voxel points of the second key point type, namely the left and right tragus points, on the first face and neck feature map is calculated, and the calculation result is used as the corresponding first face width; the straight-line distance between two voxel points of the second key point type, namely the left and right infraorbital points, on the first face and neck feature map is calculated, and the calculation result is used as the corresponding first eye width; and the first face width / eye width ratio is calculated based on the first face width and the first eye width, where the first face width / eye width ratio = first face width : first eye width; The five voxel points on the first facial and neck feature map, namely the glabella point, nasal root point, prechin point, left infraorbital point, and right infraorbital point, are designated as the corresponding glabella, nasal root, prechin, left infraorbital, and right infraorbital voxel points. The plane containing the glabella, nasal root, and prechinal voxel points is designated as the corresponding first sagittal section. The section passing through the left and right infraorbital voxel points and perpendicular to the first sagittal section is designated as the corresponding first transverse section. The section passing through the prechinal voxel point, perpendicular to the first sagittal section, and parallel to the first transverse section is designated as the corresponding second transverse section. The first vertical line is marked as the first vertical line, and the direction towards the inside and outside of the face on the first sagittal section is marked as the back and front of the first vertical line; the second vertical line is marked as the second vertical line, and the line between the first and second vertical lines is marked as the second vertical line, and the distance between the first and second vertical lines is calculated to obtain the corresponding first distance d. When the second vertical line is in front of the first vertical line, the first chin protrusion is set to d, and when the second vertical line is behind the first vertical line, the first chin protrusion is set to -d. The two voxel points on the first face and neck feature map, whose second key point types are subchin point and neck point respectively, are denoted as the corresponding subchin and neck voxel points; the spatial vector from the nasal root voxel point to the anterior chin voxel point is denoted as the corresponding vector A, and the spatial vector from the subchin voxel point to the neck voxel point is denoted as the corresponding vector B; the vector angle θ is calculated based on vectors A and B; and the corresponding first chin-neck angle is calculated based on the vector angle θ. α = 180° - θ + δ, where α is the first chin-neck angle and δ is a preset compensation angle; The first face width / eye width ratio, the first chin protrusion, and the first chin-neck angle obtained in this study will be output as the results of this measurement.

7. The method for predicting the risk of obstructive sleep apnea according to claim 2, characterized in that, The step of using the pharyngeal semantic segmentation model to calculate the length of the soft palate and the three-dimensional dimensions of the uvula in the second oral cavity image to obtain the corresponding length of the first soft palate and the dimensions of the first uvula specifically includes: The second oral cavity image is used as a corresponding second feature image and input into the pharyngeal semantic segmentation model for processing to obtain the corresponding second semantic image; and the second semantic image obtained in this process is subjected to point cloud transformation processing to obtain the corresponding oral cavity internal point cloud; the point features of the oral cavity internal point cloud include the second semantic type; The points in the oral cavity point cloud with the second semantic type being soft palate or uvula are recorded as the corresponding soft palate points or uvula points. The center coordinates of all soft palate points are calculated to obtain the corresponding first center point, and the center coordinates of all uvula points are calculated to obtain the corresponding second center point. A straight line is drawn based on the first center point and the second center point to obtain the corresponding first center line. The soft palate point with the highest height among all soft palate points on the first center line is taken as the soft palate vertex. The two soft palate points with the highest and lowest height among all uvula points on the first center line are taken as the corresponding uvula base center point and uvula tip point. The straight-line distance from the soft palate vertex to the uvula tip point is calculated, and the result is taken as the corresponding first soft palate length. The straight-line distance from the uvula base center point to the uvula tip point is calculated, and the result is taken as the corresponding uvula length. The leftmost and rightmost two uvula points are taken as the corresponding left and right endpoints; the vertical distances from the left and right endpoints to the first center line are calculated to obtain the corresponding left vertical distance and right vertical distance; the left and right vertical distances are summed and the result is taken as the corresponding uvula width. The two closest and farthest uvula points among all the uvula points are taken as the corresponding proximal and distal endpoints; the vertical distances from the proximal and distal endpoints to the first centerline are calculated to obtain the corresponding proximal vertical distance and distal vertical distance; the proximal and distal vertical distances are summed and the calculation result is taken as the corresponding uvula thickness. The first uvula dimension is formed by the obtained uvula length, uvula width, and uvula thickness; The obtained first soft palate length and first uvula size are then output as the results of this measurement.

8. An apparatus for performing the obstructive sleep apnea risk prediction method according to any one of claims 1-7, characterized in that, The device includes: a tool model preparation module, a subject image acquisition module, a face and neck image processing module, a mouth image processing module, an index and risk level prediction module, and a prediction report generation module. The tool model preparation module is used to pre-set four artificial intelligence models and train the four artificial intelligence models respectively. The four artificial intelligence models include a face and neck semantic segmentation model, a throat semantic segmentation model, a face and neck key point recognition model, and an apnea-hypopnea index prediction model. The subject image acquisition module is used to capture images of the subject's front, left and right sides, and inside the oral cavity using the same camera according to preset image capture requirements after the training of the four artificial intelligence models is completed, thereby obtaining the corresponding first front image, first left side image, first right side image, and first oral cavity image; and to perform pixel-level depth feature addition processing on the first front image, first left side image, first right side image, and first oral cavity image to obtain the corresponding second front image, second left side image, second right side image, and second oral cavity image. The face and neck image processing module is used to add face and neck semantic and key point features to the second frontal image, the second left image, and the second right image using the face and neck semantic segmentation model and the face and neck key point recognition model. It then rotates the three images after feature addition, performs point cloud conversion and registration on the rotated images, and constructs a first face and neck feature map based on the registered three-dimensional point cloud. Finally, it calculates the face width / eye width ratio, chin protrusion, and chin-neck angle based on the first face and neck feature map. The oral cavity image processing module is used to calculate the corresponding first soft palate length and first uvula size by using the pharyngeal semantic segmentation model to measure the soft palate length and uvula three-dimensional dimensions in the second oral cavity image; the first uvula size includes uvula length, uvula width and uvula thickness. The index and risk level prediction module is used to form a first feature vector composed of the first face width / eye width ratio, the first chin projection, the first chin-neck angle, the first soft palate length, and the first uvula size; and inputs the first feature vector and the first face-neck feature map into the apnea-hypopnea index prediction model to predict the apnea-hypopnea index and obtain the corresponding first predicted index; and based on the preset index-risk level correspondence, identifies the obstructive sleep apnea risk level according to the first predicted index to obtain the corresponding first risk level; the index-risk level correspondence includes multiple correspondence records; each correspondence record consists of an index value range and a corresponding risk level. The prediction report generation module is used to compose a first prediction report for the current subject, which consists of the first feature vector, the first prediction index, and the first risk level.

9. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-7; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-7.

Images