Multi-view face image processing system

Abstract

The invention provides a multi-view face image processing system, and relates to the technical field of image processing. The system comprises an image acquisition module used for shooting a front side face image, a first mouth image and a second mouth image; the preprocessing module is used for performing normalization and illumination enhancement on the multi-view face image; the feature quantitative calculation module is used for generating geometric features such as mandibular bones, tongue space ratio, neck-to-face ratio, facial geometric contour and eyelid texture based on the processed front and side face and first mouth image; the deep learning reasoning module is used for generating semantic features such as hard palate form probability distribution, tonsil grading, occlusal relationship and nose wing form based on the processed side face image, the first mouth image and the second mouth image; and the multi-modal feature fusion module is used for carrying out weighted fusion and nonlinear mapping on the geometric and semantic features and the physiological parameters of the user to obtain a fusion feature map. According to the invention, sleep apnea prediction is carried out directly based on the fused feature map, and the prediction efficiency and prediction precision of sleep apnea are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a multi-view facial image processing system. Background Technology

[0002] Currently, sleep apnea prediction is typically made directly based on multi-angle facial images of users. However, this process relies heavily on manual observation, labeling, and integration of discrete morphological cues, which is not only time-consuming and labor-intensive but also lacks a standardized quantitative prediction system, resulting in low prediction efficiency and insufficient consistency. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a multi-view facial image processing system.

[0004] This invention provides a multi-view facial image processing system, which includes an image acquisition module, a preprocessing module, a feature quantization calculation module, a deep learning inference module, and a multimodal feature fusion module; wherein: The image acquisition module is used to guide the user to capture multi-view facial images according to preset instructions. The multi-view facial images include a frontal face image, a side face image, a first mouth image, and a second mouth image. The preprocessing module is used to preprocess the multi-view facial images, wherein the preprocessing includes normalization processing and illumination enhancement processing. The feature quantization calculation module is used for: Based on the preprocessed profile image, mandibular bone features are generated; Based on the preprocessed first mouth image, tongue body spatial proportion features are generated; The neck region is segmented from the preprocessed frontal face image, and the ratio of the narrowest pixel width of the neck to the widest pixel width of the cheekbones is calculated to obtain the neck-face ratio feature. Based on the preprocessed side profile image and the preprocessed frontal profile image, the aspect ratio of the vertical height and horizontal width of the face is calculated, and the lip closure gap in a natural state is obtained to obtain the facial geometric contour features. The region below the eyelids is cropped from the preprocessed frontal image, and eyelid texture features are extracted from the region below the eyelids using local binary mode. The deep learning inference module is used for: The hard palate region is extracted from the preprocessed first mouth image, and the hard palate morphological probability distribution feature vector is extracted from the hard palate region through a first convolutional neural network. The pharyngeal isthmus region is extracted from the preprocessed first mouth image, and the tonsil hierarchical feature vector is extracted from the pharyngeal isthmus region through a classification network with an attention mechanism. The labial and dental region is extracted from the preprocessed second mouth image, and the occlusal relationship feature vector is extracted from the labial and dental region using a visual classification model; The nasal alar region is cropped from the preprocessed side profile image, and the nasal alar morphology feature vector is extracted from the nasal alar region using a second convolutional neural network; The multimodal feature fusion module is used to concatenate the mandibular bone features, tongue spatial proportion features, neck-face ratio features, facial geometric contour features, eyelid texture features, hard palate morphological probability distribution feature vector, tonsil grading feature vector, occlusal relationship feature vector, nasal wing morphological feature vector, and user physiological parameters to obtain a concatenated vector. The concatenated vector is then input into a multimodal fusion network for weighted fusion and nonlinear mapping to obtain a fused feature map.

[0005] Optionally, the feature quantization calculation module is specifically used to generate mandibular bone features based on the preprocessed profile image in the following manner: Key points of the mandible region are extracted from a preprocessed profile image using a deep generative model. These key points include the tragus point, the angle of the mandible, the premental point, and the lower lip protrusion. Based on the key points of the mandibular region, the mandibular plane angle and the depth of the mentolabial sulcus are calculated as mandibular skeletal features.

[0006] Optionally, the first mouth image is an image of the open mouth and pharynx; The feature quantization calculation module is specifically used to generate tongue spatial proportion features based on the preprocessed first mouth image in the following manner: A semantic segmentation network is used to extract the oral cavity and tongue regions from the preprocessed first mouth image; The ratio of the number of pixels in the tongue region to the number of pixels in the visible area of ​​the oral cavity is calculated as the spatial proportion feature of the tongue.

[0007] Optionally, the feature quantization calculation module is specifically used to calculate the aspect ratio of the vertical height to the horizontal width of the face based on the preprocessed side profile image and the preprocessed frontal face image, and to obtain the lip closure gap in a natural state, thereby obtaining the facial geometric contour features: The nasal root point, submental point, left cheekbone point, right cheekbone point, upper lip edge point, and lower lip edge point are extracted from the preprocessed side profile image and the preprocessed frontal profile image. The absolute value of the difference between the ordinate of the subchinus point and the ordinate of the nasal root point is determined as the face height, and the absolute value of the difference between the abscissa of the left cheekbone point and the abscissa of the right cheekbone point is determined as the face width. The ratio of the face height to the face width is determined as the aspect ratio of the vertical height to the horizontal width of the face. Calculate the distance between the upper lip edge point and the lower lip edge point, and use it as the lip closure gap in the natural state; The aspect ratio of the vertical height to the horizontal width of the face and the lip closure gap in the natural state are used as facial geometric contour features.

[0008] Optionally, the feature quantization calculation module is specifically used to extract the region below the eyelids from the preprocessed frontal face image in the following manner, and extract eyelid texture features from the region below the eyelids using a local binary mode: Extract key points of the lower eyelid from the preprocessed frontal image; Extend a predetermined number of pixels downwards from the key points of the lower eyelid, and extract a rectangular strip area covering the tear trough and the infraorbital vein area as the area below the eyelid; Calculate the local binary pattern texture histogram of the region below the eyelid to quantify the degree of pigmentation and vascular congestion in dark circles, and use it as an eyelid texture feature.

[0009] Optionally, the first mouth image is an image of the open mouth and pharynx; The deep learning inference module is specifically used to extract the hard palate region from the preprocessed first mouth image in the following manner, and to extract the hard palate morphological probability distribution feature vector from the hard palate region using a first convolutional neural network: Extract the set of key points at the lower edge of the upper lip and the set of key points at the base of the nose from the preprocessed first mouth image; Calculate the geometric center of the key points in the set of key points at the lower edge of the upper lip and the set of key points at the base of the nose, and construct a rectangular frame with the geometric center to obtain the hard palate region. The width of the rectangular frame is a first preset multiple of the distance between the left and right corners of the mouth, and the height of the rectangular frame is a second preset multiple of the distance from the base of the nose to the lower edge of the upper lip. The first preset multiple is less than 1, and the second preset multiple is greater than 1. Adjust the size of the hard palate region; The resized hard palate region is input into the first convolutional neural network to obtain the hard palate morphological probability distribution feature vector.

[0010] Optionally, the first mouth image is an image of the open mouth and pharynx; The deep learning inference module is specifically used to extract the pharyngeal isthmus region from the preprocessed first mouth image in the following manner, and extract tonsil hierarchical feature vectors from the pharyngeal isthmus region through a classification network with an attention mechanism: Extract the key points of the intraoral contour from the preprocessed first mouth image; Based on the key points of the oral cavity contour, calculate the minimum bounding rectangle of the oral cavity contour; Keeping the center of the minimum bounding rectangle unchanged, expand its length and width outwards by a preset proportion to obtain the pharyngeal region; The pharyngeal region is input into a classification network with spatial attention and channel attention mechanisms. The feature map before the global average pooling layer is extracted and flattened to obtain the tonsil hierarchical feature vector. The spatial attention mechanism is used to guide the classification network to suppress the weights of the tongue region and focus on the tonsil region on both sides and deep within the image. The channel attention mechanism is used to enhance the feature response to red congestion and tissue protrusion texture.

[0011] Optionally, the second mouth image is a closed bite image; The visual classification model includes a spatial transformation network and a third convolutional neural network; The deep learning inference module is specifically used to extract the labial and dental region from the preprocessed second mouth image in the following manner, and extract the occlusal relationship feature vector from the labial and dental region through a visual classification model: Extract key points of the left and right corners of the mouth and key points of the outer edges of the upper and lower lips from the preprocessed second mouth image; Using the line connecting the key points of the left and right corners of the mouth as the horizontal axis, a rectangular area containing the contact surface of the upper and lower lips and teeth is extracted from the preprocessed second mouth image based on the key points of the outer edges of the upper and lower lips. The rectangular region is corrected using the spatial transformation network to ensure that the occlusal plane is horizontally aligned. The corrected rectangular region is input into the third convolutional neural network to obtain the bite relationship feature vector.

[0012] Optionally, the deep learning inference module is specifically used to extract the nasal alar region from the preprocessed profile image in the following manner, and extract the nasal alar morphological feature vector from the nasal alar region using a second convolutional neural network: In the preprocessed profile image, a square region is constructed with the line connecting the tip of the nose and the base of the nasal ala as the diagonal, covering the outer edge of the nasal ala to obtain the nasal ala region; The nasal alar region is input into a second convolutional neural network to extract nasal alar morphological feature vectors.

[0013] Optionally, the first mouth image is an image of the open mouth and pharynx; The preprocessing module is specifically used to perform illumination enhancement processing on the profile image and the first mouth image in the following manner: The profile image is converted from RGB space to HSV space while keeping the hue and saturation components unchanged. The luminance component is subjected to multi-scale convolution using a Gaussian wrap function to estimate the first illumination component and then remove the first illumination component to obtain the first intermediate image. The first intermediate image is then converted from HSV space to RGB space. The first mouth image is converted from the RGB space to the HSV space, keeping the hue and saturation components unchanged. The luminance component is subjected to multi-scale convolution using a Gaussian wrap function. The second illumination component is estimated and then removed to obtain the second intermediate image. The second intermediate image is then converted from the HSV space to the RGB space.

[0014] In summary, the solution proposed in this invention has the following technical effects: The multi-view facial image processing system acquires facial images from different perspectives, including frontal, lateral, and both mouth views, and calculates explicit geometric features such as the mandible, tongue proportion, neck-to-face ratio, facial aspect ratio, facial geometric contour features, and eyelid texture features. Simultaneously, it uses a deep convolutional neural network to encode features of the hard palate morphology, tonsil grading, occlusal relationships, and nasal alar morphology, extracting implicit semantic features. Finally, the geometric features, implicit semantic features, and user physiological parameters are input into a multimodal fusion network for weighted fusion and nonlinear mapping to obtain a fused feature map. This allows for direct prediction of sleep apnea based on this fused feature map, eliminating the time required for manual observation, labeling, and integration of discrete morphological cues, significantly improving the prediction efficiency and consistency of sleep apnea, and reducing human error and experience barriers. Furthermore, multimodal feature fusion significantly improves the prediction accuracy of sleep apnea, overcoming the limitation of a single perspective in simultaneously considering both maxillofacial bones (lateral) and soft tissue texture (frontal / open mouth). Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a block diagram of a multi-view facial image processing system according to an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This invention discloses a multi-view facial image processing system. Figure 1 This is a block diagram of a multi-view facial image processing system according to an embodiment of the present invention, such as... Figure 1 As shown, the multi-view facial image processing system 10 includes: Image acquisition module 100, preprocessing module 200, feature quantization calculation module 300, deep learning inference module 400, and multimodal feature fusion module 500; The image acquisition module 100 is used to guide the user to capture multi-view facial images according to preset instructions. The multi-view facial images include a frontal face image, a side face image, a first mouth image, and a second mouth image. The preprocessing module 200 is used to preprocess the multi-view facial images, wherein the preprocessing includes normalization processing and illumination enhancement processing. The feature quantization calculation module 300 is used for: Based on the preprocessed profile image, mandibular bone features are generated; Based on the preprocessed first mouth image, tongue body spatial proportion features are generated; The neck region is segmented from the preprocessed frontal face image, and the ratio of the narrowest pixel width of the neck to the widest pixel width of the cheekbones is calculated to obtain the neck-face ratio feature. Based on the preprocessed side profile image and the preprocessed frontal profile image, the aspect ratio of the vertical height and horizontal width of the face is calculated, and the lip closure gap in a natural state is obtained to obtain the facial geometric contour features. The region below the eyelids is cropped from the preprocessed frontal image, and eyelid texture features are extracted from the region below the eyelids using local binary mode. The deep learning inference module 400 is used for: The hard palate region is extracted from the preprocessed first mouth image, and the hard palate morphological probability distribution feature vector is extracted from the hard palate region through a first convolutional neural network. The pharyngeal isthmus region is extracted from the preprocessed first mouth image, and the tonsil hierarchical feature vector is extracted from the pharyngeal isthmus region through a classification network with an attention mechanism. The labial and dental region is extracted from the preprocessed second mouth image, and the occlusal relationship feature vector is extracted from the labial and dental region using a visual classification model; The nasal alar region is cropped from the preprocessed side profile image, and the nasal alar morphology feature vector is extracted from the nasal alar region using a second convolutional neural network; The multimodal feature fusion module 500 is used to concatenate the mandibular bone features, the tongue body spatial proportion features, the neck-face ratio features, the facial geometric contour features, the eyelid texture features, the hard palate morphological probability distribution feature vector, the tonsil grading feature vector, the occlusal relationship feature vector, the nasal wing morphological feature vector, and user physiological parameters to obtain a concatenated vector. The concatenated vector is then input into a multimodal fusion network for weighted fusion and nonlinear mapping to obtain a fused feature map.

[0019] Specifically, the user's physiological parameters include body mass index, age, gender, and other information. The multi-view facial image processing system 10 can be deployed on a smartphone, tablet, or computer terminal equipped with a camera. The image acquisition module 100 can guide the user through voice or a front-end interface, prompting them to take four facial images in sequence: a frontal image, a side profile image, a first mouth image, and a second mouth image.

[0020] The front-facing image should be a natural, frontal image, requiring the user to look directly at the camera with a relaxed and natural expression, showing their neck.

[0021] The profile image is a side profile image, requiring the user to turn their head 90 degrees (to the left or right) to fully expose the tragus, jaw angle, and nose tip contour.

[0022] The first image of the mouth is an open-mouth pharyngeal cavity image. Users are required to open their mouths as wide as possible in a well-lit area and lower their tongues as much as possible to expose the uvula and pharyngeal isthmus.

[0023] The second mouth image is a closed bite image, requiring the user to bite down on their upper and lower teeth and use their fingers to help open the corners of their mouth to expose the bite relationship.

[0024] After data acquisition, the key points of the four images, including the left eye, were extracted using a multi-task cascaded convolutional neural network (MTCNN). right eye ,tip of the nose Left corner of the mouth Right corner of the mouth Calculate the facial deflection angle ,in, and These represent the coordinates of the left and right eye centers, respectively. Scaling factor. , It is the preset standard binocular pixel distance (e.g., fixed at 50 pixels); scaling factor The denominator is the measured binocular distance calculated using the Euclidean distance formula, and the scaling factor is... This determines whether the image is enlarged or reduced. Finally, it is integrated through affine transformation ( (x, y) are the pixel coordinates of the original captured facial image. These are the new pixel coordinates after correction and normalization. It is a translation vector used to ensure that the rotated and scaled face can be accurately aligned to the center area of ​​the output canvas. It realizes rotation correction and normalization, eliminating problems such as physical head tilt and inconsistent shooting distance caused by improper shooting by the user, and thus achieving face standardization.

[0025] In other embodiments, the multi-scale Retinex color restoration algorithm (MSRCR) can be used to enhance the illumination of the profile image and the first mouth image. Specifically, the preprocessing module is used to enhance the illumination of the profile image and the first mouth image in the following manner: The profile image is converted from RGB space to HSV space, keeping the hue component (H) and saturation component (S) unchanged, and the luminance component (V) is subjected to multi-scale convolution using a Gaussian wrap function to estimate the first illumination component and remove the first illumination component to obtain the first intermediate image; the first intermediate image is then converted from HSV space to RGB space. The first mouth image is converted from the RGB space to the HSV space, keeping the hue component (H) and saturation component (S) unchanged, and the luminance component (V) is subjected to multi-scale convolution using a Gaussian wrap function. The second illumination component is estimated and removed to obtain the second intermediate image; the second intermediate image is then converted from the HSV space to the RGB space.

[0026] In this invention, the feature quantization calculation module 300 and the deep learning inference module 400 are used to extract dual-stream features. The extraction of dual-stream features adopts a parallel architecture of "explicit geometric computation flow" and "implicit semantic awareness flow," with the feature quantization calculation module 300 extracting the explicit geometric computation flow and the deep learning inference module 400 extracting the implicit semantic awareness flow.

[0027] In other embodiments, the feature quantization calculation module is specifically used to generate mandibular bone features based on the preprocessed profile image in the following manner: Key points of the mandible region are extracted from a preprocessed profile image using a deep generative model. These key points include the tragus point, the angle of the mandible, the premental point, and the lower lip protrusion. Based on the key points of the mandibular region, the mandibular plane angle and the depth of the mentolabial sulcus are calculated as mandibular skeletal features.

[0028] Specifically, in the preprocessed profile image, the PRNet deep generative model is used to extract key points in the jaw region, including the tragus point. mandibular angle point Prechin and lower lip protrusion .

[0029] Calculate the mandibular plane angle (MPA): Connect the tragus point With the angle of the mandible This forms the tangent to the mandibular ascending ramus and connects the mandibular angle point. With the anterior point of the chin The mandibular ramus tangent is formed, and the angle between the mandibular ramus tangent and the mandibular body tangent is calculated. The slope of the tangent of the mandibular ramus and the slope of the tangent to the mandibular body included angle The larger the angle (e.g., >28°), the higher the risk of high-angle facial features and airway stenosis. For the tragus coordinates Mandibular angle point coordinates Prechin The coordinates.

[0030] Calculate the depth of the mentolabial sulcus: Calculate the lower lip protrusion. With the anterior point of the chin The shortest Euclidean distance from the midpoint of the line to the soft tissue contour , Lower lip protrusion With the anterior point of the chin Midpoint of the line , Lower lip protrusion coordinates Let be the coordinates of any point on the soft tissue contour line. If the value is too small, for example, less than a preset threshold, it indicates that the soft tissue is too flat or has abnormal tension. The shortest Euclidean distance is found by traversing every point (x1, y1) on the soft tissue contour line at the mentolabial sulcus.

[0031] In other embodiments, the first mouth image is an image of an open-mouthed pharynx; The feature quantization calculation module is specifically used to generate tongue spatial proportion features based on the preprocessed first mouth image in the following manner: A semantic segmentation network is used to extract the oral cavity and tongue regions from the preprocessed first mouth image; The ratio of the number of pixels in the tongue region to the number of pixels in the visible area of ​​the oral cavity is calculated as the spatial proportion feature of the tongue.

[0032] Specifically, the semantic segmentation network can be a lightweight U-Net network. The encoder uses MobileNetV2 for downsampling to extract features, and the decoder uses upsampling to restore resolution. Segmentation task: The semantic segmentation network outputs a three-channel mask, including the background, the visible oral cavity region (including the uvula, tonsils, and tongue), and the tongue region.

[0033] In other embodiments, the feature quantization calculation module is specifically used to calculate the aspect ratio of the vertical height to the horizontal width of the face based on the preprocessed side profile image and the preprocessed frontal face image, and to obtain the lip closure gap in a natural state, thereby obtaining the facial geometric contour features: The nasal root point, submental point, left cheekbone point, right cheekbone point, upper lip edge point, and lower lip edge point are extracted from the preprocessed side profile image and the preprocessed frontal profile image. The absolute value of the difference between the ordinate of the subchinus point and the ordinate of the nasal root point is determined as the face height, and the absolute value of the difference between the abscissa of the left cheekbone point and the abscissa of the right cheekbone point is determined as the face width. The ratio of the face height to the face width is determined as the aspect ratio of the vertical height to the horizontal width of the face. Calculate the distance between the upper lip edge point and the lower lip edge point, and use it as the lip closure gap in the natural state; The aspect ratio of the vertical height to the horizontal width of the face and the lip closure gap in the natural state are used as facial geometric contour features.

[0034] In other embodiments, the feature quantization calculation module is specifically used to extract the region below the eyelids from the preprocessed frontal face image in the following manner, and extract eyelid texture features from the region below the eyelids using a local binary mode: Extract key points of the lower eyelid from the preprocessed frontal image; A rectangular strip area covering the tear trough and the infraorbital vein area is extracted from the lower eyelid key point by extending a predetermined number (e.g., 20) pixels downwards; this area is designated as the area below the eyelid. Calculate the local binary pattern texture histogram of the region below the eyelid to quantify the degree of pigmentation and vascular congestion of dark circles (as a physiological marker of long-term hypoxia) and use it as an eyelid texture feature.

[0035] In other embodiments, the first mouth image is an image of an open-mouthed pharynx; The deep learning inference module is specifically used to extract the hard palate region from the preprocessed first mouth image in the following manner, and to extract the hard palate morphological probability distribution feature vector from the hard palate region using a first convolutional neural network: Extract the set of key points at the lower edge of the upper lip and the set of key points at the base of the nose from the preprocessed first mouth image; Calculate the geometric center of the key points in the set of key points at the lower edge of the upper lip and the set of key points at the base of the nose. Construct a rectangular frame with the geometric center to obtain the hard palate region. The width of the rectangular frame is a first preset multiple of the distance between the left and right corners of the mouth, and the height of the rectangular frame is a second preset multiple of the distance from the base of the nose to the lower edge of the upper lip. The first preset multiple is less than 1, for example, 0.8; the second preset multiple is greater than 1, for example, 1.2. Adjust the size of the hard palate region; The resized hard palate region is input into the first convolutional neural network to obtain the hard palate morphological probability distribution feature vector.

[0036] The first convolutional neural network can be a ResNet-18 backbone network with fully connected layers removed from the branches, extracting only the semantic feature vector from the output of the last convolutional layer, specifically the hard palate morphological probability distribution feature vector. This first convolutional neural network is pre-trained and focuses on recognizing the "high arch" and "narrow" curvature features of the hard palate.

[0037] In other embodiments, the first mouth image is an image of an open-mouthed pharynx; The deep learning inference module is specifically used to extract the pharyngeal isthmus region (i.e., the region enclosed by the inner edges of the upper and lower lips) from the preprocessed first mouth image in the following manner, and to extract tonsil hierarchical feature vectors from the pharyngeal isthmus region through a classification network with an attention mechanism: Extract the key points of the intraoral contour from the preprocessed first mouth image; Based on the key points of the oral cavity contour, calculate the minimum bounding rectangle of the oral cavity contour; Keeping the center of the minimum circumscribed rectangle unchanged, expand its length and width outwards by a predetermined proportion (e.g., 10%) to obtain the pharyngeal isthmus region, which accommodates the pharyngeal lateral walls that may shift due to the shooting angle. The pharyngeal isthmus region completely includes the uvula, pharyngeal isthmus, and bilateral tonsillar fossae.

[0038] The pharyngeal region is input into a classification network with spatial and channel attention mechanisms. The feature map before the global average pooling layer is extracted and flattened to obtain the tonsil grading feature vector. The spatial attention mechanism guides the classification network to suppress the weights of the tongue region (usually located at the bottom of the image and with different textures), focusing instead on the tonsil regions on both sides and deep within the image. The channel attention mechanism enhances the feature response to red congestion and tissue protrusion textures. The tonsil grading feature vector implicitly contains information on the tonsil hypertrophy grade (I-IV).

[0039] In other embodiments, the second mouth image is a closed bite image; The visual classification model includes a spatial transformation network and a third convolutional neural network; The deep learning inference module is specifically used to extract the labial and dental region from the preprocessed second mouth image in the following manner, and extract the occlusal relationship feature vector from the labial and dental region through a visual classification model: Extract key points of the left and right corners of the mouth and key points of the outer edges of the upper and lower lips from the preprocessed second mouth image; Using the line connecting the key points of the left and right corners of the mouth as the horizontal axis, a rectangular area containing the contact surface of the upper and lower lips and teeth is extracted from the preprocessed second mouth image based on the key points of the outer edge of the upper and lower lips. The rectangular region is corrected using the spatial transformation network to ensure that the occlusal plane is horizontally aligned. The corrected rectangular region is input into the third convolutional neural network to obtain the bite relationship feature vector.

[0040] Specifically, to prevent misjudgments caused by phone tilt during shooting, the aforementioned rectangular cropping area must vertically include the region between the tip of the nose and the midpoint of the chin, preserving sufficient contextual information for subsequent correction. The spatial transformation network contains a regression subnetwork that predicts the affine transformation matrix, automatically correcting tilted tooth images to a horizontally aligned state, eliminating interference from the user tilting their head during shooting. Feature extraction: The corrected feature map (i.e., the corrected rectangular region) is input into the third convolutional neural network to extract morphological features of deep overbite or open bite, obtaining an occlusal relationship feature vector.

[0041] In other embodiments, the deep learning inference module is specifically used to extract the nasal alar region from the preprocessed profile image in the following manner, and to extract the nasal alar morphological feature vector from the nasal alar region using a second convolutional neural network: In the preprocessed profile image, a square region is constructed with the line connecting the tip of the nose and the base of the nasal ala as the diagonal, covering the outer edge of the nasal ala to obtain the nasal ala region; The nasal alar region is input into a second convolutional neural network to extract nasal alar morphological feature vectors.

[0042] In summary, the solution proposed in this invention has the following technical effects: The multi-view facial image processing system acquires facial images from different perspectives, including frontal, lateral, and both mouth views, and calculates explicit geometric features such as the mandible, tongue proportion, neck-to-face ratio, facial aspect ratio, facial geometric contour features, and eyelid texture features. Simultaneously, it uses a deep convolutional neural network to encode features of the hard palate morphology, tonsil grading, occlusal relationships, and nasal alar morphology, extracting implicit semantic features. Finally, the geometric features, implicit semantic features, and user physiological parameters are input into a multimodal fusion network for weighted fusion and nonlinear mapping to obtain a fused feature map. This allows for direct prediction of sleep apnea based on this fused feature map, eliminating the time required for manual observation, labeling, and integration of discrete morphological cues, significantly improving the prediction efficiency and consistency of sleep apnea, and reducing human error and experience barriers. Furthermore, multimodal feature fusion significantly improves the prediction accuracy of sleep apnea, overcoming the limitation of a single perspective in simultaneously considering both maxillofacial bones (lateral) and soft tissue texture (frontal / open mouth).

[0043] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-view facial image processing system, characterized in that, The multi-view facial image processing system includes an image acquisition module, a preprocessing module, a feature quantization calculation module, a deep learning inference module, and a multimodal feature fusion module; wherein: The image acquisition module is used to guide the user to capture multi-view facial images according to preset instructions. The multi-view facial images include a frontal face image, a side face image, a first mouth image, and a second mouth image. The preprocessing module is used to preprocess the multi-view facial images, wherein the preprocessing includes normalization processing and illumination enhancement processing. The feature quantization calculation module is used for: Based on the preprocessed profile image, mandibular bone features are generated; Based on the preprocessed first mouth image, tongue body spatial proportion features are generated; The neck region is segmented from the preprocessed frontal face image, and the ratio of the narrowest pixel width of the neck to the widest pixel width of the cheekbones is calculated to obtain the neck-face ratio feature. Based on the preprocessed side profile image and the preprocessed frontal profile image, the aspect ratio of the vertical height and horizontal width of the face is calculated, and the lip closure gap in a natural state is obtained to obtain the facial geometric contour features. The region below the eyelids is cropped from the preprocessed frontal image, and eyelid texture features are extracted from the region below the eyelids using local binary mode. The deep learning inference module is used for: The hard palate region is extracted from the preprocessed first mouth image, and the hard palate morphological probability distribution feature vector is extracted from the hard palate region through a first convolutional neural network. The pharyngeal isthmus region is extracted from the preprocessed first mouth image, and the tonsil hierarchical feature vector is extracted from the pharyngeal isthmus region through a classification network with an attention mechanism. The labial and dental region is extracted from the preprocessed second mouth image, and the occlusal relationship feature vector is extracted from the labial and dental region using a visual classification model; The nasal alar region is cropped from the preprocessed side profile image, and the nasal alar morphology feature vector is extracted from the nasal alar region using a second convolutional neural network; The multimodal feature fusion module is used to concatenate the mandibular bone features, tongue spatial proportion features, neck-face ratio features, facial geometric contour features, eyelid texture features, hard palate morphological probability distribution feature vector, tonsil grading feature vector, occlusal relationship feature vector, nasal wing morphological feature vector, and user physiological parameters to obtain a concatenated vector. The concatenated vector is then input into a multimodal fusion network for weighted fusion and nonlinear mapping to obtain a fused feature map.

2. The multi-view facial image processing system according to claim 1, characterized in that, The feature quantization calculation module is specifically used to generate mandibular bone features based on the preprocessed profile image in the following manner: Key points of the mandible region are extracted from a preprocessed profile image using a deep generative model. These key points include the tragus point, the angle of the mandible, the premental point, and the lower lip protrusion. Based on the key points of the mandibular region, the mandibular plane angle and the depth of the mentolabial sulcus are calculated as mandibular skeletal features.

3. The multi-view facial image processing system according to claim 1, characterized in that, The first mouth image is an image of the open-mouth pharynx; The feature quantization calculation module is specifically used to generate tongue spatial proportion features based on the preprocessed first mouth image in the following manner: A semantic segmentation network is used to extract the oral cavity and tongue regions from the preprocessed first mouth image; The ratio of the number of pixels in the tongue region to the number of pixels in the visible area of ​​the oral cavity is calculated as the spatial proportion feature of the tongue.

4. The multi-view facial image processing system according to claim 1, characterized in that, The feature quantization calculation module is specifically used to calculate the aspect ratio of the vertical height and horizontal width of the face based on the preprocessed side face image and the preprocessed front face image, and to obtain the lip closure gap in a natural state, thereby obtaining the facial geometric contour features: The nasal root point, submental point, left cheekbone point, right cheekbone point, upper lip edge point, and lower lip edge point are extracted from the preprocessed side profile image and the preprocessed frontal profile image. The absolute value of the difference between the ordinate of the subchinus point and the ordinate of the nasal root point is determined as the face height, and the absolute value of the difference between the abscissa of the left cheekbone point and the abscissa of the right cheekbone point is determined as the face width. The ratio of the face height to the face width is determined as the aspect ratio of the vertical height to the horizontal width of the face. Calculate the distance between the upper lip edge point and the lower lip edge point, and use it as the lip closure gap in the natural state; The aspect ratio of the vertical height to the horizontal width of the face and the lip closure gap in the natural state are used as facial geometric contour features.

5. The multi-view facial image processing system according to claim 1, characterized in that, The feature quantization calculation module is specifically used to extract the region below the eyelids from the preprocessed frontal face image in the following way, and extract eyelid texture features from the region below the eyelids using local binary mode: Extract key points of the lower eyelid from the preprocessed frontal image; Extend a predetermined number of pixels downwards from the key points of the lower eyelid, and extract a rectangular strip area covering the tear trough and the infraorbital vein area as the area below the eyelid; Calculate the local binary pattern texture histogram of the region below the eyelid to quantify the degree of pigmentation and vascular congestion in dark circles, and use it as an eyelid texture feature.

6. The multi-view facial image processing system according to claim 1, characterized in that, The first mouth image is an image of the open-mouth pharynx; The deep learning inference module is specifically used to extract the hard palate region from the preprocessed first mouth image in the following manner, and to extract the hard palate morphological probability distribution feature vector from the hard palate region using a first convolutional neural network: Extract the set of key points at the lower edge of the upper lip and the set of key points at the base of the nose from the preprocessed first mouth image; Calculate the geometric center of the key points in the set of key points at the lower edge of the upper lip and the set of key points at the base of the nose, and construct a rectangular frame with the geometric center to obtain the hard palate region. The width of the rectangular frame is a first preset multiple of the distance between the left and right corners of the mouth, and the height of the rectangular frame is a second preset multiple of the distance from the base of the nose to the lower edge of the upper lip. The first preset multiple is less than 1, and the second preset multiple is greater than 1. Adjust the size of the hard palate region; The resized hard palate region is input into the first convolutional neural network to obtain the hard palate morphological probability distribution feature vector.

7. The multi-view facial image processing system according to claim 1, characterized in that, The first mouth image is an image of the open-mouth pharynx; The deep learning inference module is specifically used to extract the pharyngeal isthmus region from the preprocessed first mouth image in the following manner, and to extract tonsil hierarchical feature vectors from the pharyngeal isthmus region through a classification network with an attention mechanism: Extract the key points of the intraoral contour from the preprocessed first mouth image; Based on the key points of the oral cavity contour, calculate the minimum bounding rectangle of the oral cavity contour; Keeping the center of the minimum bounding rectangle unchanged, expand its length and width outwards by a preset proportion to obtain the pharyngeal region; The pharyngeal region is input into a classification network with spatial attention and channel attention mechanisms. The feature map before the global average pooling layer is extracted and flattened to obtain the tonsil hierarchical feature vector. The spatial attention mechanism is used to guide the classification network to suppress the weights of the tongue region and focus on the tonsil region on both sides and deep within the image. The channel attention mechanism is used to enhance the feature response to red congestion and tissue protrusion texture.

8. The multi-view facial image processing system according to claim 1, characterized in that, The second mouth image is a closed bite image; The visual classification model includes a spatial transformation network and a third convolutional neural network; The deep learning inference module is specifically used to extract the labial and dental region from the preprocessed second mouth image in the following manner, and extract the occlusal relationship feature vector from the labial and dental region through a visual classification model: Extract key points of the left and right corners of the mouth and key points of the outer edges of the upper and lower lips from the preprocessed second mouth image; Using the line connecting the key points of the left and right corners of the mouth as the horizontal axis, a rectangular area containing the contact surface of the upper and lower lips and teeth is extracted from the preprocessed second mouth image based on the key points of the outer edge of the upper and lower lips. The rectangular region is corrected using the spatial transformation network to ensure that the occlusal plane is horizontally aligned. The corrected rectangular region is input into the third convolutional neural network to obtain the bite relationship feature vector.

9. The multi-view facial image processing system according to claim 1, characterized in that, The deep learning inference module is specifically used to extract the nasal alar region from the preprocessed profile image in the following manner, and to extract the nasal alar morphological feature vector from the nasal alar region using a second convolutional neural network: In the preprocessed profile image, a square region is constructed with the line connecting the tip of the nose and the base of the nasal ala as the diagonal, covering the outer edge of the nasal ala to obtain the nasal ala region; The nasal alar region is input into a second convolutional neural network to extract nasal alar morphological feature vectors.

10. The multi-view facial image processing system according to claim 1, characterized in that, The first mouth image is an image of the open-mouth pharynx; The preprocessing module is specifically used to perform illumination enhancement processing on the profile image and the first mouth image in the following manner: The profile image is converted from RGB space to HSV space while keeping the hue and saturation components unchanged. The luminance component is subjected to multi-scale convolution using a Gaussian wrap function to estimate the first illumination component and then remove the first illumination component to obtain the first intermediate image. The first intermediate image is then converted from HSV space to RGB space. The first mouth image is converted from the RGB space to the HSV space, keeping the hue and saturation components unchanged. The luminance component is subjected to multi-scale convolution using a Gaussian wrap function. The second illumination component is estimated and then removed to obtain the second intermediate image. The second intermediate image is then converted from the HSV space to the RGB space.

Images