Method, apparatus, terminal device, and readable storage medium for virtually wearing a mask

The method and apparatus address the issue of improper mask selection by using facial image analysis and virtual reality to ensure accurate fitting of masks for non-invasive ventilation, enhancing comfort and effectiveness by simulating the fitting process.

JP2024543707AInactive Publication Date: 2024-11-21BMC MEDICAL CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2024536987
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-31
Filing Date
2022-12-08
Publication Date
2024-11-21
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Conventional methods for selecting masks for non-invasive ventilation treatments, such as those for obstructive sleep apnea syndrome and chronic obstructive emphysema, often result in errors due to improper sizing, leading to air leakage and reduced therapeutic effectiveness, as they rely on subjective judgments based on photos and measurements rather than accurate facial feature analysis.

Method used

A method and apparatus that utilizes facial image acquisition, processing, and virtual reality to determine and display suitable mask models based on actual facial features, allowing users to select and simulate mask fitting virtually, ensuring accurate mask selection through real-time facial image analysis and virtual reality simulation.

Benefits of technology

Enables precise matching of masks to individual facial features, improving comfort and therapeutic efficacy by reducing air leakage and enhancing the user's ability to select the appropriate mask type and size accurately.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543707000001_ABST
    Figure 2024543707000001_ABST
Patent Text Reader

Abstract

A method, an apparatus, a terminal device, and a readable storage medium for virtually wearing a mask are provided. The method includes the steps of: acquiring a face image of a user (101); determining actual facial feature data based on the face image (102); determining and displaying one or more first masks of various matched models based on the actual facial feature data (103); determining a target mask from each of the first masks according to a first input received by the user (104); and generating and displaying wearing screen information based on the face image and the target mask (105). Not only can multiple first masks of various types and sizes suitable for the user's facial features be accurately matched based on the user's facial feature data, but also generating and displaying wearing screen information using a technology such as virtual reality to simulate the effect of the user wearing the target mask, so that the user can have a sense of realism as if he or she were actually in the place, and it is easy for the user to select a mask that satisfies him or her.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present application relates to the technical field of medical equipment, and in particular to a method, apparatus, terminal device, and readable storage medium for virtually wearing a mask.

[0002] <CROSS REFERENCE TO RELATED APPLICATIONS> This application claims priority to a Chinese patent application filed with the China Patent Office on December 31, 2021, bearing application number 202111678051.6 and titled "Method, apparatus, terminal device, and readable storage medium for virtually wearing a mask," the entire contents of which are incorporated herein by reference. [Background technology]

[0003] In recent years, noninvasive positive pressure ventilation has been widely used for the treatment of obstructive sleep apnea (OSA) and chronic obstructive emphysema (COPD).

[0004] As a non-invasive ventilation treatment, a mask includes a facial interface device on the patient's face and can form a sealed breathing space around the nose and mouth of the patient's face. According to the contact method, masks are usually classified into four types: nose masks, mouth masks, full face masks, and nose pad masks, among which nose masks cover only the nose, mouth masks cover only the mouth, mouth-nose masks cover both the mouth and nose, and nose pad masks are inserted into the nostrils. In addition, to fit various face sizes, masks are set in different models such as large size, medium size, small size, and narrow size.

[0005] To provide more effective treatment, it is necessary to select a mask that is suitable for the patient's face size. Selecting a mask that does not fit the face size may cause phenomena such as air leakage, affect the comfort of wearing it, and reduce the therapeutic effect.

[0006] In the prior art, mask type selection is usually done by using photos, videos, etc., combining with personal aesthetics to select the mask type and appearance, and then using nose calipers to measure the nose width to select a suitable mask model. In practical application, the above method often has a large deviation value, poor experience, misjudgment, misjudgment, and inappropriate positioning of the nose bridge and chin, resulting in frequent returns and exchanges. Summary of the Invention [Problem to be solved by the invention]

[0007] In view of the above problems, embodiments of the present application are proposed to provide a method, an apparatus, a terminal device, and a readable storage medium for virtually wearing a mask that overcome or at least partially solve the above problems. [Means for solving the problem]

[0008] According to a first aspect of the present application, acquiring a facial image of a user; determining actual facial feature data based on the facial image; determining and displaying one or more first masks of the various matched models based on the actual facial feature data; determining a target mask from each of the first masks in response to the received first user input; and generating wearing screen information based on the face image and the target mask.

[0009] According to a second aspect of the present application, an acquisition module for acquiring a face image of a user; an image processing module for determining actual facial feature data based on the facial image; a display module for displaying one or more first masks of the matched models based on the actual facial feature data; an operational module for determining a target mask from each of the first masks in response to the received first user input; A determination module for generating wearing screen information based on a face image and a target mask, The display module further provides a virtual mask wearing device, which is used to display wearing screen information.

[0010] According to a third aspect of the present application, There is provided a terminal device comprising a processor, a memory and a program or instructions stored in the memory and executable by the processor, the program or instructions, when executed by the processor, implementing the steps of the method according to the first aspect.

[0011] According to a fourth aspect of the present application, A readable storage medium having a program or instruction stored thereon, A program or instructions are provided on a readable storage medium which, when executed by a processor, implements the steps of the method according to the first aspect. Effect of the Invention

[0012] The embodiments of the present application include the following advantages:

[0013] The method obtains a face image of a user, determines actual facial feature data based on the face image, determines and displays one or more first masks of various matched models based on the actual facial feature data, determines a target mask from each first mask according to the received first input by the user, and generates and displays wearing screen information based on the face image and the target mask. Since the actual facial feature data is determined based on the real-time face image of the user, the actual facial feature data reflects the actual facial features of the user, and therefore, a plurality of first masks of various types and sizes suitable for the facial features of the user can be accurately matched based on the facial feature data. Then, after the user determines the target mask through the first input, the wearing screen information is generated and displayed based on the target mask and the face image of the user using a technology such as virtual reality (VR), thereby simulating the effect of the user wearing the target mask, allowing the user to have a sense of realism as if he or she were actually in the place, and making it easier for the user to select a mask that satisfies him or her. [Brief description of the drawings]

[0014] [Figure 1] 1 is a flowchart of a virtual wearing method of a mask according to an embodiment of the present application. [Diagram 2] FIG. 2 is a schematic diagram of a reference classification of masks according to an embodiment of the present application. [Diagram 3] In the examples of the present application, the applicable size standards for the "F Mask-A Type" are classified according to the distance from the tip of the nose to the lower chin. [Figure 4] FIG. 2 is a schematic diagram of the geometric relationship of the locations of key features in an embodiment of the present application. [Diagram 5] 1 is a schematic flow chart of a mask recommendation by identifying facial hair features of a user in an embodiment of the present application; [Figure 6] FIG. 2 is a block diagram of a virtual fitting device for a mask according to an embodiment of the present application. [Figure 7] FIG. 13 is a block diagram of a virtual fitting device for another mask according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In order to make the above objects, features and advantages of the present application more clearly and comprehensibly, the present application will be described in more detail below with reference to the drawings and specific embodiments.

[0016] In some embodiments, a flowchart of a virtual wearing method of a mask is shown with reference to FIG. 1, and the method may specifically include the following steps 101 to 105.

[0017] The method for virtually wearing a mask according to some embodiments of the present application is applied to a terminal device having a display device such as a screen, and the terminal device is a device such as a mask sales terminal device, through which a user can select a specific model of mask.

[0018] Step 101: Acquire a face image of the user.

[0019] In this step, the user is a patient who needs to select a mask, and this step is to collect the user's face image in real time by the image collection module with the user's permission, and the face image can be a two-dimensional image or a three-dimensional image, or a still photo or a video. In practical application, the user's face image is obtained by automatically and real-timely collecting the user's face image by the image collection module.

[0020] The image collection module may include any known device capable of collecting two-dimensional image information, such as, but not limited to, a camera, a video camera, an infrared imager, a monitor, and the like. The image collection module may include any known device capable of collecting three-dimensional image information, such as, but not limited to, a 3D scanner, a stereo imager, and the like. The three-dimensional image information may be a dynamic or static three-dimensional image collected by a device capable of collecting the three-dimensional image information, a point cloud, or a three-dimensional image obtained by 2D data analysis.

[0021] Step 102: Determine actual facial feature data based on the facial image.

[0022] In this step, based on the collected face images of the user, the facial features of the user are identified to obtain the actual facial feature data of the user.

[0023] In practical applications, the above-mentioned actual facial feature data includes, but is not limited to, at least one of facial hair feature information, facial contour feature information, and chin type feature information, and further includes position information and size information of main features, where the main features include, but are not limited to, eyes, center of the glabella, nose, philtrum, and chin, the eye size information includes interocular distance information, and the nose size information includes nose width information, nose bridge height information, and distance information from the nose tip to the chin.

[0024] Step 103: Determine and display one or more first masks of the various matched models based on the actual facial feature data.

[0025] In this step, the above-mentioned multiple kinds of facial feature data reflect the user's actual facial situation, and the facial features and size ranges of the facial features applied to the masks of various models can be preset or determined by big data analysis, so that various models of first masks including different types and different sizes and suitable for the user's face can be determined based on the facial feature data determined in step 102, and each of the above first masks can be displayed by the display module for the user to select and display the wearing effect.

[0026] A mask model is composed of a mask type, an applicable facial feature, and an applicable size. That is, each mask model includes information such as a mask type, an applicable facial feature, and an applicable size, where the mask type includes a nose mask, a mouth mask, a full face mask, and a nose pad mask, the applicable facial feature includes facial hair feature information, facial contour feature information, chin type feature information, etc., and the applicable size is divided into various sizes such as a large size, a medium size, a small size, and a narrow size, and different sizes correspond to different facial feature data ranges.

[0027] In some embodiments, a schematic diagram of a reference classification of masks is shown with reference to Figure 2. As shown in Figure 2, F mask-A type-L size mask indicates a full face type, no facial hair, large size mask model.

[0028] Here, the applicable size is a size range that fits the main features. For example, in the case of a range of size b of the distance from the nose tip to the lower chin, the database stores the range of size b of the distance from the nose tip to the lower chin of the patient's face to be applied for each mask model. Specifically, referring to FIG. 3, a part of the applicable size standard of "F mask-A type" classified according to size b in the embodiment of the present application is shown. When a mask that matches the user's actual facial feature data is determined, when a call command is issued to the database, the applicable range information of size b of the mask of a specific model is output to the terminal device.

[0029] Of course, it is understood that the patient interface compatible size database may also store multiple size range information of the applicable face for each mask model in the above classification mark, for example, nose width information, interocular distance information, nose bridge height information, etc. of the applicable face. By commonly referring to multiple size information ranges, the final selection of the mask device becomes more accurate.

[0030] Step 104: Determine a target mask from each of the first masks in response to the received first user input.

[0031] In this step, the first input is an input by the user to select the target mask from each of the first masks, and specifically may be an operation such as touching, clicking, or cursor locking on the target mask.

[0032] Step 105: Generate and display wearing screen information based on the face image and the target mask.

[0033] In this step, by utilizing image synthesis technology, the target mask can be added to the above-mentioned face image to generate image information of the user wearing the target mask, and then this wearing image information can be displayed to simulate an image of the user wearing the target mask, thereby displaying the wearing effect to the user and enabling the user to know the mask wearing effect in advance at the selection stage, thereby avoiding misjudgment or incorrect selection of a mask of a type and appearance that does not suit the user's actual situation.

[0034] Some embodiments of the present application include the following advantages:

[0035] The method obtains a face image of a user, determines a plurality of face feature data based on the face image, determines and displays one or a plurality of first masks of various matched models based on each face feature data, determines a target mask from each first mask according to the received first input by the user, and generates and displays wearing screen information based on the face image and the target mask. Since the face feature data is determined based on the real-time face image of the user, the face feature data reflects the actual face features of the user, and therefore, a plurality of first masks of various types and sizes suitable for the face features of the user can be accurately matched based on the face feature data. Then, after the user determines the target mask through the first input, the wearing screen information is generated and displayed based on the target mask and the face image of the user using a technology such as virtual reality (VR), thereby simulating the effect of the user wearing the target mask, allowing the user to have a sense of realism as if he or she were actually in the place, and making it easier for the user to select a mask that satisfies him or her.

[0036] In some embodiments of the present application, a user photographs his / her face through the image collection module of the terminal device, and views the image of his / her face in real time on the display module. The system prompts the user to move his / her face to match the camera and collect size information. The system identifies the patient's facial features and recommends the type of mask to be applied. The user then selects a model of mask and realizes VR wearing on the display module. Through the terminal device, the user can intuitively experience the appearance, size, performance, and wearing and usage method of each mask model, and can watch video tutorials to order and purchase, which is convenient and efficient.

[0037] It will be understood that the method according to the embodiment of the present application may be configured as a mobile phone application app, WeChat plug-in, etc. for easy use by customers, and may be used as a standalone device or in combination.

[0038] In some embodiments, the method according to the embodiments of the present application further includes steps 106-108 after step 102 above.

[0039] Step 106: Based on the actual facial feature data, determine the geometric relationship between the positions of each key feature.

[0040] In this step, the facial feature data includes position information of key features, including but not limited to the eyes, center of the eyebrow, nose, philtrum, upper jaw and lower jaw, so that the geometric relationship between the positions of each key feature can be determined.

[0041] Specifically, the distance between the key features is calculated from the alignment mark of each key feature to further determine the geometric relationship between the key features.

[0042] Step 107: Determine whether the user's face is distorted based on the geometric relationships between the locations of the key features.

[0043] In a natural frontal face, the positional relationships of the key features of a person's face are fixed, i.e., the position of each key feature is in a fixed geometric relationship, so it is possible to determine whether the user's current face is distorted or not based on the actual geometric relationships between the positions of the key features.

[0044] For example, in the five points located on the user's face, the left eyeball, the right eyeball, the center of the eyebrows, the tip of the nose, and the lower jaw, as shown in FIG. 4, the line connecting the left eyeball and the right eyeball is defined as l1, and the line connecting the center of the eyebrows, the tip of the nose, and the lower jaw is defined as l2. When the position information of the above five points is received, the distance between the eyes (size a) and the distance from the tip of the nose to the lower jaw (size b) are automatically calculated, and the geometric positional relationship of the lines l1, l2, size a, and size b is checked. Only when the check result shows that the lines l1 and l2 are perpendicular to each other and the size a is divided into two equal parts by l2, is it determined that the patient's face is in a frontal face state without distortion or rotation at that time, and the size b is output, and then the data is provided for determining the second matching score of the matching mask. On the other hand, if the lines l1 and l2 are not perpendicular, or the size a cannot be divided into two equal parts by l2, it indicates that the user is not currently facing the image collection module of the terminal device in a frontal face state, that is, the user's face is in a distorted state.

[0045] Step 108: If it is determined that the user's face is distorted, generate presentation information to prompt the user to adjust the facial posture.

[0046] In this step, presentation information needs to be sent to guide the patient to rotate their face into a specific position, since if the user's face is distorted, it is not difficult to collect data and recommend a fitting mask.

[0047] In some embodiments, the above step 103 includes steps 301-304.

[0048] Step 301: Obtain a preset correspondence between masks and facial feature data of different models.

[0049] In this step, the facial features and facial feature sizes to be applied to the masks of various models are determined in advance through big data analysis, and the range of facial feature data to be applied to the masks of various models is determined, thereby determining the above correspondence relationship.

[0050] In some specific embodiments, the facial feature data includes, but is not limited to, facial hair feature information, facial contour feature information, chin type feature information, key feature position information and size information, where the key features include, but are not limited to, one or more of eyes, center of the glabella, nose, philtrum, upper jaw and lower jaw, and the eye size information includes interocular distance information.

[0051] In some specific embodiments, the facial feature data includes nose size information, including nose width information, nose bridge height information, nose tip to upper chin distance information, and nose tip to lower chin distance information. Here, the nose tip to upper chin distance information is used for selecting a nose mask or a nose pad mask, specifically referring to the distance from the nose tip to the highest point of the upper chin. Meanwhile, the nose tip to lower chin distance information is used for a user to select a full face mask, specifically referring to the distance from the nose tip to the lowest point of the chin.

[0052] In some specific embodiments, the facial feature data includes nose size information including nose width information, nose bridge height information, and distance information from nose tip to upper jaw.

[0053] In some specific embodiments, the facial feature data includes nose size information including nose width information, nose bridge height information, and distance information from nose tip to chin.

[0054] Here, the above correspondence may be stored in a data system managed and maintained by the service side, and the data system may be bound to the above terminal device and may be downloaded and accessed by the above terminal device. For example, the data information of the data system and the system files of the above terminal device may be used to form an APP applied to the terminal device, or the APP may be stored in a cloud, a base station, hardware storage (hard drive or USB device), etc., and called up when in use, but is not limited to these.

[0055] Step 302: Determine a first mask of various models corresponding to the actual facial feature data according to a preset correspondence relationship.

[0056] In this step, since a model of a mask that corresponds to and matches each facial feature data is predetermined based on the above-mentioned preset correspondence relationship, a model of a mask corresponding to the actual facial feature data can be determined based on the above-mentioned preset correspondence relationship, i.e., the above-mentioned first mask can be determined.

[0057] In practical applications, the above first mask corresponding to the actual facial feature data may be one or more.

[0058] Step 303: Display each first mask.

[0059] Each first mask determined in step 302 is displayed for the user to select and confirm.

[0060] In some particular embodiments, the above-mentioned pre-set correspondence relationship includes a first sub-correspondence relationship between the type of mask and the facial features, and a second sub-correspondence relationship between the size of the mask and the size range of the main features, and the above-mentioned step 302 includes steps 3021 to 3023.

[0061] Step 3021: Determine a first mask type according to the first sub-correspondence relationship and the facial features in the actual facial feature data.

[0062] In this step, the user's actual facial features are extracted from the actual facial feature data, and then based on the actual facial features, in combination with the above-mentioned first sub-correspondence relationship, a suitable mask type, i.e., the above-mentioned first mask type, is determined.

[0063] Step 3022: Determine a first mask size according to the second sub-correspondence relationship and the size information of the main features in the actual facial feature data.

[0064] In this step, size information of the user's main features is extracted from the actual facial feature data, and then based on this size information of the main features, in combination with the above-mentioned second sub-correspondence relationship, a suitable mask size, i.e., the above-mentioned first mask size, is determined.

[0065] Step 3023: Determine a first mask based on the first mask type and the first mask size.

[0066] In this step, the first mask type determines the specific type of the mask, and the first mask size limits the specific size of the mask, so that by combining the first mask type and the first mask size, a specific model of mask can be determined, that is, the above-mentioned first mask that matches the user's actual facial feature data can be determined.

[0067] In some specific embodiments, the method according to the embodiments of the present application further includes steps 3024 to 3027 after the above step 3023.

[0068] Step 3024: Determine a first matching score for each first mask based on the actual facial feature data and the first sub-correspondence.

[0069] In this step, since the first sub-correspondence relationship determines the correspondence relationship between the mask type and the facial features, a corresponding matching score, i.e., the above-mentioned first matching score, can be determined based on the user's actual facial features in the actual facial feature data, and the higher the score, the more suitable the mask model is determined to be for the user's facial features from the perspective of the user's features, and the lower the score, the less suitable the mask model is determined to be for the user's facial features from the perspective of the user's features.

[0070] For example, if the actual facial feature data identifies the user's face as having no facial hair and outputs a recommendation to use an A-type mask, and a data system, e.g., a database, outputs an "F mask-A type-M size" model mask, the model mask is determined to be suitable for the facial features and is given a higher feature score, e.g., 95 points, which is the first matching score described above.

[0071] Step 3025: Determine a second matching score for each first mask based on the actual facial feature data and the second sub-correspondence.

[0072] In this step, since the second sub-correspondence relationship determines the correspondence relationship between the mask size and the size range of the main feature, a corresponding matching score, i.e., the above-mentioned second matching score, can be determined based on the size information of the user's main feature in the actual facial feature data, and the higher the score, the more suitable the mask model is determined to be for the user's facial shape features from the perspective of the user's face size, and the lower the score, the less suitable the mask model is determined to be for the user's face size from the perspective of the user's features.

[0073] For example, if image analysis outputs b=70 mm, i.e., it is determined that the distance from the tip of the user's nose to the lower chin is 70 mm, and the database outputs "F Mask-A Type-M Size", it is easy to see from Figure 3 that b=70 mm is within the applicable b size range of "F Mask-A Type-M Size", and this "F Mask-A Type-M Size" mask is determined to have a high size score for the user's face, i.e., a high second matching score, for example 95 points.

[0074] Similarly, if image analysis outputs b=88 mm and the database outputs "F Mask-A Type-M Size", b=88 mm is not within the applicable b size range of "F Mask-A Type-M Size", so this "F Mask-A Type-M Size" mask is determined to have a low size score for the user's face, for example 30 points. In this way, a size score for a mask for a user's face is obtained, and the above size score is further displayed.

[0075] Step 3026: Determine an overall matching score for each first mask based on the first matching score and the second matching score.

[0076] In this step, the first matching score and the second matching score are weighted to obtain a total score of the matching between each first mask and the patient's face, i.e., the total matching score mentioned above. A higher total score indicates a better match.

[0077] Step 3027: Display the first matching score, the second matching score, and the overall matching score for each first mask.

[0078] In this step, the above matching scores are displayed to facilitate the user to understand the specific matching situation between each first mask and his / her own face, and to facilitate the user to select and compare.

[0079] In some embodiments, the above step 105 includes steps 501-502.

[0080] Step 501: The face image is marked based on the position information of each main feature.

[0081] In this step, one or more of the position information of the main features such as the tip of the nose, the center of the glabella, the philtrum, and the eyes are marked and positioned to realize real-time positioning and marking for the face image. The positioning and marking may be performed in real time according to the movement of the patient's face.

[0082] Step 502: Based on the positioning mark, the VR graphic information of the target mask is positioned on the face image to generate wearing screen information.

[0083] In this step, when the patient's face moves, the above-mentioned positioning marks can position the positions of the main features of the patient's face in real time, so that the VR graphic information of the target mask is obtained through the database, and then the VR graphic information of the mask is matched with the above-mentioned positioning marks for positioning, and the wearing screen information is generated and displayed, so that the VR graphic information of the mask can be positioned on the patient's face in real time. It can be understood that the VR graphic information of the mask moves according to the movement of the patient's face, and the VR effect of "wearing in real time" can be achieved.

[0084] In practical application, the positioning is performed by marking and positioning one or more main features of the user's face in real time, receiving the VR graphic information of the mask output from the database, and then matching and positioning the VR graphic information of the mask with the positioning mark, and then positioning the VR graphic information of the mask on the patient's face in real time. When matching and positioning, it may be individual matching of each feature, or a comprehensive matching of several features, such as matching the distance between the tip of the nose and the outermost part of the upper alveolus, the inclination angle, etc.

[0085] In practical application, the above-mentioned wearing screen is displayed through an interface interaction window of a terminal device configured to output information, display related content, receive operation commands input by a user, and display presentation information as necessary. For example, information such as a real-time face image of a user acquired, real-time positioning, etc. can be displayed in real time, allowing a user to view his / her facial features in real time. A VR image of a mask can be displayed, and the VR image of the mask can be virtually worn on a patient's face based on the real-time positioning information. It is understood that the mask VR image is stationary relative to the face even if the face moves, that is, similar to the actual wearing effect.

[0086] In some embodiments, the above interface interaction window may further display performance information of the selected target mask, such as, but not limited to, the weight of the mask, the volume of the dead cavity of the mask, the applied pressure of the mask, and information on wearing and using the mask, such as, but not limited to, the steps of wearing the mask, how to use and clean it, and may further display the fit information of the face mask, such as, for example, when the overall score of face matching is low, the reason for the low score may be "The mask is too large, so we recommend using a smaller size mask." It is also possible to display, but is not limited to, a message such as, "The mask is too small, we recommend using a larger size mask," and to provide the user with presentation information as necessary. For example, when calculating the distance size from the tip of the nose to the lower chin, presentation information can be provided to encourage the user to adjust the position of the face, but is not limited to this. It is also possible to display dynamic presentation information. For example, if the mask selected by the user is a nasal mask or a nasal pad mask and the user opens his / her mouth when using it, a dynamic line showing the user's mouth exhaust may be displayed to inform the user that he / she should not open his / her mouth when using a nasal mask or a nasal pad mask.

[0087] In some embodiments, the method according to the embodiments of the present application may receive a second input by the user to select a different matching size reference, such as, but not limited to, the distance size b from the tip of the nose to the chin on the patient's face, and may interrupt the transmission of the applicable size information, i.e., the above-mentioned second sub-correspondence relationship, from the database to the terminal device if necessary, in which case the user may selectively decide whether or not to make the size score a necessary reference condition for the final overall matching score.

[0088] In some embodiments, the method according to the embodiments of the present application may receive a third input by the user to manage the feature score instruction and interrupt the above-mentioned operation of determining the second matching score as necessary, in which case the user may selectively determine whether or not to use the second matching score as a necessary reference condition for the final overall matching score through the second input operation. The user may also determine the category of the specifically identified facial feature of the user through the third input, for example, identifying the facial features of the beard and the features of the face shape individually or simultaneously. The user may also determine the mark positioning of the main features through the third input operation. For example, some features among the eyes, the center of the eyebrow, the philtrum, the upper lip, the lower chin, and the tip of the nose are marked and positioned.

[0089] Referring to FIG. 5, a schematic flow chart of mask recommendation by identifying facial hair features of a user in an embodiment of the present application is shown.

[0090] As shown in FIG. 5, in step 221, image information collected by an image collection module is received.

[0091] In step 222, a selection program is run to identify whisker features.

[0092] In step 223, based on the specific beard features, matching and recommending a corresponding model mask, then proceeding to step 224, for example, recommending the user to use type A mask if the user's face is identified as having no beard, recommending the user to use type B mask if the user's face is identified as having sideburns, and recommending the user to use type C mask if the user's face is identified as having goatee.

[0093] In step 224, the recommendation results are output to a display module, which then receives and displays the recommendation results.

[0094] Of course, when identifying the patient's facial contour characteristics, the patient's jaw type characteristics, etc., there are the above-mentioned discrimination and recommendation processes, but we will not go into them further here.

[0095] For example, it will be appreciated that multiple features of a patient's face may be simultaneously identified, such as, but not limited to, beard features, facial contour features, chin type features, etc., which may be integrated to provide a recommended mask type.

[0096] In some embodiments, referring to FIG. 6, a block diagram of a virtual fitting device for a mask is shown, which specifically includes: an image collection module 61 for acquiring a face image of a user; an image processing module 62 for determining actual facial feature data based on the facial image; a determining module 63 for determining one or more first masks of the matched models based on actual facial feature data; a display module 64 for displaying each first mask; and an operation module 65 for determining a target mask from each of the first masks in response to the received first user input. The determination module 63 is further used for generating wearing screen information according to the face image and the target mask; The display module 64 is further used to display the installation screen information.

[0097] Some embodiments of the present application include the following advantages:

[0098] The method obtains a face image of a user, determines actual facial feature data based on the face image, determines and displays one or more first masks of various matched models based on the actual facial feature data, determines a target mask from each first mask according to the received first input by the user, and generates and displays wearing screen information based on the face image and the target mask. Since the actual facial feature data is determined based on the real-time face image of the user, the actual facial feature data reflects the actual facial features of the user, and therefore, a plurality of first masks of various types and sizes suitable for the facial features of the user can be accurately matched based on the facial feature data. Then, after the user determines the target mask through the first input, the wearing screen information is generated and displayed based on the target mask and the face image of the user using a technology such as virtual reality (VR), thereby simulating the effect of the user wearing the target mask, allowing the user to have a sense of realism as if he or she were actually in the place, and making it easier for the user to select a mask that satisfies him or her.

[0099] In some embodiments, the determination module 63 is specifically used for obtaining preset correspondence relationships between masks of different models and facial feature data, and determining first masks of various models corresponding to actual facial feature data according to the preset correspondence relationships.

[0100] In some embodiments, the facial feature data includes, but is not limited to, at least one of facial hair feature information, facial contour feature information, and chin type feature information, and further includes position information and size information of key features, including, but not limited to, one or more of eyes, glabella center, nose, philtrum, upper jaw, and lower jaw. The eye size information includes interocular distance information.

[0101] The nose size information includes nose width information, nose bridge height information, distance information from the nose tip to the upper chin and distance information from the nose tip to the lower chin, or the nose size information includes nose width information, nose bridge height information and distance information from the nose tip to the upper chin, or the nose size information includes nose width information, nose bridge height information and distance information from the nose tip to the lower chin.

[0102] In some embodiments, the image processing module 62 includes a size data processing unit, which: a first determination subunit for determining a geometric relationship between the positions of each main feature based on the actual facial feature data after determining facial feature data based on the facial image; a second determination subunit for determining whether the user's face is distorted based on a geometric relationship between the positions of the main features; and a presentation subunit for generating presentation information for prompting the user to adjust the facial posture when it is determined that the user's face is distorted.

[0103] In some embodiments, the image processing module 62 includes: further comprising an image registration unit for marking the facial image based on the position information of each of the main features; The decision module 63 is The apparatus includes a VR matching determination subunit for positioning the VR graphic information of the target mask to the face image based on the positioning mark, and generating wearing screen information.

[0104] In some embodiments, the preset correspondence of the device includes a first sub-correspondence between the type of mask and the facial features, and a second sub-correspondence between the size of the mask and the size range of the main features; The decision module 63 is a type determining unit for determining a first mask type according to the first sub-correspondence relationship and the facial features in the actual facial feature data; a size determining unit for determining a first mask size according to the second sub-correspondence relationship and size information of the main features in the actual facial feature data; and a mask determination unit for determining the first mask based on the first mask type and the first mask size.

[0105] In some embodiments, the determination module 63 is a feature score determining unit for determining a first matching score of each first mask according to the actual facial feature data and the first sub-correspondence relationship after determining a first mask according to the first mask type and the first mask size; a size score determining unit for determining a second matching score of each first mask according to the actual facial feature data and the second sub-correspondence; and an overall score determining unit for determining an overall matching score of each first mask based on the first matching score and the second matching score; The display module is further used for displaying the first matching score, the second matching score and the overall matching score of each first mask.

[0106] Referring to FIG. 7, a block diagram of another mask virtual fitting apparatus according to an embodiment of the present application is shown.

[0107] 7, the above-mentioned image processing module 62 functions to receive the image information collected by the image collection module 61, and identify, locate and process specific size data. The image processing module 62 further includes a user feature identification unit 621, an image positioning unit 622 and a size data processing unit 623.

[0108] Specifically, the user feature identification unit 621 can identify the facial features of the patient, such as, but not limited to, facial hair features of the patient (such as the presence or absence of hair, the distribution and density of hair), facial contour features of the patient (such as round face or oval face, the presence or absence of facial defects), and key feature positions (such as the tip of the nose, the center of the glabella, the philtrum, the eyes, etc.). The user feature identification unit 621 can identify the facial features of the patient and output a mask type that matches the facial features of the patient to the display module 64.

[0109] The above image positioning unit 622 can receive the key feature positions (e.g., nose tip, center of eyebrow, philtrum, eyes, etc.) identified by the user feature identification unit 621, and mark and position one or more of the above key feature positions. Furthermore, the image positioning unit 622 is configured to perform marking and positioning in real time according to the movement of the user's face. It is understood that as the patient's face moves, the image positioning unit 622 marks and positions the key feature positions of the patient's face in real time, and outputs them to the display module 64 to provide a basis for positioning for subsequent VR image display. Furthermore, the image positioning unit 622 may output the marked key feature positions to the size data processing unit 623.

[0110] The above-mentioned size data processing unit 623 is configured to receive information of one or more key feature positions of the user's face from the image positioning unit 622, calculate distance data between certain marks, and then transmit the distance data to the determination module 63.

[0111] The database 66 is a data system for selecting a mask model, and is managed and maintained by the service side. The database 66 may be bound to system data and may be downloaded and accessed together with the virtual wearing device of the mask. For example, the database 66 may be used to form an APP applied to a mobile terminal using the data information of the database 66 and the system file of the selection device, or may be stored in a cloud, a base station, a hardware storage (hard drive or USB device), etc., and called up when used, but is not limited thereto. Furthermore, the database 66 is composed of a patient interface device 3D model 661, a patient interface compatible size database 662, a patient interface VR graphic database 663, etc. The patient interface device 3D model 661 is a 3D model of all selectable masks that can be provided by the service side, and is classified and marked according to the contact method with the face, the applicable facial features, the applicable size, etc.

[0112] The patient interface fitting size database 662 stores certain size range information of the face applicable to each model of mask classified and marked above, and outputs this size range information to the determination module 63 to provide second data for determining the size score of the fitting mask in the determination module 63. For example, in this embodiment, the patient interface fitting size database 662 stores a range of sizes b of the distance from the tip of the nose to the lower chin on the patient's face applicable to each model of mask, and when the operation module issues a call command, the application range information of the size b of the mask of a specific model is output to the determination module 63.

[0113] The patient interface VR graphics database 663 stores the VR graphics data for each mask model classified and marked above, and outputs the above VR image data to the decision module 63 so that the decision module 63 makes a matching decision on the VR data.

[0114] The above-mentioned determination module 63 is a logic execution unit of the VR patient interface device selection system. It functions to receive the data information in the image processing module 62 and the database 66, perform size score determination and VR matching determination, and further output the result to the display module 64. Specifically, the determination module 63 includes a size score determination unit 631, a VR matching determination unit 632, and a feature score determination unit 633. In this embodiment, the size score determination unit 631 simultaneously receives the first data (for example, in this embodiment, the size b of the distance from the nose tip to the lower chin) output from the image processing module 62 and the second data (for example, in this embodiment, the size b range information applied to the mask of each model) output from the database 66, and compares the first data with the second data to determine the size score of the mask. In this way, in the size score determination unit 631, the size score of the mask of a certain model for the face of a certain patient is obtained, and is output to the display module 64 for display.

[0115] The VR matching determination unit 632 simultaneously receives the VR graphic information output from the database 66 and the image positioning information output from the image processing module 62, and functions to position the above VR graphic information on the patient's face in the image in real time. Specifically, the VR matching determination unit 632 receives real-time mark positioning (e.g., the position of the eyes, nose tip, philtrum, etc.) of one or more main feature positions of the patient's face output from the image positioning unit 622, receives the VR graphic information of the mask output from the patient interface VR graphic database, and then matches and positions the VR graphic information of the mask with the mark positioning, and further positions the VR graphic information of the mask on the patient's face in real time. The result is output to the display module 64 in real time, and the above wearing effect is displayed on the display module 64 in real time, where the matching of features may be individual matching of individual features or comprehensive matching of several features, such as matching the distance between the nose tip and the outermost part of the upper alveolus, the inclination angle, etc.

[0116] The feature score determination unit 633 performs matching based on the user feature information output from the image processing module 62 and the mask model information output from the database 66, and functions to give a corresponding feature score, where the higher the score, the more suitable the mask model is for the face shape features of the user from the viewpoint of the user feature, and the lower the score, the less suitable the mask model is for the face shape features of the user from the viewpoint of the user feature. Specifically, the feature score determination unit 633 simultaneously receives the mask type recommendation information output from the user feature identification unit 621 and the mask model information output from the patient interface suitable size database 662 to perform matching. For example, when the user feature identification unit 621 identifies that the user has no facial hair and outputs a recommendation to use an A type mask, and outputs a mask of the type "F mask-A type-M size" according to the patient interface suitable size database 662, the feature score determination unit 633 determines that the mask of this model is suitable for the face features and gives a higher feature score (e.g., 95 points). In the opposite case, a lower feature score is given. Furthermore, the feature score determination unit 633 may output the feature score to the display module 64 for display.

[0117] The above operation module 65 is a link connecting the image processing module 62, the database 66, and the display module 64. A user can give operation instructions to other modules through the operation module 65. Furthermore, the operation module 65 includes a first operation unit 651 and a second operation unit 652. The first operation unit 651 can simultaneously send operation instructions to the image processing module 62 and the display module 64, and manage the transmission instructions sent or received by the image processing module 62 and the display module 64. Furthermore, for example, the first operation unit 651 can control the feature score instruction transmitted from the user feature identification unit 621 to the feature score determination unit 633, and interrupt the transmission as necessary, in which case the user can selectively determine whether or not to make the feature score a necessary reference condition for the final overall score through the first operation unit 651. The user can also determine the category of the patient's facial features identified by the user feature identification unit 621 through the first operation unit 651, for example, beard features and face shape features can be identified separately or simultaneously, and the user can also determine the mark positioning of the image positioning unit 622 through the first operation unit 651, for example, mark positioning for any of the features of the eye, the center of the glabella, the philtrum, the upper lip, the lower chin, and the tip of the nose. Furthermore, the first operation unit 651 may further include an operation interface having touch panel operation keys and similar functions displayed on the display module 64.

[0118] Similarly, the second operation unit 652 can simultaneously send operation instructions to the database 66 and the display module 64, and manage the transmission instructions sent or received by the database 66 and the display module 64. In addition, the user can select a different model of patient interface device to call up a VR graphic of a different model of mask through the second operation unit 652, and can also select a different fit size reference (for example, but not limited to, the distance size b from the nose tip to the lower chin on the patient's face) through the second operation unit 652, and can also interrupt the transmission of the patient interface fit size database 662 to the determination module 64 as necessary, in which case the user can selectively determine whether or not to make the size score a necessary reference condition for the final overall score through the second operation unit 652. In addition, the second operation unit 652 can further include an operation interface having a touch panel operation key and similar functions displayed on the display module 64.

[0119] The above-mentioned display module 64 is an interface interaction window of the device, and is configured to receive the output information of the image processing module 62, the decision module 63, and the operation command of the operation module 65 to display related content, and to display presentation information as necessary. Furthermore, the display module 64 includes a real-time image display module 641, a total score display module 642, a VR image display module 643, and an information display module 644. The real-time image display module 641 can receive information such as real-time images and real-time positioning output from the image processing module 62 and display them in real time. That is, the user can see his / her facial features in real time in the real-time image display module 641. The VR image display module 643 can display a VR image of a mask, and "wear" the mask VR image on the patient's face based on the real-time positioning information. It can be understood that the mask VR image is stationary relative to the face even if the face moves, that is, similar to the actual wearing effect. The overall score display module 642 can receive the size score output from the size score determination unit 631 and / or the feature score output from the feature score determination unit 633, and perform a weighting operation to obtain and display an overall score of the matching of the selected mask to the patient's face, where a higher overall score indicates a better match.

[0120] The information display module 644 may display performance information of the selected mask (e.g., but is not limited to, information such as the weight of the mask, the volume of the dead space of the mask, and the applied pressure of the mask), may display information on wearing and using the mask (e.g., but is not limited to, information on the procedure for wearing the mask, the method for using and cleaning the mask), may display information on the fit of the mask (e.g., if the overall score for fit to the face is low, the reason for the low score may be displayed as "the mask is too large, so we recommend using a smaller size mask" or "the mask is too small, so we recommend using a larger size mask", etc.), and is configured to display presentation information to the user as necessary (e.g., but is not limited to, information to prompt the user to adjust the position of the face when calculating the distance size b from the tip of the nose to the lower chin, etc.).

[0121] The device embodiment is basically similar to the method embodiment, so the description will be relatively simple, but for related points, please refer to the method embodiment.

[0122] In some embodiments of the present application, a terminal device is provided that includes a display, a processor, a memory, and a computer program stored in the memory and executable by the processor, which computer program, when executed by the processor, realizes steps in a method for virtually fitting a mask and can achieve the same technical effect, but to avoid repetition, will not be mentioned further here.

[0123] In some embodiments of the present application, a readable storage medium having a computer program stored thereon is also provided, which, when executed by a processor, performs various processes of steps in a virtual fitting method for a mask and can achieve the same technical effect, but to avoid repetition, will not be mentioned further here.

[0124] The various embodiments herein are described in a progressive manner, with each embodiment being described by focusing on the differences from other embodiments, and similar parts of the various embodiments may be referenced to one another.

[0125] Those skilled in the art can appreciate that the embodiments of the present application may be provided as a method, an apparatus, or a computer program product. Thus, the embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of the present application may take the form of a computer program product embodied on one or more computer usable storage mediums (including, but not limited to, disk memories, CD-ROMs, optical memories, etc.) containing computer usable program code.

[0126] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent, random access memory (RAM) and / or non-volatile forms of computer-readable media, such as read-only memory (ROM) or flash RAM (Flash RAM). Memory is an example of a computer-readable medium. Computer-readable media includes persistent and non-persistent, removable and non-removable media, and may implement information storage by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, read only optical disk read only memory (CD-ROM), digital versatile optical disk (DVD) or other optical storage device, magnetic cartridge, magnetic tape magnetic disk storage device or other magnetic storage device, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transitory media), such as modulated data signals and carrier waves.

[0127] The embodiments of the present application are described with reference to flowcharts and / or block diagrams of a method, a terminal device (system), and a computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, whereby the instructions executed by the processor of the computer or other programmable data processing terminal device can generate means for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0128] These computer program instructions may be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a particular manner, whereby the instructions stored in the computer readable memory produce an article of manufacture including an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device to execute a series of operational steps on the computer or other programmable data processing terminal device to generate a computer-implemented process, whereby the instructions executing on the computer or other programmable data processing terminal device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0130] Although the preferred embodiments of the present application have been described, additional changes and modifications can be made to these embodiments by those skilled in the art upon learning the basic inventive concept. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0131] Finally, it should be noted that in this specification, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that such an actual relation or order exists between these entities or operations. Furthermore, terms such as "comprises," "includes," or any other variant are intended to cover a non-exclusive inclusion, whereby a process, method, article, or terminal device that includes a set of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element qualified by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0132] The above describes the virtual mask wearing method and the virtual mask wearing device according to the present application in detail, and the present application describes the principles and embodiments of the present application by applying specific examples. However, the above description of the examples is only used to help understand the method and the core idea of ​​the present application. In addition, those skilled in the art should not understand that the content of the present application limits the present application, since there are changes in the specific embodiments and application scope according to the concept of the present application.

Claims

1. acquiring a facial image of a user; determining actual facial feature data based on the facial image; determining and displaying one or more first masks of the various matched models based on the actual facial feature data; determining a target mask from each of the first masks in response to the received first user input; generating and displaying wearing screen information based on the face image and the target mask.

2. determining actual facial feature data based on the facial image; determining a geometric relationship between the location of each key feature based on the actual facial feature data; determining whether the user's face is distorted based on a geometric relationship between the locations of the key features; 10. The method of claim 1, further comprising: if determining that the user's face is distorted, generating presentation information to prompt the user to adjust the facial posture.

3. determining and displaying a first mask of each of the matched models based on the actual facial feature data; Obtaining a pre-established correspondence between masks and facial feature data of different models; determining a first mask of various models corresponding to the actual facial feature data based on the preset correspondence; and displaying each of the first masks.

4. The preset correspondence relationship includes a first sub-correspondence relationship between a mask type and a facial feature, and a second sub-correspondence relationship between a mask size and a size range of a main feature; The step of determining a first mask of various models corresponding to the actual facial feature data based on the preset correspondence relationship includes: determining a first mask type based on the first sub-correspondence and facial features in the actual facial feature data; determining a first mask size based on the second sub-correspondence relationship and size information of a main feature in the actual facial feature data; 4. The method of claim 3, further comprising: determining the first mask based on the first mask type and the first mask size.

5. determining the first mask based on the first mask type and the first mask size; determining a first matching score for each of the first masks based on the actual facial feature data and the first sub-correspondences; determining a second matching score for each of the first masks based on the actual facial feature data and the second sub-correspondence; determining an overall matching score for each of the first masks based on the first matching scores and the second matching scores; 5. The method of claim 4, further comprising: displaying the first matching score, the second matching score, and the overall matching score for each of the first masks.

6. The facial feature data includes nose size information including nose width information, nose bridge height information, distance information from the nose tip to the upper jaw, and distance information from the nose tip to the lower jaw, or The facial feature data includes nose size information including nose width information, nose bridge height information, and distance information from the nose tip to the upper jaw, or The method of claim 3 , wherein the facial feature data includes nose size information including nose width information, nose bridge height information, and distance information from nose tip to lower chin information.

7. 7. The method of claim 6, wherein the facial feature data includes at least one of facial hair feature information, facial contour feature information, and chin type feature information, and further includes position information and size information of the main features, the main features including one or more of eyes, center of the glabella, nose, philtrum, upper jaw, and lower jaw, and the eye size information includes interocular distance information.

8. The step of generating wearing screen information based on the face image and the target mask includes: marking said facial image based on position information of each of said primary features; The method according to claim 6 , further comprising: a step of positioning VR graphic information of the target mask on the face image based on the positioning mark to generate the wearing screen information.

9. The step of acquiring a face image of a user includes:

7. The method of claim 6, further comprising automatically acquiring a facial image of the user.

10. an image collection module for acquiring a facial image of a user; an image processing module for determining actual facial feature data based on the facial image; a determining module for determining one or more first masks of the matched models based on the actual facial feature data; a display module for displaying each said first mask; and an operation module for determining a target mask from each of the first masks in response to a received first input by the user; The determination module is further used for generating wearing screen information based on the face image and the target mask; The virtual wearing device for a mask, wherein the display module is further used to display the wearing screen information.

11. A terminal device including a display, a processor, a memory, and a program or instruction stored in the memory and executable by the processor, A terminal device, characterized in that the program or instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9.

12. A readable storage medium having a program or instruction stored thereon, A readable storage medium, characterized in that the program or instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Virtual intelligent display system for trying on mask

    CN113674053A

  • Mask fitting system and method

    JP2008501375A

  • Information processing apparatus, display control method, and program

    JP2013101526A

  • 3D Patient Interface Device Selection System and Method

    JP2016520384A

  • System and method for selecting a respirator

    US20140278319A1