Method for predicting life vitality state based on facial image, computing device and storage medium

Abstract

The embodiment of the invention relates to a method, equipment and medium for predicting a life vitality state based on a face image. The method comprises the following steps: inputting obtained target facial image data of a to-be-detected object into a prediction model, wherein the prediction model comprises a first network model, a second network model and a third network model; extracting a first feature of the target facial image data via a first network model; extracting a second feature of the target facial image data based on the first feature of the target facial image data via a second network model; and through a third network model, based on the first feature of the target face image data and the second feature of the target face image data, generating a predicted life vitality state about the to-be-detected object for the target face image, wherein the life vitality state is at least related to the age and health state of the to-be-detected object. Therefore, the life vitality state of the to-be-detected object can be predicted based on the face image of the to-be-detected object, and the comprehensive state of age and health of the to-be-detected object is reflected.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of facial image analysis, and more specifically to a method, computing device, and storage medium for predicting vitality status based on facial images. Background Technology

[0002] Traditional methods for analyzing human facial images, such as age prediction based on facial images and health status prediction based on facial images, only provide an age prediction result, while health status prediction only predicts the health status of the subject. Currently, there is no method that can comprehensively predict the age and health status of the subject based on facial images.

[0003] Furthermore, traditional facial image analysis methods, such as facial age prediction models based on convolutional neural networks (e.g., ResNet50, VGG19, Se-Resnext50), suffer from insufficient generalization ability when used for facial analysis, and their model capacity limitations also make it difficult to support training models with large amounts of data.

[0004] In summary, the shortcomings of traditional facial image analysis methods are that they cannot simultaneously consider the age and health status of the subject during analysis or prediction, and the models lack generalization ability. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure provides a method, computing device, and storage medium for predicting vitality status based on facial images, capable of predicting the vitality of a subject based on facial images, where vitality is at least related to the subject's age and health status.

[0006] According to a first aspect of this disclosure, a method for predicting vitality status based on facial images is provided, comprising: inputting target facial image data of a subject to be tested into a prediction model, the prediction model including a first network model, a second network model, and a third network model; extracting a first feature from the target facial image data via the first network model; extracting a second feature from the target facial image data based on the first feature via the second network model; and generating a predicted vitality status of the subject to be tested based on the target facial image via the third network model, based on the first feature and the second feature; wherein the vitality status is at least related to the age and health status of the subject to be tested.

[0007] According to a second aspect of this disclosure, a computing device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the first aspect of this disclosure.

[0008] In a third aspect of this disclosure, a computer-readable storage medium is provided having computer program code stored thereon, which, when executed, performs the method of the first aspect of this disclosure. In some embodiments, a first network model is constructed based on a Swin-Transformer architecture, a second network model is constructed based on a multilayer perceptron architecture, and a third network model is constructed based on a long short-term memory network architecture; and the first network model, the second network model, and the third network model are jointly trained based on a constructed facial image dataset, the facial image dataset including facial image data of multiple objects at multiple time periods.

[0009] In some embodiments, the prediction model is trained by the following steps: extracting multi-scale features for each facial image in each set of temporal data from a facial image dataset via a Swin-Transformer backbone network of a first network; obtaining a first predicted age and hidden features for each facial image in each set of temporal data based on the extracted multi-scale features via a multilayer perceptron of a second network; fusing the extracted first predicted age and hidden features for each facial image in each set of temporal data into a fused feature for the corresponding facial image, so as to obtain the predicted vitality status of the object associated with the facial image in each set of temporal data via a long short-term memory network based on the fused feature of the facial image; and training the prediction model based on a joint loss function constructed from the first predicted age and the predicted vitality status.

[0010] In some embodiments, obtaining the first predicted age and hidden features of each facial image in each set of facial image time series data based on the extracted multi-scale features of the facial images via the multilayer perceptron of the second network includes: obtaining the first predicted age of each facial image in each set of facial image time series data based on the extracted multi-scale features of the facial images via the fully connected layer of the multilayer perceptron; and obtaining the hidden features of each facial image in each set of facial image time series data based on the extracted multi-scale features of the facial images via the hidden layer of the fully connected layer of the multilayer perceptron.

[0011] In some embodiments, the facial image dataset is constructed by: collecting facial image data of multiple objects at multiple times to form multiple sets of facial image time-series data; wherein each set of facial image time-series data includes: multiple facial images of the same object arranged in chronological order; processing the collected facial image data to determine the maximum age of the object associated with each set of facial image time-series data; and for each facial image in each set of facial image time-series data, adding at least the current age feature label and the maximum age feature label of the corresponding object to construct the facial image dataset.

[0012] In some embodiments, constructing a facial image dataset further includes: assigning a total vitality value to the corresponding object based on the maximum age of the corresponding object in each set of facial image time series data; adding a total vitality value feature label to each facial image in each set of facial image time series data; wherein, the total vitality value of an object is proportional to its maximum age; and determining the time information corresponding to different health status types based on the health status information of the corresponding object in each set of facial image time series data, so as to add health status feature labels to the facial images of the corresponding object associated with the time information.

[0013] In some embodiments, generating a predicted vitality status of a test subject based on a target facial image includes: generating a total predicted vitality value of the test subject based on the target facial image; obtaining a first predicted age of the test subject based on a first feature of the target facial image data via a second network model; and generating a current predicted vitality value of the test subject based on the first predicted age and the total predicted vitality value of the test subject.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0016] Figure 1 A schematic diagram of a system for implementing a method for predicting vital status based on facial images according to embodiments of the present disclosure is shown.

[0017] Figure 2 A flowchart of a method 200 for predicting vitality status based on facial images, according to an embodiment of the present disclosure, is shown.

[0018] Figure 3 A schematic diagram of the network structure of a prediction model according to an embodiment of the present disclosure is shown.

[0019] Figure 4 A flowchart of a method 400 for constructing a facial image dataset according to an embodiment of the present disclosure is shown.

[0020] Figure 5 A flowchart of a method 500 for joint training of network parameters according to an embodiment of the present disclosure is shown.

[0021] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] As described above, traditional methods for analyzing human facial images, such as age prediction based on facial images and health status prediction based on facial images, only provide an age prediction result, while health status prediction only predicts the health status of the subject. Currently, there is no method that can comprehensively predict the age and health status of the subject based on facial images.

[0025] In summary, the shortcomings of traditional facial image analysis methods are that they cannot simultaneously consider the age and health status of the subject during analysis or prediction.

[0026] To at least partially address one or more of the aforementioned problems and other potential issues, exemplary embodiments of this disclosure propose a scheme for predicting vitality status based on facial images. In this scheme, target facial image data of the subject to be tested is input into a prediction model, which includes a first network model, a second network model, and a third network model. A first feature is extracted from the target facial image data via the first network model. A second feature is extracted from the target facial image data based on the first feature via the second network model. A predicted vitality status of the subject is generated based on the first and second features of the target facial image data via the third network model. The first feature (e.g., a multi-scale feature reflecting the geometric distribution of the face) is extracted by the first network, and a higher-order second feature (e.g., a hidden feature reflecting the physiological state of the face) is extracted by the second network based on the first feature. The third network integrates both to generate a comprehensive prediction. This avoids the limitations of single feature extraction and achieves a more accurate assessment of age and health status.

[0027] Vital state is related to at least the age and health status of the subject, integrating both factors. The predicted vital state output by the model is a multi-dimensional prediction, which is helpful for personalized health management. Therefore, this disclosed scheme constructs a three-stage model framework to separate and extract the subject's facial images, solving the problem that a single model cannot adequately consider both static features and dynamic changes. Based on a multi-level feature extraction and fusion mechanism, the reliability and practicality of the prediction results are ensured.

[0028] Figure 1 A schematic diagram of a system 100 (hereinafter referred to as system 100) for implementing a method for predicting vital status based on facial images according to embodiments of the present disclosure is shown. Figure 1 As shown, system 100 includes computing device 110, server 130, network 140, facial image database 150, and user terminals 160 (160-1 to 106-n). Computing device 110, server 130, facial image database 150, and user terminals 160 (160-1 to 106-n) can interact with each other via network 140 (e.g., the Internet, local area network, or wide area network).

[0029] Regarding server 130, it can be, for example, a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud databases, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0030] Regarding server 130, a method for predicting vitality status based on facial images according to embodiments of the present disclosure is deployed thereon. For example, it can obtain facial image data of an object through network 140 and send the facial image data of the object to computing device 110 for training the prediction model of the present invention and implementing the method for predicting vitality status based on facial images.

[0031] Regarding the user terminal 160, which may be a mobile device, personal terminal, desktop computer, smartwatch, tablet computer, interactive device, etc., the user can upload the target facial image data of the subject to be tested through the user terminal 160 via the network 140 so that the server 130 and / or computing device 110 can obtain this facial image data; the server 130 and / or computing device 110 can also send the predicted vitality status of the subject to be tested to the user terminal 160 via the network 140.

[0032] Regarding the computing device 110, it may be used for, for example, training a network model for a method of predicting vitality status based on facial images according to embodiments of the present disclosure, training and / or deploying a prediction model, and deploying a facial image database 150. The computing device 110 may have one or more processing units, including dedicated processing units such as GPUs (Graphics Processing Units), FPGAs (Field-Programmable Gate Arrays), and ASICs (Application-Specific Integrated Circuits), as well as general-purpose processing units such as CPUs. The computing device 110 may be an integration of multiple physical servers, an integration of multiple processing units, etc. Additionally, one or more virtual machines may run on each computing device 110. In some embodiments, the computing device 110 and the server 130 may be integrated together or separately configured. In some embodiments, the computing device 110 may include, for example, an image data input module 112, a first feature extraction module 114, a second feature extraction module 116, and a vitality prediction module 118.

[0033] Regarding the image data input module 112, it is used to input the acquired target facial image data of the test object into the prediction model, which includes a first network model, a second network model, and a third network model.

[0034] Regarding the first feature extraction module 114, it is used to extract the first feature of the target facial image data via the first network model.

[0035] Regarding the second feature extraction module 116, it is used to extract a second feature of the target facial image data based on the first feature of the target facial image data via a second network model.

[0036] Regarding the vitality prediction module 118, it is used to generate a predicted vitality status of the subject based on the first feature and the second feature of the target facial image data via a third network model; wherein the vitality status is at least related to the age and health status of the subject.

[0037] Figure 2 A flowchart of a method 200 for predicting vitality status based on facial images, according to an embodiment of the present disclosure, is shown. Method 200 may be derived from, for example... Figure 1 The computing device 110 shown can be used for execution, and can also be used in Figure 6 The method is performed at the illustrated electronic device 600. It should be understood that method 200 may also include additional steps not shown and / or the steps shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0038] In step 202, the computing device 110 inputs the acquired target facial image data of the object to be tested into the prediction model, which includes a first network model, a second network model, and a third network model.

[0039] In some embodiments, the first network model is built on the Swin-Transformer architecture, the second network model is built on the Multilayer Perceptron (MLP) architecture, and the third network model is built on the Long Short-Term Memory (LSTM) architecture.

[0040] In some embodiments, based on the constructed facial image dataset, a first network model, a second network model, and a third network model are jointly trained. The facial image dataset includes facial image data of multiple objects at multiple time periods. For example, the facial image dataset includes multiple sets of temporal data of facial images, each set of temporal data of facial images including multiple facial images of the same object arranged in chronological order.

[0041] Please refer to Figure 3 This illustration shows the network structure of a prediction model provided in an embodiment of the present disclosure. The prediction model may include multiple sets of branch networks, each set of branch networks including a first branch network and a second branch network, and the outputs of the branch networks converge into a general third network (e.g., Figure 3 The outputs of each branch network are converged into a third network constructed using a unified LSTM architecture. Figure 3The first group of branch networks includes: a first branch network built based on Swin-Transformer-01, and a second branch network built based on MLP-01 (Multilayer Perceptron 01); the second branch network and the nth branch network are as follows: Figure 3 This will not be elaborated further; the training input data for each branch network is a set of facial temporal data (such as...). Figure 3 The dataset contains facial temporal data 01, facial temporal data 02, ..., facial temporal data n. Based on the facial image dataset, the first network model, the second network model, and the third network model are jointly trained, including sharing the parameters of each group of branch networks, so as to obtain the training of the complete prediction model.

[0042] The following will combine Figure 3 and Figure 5 A detailed explanation of the method for joint training of network parameters; and the following text will also combine... Figure 4 The method for constructing the facial image dataset is explained in detail; however, it will not be repeated here.

[0043] In step 204, the computing device 110 extracts the first feature of the target facial image data via the first network model.

[0044] For example, when the first network model is built based on the Swin-Transformer architecture, the first feature is the multi-scale feature extracted by the Swin-Transformer backbone network. The multi-scale feature is, for example, a multi-scale spatial feature (such as skin texture, facial feature position distribution, etc.).

[0045] In step 206, the computing device 110 extracts a second feature of the target facial image data based on a first feature of the target facial image data via a second network model.

[0046] For example, the second network model is built on a multilayer perceptron (MLP) architecture, and the second feature is a high-level abstract feature extracted by the MLP through nonlinear transformation (such as hidden features that cannot be directly reflected at the geometric level of the facial image).

[0047] In step 208, the computing device 110 generates a predicted vitality status of the subject based on the first feature and the second feature of the target facial image data via a third network model; wherein the vitality status is at least related to the age and health status of the subject.

[0048] Regarding the object, it is, for example, a living being, such as a cat, dog, human, or other creature with a face.

[0049] Vitality status is a novel concept introduced in this publication. It is, for example, a dynamic balance indicator that comprehensively reflects an individual's physiological age and overall life activities, and it is correlated with an individual's physical functions and health status. Furthermore, it is at least related to an individual's age and health status (which may include both physiological and psychological factors), and can also be correlated with other factors such as an individual's emotions and body language. For example, vitality status... y =Total Vitality Value / x ,in, x Indicates age, such as vitality status. y The value range is 0-1, where 0 indicates that vitality has almost ceased and 1 indicates that vitality is abundant. It should be understood that the above is just a simple illustration. The actual vitality status is more complex, and the value range can also be 0-10000, 0-1000, etc., adjusted according to the actual assessment granularity required. Furthermore, vitality status is affected by many other factors besides age (such as health status factors), and cannot be simply represented by total vitality value / age, as the two are not necessarily evenly distributed. Therefore, the above formula is only a quantitative illustration.

[0050] For example, two cats, a 3-year-old domestic cat A and a 3-year-old stray cat B, are the same age, but their health and emotional states are different. Therefore, the predicted vitality of the 3-year-old domestic cat A may be higher than that of the 3-year-old stray cat B.

[0051] For example, Zhang San and Li Si, both 40 years old, have very different predicted life force statuses; similarly, Wang Wu and Li Si, both 30 years old, have almost identical predicted life force statuses.

[0052] Therefore, the above scheme can integrate the first feature extracted by the first network and the second feature extracted by the second network, and perform feature fusion through the third network, thereby constructing a three-stage model framework to separate and extract the facial image of the object, solving the problem that a single model cannot take into account both static features and dynamic changes; based on the multi-level feature extraction and fusion mechanism, the reliability and practicality of the prediction results of vitality status are ensured.

[0053] For example, the third network model is built on a Long Short-Term Memory (LSTM) architecture. The LSTM architecture allows the spatial features (dimension N) of the first network's Swin Transformer and the hidden features (dimension M) of the MLP to be aligned and concatenated through a fully connected layer, forming a spatiotemporal joint input vector (dimension N+M). Then, selective fusion is performed based on the LSTM network's controllable gates (e.g., filtering out irrelevant and interfering features from facial image features and increasing the weight of key features). This allows for the selective fusion of the multi-scale features of the facial image extracted by the first network's Swin Transformer and the hidden features of the facial image extracted by the second network's MLP, based on the training objective, resulting in a more accurate prediction model and thus a more precise prediction of vitality status.

[0054] In the above scheme, a prediction model is constructed by combining three network model frameworks. First, the first and second features are initially extracted (e.g., multi-scale features and hidden features are obtained). Then, complementary features are formed by the first and second features. Finally, the two features are fused together based on the third network to improve the prediction accuracy. Furthermore, a life vitality state quantification parameter is proposed to comprehensively quantify the age and health status of living organisms.

[0055] Furthermore, the specified model architecture (e.g., Swin-transformer+MLP+LSTM) can separate and extract spatiotemporal features, improve the accuracy of LSTM temporal networks through the complementarity of multi-scale features and hidden features; and optimize gradient propagation efficiency through joint training of the three-layer network structure, avoiding error accumulation in multi-model step-by-step training; using LSTM as the third network can enable the prediction model to achieve better improvements in spatiotemporal feature fusion, long-term capture of aging and health, and parameter interpretability.

[0056] Therefore, by using the multi-model collaborative deep learning framework provided by the above scheme, spatiotemporal features can be extracted from facial images to achieve quantitative prediction of the "vitality status" of living organisms; age prediction and health status assessment can be integrated into a unified biometric, and time series modeling can be combined to improve the prediction accuracy of vitality status.

[0057] Figure 4 A flowchart of a method 400 for constructing a facial image dataset according to an embodiment of the present disclosure is shown. Method 400 may be derived from, for example... Figure 1 The computing device 110 shown can be used for execution, and can also be used in Figure 6The method is performed at the illustrated electronic device 600. It should be understood that method 400 may also include additional steps not shown and / or the steps shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0058] In step 402, the computing device 110 collects facial image data of multiple objects at multiple periods to form multiple sets of facial image time-series data; wherein, each set of facial image time-series data includes: multiple facial images of the same object arranged in chronological order.

[0059] For example, in Figure 3 In this context, facial image time series data 01-0n represents a set of facial image time series data, such as several facial images of object 01 between 2010 and 2025. These facial image data are sorted in chronological order to form facial image time series data 01-0n.

[0060] For example, generating each set of facial image time-series data includes at least: collecting a predetermined number of facial images of the same object in each of multiple preset age ranges. For example, for object 01, at least one facial image for each year from 2010 to 2025, or at least five facial images for every three years from 2010 to 2025.

[0061] In step 404, the computing device 110 processes the collected facial image data to determine the maximum age of the object associated with each set of facial image time-series data in a plurality of sets of facial image time-series data.

[0062] For example, in facial image time series data group B2, there is facial image time series data of object 02 (e.g., 02 is a cat, breed is tabby cat), from 0 years old to 15 years old, and it is determined that the maximum age of the object associated with facial image time series data 02 is 15 years old.

[0063] For example, in facial image time series data group A3, there is facial image time series data of object 03 (e.g., 03 is human Zhang San) from age 0 to 80. It is determined that the maximum age of the object associated with facial image time series data 03 is 80 years old.

[0064] For example, in facial image time series data group A4, there is facial image time series data of object 04 (e.g., 04 is human Li Si) from age 3 to 46. It is determined that the maximum age of the object associated with facial image time series data 04 is 46 years old.

[0065] For example, in facial image time series data group C5, there is facial image time series data of object 05 (e.g., 05 is a dog, breed is husky) from 0 years old to 20 years old. It is determined that the maximum age of the object associated with facial image time series data 04 is 20 years old.

[0066] It is worth noting that when training the model, the biological type of each object in the constructed facial image dataset should be the same, based on the type of the object to be predicted. For example, to predict the vitality status of a cat, a facial image dataset of cats needs to be constructed; to predict the vitality status of a dog, a facial image dataset of dogs needs to be constructed; and to predict the vitality status of a human, a facial image dataset of humans needs to be constructed.

[0067] In step 406, the computing device 110 adds at least the current age feature label and the maximum age feature label of the corresponding object to each facial image in each set of facial image time series data to construct a facial image dataset.

[0068] For example, feature tags can be constructed based on various factors that may affect the vitality of an object, such as its type, breed (e.g., Persian cat, tabby cat, Maine Coon; Husky, Akita, Samoyed, etc.), living area (Arctic, Tibet, Yunnan, Egypt, etc.), job type, emotional state, and health status (e.g., suffering from a certain disease, such as feline panleukopenia, depression, bronchitis, pneumonia, etc.).

[0069] It is worth noting that the facial image dataset in this disclosure is constructed from facial images sourced from publicly available facial image databases that have been authorized, or from authorized sources of the associated subject, or from authorized sources of the associated subject's caregiver / guardian, or from other authorized facial image databases.

[0070] For example, continuing the example above, in the facial image time series data group B2, since the maximum age of object 02 is 15 years old, at least the labels "maximum age 15 years old" and "age x years old" are added to each facial image.

[0071] Therefore, the above scheme proposes to construct facial image time series data with maximum age label, which can reflect the facial images of an object during a life cycle and associate the changes in an individual's facial state during a life cycle.

[0072] In some embodiments, constructing a facial image dataset further includes: assigning a total vitality value to the corresponding object based on the maximum age of the corresponding object in each set of facial image time series data, and adding a total vitality value feature label to each facial image in each set of facial image time series data; wherein, the total vitality value of an object is proportional to its maximum age.

[0073] For example, in facial image time series data group B2, the maximum age of object 02 is 15 years old, and its total vitality value is assigned to 1500; for example, in facial image time series data 03, the maximum age of object 03 is 80 years old, and its total vitality value is assigned to 8000.

[0074] In some embodiments, based on the health status information of the corresponding object in each set of facial image time-series data, the time information corresponding to different health status types is determined, so as to add health status feature labels to the facial images of the corresponding object associated with the time information.

[0075] For example, in facial image time series data group B2, subject 02 contracted feline influenza at age 10, and corresponding health status feature labels were added to his facial images at age 10; for example, in facial image time series data group 03, subject 03 developed depression between the ages of 40 and 50, and corresponding health status feature labels were added to his facial images between the ages of 40 and 45; for example, in facial image time series data group A4, subject 04 developed ALS after age 35, and corresponding health status feature labels were added to his facial images after age 35.

[0076] Therefore, the above scheme defines total vitality score and health status label, quantifying the abstract vitality state into a calculable indicator. Based on this, the resulting facial image time-series data can provide predictive models with more time-related facial image data, enabling the discovery of the relationship between an individual's health status, physical decline, and age. Furthermore, the triple labeling system consisting of age + maximum age + health status also helps to integrate the time and physiological dimensions into a comprehensive biological indicator (such as "vitality status" in this scheme), thereby accurately reflecting an individual's physiological state (such as the degree of decline).

[0077] Figure 5 A flowchart of a method 500 for joint training of network parameters according to an embodiment of the present disclosure is shown. Method 500 may be performed by, for example... Figure 1 The computing device 110 shown can be used for execution, and can also be used in Figure 6 The method is performed at the illustrated electronic device 600. It should be understood that method 500 may also include additional steps not shown and / or the steps shown may be omitted; the scope of this disclosure is not limited in this respect. The following is in conjunction with… Figure 3 and Figure 5 The method for joint training of network parameters is further described.

[0078] In step 502, the computing device 110 extracts multi-scale features for each facial image in each set of facial image time series data via the Swin-Transformer backbone network of the first network for multiple sets of facial image time series data in the facial image dataset.

[0079] Regarding multi-scale features, which correspond to the first feature mentioned above, they are multi-layered (shallow basic features, mid-level structural features, and deep semantic features). These features mainly concern the geometric structure and color texture of facial images. For example, shallow basic features include: local edges and textures (such as skin wrinkles and pores), and the distribution of small spots or blood vessels; mid-level structural features include: facial contours (eye shape, nose bridge curvature, lip symmetry), and the location of key facial points (corners of the eyes and mouth); deep semantic features include: overall facial geometry, global illumination and shadow distribution, and other facial features.

[0080] In step 504, the computing device 110, via the multilayer perceptron of the second network, acquires the first predicted age and hidden features of each facial image in each set of facial image time series data based on the extracted multiscale features of the facial images.

[0081] Regarding the hidden features that MLP can extract, which correspond to the second feature mentioned above, for example, physiological state-related features: skin elasticity and the degree of collagen loss (reflecting aging), facial blood flow features (such as paleness / rosiness implying a healthy state); health indicator mapping features: chronic disease manifestations (such as edema suggesting kidney problems, eye bags suggesting metabolic abnormalities), fatigue or stress levels (through fine lines around the eyes, muscle relaxation); age-related degenerative features: soft tissue sagging caused by bone resorption, pigmentation and age spot distribution patterns; and various other more hidden facial feature information.

[0082] In some embodiments, obtaining the first predicted age and hidden features of each facial image in each set of facial image time series data based on the extracted multi-scale features of the facial images via the multilayer perceptron of the second network includes: obtaining the first predicted age of each facial image in each set of facial image time series data based on the extracted multi-scale features of the facial images via the fully connected layer of the multilayer perceptron; and obtaining the hidden features of each facial image in each set of facial image time series data based on the extracted multi-scale features of the facial images via the hidden layer of the fully connected layer of the multilayer perceptron.

[0083] Therefore, the above scheme, based on the MLP network, can obtain age prediction results and hidden features respectively. The hidden features carry age-related (such as aging features) and / or non-age-related health information (such as pathological features), providing key inputs for subsequent feature fusion.

[0084] In step 506, the computing device 110 fuses the hidden features and multi-scale features of each facial image in each set of facial image time series data into the corresponding fused features of the facial image, so as to obtain the predicted vitality status of the object associated with the facial image in each set of facial image time series data through the fused features of the facial image via the long short time memory network.

[0085] Therefore, by fusing multi-scale features and hidden features of facial images based on LSTM networks, it is possible to capture facial changes of the same object over time (such as aging trajectory, such as disease progression). Among them, the multi-scale feature output of Swing Transformer (such as organ contour features, etc.) is used as input to MLP and fused into hidden features through the fully connected layer of MLP, thus forming feature complementarity. For example, spatial details (such as local wrinkles) are captured by the first network, and their implicit association with health / age (such as the quantitative relationship between wrinkle depth and elastin loss) is inferred by the second network. Swing Transformer's window shifting mechanism solves the interference of facial pose changes, and LSTM's long-term memory overcomes the misalignment problem of cross-period images.

[0086] In step 508, the computing device 110 trains a prediction model based on a joint loss function constructed from the first predicted age and the predicted vitality status.

[0087] Therefore, the above method introduces the fusion of multi-scale features and hidden features, and uses a joint loss function based on the dual objectives of age prediction and vitality status prediction to jointly train the prediction model composed of three models. This can optimize gradient propagation efficiency, avoid error accumulation in multi-model step-by-step training, improve the accuracy of LSTM time series modeling through the complementarity of the first and second features, and enhance the model's generalization ability by combining the constraints of the dual loss function.

[0088] Please refer to Figure 3 To further explain the model training process, facial temporal data 01...facial temporal data 0n represent one set of temporal data, which are used as input data for the corresponding branch networks, such as... Figure 3 Each image from the first set of facial temporal data is input into the corresponding branch of the Swin-Transformer network (i.e., the first branch network). Taking the first branch network as an example (e.g.) Figure 3 It includes models such as Swin-Transformer-01 and MLP-01, and describes the training process of the prediction model: Step 311: Input facial temporal data 01 into Swin-Transformer-01.

[0089] Step 312: Extract the first feature of the facial temporal data 01 via Swin-Transformer-01.

[0090] Step 313: Input the first feature of the facial temporal data 01 into MLP-01.

[0091] Step 314: The age of the object corresponding to the face image is predicted to be lower than the predicted age via MLP-01.

[0092] Step 315: Extract the second feature of the facial temporal data 01 via MLP-01.

[0093] Step 316: Input the first feature and hidden feature of the facial temporal data 01 into the LSTM.

[0094] Step 317: Predict the predicted vitality status of the objects corresponding to each facial image in the facial time series data 01 using LSTM.

[0095] For ease of understanding, the following illustrations illustrate the input data for each stage: Input data for Swin-Transformer-01, facial temporal data 01 (time series data s), time series data , Facial images representing a specific point in time (e.g., each image's label includes at least the current age and vital signs). Each x The input is a 3×H×W matrix, where the first dimension represents the number of color channels (RGB), and H and W correspond to the height and width of the image matrix, respectively. s The first features of different facial images are obtained by inputting them into a shared parameter Swin-Transformer module (containing Swin-Transformer-01 to Swin-Transformer-n). . The first predicted age is obtained by passing through a fully connected MLP-01 layer on the facial temporal data 01. In a fully connected layer, the hidden layers represent hidden features (i.e., second features). The second feature is added to the first feature, and then passed through an LSTM network to obtain the predicted vitality status for each facial image in the facial temporal data 01. ,in, In training the prediction model, this invention employs a joint loss function for the first predicted age and vitality status, as shown in the following formula: in, , These represent the true age and vitality status of the sample facial images, respectively (refer to the above section on the correlation between the total vitality value and the maximum age of each group of facial image time series data, and the correlation between vitality status and the age of the current facial image data).

[0096] In some embodiments, a total predicted vitality value for the subject is generated based on the target facial image; a first predicted age of the subject is obtained based on a first feature of the target facial image data via a second network model; and a current predicted vitality value (i.e., the predicted current vitality state) is generated based on the first predicted age and the total predicted vitality value of the subject.

[0097] The predicted current vitality value of the test subject can be, for example, the predicted current vitality value corresponding to the current input facial image; for example, if a facial image a1 of the test subject is input, the predicted current vitality value of facial image a1 is generated; if multiple facial images of the test subject are input, the predicted current vitality value corresponding to each facial image can be output separately. The current vitality value reflects the current vitality state.

[0098] Therefore, the above scheme can reflect an individual's current physiological state by outputting the current value of vitality, thereby reflecting the degree of physiological decline and quantifying individual physiological decline. This scheme can be applied to the field of health management to achieve physiological state assessment.

[0099] Figure 6 A schematic step diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. For example, as... Figure 1 The computing device 110 shown can be implemented by an electronic device 600. As shown, the electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 602 or loaded from storage unit 608 into random access memory (RAM) 603. The random access memory 603 may also store various programs and data required for the operation of the electronic device 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0100] Multiple components in electronic device 600 are connected to input / output interface 605, including: input unit 606, such as keyboard, mouse, microphone, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] The various processes and handling described above, such as methods 300 to 500, can be executed by the central processing unit 601. For example, in some embodiments, methods 300 to 500 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via read-only memory 602 and / or communication unit 609. When the computer program is loaded into random access memory 603 and executed by the central processing unit 601, one or more actions of methods 300 to 500 described above can be performed.

[0102] This disclosure relates to methods, apparatus, systems, electronic devices, computer-readable storage media, and / or computer program products. A computer program product may include computer-readable program instructions for performing various aspects of this disclosure.

[0103] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0104] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge computing devices. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media within the respective computing / processing device.

[0105] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0106] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or step diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each step in the flowchart illustrations and / or step diagrams, as well as combinations of steps in the flowchart illustrations and / or step diagrams, can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more steps of the flowchart and / or diagram of steps. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more steps of the flowchart and / or diagram of steps.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more steps of a flowchart and / or a diagram of steps.

[0109] The flowcharts and step diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each step in a flowchart or step diagram may represent a module, segment, or part of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the step diagrams may occur in a different order than those indicated in the drawings. For example, two consecutive step diagrams may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each step in the step diagrams and / or flowcharts, and combinations of step diagrams in the step diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0110] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for predicting vitality status based on facial images, comprising: The target facial image data of the object to be tested is input into the prediction model, which includes a first network model, a second network model and a third network model. The first feature of the target facial image data is extracted via the first network model; The second feature of the target facial image data is extracted based on the first feature of the target facial image data via the second network model; as well as Through the third network model, based on the first feature and the second feature of the target facial image data, a predicted vitality status of the subject is generated for the target facial image; wherein the vitality status is at least related to the age and health status of the subject.

2. The method according to claim 1, characterized in that, Also includes: The first network model is built based on the Swin-Transformer architecture, the second network model is built based on the multilayer perceptron architecture, and the third network model is built based on the long short-term memory network architecture. as well as Based on the constructed facial image dataset, the first network model, the second network model, and the third network model are jointly trained. The facial image dataset includes facial image data of multiple objects at multiple time periods.

3. The method according to claim 2, characterized in that, The prediction model is trained by the following steps: Through the Swin-Transformer backbone network of the first network, multi-scale features of each facial image in each set of facial image time series data are extracted for multiple sets of facial image time series data in the facial image dataset. Through the multilayer perceptron of the second network, based on the multi-scale features of the extracted facial images, the first predicted age and hidden features of each facial image in each set of facial image time series data are obtained respectively. The hidden features and multi-scale features of each facial image in each set of temporal facial image data are fused into a corresponding fused feature for the facial image. This fused feature is then used by a Long Short-Term Memory (LSTM) network to obtain the predicted vitality status of the object associated with each facial image in the temporal facial image data set. The prediction model is trained based on the joint loss function constructed from the first predicted age and predicted vitality status.

4. The method according to claim 3, characterized in that, Through the multilayer perceptron of the second network, based on the extracted multi-scale features of the facial images, the first predicted age and hidden features of each facial image in each set of temporal data are obtained, including: Through the fully connected layer of the multilayer perceptron, based on the extracted multi-scale features of the facial images, the first predicted age of each facial image in each set of temporal data is obtained; and The hidden features of each facial image in each set of temporal data are obtained through the hidden layer of the fully connected layer of the multilayer perceptron, based on the multi-scale features of the extracted facial images.

5. The method according to claim 1, characterized in that, The facial image dataset was constructed using the following methods: Facial image data of multiple objects at multiple time periods are collected to form multiple sets of temporal data of facial images; each set of temporal data of facial images includes: multiple facial images of the same object arranged in chronological order; and The collected facial image data is processed to determine the maximum age of the object associated with each set of facial image time-series data in the plurality of sets of facial image time-series data; and For each facial image in each set of time-series facial image data, at least the current age feature label and the maximum age feature label of the corresponding object are added to construct a facial image dataset.

6. The method according to claim 5, characterized in that, Building a facial image dataset also includes: Based on the maximum age of the corresponding object in each set of facial image time-series data, a total vitality value is assigned to the corresponding object. For each facial image in each set of facial image time-series data, a total vitality value feature label is added; wherein, the total vitality value of an object is directly proportional to its maximum age; and Based on the health status information of the corresponding object in each set of facial image time-series data, the time information corresponding to different health status types is determined, so as to add health status feature labels to the facial images of the corresponding object that are associated with the time information.

7. The method according to any one of claims 1 to 6, wherein generating a predicted vitality state of the subject based on the target facial image comprises: Generate a predicted total vitality value for the subject of the test based on the target facial image; The first predicted age of the subject is obtained by means of the second network model based on the first feature of the target facial image data; as well as Based on the first predicted age and total predicted vitality of the subject, a current predicted vitality value for the subject is generated.

8. A computing device, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1 to 7 when executed by a machine.

10. A computer program product comprising a computer program that, when executed by a machine, performs the method according to any one of claims 1-7.

Images