Psychological health analysis method, device and equipment based on multi-angle micro-expression recognition

Abstract

The present application relates to a kind of mental health analysis method, device and equipment based on multi-angle micro-expression recognition.The method determines first feature point position set and second feature point position set by feature point detection mode and time sequence prediction mode respectively, then interpolates according to first feature point position set and second feature point position set, determines third feature point position set, can improve the accuracy of feature point positioning, and according to the feature point comparison between the face images collected by multi-angle acquisition device, the confidence of feature point in the face image of non-target image is determined, facial movement unit detection is combined with confidence, can effectively reduce misjudgment, and then improve the accuracy of micro-expression recognition, to improve the reliability of target patient mental health analysis result.

Description

TECHNICAL FIELD

[0001] The present application is suitable for the field of artificial intelligence, and in particular relates to a psychological health analysis method, device and equipment based on multi-angle micro-expression recognition. BACKGROUND

[0002] At present, by analyzing the psychological health of a patient, a medical staff can effectively determine a treatment or health improvement plan for the patient. For example, when a patient gradually starts to express positive emotions during treatment, it may mean that the treatment is working and the psychological state of the patient is gradually improving.

[0003] In the prior art, it is usually necessary to recognize the micro-expression of a patient, and then determine the psychological health analysis result of the patient according to the micro-expression recognition result. Micro-expression is a reaction of facial muscles of a human being after being stimulated by the outside world, which is usually difficult to be consciously controlled and can reflect the real emotions and emotional state of an individual to a certain extent.

[0004] However, since the duration of micro-expression is short, usually between 1 / 25 second and 1 / 5 second, and micro-expression recognition can be disturbed by various environmental factors such as light changes and head posture changes. These interference factors will affect the quality of the face image and the accuracy of feature extraction, thereby reducing the accuracy of micro-expression recognition. Therefore, the misjudgment probability of micro-expression recognition in the prior art is high, and the recognition accuracy is low, so that the psychological health analysis result of the patient cannot be effectively determined, and a reliable psychological health analysis result cannot be given to assist the treatment or health improvement plan of the patient.

[0005] Therefore, how to improve the accuracy of micro-expression recognition has become a problem to be solved. SUMMARY

[0006] Therefore, the embodiments of the present application provide a psychological health analysis method, device and equipment based on multi-angle micro-expression recognition to solve the problem.

[0007] In a first aspect, a psychological health analysis method based on multi-angle micro-expression recognition is provided, and the method comprises:

[0008] Obtaining a face image of a target patient collected by each collection device at each preset time point;

[0009] For any face image, inputting the current face image into a trained face feature point detection model to obtain a first feature point position set of the current face image;

[0010] For any face image, a preset time point corresponding to the face image is used as a reference time point, and the acquisition device corresponding to the face image is used as a reference device. Based on the first feature point position sets corresponding to M face images acquired by the reference device at M preset time points before the reference time point, the trained feature point position prediction model is used to predict the feature point position of the face image to obtain the second feature point position set of the face image, where M is a positive integer.

[0011] Based on the first feature point location set and the second feature point location set of the face image, the third feature point location set of the face image is determined, wherein the third feature point location set includes third position information corresponding to several feature points respectively;

[0012] The face image acquired by the preset target acquisition device is determined as the target image. For any feature point, if the current face image to which the feature point belongs is not the target image, and the target image also contains the feature point, then the confidence level of the feature point in the current face image is determined based on the third position information of the feature point in the target image and the third position information of the feature point.

[0013] Based on the third location information and confidence level of each feature point in each face image, the detection result of each facial motion unit is determined.

[0014] The micro-expression recognition result is determined based on the detection results of each facial motion unit;

[0015] Based on the micro-expression recognition results and the target patient's state information, the psychological health analysis results of the target patient are determined, and the psychological health analysis results are used to assist in optimizing the treatment plan for the target patient.

[0016] Secondly, a mental health analysis device based on multi-angle micro-expression recognition is provided, the device comprising:

[0017] The image acquisition module is used to acquire the facial images of the target patient captured by each acquisition device at each preset time point;

[0018] The feature point detection module is used to input the current face image into the trained face feature point detection model for any face image, and obtain the first feature point location set of the current face image;

[0019] The feature point prediction module is configured to, for any one face image, take a preset time point corresponding to the face image as a reference time point, take a collection device corresponding to the face image as a reference device, use a trained feature point position prediction model to predict the feature point position of the face image according to a first feature point position set corresponding to M face images collected by the reference device at M preset time points before the reference time point, and obtain a second feature point position set of the face image, where M is a positive integer.

[0020] The feature point interpolation module is configured to determine a third feature point position set of the face image according to the first feature point position set and the second feature point position set of the face image, where the third feature point position set includes third position information corresponding to each feature point.

[0021] The confidence determination module is configured to determine a face image collected by a preset target collection device as a target image, for any one feature point, if a current face image to which the feature point belongs is not the target image and the target image also contains the feature point, determine the confidence of the feature point in the current face image according to the third position information of the feature point in the target image and the third position information of the feature point.

[0022] The motion unit detection module is configured to determine the detection result of each facial motion unit according to the third position information of each feature point in each face image and the confidence.

[0023] The micro-expression recognition module is configured to determine a micro-expression recognition result according to the detection result of each facial motion unit.

[0024] The mental health analysis module is configured to determine a mental health analysis result of the target patient according to the micro-expression recognition result and state information of the target patient.

[0025] In a third aspect, an embodiment of the present application provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the mental health analysis method based on multi-angle micro-expression recognition when executing the computer program.

[0026] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the mental health analysis method based on multi-angle micro-expression recognition.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] The first feature point position set and the second feature point position set are respectively determined through a feature point detection manner and a time sequence prediction manner, and then interpolation is performed according to the first feature point position set and the second feature point position set to determine the third feature point position set, so that the accuracy of feature point positioning can be improved. Moreover, the confidence of the feature points in the face image of the non-target image is determined through feature point comparison between the face images collected by the multi-angle acquisition device, and the face motion unit detection is performed in combination with the confidence, so that the misjudgment can be effectively reduced, and the accuracy of micro-expression recognition is improved, thereby improving the reliability of the target patient psychological health analysis result. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 is an application environment schematic diagram of a psychological health analysis method based on multi-angle micro-expression recognition provided by the first embodiment of the present application;

[0031] Figure 2 is a flowchart of a psychological health analysis method based on multi-angle micro-expression recognition provided by the first embodiment of the present application;

[0032] Figure 3 is a structural schematic diagram of a psychological health analysis device based on multi-angle micro-expression recognition provided by the second embodiment of the present application;

[0033] Figure 4 is a structural schematic diagram of a computer device of a psychological health analysis method based on multi-angle micro-expression recognition provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0034] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments without these specific details. In other instances, well-known systems, devices, circuits, and methods have not been described in detail in order to avoid obscuring the present application.

[0035] It should be understood that the word "comprise" or "comprising" when used in this specification and the appended claims specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0036] It should also be understood that the term "and / or" when used in this specification and the appended claims, unless otherwise stated, means any one of the associated listed items or a combination of any combination of at least one of the associated listed items.

[0037] As used in this specification and the appended claims, the term "if" can be interpreted as meaning "when" or "once" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "once it is determined" or "in response to a determination" or "once [the described condition or event] is detected" or "in response to detecting [a described condition or event]" depending on the context.

[0038] In addition, the terms "first", "second", "third", etc. are used in this description and the appended claims only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0039] Reference in this specification to "one embodiment" or "some embodiments" etc. means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in additional embodiments", etc. in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise are specifically stated to be such. The terms "comprise", "comprising", "having", "including", and their variants mean "including but not limited to", unless otherwise specifically stated to be such.

[0040] It should be understood that the size of the serial number of each step in the following embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0041] In order to illustrate the technical solutions of the application, the following specific embodiments are described.

[0042] The psychological health analysis method based on multi-angle micro-expression recognition provided by the embodiment one of the application can be applied to, for example, Figure 1The application environment is shown in FIG. 1. The server communicates with the client. The client includes, but is not limited to, a palm computer, a desktop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cloud terminal device, a personal digital assistant (PDA), and the like. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0043] Referring to Figure 2 FIG. 1 is a flowchart of a psychological health analysis method based on multi-angle micro-expression recognition according to an embodiment of the present application. The psychological health analysis method can be applied to the server in Figure 1 The server can obtain the face images of the target patient collected by each collection device. The server is internally deployed with a trained face feature point detection model, a trained feature point position prediction model, and the like to support micro-expression recognition and psychological health analysis of the face images of the target patient. The psychological health analysis result is sent to the client. The client can be bound to a medical staff. The medical staff can obtain the psychological health analysis result through the client, and then determine or optimize the treatment or health improvement plan of the target patient in combination with the psychological health analysis result. As shown in FIG. 1, the psychological health analysis method can include the following steps: Figure 2

[0044] In step S201, the face images of the target patient collected by each collection device at each preset time point are obtained.

[0045] The collection device can be a depth camera. The image collection angles of different collection devices are different. As an example, the number of collection devices can be three, which correspond to the front view angle, the left view angle, and the right view angle of the face of the target patient. It should be noted that the intrinsic and extrinsic parameters of each collection device are known priori information. The intrinsic parameters of the collection device can be determined by a calibration method. The extrinsic parameters of the collection device can be determined according to the inertial measurement unit built in the collection device. The specific implementation of the acquisition of the intrinsic and extrinsic parameters of the collection device is not described here.

[0046] Since the duration of micro-expression is short, the collection frequency of the collection device should be high. As an example, the collection frequency of the collection device can be 60 frames per second or 120 frames per second. Taking 120 frames per second as an example, the time interval between adjacent preset time points is 1 / 120 seconds.

[0047] ​The target patient can refer to a patient in need of mental health analysis. It should be noted that the present application does not diagnose and treat the target patient, but performs mental health analysis on the target patient, and then provides reference information to assist medical personnel in determining or adjusting the treatment or health improvement plan for the target patient.

[0048] In step S202, for each face image, the current face image is input into the trained face feature point detection model to obtain a first feature point position set of the current face image.

[0049] The face feature point detection model can be used to output the face feature points contained in the input face image according to the input face image. The face feature points include different feature point categories. As an example, the face feature points can include chin vertex, cheek key point, eye corner point, eyelid point, eyebrow starting point, eyebrow ending point, eyebrow peak point, nose tip point, nose root point, nose wing point, and mouth corner point.

[0050] The first feature point position set can include a plurality of feature points and corresponding first position information respectively. The first position information can refer to the coordinate information of the corresponding feature point in the current face image.

[0051] Specifically, the face feature point detection model can be selected from HRNet, MTCNN, etc. Since the present application processes multi-angle face images, the training set of the face feature point detection model can be selected from a training dataset with multi-angle sample face images such as RaFD training set.

[0052] Optionally, the current face image is input into the trained face feature point detection model to obtain the first feature point position set of the current face image, including:

[0053] The current face image is input into the trained face feature point detection model to obtain a heat map corresponding to the current face image. The heat map contains a plurality of channels, and each channel corresponds to a feature point.

[0054] Each channel in the heat map is searched for a maximum value, and the coordinates corresponding to the maximum value search result of each channel are taken as the first position information of the corresponding feature point.

[0055] The first position information of each feature point corresponding to the current face image forms the first feature point position set of the current face image.

[0056] For any channel contained in the heat map, the channel contains a heat distribution of the corresponding feature point, and the heat distribution is a Gaussian distribution. The heat value of each position in the heat distribution can represent the probability that each position is the corresponding feature point.

[0057] The maximum value search can be implemented by using argmax, soft-argmax, or the like.

[0058] In step S203, for each face image, a preset time point corresponding to the face image is taken as a reference time point, a collection device corresponding to the face image is taken as a reference device, and a trained feature point position prediction model is used to predict the feature point positions of the face image according to the first feature point position sets of M face images collected by the reference device at M preset time points before the reference time point, to obtain a second feature point position set of the face image.

[0059] M is a positive integer, for each face image, there are a preset time point and a collection device for collecting the face image, and each preset time point has a time sequence order, so that the M preset time points before the reference time point can be determined according to the reference time point.

[0060] The feature point position prediction model can be a time series prediction model, which can be an LSTM model, a TCN model, or the like.

[0061] The second feature point position set includes a plurality of feature points in the face image and second position information corresponding to each feature point.

[0062] Optionally, using the trained feature point position prediction model to predict the feature point positions of the face image according to the first feature point position sets of M face images collected by the reference device at M preset time points before the reference time point, to obtain a second feature point position set of the face image, includes:

[0063] For any feature point in the face image, the trained feature point position prediction model is used to predict the position of the feature point according to the first position information of the feature point in the M face images collected by the reference device at the M preset time points before the reference time point, to obtain the second position information of the feature point in the face image.

[0064] All feature points in the face image are traversed to obtain the second position information of each feature point contained in the face image, to form a second feature point position set of the face image.

[0065] When the feature point position prediction model is used, the feature points are processed as objects, for each feature point, the first position information of the feature point at the M preset time points is input into the trained feature point position prediction model, and the predicted position information of the feature point at the reference time point is output as the second position information.

[0066] It should be noted that if there are only N preset time points before the reference time point, N is less than M, then M-N preset position information is supplemented before the first position information corresponding to the first preset time point to ensure the consistency of the input data size of the trained feature point position prediction model, and the horizontal and vertical coordinates of the preset position information can be 0.

[0067] In step S204, a third feature point position set of the face image is determined according to the first feature point position set and the second feature point position set of the face image.

[0068] The third feature point position set includes third position information corresponding to a plurality of feature points.

[0069] Optionally, the third feature point position set of the face image is determined according to the first feature point position set and the second feature point position set of the face image, including:

[0070] For any feature point in the face image, the first position information and the second position information of the feature point are calculated by interpolation to obtain the third position information of the feature point.

[0071] All feature points in the face image are traversed to obtain third position information corresponding to all feature points in the face image, and a third feature point position set of the face image is formed.

[0072] The interpolation calculation can calculate the horizontal coordinate mean value and the vertical coordinate mean value according to the first position information and the second position information of the feature point, and determine the third position information according to the horizontal coordinate mean value and the vertical coordinate mean value.

[0073] Specifically, since the pose of the acquisition device is fixed during the acquisition process and the acquisition frequency is high, the feature points in the face images collected by the same acquisition device at different preset time points are consistent by default.

[0074] Since the feature point position is determined only according to the current face image in face feature point detection, and the feature point time sequence prediction is only based on the feature point position prediction of the face image before the reference image, both may have false positives and cause low accuracy, therefore, the third position information corresponding to the feature point is determined by combining the results of face feature point detection and feature point time sequence prediction, thereby improving the reliability and accuracy of feature point positioning.

[0075] In step S205, it is determined that the face image collected by the preset target collection device is a target image. For any feature point, if the current face image to which the feature point belongs is not the target image, and the target image also contains the feature point, then the confidence of the feature point in the current face image is determined according to the third position information of the feature point in the target image and the third position information of the feature point.

[0076] In the embodiment, the collection device corresponding to the front view angle is taken as the target collection device.

[0077] Specifically, the preset time point corresponding to the current face image to which the feature point belongs is taken as a target time point. For any feature point, if the current face image to which the feature point belongs at the target time point is not the target image collected at the target time point, and the target image collected at the target time point also contains the feature point, that is, the current face image collected at the target time point and the target image both contain the feature point, then the confidence of the feature point in the current face image collected at the target time point is determined according to the third position information of the feature point in the target image collected at the target time point and the third position information of the feature point. The confidence can represent the reliability of the third position of the corresponding feature point in the current face image.

[0078] Optionally, each collection device has a depth collection function.

[0079] For any feature point, if the current face image to which the feature point belongs is not the target image, and the target image also contains the feature point, then the confidence of the feature point in the current face image is determined according to the third position information of the feature point in the target image and the third position information of the feature point, including:

[0080] For any feature point, if the current face image to which the feature point belongs is not the target image, and the target image also contains the feature point, then the first camera coordinate information corresponding to the feature point is determined according to the depth information and the third position information of the feature point in the current face image.

[0081] The second camera coordinate information corresponding to the feature point is determined according to the depth information and the third position information of the feature point in the target image.

[0082] The second camera coordinate information is mapped to third camera coordinate information in the camera coordinate system of the collection device corresponding to the current face image according to the mapping relationship between the collection device corresponding to the current face image and the collection device corresponding to the target image.

[0083] The mapping difference distance corresponding to the feature point is determined according to the third camera coordinate information and the first camera coordinate information.

[0084] According to the mapping difference distance, the confidence of the feature point in the current face image is determined.

[0085] Each of the acquisition devices is a depth camera to acquire depth information of the face.

[0086] The first camera coordinate information can be coordinate information of the feature point in a camera coordinate system of an acquisition device corresponding to the current face image, the second coordinate information can be coordinate information of the feature point in a camera coordinate system of an acquisition device corresponding to the target image, since the intrinsic parameters of each acquisition device are known, the corresponding camera coordinate information in the camera coordinate system can be determined according to the third position information and the corresponding depth information, since the extrinsic parameters of each acquisition device are known, the mapping relationship between the camera coordinate systems of any two acquisition devices can be determined, and the third coordinate information can be coordinate information of the feature point in the camera coordinate system of the acquisition device corresponding to the target image mapped into the camera coordinate system of the acquisition device corresponding to the current face image.

[0087] The mapping difference distance can be calculated by the Euclidean distance, and the confidence of the feature point in the current face image can be obtained by normalizing the mapping difference distance. The specific normalization method can be normalization processing according to the max-min method according to the mapping difference distances of all feature points contained in the current face image and the target image.

[0088] Optionally, after determining the mapping difference distance of the feature point according to the third camera coordinate information and the first camera coordinate information, the method further includes:

[0089] According to the third position information of the feature point in the target image, and in combination with the channel of the feature point in the heat map corresponding to the target image, a first reference confidence of the feature point is determined.

[0090] According to the third position information of the feature point in the current face image, and in combination with the channel of the feature point in the heat map corresponding to the current face image, a second reference confidence of the feature point is determined.

[0091] Accordingly, the confidence of the feature point in the current face image is determined according to the mapping difference distance, including:

[0092] The confidence of the feature point in the current face image is determined according to the mapping difference distance, the first reference confidence and the second reference confidence.

[0093] The first reference confidence can represent the reliability of the third position information of the feature point in the target image, and the second reference confidence can represent the reliability of the third position information of the feature point in the current face image.

[0094] Specifically, the mapping difference distance is normalized to obtain a normalized result, the normalized result is multiplied by the first reference confidence, and then multiplied by the second reference confidence, and the final multiplication result is taken as the confidence of the feature point in the current face image.

[0095] In step S206, the detection result of each facial movement unit is determined according to the third position information and the confidence of each feature point in each face image.

[0096] The facial movement unit can be a basic unit in a facial action coding system, and a single facial movement unit corresponds to a specific facial muscle movement. For example, AU1 represents inner eyebrow up, which is mainly caused by the contraction of the medial frontalis bundle; AU4 represents frowning, which is caused by the contraction of the corrugator supercilii and procerus muscles.

[0097] The detection result of a single facial movement unit can include presence and absence.

[0098] Optionally, the detection result of each facial movement unit is determined according to the third position information and the confidence of each feature point in each face image.

[0099] For any acquisition device, if the feature points associated with any facial movement unit are present in each face image collected by the acquisition device, the initial judgment result and the initial judgment confidence of the facial movement unit are determined according to the third position information and the confidence of each feature point in each face image collected by the acquisition device.

[0100] All acquisition devices are traversed, and the detection result of the facial movement unit is determined according to all initial judgment results and initial judgment confidences of the facial movement unit.

[0101] Since the acquisition angles of the acquisition devices are different, there can be a situation where the feature points contained in the face image are not complete. When the feature points associated with any facial movement unit are present in each face image collected by the acquisition device at different preset time points, it can be considered that the detection of the facial movement unit can be determined according to the third position information and the confidence of each feature point in each face image collected by the acquisition device.

[0102] Specifically, the third position information of each feature point in each face image collected by the acquisition device is input into the trained movement unit detection model corresponding to the facial movement unit, and the detection result of the facial movement unit is output. The output result is taken as the initial judgment result, and the average of the confidences of the third position information of each feature point is taken as the initial judgment confidence. Each facial movement unit corresponds to a trained movement unit detection model, and the trained movement unit detection model can be a binary classification model, and the classification result is presence and absence.

[0103] It can be known that the feature points associated with the same facial motion unit can exist in the face images collected by multiple collection devices, and therefore, a single facial motion unit can correspond to multiple preliminary determination results. Since the trained motion unit detection model is a binary classification model, the reference confidence of the facial motion unit can be determined according to the preliminary determination result and the confidence thereof. That is, if the preliminary determination result is that the facial motion unit exists, the confidence of the preliminary determination result is the reference confidence. If the preliminary determination result is that the facial motion unit does not exist, the difference between 1 and the confidence of the preliminary determination result is the reference confidence. The mean value of the reference confidence of all preliminary determination results is calculated. If the mean value is greater than a preset threshold, it is determined that the detection result of the facial motion unit is that the facial motion unit exists. Otherwise, it is determined that the detection result of the facial motion unit is that the facial motion unit does not exist. As an example, the preset threshold can be 0.5.

[0104] In step S207, the micro-expression recognition result is determined according to the detection result of each facial motion unit.

[0105] The micro-expression recognition result can be determined according to the facial motion unit whose detection result is that the facial motion unit exists. The corresponding relationship between the micro-expression and the facial motion unit associated with the micro-expression is known. Therefore, the micro-expression recognition result can be determined according to the facial motion unit whose detection result is that the facial motion unit exists. The micro-expression recognition result can include multiple micro-expression types. As an example, the micro-expression types can include surprise, fear, anger, disgust, sadness, joy, contempt, and the like.

[0106] In step S208, the mental health analysis result of the target patient is determined according to the micro-expression recognition result and the state information of the target patient.

[0107] The mental health analysis result is used to assist in optimizing the treatment plan of the target patient. The state information can include the disease type of the target patient, the treatment cycle, the interaction information when the face image is collected, and the like. The mental health analysis result can include abnormal and normal.

[0108] Specifically, the micro-expression recognition result of the target patient can be determined according to the micro-expression recognition result and the state information of the target patient. When the micro-expression recognition result of the target patient does not conform to the expected psychological state, the mental health analysis result is abnormal. When the micro-expression recognition result of the target patient conforms to the expected psychological state, the mental health analysis result is normal. The medical staff can adjust, formulate, or maintain the current treatment plan of the target patient according to the mental health analysis result. As an example, when the disease type of the target patient is post-traumatic stress disorder, the interaction information when the face image is collected is a traumatic event, and the micro-expression recognition result is a fear type, it can be considered that the micro-expression recognition result of the target patient does not conform to the expected psychological state, and the mental health analysis result of the target patient is abnormal.

[0109] In this embodiment, the first feature point position set and the second feature point position set are respectively determined through the feature point detection mode and the time sequence prediction mode, and then the third feature point position set is determined through interpolation according to the first feature point position set and the second feature point position set, so as to improve the accuracy of feature point positioning. Moreover, the confidence of the feature points in the face image of the non-target image is determined through feature point comparison between the face images collected by the multi-angle collection device, and the face motion unit detection is performed in combination with the confidence, so as to effectively reduce the misjudgment and further improve the accuracy of micro-expression recognition, thereby improving the reliability of the psychological health analysis result of the target patient.

[0110] corresponding to the method of the above embodiment, Figure 3 A structure schematic diagram of a psychological health analysis device based on multi-angle micro-expression recognition provided by Embodiment Two of the present application is shown. The psychological health analysis device is applied to a server. The server can obtain the face images of the target patient collected by each collection device. The server is internally deployed with a trained face feature point detection model, a trained feature point position prediction model, etc., to support micro-expression recognition and psychological health analysis on the face images of the target patient, and then send the psychological health analysis result to a client. The client can be bound with medical staff. The medical staff can obtain the psychological health analysis result through the client, and then determine or optimize the treatment or health improvement plan of the target patient in combination with the psychological health analysis result. For ease of illustration, only parts related to the embodiments of the present application are shown.

[0111] Referring to Figure 3 The psychological health analysis device comprises:

[0112] An image acquisition module 31 is configured to acquire the face images of the target patient collected by each collection device at each preset time point.

[0113] A feature point detection module 32 is configured to, for any face image, input the current face image into the trained face feature point detection model to obtain a first feature point position set of the current face image.

[0114] A feature point prediction module 33 is configured to, for any face image, take the preset time point corresponding to the face image as a reference time point, take the collection device corresponding to the face image as a reference device, and use the trained feature point position prediction model to perform feature point position prediction on the face image according to the first feature point position sets of the M face images collected by the reference device at the M preset time points before the reference time point, to obtain a second feature point position set of the face image, where M is a positive integer.

[0115] The feature point interpolation module 34 is configured to determine a third feature point position set of the face image according to the first feature point position set and the second feature point position set of the face image, the third feature point position set including third position information corresponding to each feature point;

[0116] The confidence determination module 35 is configured to determine a face image collected by a preset target collection device as a target image, and for any feature point, if the current face image to which the feature point belongs is not the target image and the target image also contains the feature point, determine a confidence of the feature point in the current face image according to the third position information of the feature point in the target image and the third position information of the feature point.

[0117] The motion unit detection module 36 is configured to determine a detection result of each facial motion unit according to the third position information of each feature point in each face image and the confidence.

[0118] The micro-expression recognition module 37 is configured to determine a micro-expression recognition result according to the detection result of each facial motion unit.

[0119] The mental health analysis module 38 is configured to determine a mental health analysis result of the target patient according to the micro-expression recognition result and state information of the target patient.

[0120] Optionally, the feature point detection module 32 includes:

[0121] The heatmap regression unit is configured to input the current face image into the trained face feature point detection model to obtain a heatmap corresponding to the current face image, the heatmap including a plurality of channels, and each channel corresponding to a feature point.

[0122] The first position determination unit is configured to perform maximum value searching on each channel of the heatmap respectively, and take coordinates corresponding to a maximum value searching result of each channel as first position information of a corresponding feature point.

[0123] The first set determination unit is configured to form the first feature point position set of the current face image from the first position information corresponding to each feature point included in the current face image.

[0124] Optionally, the feature point prediction module 33 includes:

[0125] The second position determination unit is configured to, for any feature point in the face image, perform position prediction of the feature point using the trained feature point position prediction model according to the first position information corresponding to the feature point in the M face images collected by the reference device at the M preset time points before the reference time point, to obtain second position information of the feature point in the face image.

[0126] The second set determination unit is configured to traverse all the feature points in the face image to obtain second position information of each feature point contained in the face image, and form a second feature point position set of the face image.

[0127] Optionally, the feature point interpolation module 34 comprises:

[0128] The third position determination unit is configured to, for any feature point in the face image, perform interpolation calculation according to the first position information and the second position information of the feature point, and take the interpolation calculation result as third position information of the feature point.

[0129] The third set determination unit is configured to traverse all the feature points in the face image to obtain third position information corresponding to each feature point in the face image, and form a third feature point position set of the face image.

[0130] Optionally, each acquisition device has a depth acquisition function.

[0131] The confidence determination module 35 comprises:

[0132] The first coordinate conversion unit is configured to, for any feature point, if the current face image to which the feature point belongs is not the target image and the target image also contains the feature point, determine first camera coordinate information corresponding to the feature point according to the depth information and the third position information of the feature point in the current face image.

[0133] The second coordinate conversion unit is configured to determine second camera coordinate information corresponding to the feature point according to the depth information and the third position information of the feature point in the target image.

[0134] The third coordinate conversion unit is configured to map the second camera coordinate information to third camera coordinate information in a camera coordinate system of an acquisition device corresponding to the current face image according to a mapping relationship between the acquisition device corresponding to the current face image and an acquisition device corresponding to the target image.

[0135] The distance determination unit is configured to determine a mapping difference distance corresponding to the feature point according to the third camera coordinate information and the first camera coordinate information.

[0136] The confidence determination unit is configured to determine a confidence of the feature point in the current face image according to the mapping difference distance.

[0137] Optionally, the confidence determination module 35 further comprises:

[0138] The first reference confidence determination subunit is used to determine the first reference confidence of the feature point based on the third position information of the feature point in the target image and the channel of the feature point in the heat map corresponding to the target image.

[0139] The second reference confidence determination subunit is used to determine the second reference confidence of the feature point based on the third position information of the feature point in the current face image and the channel of the feature point in the heat map corresponding to the current face image.

[0140] Accordingly, the confidence level determination unit mentioned above includes:

[0141] The confidence level comprehensive determination subunit is used to determine the confidence level of the feature point in the current face image based on the mapping difference distance, the first reference confidence level, and the second reference confidence level.

[0142] Optionally, the motion unit detection module 36 mentioned above includes:

[0143] The motion initial judgment unit is used to determine the initial judgment result and initial judgment confidence of the facial motion unit based on the third position information and confidence of each feature point in each face image acquired by the acquisition device, if all the feature points associated with any facial motion unit exist in each face image acquired by the acquisition device.

[0144] The motion detection unit is used to traverse all acquisition devices and determine the detection result of the facial motion unit based on all preliminary judgment results and preliminary judgment confidence levels.

[0145] It should be noted that the information interaction and execution process between the above modules, units, and sub-units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0146] Figure 4 This is a schematic diagram of the structure of a computer device for a mental health analysis method based on multi-angle micro-expression recognition, as provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and capable of running on at least one processor, which, when executing the computer program, implements the steps in any of the above embodiments of the mental health analysis method based on multi-angle micro-expression recognition.

[0147] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4The computer device is merely an example and does not limit the computer device, which can include more or less components than shown, or combine some components, or have different components, such as a network interface, a display screen, an input device, and the like.

[0148] The processor can be a CPU, and can also be other general-purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0149] The memory includes a readable storage medium, an internal memory, and the like. The internal memory can be a memory of the computer device, and provides an environment for running an operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of the computer device, and in other embodiments can also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, and the like. Further, the memory can include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a BootLoader, data, and other programs, such as program codes of computer programs. The memory can also be used to temporarily store data that has been output or will be output.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above device can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here. If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the above-mentioned embodiment methods, which can be realized by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer readable medium cannot be electric carrier signal and telecommunication signal.

[0151] The present application realizes all or part of the processes in the above-mentioned embodiment methods, which can also be completed by a computer program product. When the computer program product runs on the computer equipment, it makes the computer equipment execute the steps in the above-mentioned method embodiment.

[0152] In the above-mentioned embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0153] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0154] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other manners. For example, the described apparatus / computer device embodiments are merely schematic. For example, the division of the modules or units can be different, and each can include a plurality of sub-units. Some or all of the modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0155] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0156] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for psychological health analysis based on multi-angle micro-expression recognition, characterized in that, The method includes: Acquire the facial images of the target patient captured by each acquisition device at each preset time point; For any face image, input the current face image into the trained face feature point detection model to obtain the first feature point location set of the current face image; For any face image, a preset time point corresponding to the face image is used as a reference time point, and the acquisition device corresponding to the face image is used as a reference device. Based on the first feature point position sets corresponding to M face images acquired by the reference device at M preset time points before the reference time point, the trained feature point position prediction model is used to predict the feature point position of the face image to obtain the second feature point position set of the face image, where M is a positive integer. Based on the first feature point location set and the second feature point location set of the face image, the third feature point location set of the face image is determined, wherein the third feature point location set includes third position information corresponding to several feature points respectively; The face image acquired by the preset target acquisition device is determined as the target image. For any feature point, if the current face image to which the feature point belongs is not the target image, and the target image also contains the feature point, then the confidence level of the feature point in the current face image is determined based on the third position information of the feature point in the target image and the third position information of the feature point. Based on the third location information and confidence level of each feature point in each face image, the detection result of each facial motion unit is determined. The micro-expression recognition result is determined based on the detection results of each facial motion unit; Based on the micro-expression recognition results and the target patient's state information, the psychological health analysis results of the target patient are determined, and the psychological health analysis results are used to assist in optimizing the treatment plan for the target patient.

2. The mental health analysis method according to claim 1, characterized in that, The step of inputting the current face image into a trained face feature point detection model to obtain the first feature point location set of the current face image includes: The current face image is input into a trained face feature point detection model to obtain a heat map corresponding to the current face image. The heat map contains several channels, and each channel corresponds to a feature point. For each channel in the heatmap, a maximum value search is performed, and the coordinates corresponding to the maximum value search result for each channel are used as the first position information of the corresponding feature point. The first feature point position set of the current face image is formed by the first position information corresponding to each feature point contained in the current face image.

3. The mental health analysis method according to claim 2, characterized in that, The step involves using a trained feature point location prediction model to predict the feature point locations of the face image based on the first feature point location set corresponding to the M face images acquired by the reference device at M preset time points prior to the reference time point, thereby obtaining the second feature point location set of the face image, including: For any feature point in the face image, based on the first position information corresponding to the feature point in M ​​face images collected by the reference device at M preset time points before the reference time point, the trained feature point position prediction model is used to predict the position of the feature point, thereby obtaining the second position information of the feature point in the face image. By traversing all feature points in the face image, the second location information of each feature point contained in the face image is obtained, forming the second feature point location set of the face image.

4. The mental health analysis method according to claim 3, characterized in that, Determining the third set of feature points of the face image based on the first and second set of feature point locations includes: For any feature point in the face image, interpolation calculation is performed based on the first and second position information of the feature point, and the interpolation result is used as the third position information of the feature point. By traversing all feature points in the face image, the third position information corresponding to each feature point in the face image is obtained, forming a set of third feature point positions for the face image.

5. The mental health analysis method according to claim 4, characterized in that, Each acquisition device is equipped with depth acquisition capabilities; For any feature point, if the current face image to which the feature point belongs is not the target image, but the target image also contains the feature point, then the confidence level of the feature point in the current face image is determined based on the third position information of the feature point in the target image and the third position information of the feature point, including: For any feature point, if the current face image to which the feature point belongs is not the target image, and the target image also contains the feature point, then the first camera coordinate information corresponding to the feature point is determined based on the depth information and third position information of the feature point in the current face image. Based on the depth information and third position information of the feature point in the target image, determine the second camera coordinate information corresponding to the feature point; Based on the mapping relationship between the acquisition device corresponding to the current face image and the acquisition device corresponding to the target image, the second camera coordinate information is mapped to the third camera coordinate information in the camera coordinate system of the acquisition device corresponding to the current face image; Based on the coordinate information of the third camera and the coordinate information of the first camera, determine the mapping difference distance corresponding to the feature point; The confidence level of the feature point in the current face image is determined based on the mapping difference distance.

6. The mental health analysis method according to claim 5, characterized in that, After determining the mapping difference distance corresponding to the feature point based on the third camera coordinate information and the first camera coordinate information, the method further includes: Based on the third location information of the feature point in the target image, and combined with the channel of the feature point in the heat map corresponding to the target image, the first reference confidence level of the feature point is determined. Based on the third location information of the feature point in the current face image, and combined with the channel of the feature point in the heatmap corresponding to the current face image, the second reference confidence of the feature point is determined. Accordingly, determining the confidence level of the feature point in the current face image based on the mapping difference distance includes: The confidence level of the feature point in the current face image is determined based on the mapping difference distance, the first reference confidence level, and the second reference confidence level.

7. The mental health analysis method according to claim 1, characterized in that, The step of determining the detection result of each facial motion unit based on the third location information and confidence level of each feature point in each face image includes: For any acquisition device, if the feature points associated with any facial motion unit are present in all the face images acquired by the acquisition device, then the preliminary judgment result and the preliminary judgment confidence of the facial motion unit are determined based on the third position information and confidence of each feature point in each face image acquired by the acquisition device. By traversing all acquisition devices and based on all preliminary judgment results and the preliminary judgment confidence level of the facial motion unit, the detection result of the facial motion unit is determined.

8. A mental health analysis device based on multi-angle micro-expression recognition, characterized in that, The device includes: The image acquisition module is used to acquire the facial images of the target patient captured by each acquisition device at each preset time point; The feature point detection module is used to input the current face image into the trained face feature point detection model for any face image, and obtain the first feature point location set of the current face image; The feature point prediction module is used to predict the feature point positions of any face image by taking a preset time point corresponding to the face image as a reference time point and the acquisition device corresponding to the face image as a reference device, and using a trained feature point position prediction model to predict the feature point positions of the face image based on the first feature point position sets corresponding to M face images acquired by the reference device at M preset time points before the reference time point, respectively, so as to obtain the second feature point position set of the face image, where M is a positive integer; The feature point interpolation module is used to determine the third feature point location set of the face image based on the first feature point location set and the second feature point location set of the face image. The third feature point location set includes the third location information corresponding to several feature points respectively. The confidence determination module is used to determine that the face image acquired by the preset target acquisition device is the target image. For any feature point, if the current face image to which the feature point belongs is not the target image, and the target image also contains the feature point, then the confidence of the feature point in the current face image is determined based on the third position information of the feature point in the target image and the third position information of the feature point. The motion unit detection module is used to determine the detection results of each facial motion unit based on the third position information and confidence level of each feature point in each face image. The micro-expression recognition module is used to determine the micro-expression recognition result based on the detection results of each facial motion unit; The mental health analysis module is used to determine the mental health analysis results of the target patient based on the micro-expression recognition results and the target patient's state information.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the mental health analysis method based on multi-angle micro-expression recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the mental health analysis method based on multi-angle micro-expression recognition as described in any one of claims 1 to 7.

Images