Psychological state information identification method and device based on data fusion

Abstract

The invention provides a psychological state information recognition method and device based on data fusion, and the method comprises the steps: carrying out the continuous N times of face photographing of a target person, and obtaining a face image sequence, N being a positive integer greater than 20; acquiring a skin conductance signal of the target person to obtain a signal sequence; inputting the face image sequence into an image feature extractor to obtain face image features; the signal sequence is input into a signal feature extractor, signal features are obtained, and the signal features and the face image features belong to different feature spaces; inputting the face image features into a mapper to obtain face image mapping features; and performing psychological state recognition based on the face image mapping features and the signal features to obtain a psychological state recognition result. The method can significantly improve the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical treatment, and in particular to a method and device for identifying psychological state information based on data fusion. Background Art

[0002] Psychological state information recognition based on data fusion is a technology that uses artificial intelligence technology to collect and analyze multi-dimensional physiological signals and behavioral data to identify individual psychological states. The core of this method is to integrate data from different sensors to improve the accuracy and reliability of psychological state recognition. However, the existing technology is still relatively superficial in its research, the accuracy of psychological state recognition is not enough, and no technical solution that can meet actual needs has been formed. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a method and device for identifying psychological state information based on data fusion. The present invention is specifically implemented by the following technical solutions: On the one hand, the present application discloses a method for identifying mental state information based on data fusion, comprising: Take N consecutive facial photos of the target person to obtain a facial image sequence, where N is a positive integer greater than 20; collecting skin conductance signals of the target person to obtain a signal sequence; Inputting the facial image sequence into an image feature extractor to obtain facial image features; Inputting the signal sequence into a signal feature extractor to obtain signal features, wherein the signal features and the face image features belong to different feature spaces; Inputting the facial image features into a mapper to obtain facial image mapping features; Psychological state recognition is performed based on the facial image mapping features and the signal features to obtain a psychological state recognition result.

[0004] On the one hand, the present application discloses a mental state information recognition device based on data fusion, comprising: The data acquisition module is used to take N consecutive facial photos of the target person to obtain a facial image sequence, where N is a positive integer greater than 20; The data processing module is used to perform the following operations: collecting skin conductance signals of the target person to obtain a signal sequence; Inputting the facial image sequence into an image feature extractor to obtain facial image features; Inputting the signal sequence into a signal feature extractor to obtain signal features, wherein the signal features and the face image features belong to different feature spaces; Inputting the facial image features into a mapper to obtain facial image mapping features; Psychological state recognition is performed based on the facial image mapping features and the signal features to obtain a psychological state recognition result.

[0005] On the one hand, the present application discloses a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the aforementioned data fusion-based mental state information recognition method.

[0006] On the one hand, the present application discloses a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the mental state information recognition method based on data fusion as described above.

[0007] On the one hand, the present application discloses a computer program product, which includes computer instructions, the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the aforementioned data fusion-based psychological state information recognition method.

[0008] The present application proposes a method and device for identifying psychological state information based on data fusion. The technical solution sets a mapper to unify the signal features and facial image features that originally belong to different feature spaces in the feature space, thereby enhancing the fusion effect of data of different modalities, thereby making full use of the information related to psychological state recognition in data of different modalities, significantly improving the accuracy of psychological state recognition, and obtaining an automatic psychological state recognition solution that can fully meet actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 A method for identifying psychological state information based on data fusion provided by an embodiment of the present invention; Figure 2is a flow chart of a second extractor training method provided by an embodiment of the present invention; Figure 3 It is a flow chart of a signal feature extraction method provided by an embodiment of the present invention; Figure 4 is a flow chart of a psychological state identifier training method provided by an embodiment of the present invention; Figure 5 It is a block diagram of a mental state information recognition device based on data fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0012] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0013] The embodiment of the present invention discloses a method for identifying psychological state information based on data fusion, such as Figure 1 As shown, the method includes: S101. Take N consecutive facial photos of a target person to obtain a facial image sequence, where N is a positive integer greater than 20.

[0014] The target person is the person whose psychological state needs to be identified.

[0015] S102. Collect skin conductance signals of the target person to obtain a signal sequence.

[0016] Specifically, collecting the skin conductance signal of the target person to obtain a signal sequence includes: Collect the skin conductance signal of the target person to obtain an initial signal sequence ; This application does not limit the initial signal sequence The length can be set as needed during the specific implementation process, and S102 and S101 can be executed simultaneously or sequentially.

[0017] Based on the initial signal sequence , and get the intermediate signal sequence ,in, is the preset first adjustment parameter; based on the intermediate signal sequence , get the reference signal sequence , , They are the second adjustment parameter and the third adjustment parameter respectively; the first adjustment parameter, the second adjustment parameter and the third adjustment parameter can all be set according to actual needs. These three parameters are experimental data, which can be adjusted based on a limited number of experiments and are related to actual needs, so they do not constitute implementation obstacles.

[0018] Fusion initial signal sequence and reference signal sequence , and obtain the signal sequence The present application embodiment does not limit the fusion method, for example, it can take the average or weighted fusion, and the weight is not limited. In an exemplary implementation, the formula can be used To fuse the initial signal sequence and reference signal sequence ,in It can be set according to actual needs. It is experimental data. It can be adjusted according to a limited number of experiments and is related to actual needs. Therefore, it does not constitute an implementation obstacle. This implementation method can improve The indication sensitivity of the signal is significantly improved, which significantly improves the accuracy of psychological state prediction. Of course, the first signal of each signal sequence is the same. Then, the values ​​of other signals are recursively inferred by increasing the value of k. This process is self-explanatory.

[0019] S103. Input the facial image sequence into an image feature extractor to obtain facial image features.

[0020] Specifically, the image feature extractor includes a first extractor, a second extractor and a fusion device, the first extractor is an ArcFace face encoder, and the face image sequence is input into the image feature extractor to obtain face image features, including: inputting the face image sequence into the first extractor to obtain features corresponding to each face image output by the first extractor to form a first feature sequence; inputting the face image sequence into the second extractor to obtain features corresponding to each face image output by the second extractor to form a second feature sequence; using the fusion device to fuse the features in the first feature sequence and the second feature sequence to obtain the face image features. The fusion device completes the fusion of the two feature sequences, which can be an architecture and fusion method using existing technology, and its structure and fusion method do not constitute an implementation obstacle.

[0021] The first extractor in this application is the ArcFace face encoder, and the second extractor is specially trained in this application, has strong feature extraction capabilities, and is particularly suitable for extracting relevant information for psychological state recognition. This application embodiment discloses a training method for the second extractor, please refer to Figure 2 , the method comprising: S201. Obtain a sample face image set, wherein each sample face image in the sample face image set includes M target regions and corresponding psychological state labels.

[0022] The present application does not limit the number of target regions, for example, four target regions may be included, namely, eyes, mouth, cheeks, and eyebrows. The psychological state label indicates the psychological state, for example, normal, depressed, indifferent, and the like.

[0023] S202. For each of the sample facial images, the sample facial image is input into a feature extraction network to extract sample area features corresponding to each of the target areas, the sample area features corresponding to each of the target areas are fused using a feature fusion network to obtain sample fusion features, the sample fusion features are input into a prediction network to obtain a first psychological state prediction result; based on the difference between the first psychological state prediction result and the psychological state label, the parameters of the feature extraction network, the feature fusion network and the prediction network are adjusted to obtain a pre-trained model.

[0024] The embodiments of the present application do not limit the specific method for adjusting the parameters of the feature extraction network, the feature fusion network, and the prediction network based on the difference between the first mental state prediction result and the mental state label. For example, the gradient descent method can be used. The specific method for quantifying the loss caused by the difference between the first mental state prediction result and the mental state label is also not limited. An existing loss function can be used, and then the gradient descent method is used to adjust the parameters based on the loss. These are routine operations for model training and are not limited and do not constitute an implementation obstacle. The present application does not limit the parameter adjustment stop condition. It can be that the value of the loss function is less than a preset value or the number of parameter adjustments reaches a preset number. Of course, the present application does not limit the preset value and the preset number of times. They can be set according to actual conditions and do not constitute an implementation obstacle.

[0025] S203. For each target area of ​​each of the sample facial images, a corresponding surrounding area image is obtained, wherein the surrounding area image is an image composed of pixels in the sample facial image whose distance from the central pixel of the target area is less than a preset distance threshold, the area of ​​the surrounding area image is larger than the target area, and the pixels of the surrounding area image that overlap with the target area are set to blank.

[0026] The embodiments of the present application do not limit the preset distance threshold and do not constitute an implementation obstacle. There is a target area A in the sample face image, and the target area A corresponds to the surrounding area image B. The surrounding area image B includes the target area A and some pixels around it, and the pixels in the target area A in the surrounding area image B are erased. This will force the model to try to predict the accurate psychological state based only on the surrounding information of the target area A when it is unable to obtain the information of the target area A. This forces the model to deeply mine the surrounding information of the target area A and obtain high-quality information extraction results.

[0027] S204. Input each of the surrounding area images into the feature extraction network of the pre-trained model for feature extraction to obtain corresponding sample patch features, use the feature fusion network to fuse the features of each of the sample patch to obtain sample patch information, input the sample patch information into the prediction network to obtain a second psychological state prediction result; when the parameters of the prediction network are fixed, adjust the parameters of the feature extraction network and the feature fusion network based on the difference between the second psychological state prediction result and the psychological state label to obtain a second extractor, which only includes the feature extraction network and the feature fusion network whose parameters are adjusted.

[0028] There is no information about the corresponding target area in the sample patch features, and there is no information about each target area in the sample patch information. Therefore, the sample patch information is essentially the environmental features after digging out each target area. The psychological state is predicted based on this environmental feature, and the parameters of the feature extraction network and the feature fusion network are adjusted while fixing the parameters of the prediction network. This will force the feature extraction network and the feature fusion network to deeply extract environmental features, significantly enhancing the feature extraction capabilities of the feature extraction network and the feature fusion network. Therefore, the feature extraction capability of the second extractor for information related to the psychological state is significantly better than that of the pre-trained model.

[0029] The pre-trained model predicts the psychological state using the characteristics of the target area itself, and during the training process, the feature extraction network and the feature fusion network have preliminary psychological state feature extraction capabilities. After the target area is dug out to obtain the surrounding area image, the feature extraction network and the feature fusion network of the pre-trained model are used to extract the environmental features of the target area in the surrounding area image, and the parameters of the feature extraction network and the feature fusion network are adjusted using the environmental features, so that the feature extraction network and the feature fusion network can significantly enhance the ability to extract environmental features on the basis of the original preliminary psychological state feature extraction capabilities. That is, the feature extraction network and the feature fusion network of the second feature extractor can not only extract the psychological state features of the target area but also extract the environmental features, thereby obtaining high-quality information indicating the psychological state.

[0030] S104. Input the signal sequence into a signal feature extractor to obtain signal features, wherein the signal features and the facial image features belong to different feature spaces.

[0031] Signal features are features extracted from time series and are not in the same feature space as facial image features extracted from images.

[0032] Please refer to Figure 3 , which shows a schematic flow chart of a signal feature extraction method. The signal sequence is input into a signal feature extractor to obtain signal features, including: S301. For each subscript k in the signal sequence, determine the corresponding signal component ,in, is a preset parameter; , j is a preset parameter, which can be set according to experience.

[0033] S302. Obtain a reference signal quantity by accumulating various signal components.

[0034] Directly accumulate each signal component to obtain the characteristic basis; construct the adjustment parameter based on the preset parameter S and the number of signal components T The product of the adjustment parameter and the characteristic base quantity is used as the reference signal quantity. S is experimental data, which can be obtained by a limited number of adjustments according to actual conditions and does not constitute an implementation obstacle.

[0035] S303. Input the reference signal quantity into a signal feature extractor to obtain the signal feature.

[0036] The embodiments of the present application do not limit the structure of the signal feature extractor and do not constitute an implementation obstacle. As long as the feature extraction of the reference signal quantity can be achieved, the signal feature extractor can be constructed as needed in various networks in the artificial intelligence model.

[0037] S105. Input the facial image features into a mapper to obtain facial image mapping features.

[0038] The purpose of the mapper is to unify the feature space. The embodiment of the present application does not limit the mapper. For example, it can be obtained by cascading at least two convolutional layers.

[0039] S106. Perform psychological state recognition based on the facial image mapping features and the signal features to obtain a psychological state recognition result.

[0040] The method is implemented based on a psychological state identifier, which includes the image feature extractor, the signal feature extractor, the mapper and the fitter. The fitter is used for psychological state recognition. The image feature extractor is connected to the mapper, and the mapper and the signal feature extractor are both connected to the fitter. Each component in the psychological state identifier is not limited in detail and can be selected and built as needed from the artificial intelligence model, which does not constitute an implementation obstacle. The invention of this application does not lie in the innovation of the structure of a single component itself, but in the overall data processing process and the creative combination of components.

[0041] Please refer to Figure 4 , the mental state identifier is trained by the following method: S401. Obtaining a psychological state label of a sample person, a sample face image sequence of the sample person, and a sample signal sequence; The sample face images in the sample face image sequence may be the same as or different from the sample face images in the sample face image set mentioned above, and this does not constitute an implementation obstacle.

[0042] S402. Input the sample face image sequence into the first extractor to obtain a sample first feature sequence; input the sample face image sequence into the second extractor to obtain a sample second feature sequence; use the fusion device to fuse the features in the sample first feature sequence and the sample second feature sequence to obtain sample face image features.

[0043] Specifically, each sample face image in the sample face image sequence is input into the second extractor in turn to obtain a corresponding extraction result, wherein the extraction result includes the fusion features of the target area and the corresponding patch information, thereby obtaining a sample second feature sequence formed by each extraction result. In the application stage of the second extractor, there are also multiple target areas in the sample face image, and the target area will not be erased. Then, the second extractor can extract both the fusion features of the target area itself and the patch information representing the environmental features of the target area. The meanings of the fusion features and patch information are based on the same inventive concept as the sample fusion features and sample patch information in the previous text, and no further explanation is given.

[0044] S403. Input the sample signal sequence into the signal feature extractor to obtain sample signal features; input the sample face image features into the mapper to obtain sample face image mapping features; S404. Input the sample facial image mapping features and the sample signal features into the fitter to obtain a psychological state prediction result; based on the difference between the psychological state prediction result and the psychological state label, fix the parameters of the first extractor and the second extractor, and adjust the parameters of the fusion device, the signal feature extractor, the mapper and the fitter until the parameter adjustment is completed to obtain the psychological state identifier.

[0045] The training principle of the mental state identifier is based on the same inventive concept as the training principle of the other models mentioned above and will not be elaborated on here. The implementation details of each step of the mental state identifier are consistent with the implementation details of the application stage mentioned above and will not be elaborated on here.

[0046] Please refer to Figure 5 , which shows a block diagram of a mental state information recognition device based on data fusion, the device comprising: The data acquisition module is used to take N consecutive facial photos of the target person to obtain a facial image sequence, where N is a positive integer greater than 20; The data processing module is used to perform the following operations: collecting skin conductance signals of the target person to obtain a signal sequence; Inputting the facial image sequence into an image feature extractor to obtain facial image features; Inputting the signal sequence into a signal feature extractor to obtain signal features, wherein the signal features and the face image features belong to different feature spaces; Inputting the facial image features into a mapper to obtain facial image mapping features; Psychological state recognition is performed based on the facial image mapping features and the signal features to obtain a psychological state recognition result.

[0047] In one embodiment, the present application discloses a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the aforementioned data fusion-based psychological state information recognition method.

[0048] In one embodiment, the present application discloses a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the mental state information recognition method based on data fusion as described above.

[0049] In one embodiment, the present application discloses a computer program product, which includes computer instructions, the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the aforementioned data fusion-based psychological state information recognition method.

[0050] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0051] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0052] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0053] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying psychological state information based on data fusion, characterized in that: include: Take N consecutive facial photos of the target person to obtain a facial image sequence, where N is a positive integer greater than 20; collecting skin conductance signals of the target person to obtain a signal sequence; Inputting the facial image sequence into an image feature extractor to obtain facial image features; Inputting the signal sequence into a signal feature extractor to obtain signal features, wherein the signal features and the face image features belong to different feature spaces; Inputting the facial image features into a mapper to obtain facial image mapping features; Psychological state recognition is performed based on the facial image mapping features and the signal features to obtain a psychological state recognition result.

2. The method according to claim 1, characterized in that The step of collecting the skin conductance signal of the target person to obtain a signal sequence includes: Collect the skin conductance signal of the target person to obtain an initial signal sequence ; Based on the initial signal sequence , and get the intermediate signal sequence ,in, is the preset first adjustment parameter; Based on the intermediate signal sequence , get the reference signal sequence , , are the second adjustment parameter and the third adjustment parameter respectively; Fusion initial signal sequence and reference signal sequence , and obtain the signal sequence , the first signal of each signal sequence is the same.

3. The method according to claim 2, characterized in that The step of inputting the signal sequence into a signal feature extractor to obtain signal features comprises: For each subscript k in the signal sequence, determine the corresponding signal component ,in, It is a preset parameter; By accumulating various signal components, a reference signal quantity is obtained; The reference signal quantity is input into a signal feature extractor to obtain the signal feature.

4. The method according to claim 3, characterized in that The image feature extractor includes a first extractor, a second extractor and a fusion device, the first extractor is an ArcFace face encoder, and the face image sequence is input into the image feature extractor to obtain face image features, including: Inputting the facial image sequence into the first extractor to obtain a first feature sequence; Inputting the facial image sequence into the second extractor to obtain a second feature sequence; The features in the first feature sequence and the second feature sequence are fused using the fusion device to obtain the facial image features.

5. The method according to claim 4, characterized in that The method is implemented based on a psychological state identifier, which includes the image feature extractor, the signal feature extractor, the mapper and the fitter. The fitter is used for performing psychological state recognition, the image feature extractor is connected to the mapper, and the mapper and the signal feature extractor are both connected to the fitter.

6. The method according to claim 5, characterized in that The mental state identifier is trained by the following method: Obtaining a psychological state label of a sample person, a sample face image sequence of the sample person, and a sample signal sequence; Inputting the sample face image sequence into the first extractor to obtain a sample first feature sequence; Inputting the sample face image sequence into the second extractor to obtain a sample second feature sequence; Using the fusion device to fuse the features in the sample first feature sequence and the sample second feature sequence to obtain sample face image features; Inputting the sample signal sequence into the signal feature extractor to obtain sample signal features; Inputting the sample face image features into the mapper to obtain sample face image mapping features; Inputting the sample face image mapping features and the sample signal features into the fitter to obtain a psychological state prediction result; Based on the difference between the mental state prediction result and the mental state label, the parameters of the first extractor and the second extractor are fixed, and the parameters of the fusion device, the signal feature extractor, the mapper and the fitter are adjusted until the parameter adjustment is completed to obtain the mental state identifier.

7. A psychological state information recognition device based on data fusion, characterized in that: include: The data acquisition module is used to take N consecutive facial photos of the target person to obtain a facial image sequence, where N is a positive integer greater than 20; The data processing module is used to perform the following operations: collecting skin conductance signals of the target person to obtain a signal sequence; Inputting the facial image sequence into an image feature extractor to obtain facial image features; Inputting the signal sequence into a signal feature extractor to obtain signal features, wherein the signal features and the face image features belong to different feature spaces; Inputting the facial image features into a mapper to obtain facial image mapping features; Psychological state recognition is performed based on the facial image mapping features and the signal features to obtain a psychological state recognition result.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the mental state information recognition method based on data fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the mental state information recognition method based on data fusion as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the mental state information recognition method based on data fusion as described in any one of claims 1 to 6.

Images