Personnel state analysis method and equipment based on multi-modal data

Abstract

The invention provides a personnel state analysis method and equipment based on multi-modal data, and relates to the technical field of computers. The method comprises the following steps: firstly, extracting target image data and target audio data from target person session data; secondly, performing first semantic mining on the target image data, and outputting a target image vector; then, performing second semantic mining on the target audio data, and outputting a target audio vector; further, fusing the target image vector and the target audio vector, and outputting a global emotion semantic vector; and finally, based on the global emotion semantic vector, carrying out classification to obtain an emotion type. Based on the above content, the problem that the reliability of state analysis is relatively low in the prior art can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for analyzing personnel status based on multimodal data. Background Art

[0002] Currently, corporate employee mental health monitoring and intervention mainly rely on the following technical solutions: 1. Traditional manual assessment: This method relies on subjective experience to manually judge employees' emotional state through interviews or standardized questionnaires (such as the PHQ-9 depression scale). 2. Unimodal analysis: Text-based conversational robots extract emotional features from a single dimension; That is to say, in the prior art, due to the unreliability of subjective experience and the limitations of text data representation information, the reliability of employee status analysis is relatively low. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a personnel status analysis method and device based on multimodal data to improve the problem of relatively low reliability of status analysis in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solutions: A personnel status analysis method based on multimodal data, comprising: Extracting target image data and target audio data from the target person's conversation data, wherein the target image data is formed by an image acquisition operation of the target person during the emotional state test, and the target audio data is used to reflect the conversation content of the emotional state test; Performing first semantic mining on the target image data and outputting a target image vector, wherein the first semantic mining includes local semantic mining and global semantic mining of facial expressions; Performing second semantic mining on the target audio data and outputting a target audio vector; fusing the target image vector and the target audio vector to output a global emotion semantic vector; Classification is performed based on the global emotion semantic vector to obtain the emotion type of the target person.

[0005] In a preferred embodiment of the present application, in the above-mentioned method for analyzing personnel status based on multimodal data, the step of performing first semantic mining on the target image data and outputting a target image vector includes: Performing depth convolution processing on the target image data to output an image depth convolution vector, wherein the image depth convolution vector is used to represent facial semantic information of the target person; Performing key point detection on the target image data, and performing feature vector construction processing on each facial feature point obtained to form a feature point semantic vector corresponding to each facial feature point; The image depth convolution vector and the feature point semantic vector corresponding to each facial feature point are fused to form a target image vector corresponding to the target image data.

[0006] In a preferred embodiment of the present application, in the above-mentioned method for analyzing the personnel status based on multimodal data, the step of performing deep convolution processing on the target image data and outputting the image deep convolution vector includes: Extracting color channel data from the target image data to obtain corresponding red channel data, green channel data, and blue channel data; Convolving the red channel data, the green channel data, and the blue channel data respectively to form corresponding red channel vectors, green channel vectors, and blue channel vectors; performing two different depth mining operations inside the red channel vector, the green channel vector, and the blue channel vector, respectively, to obtain two red depth vectors, two green depth vectors, and two blue depth vectors, and correlating and aggregating the two red depth vectors to form a red aggregate vector, correlating and aggregating the two green depth vectors to form a green aggregate vector, and correlating and aggregating the two blue depth vectors to form a blue aggregate vector; Based on the preset red weight, green weight and blue weight, a weighted summation calculation is performed on the red aggregation vector, the green aggregation vector and the blue aggregation vector to obtain a corresponding image depth convolution vector, wherein the red weight, the green weight and the blue weight are formed in the process of the corresponding neural network model learning the sample data, and the neural network model has the following constraints: the red weight is greater than the green weight, and the green weight is greater than the blue weight.

[0007] In a preferred embodiment of the present application, in the above-mentioned method for analyzing the status of a person based on multimodal data, the step of performing key point detection on the target image data and constructing a feature vector for each obtained facial feature point to form a feature point semantic vector corresponding to each facial feature point includes: Use Dlib to perform face bounding box detection and facial feature point detection on the target image data to obtain the corresponding face bounding box and each facial feature point; For each facial feature point, based on the coordinate parameters of the facial feature point and the coordinate parameters of the center point of the face bounding box, calculating the center offset corresponding to the facial feature point, and based on the coordinate parameters of the facial feature point and the coordinate parameters of other adjacent facial feature points, determining the corresponding adjacent feature point distance; Based on the coordinate parameters, center offset and adjacent feature point distance of each facial feature point, a feature point semantic vector corresponding to each facial feature point is constructed.

[0008] In a preferred embodiment of the present application, in the above-mentioned method for analyzing the status of a person based on multimodal data, the step of fusing the image depth convolution vector and the feature point semantic vector corresponding to each facial feature point to form a target image vector corresponding to the target image data includes: For each of the facial feature points, concatenating the feature point semantic vector corresponding to the facial feature point and the image depth convolution vector to form a corresponding feature point concatenation vector; The feature point splicing vectors corresponding to each of the facial feature points are fused to form a target image vector corresponding to the target image data.

[0009] In a preferred embodiment of the present application, in the above-mentioned method for analyzing the status of a person based on multimodal data, the step of fusing the feature point splicing vectors corresponding to each of the facial feature points to form a target image vector corresponding to the target image data includes: Based on the facial feature point topology of Dlib, a graph structure is constructed for each facial feature point to form a corresponding target facial graph structure, wherein the facial feature point serves as a corresponding graph node in the target facial graph structure; Based on the connection relationship in the target facial graph structure, performing first association semantic mining on the feature point splicing vector corresponding to each facial feature point to form a feature point association vector corresponding to each facial feature point; Based on the connection relationship in the facial graph structure, performing a second association semantic mining on the feature point association vector corresponding to each facial feature point to form a feature point deep association vector corresponding to each facial feature point, wherein the first association semantic mining is different from the second association semantic mining; Based on the feature point depth association vector corresponding to each of the facial feature points, a target image vector corresponding to the target image data is determined.

[0010] In a preferred embodiment of the present application, in the above-mentioned method for analyzing the personnel status based on multimodal data, the step of performing first association semantic mining on the feature point splicing vector corresponding to each facial feature point based on the connection relationship in the target facial graph structure to form a feature point association vector corresponding to each facial feature point includes: For each of the facial feature points, based on the connection relationship in the target facial graph structure, determining each other facial feature point having a connection relationship with the facial feature point; Based on the feature point splicing vectors corresponding to each other facial feature point having a connection relationship, cross-attention processing is performed on the feature point splicing vectors corresponding to the facial feature point, and a facial attention vector corresponding to each other facial feature point is output; The feature point concatenation vector corresponding to the facial feature point and the facial attention vector corresponding to each other facial feature point having a connection relationship are summed up to obtain a feature point association vector corresponding to the facial feature point.

[0011] In a preferred embodiment of the present application, in the above-mentioned method for analyzing the personnel status based on multimodal data, the step of performing second association semantic mining on the feature point association vector corresponding to each facial feature point based on the connection relationship in the facial graph structure to form a feature point depth association vector corresponding to each facial feature point includes: For each facial feature point, use the facial feature point as a central feature point; Traversing the facial feature points in the facial graph structure in sequence from a direction away from the central feature point to a direction close to the central feature point; fusing the current feature point association vector of the currently traversed facial feature point into the feature point association vector corresponding to the adjacent facial feature point corresponding to the currently traversed facial feature point based on cross-attention processing to form the current feature point association vector of the adjacent facial feature point, wherein the adjacent facial feature point has a connection relationship with the currently traversed facial feature point, and the distance between the adjacent facial feature point and the central feature point is less than the distance between the currently traversed facial feature point and the central feature point; After forming the current feature point association vector of the central feature point, the current feature point association vector of the central feature point is used as the corresponding feature point depth association vector.

[0012] In a preferred embodiment of the present application, in the above-mentioned method for analyzing the state of a person based on multimodal data, the step of fusing the target image vector and the target audio vector to output a global emotion semantic vector includes: Performing text conversion processing on the target audio data to form target text data, and obtaining target physiological data of the target person during the emotional state test, wherein the target physiological data at least includes heart rate data; performing a third semantic mining on the target text data to form a target text vector, and performing a fourth semantic mining on the target physiological data to form a target physiological vector; Determining the confidence of the target image data based on a clarity index of a facial region and a stability index of facial feature points in the target image data, determining the confidence of the target audio data and the confidence of the target physiological data based on signal-to-noise ratios of the target audio data and the target physiological data, respectively, and determining the confidence of the target text data based on a dialogue coherence index represented by the target text data; Determining a weight coefficient corresponding to each semantic vector based on the confidence of the target image data, the confidence of the target audio data, the confidence of the target physiological data, and the confidence of the target text data, and in combination with the confidence of the historical image data, the confidence of the historical audio data, the confidence of the historical physiological data, and the confidence of the historical text data; Based on the weight coefficient corresponding to each semantic vector, a weighted sum calculation is performed on the target image vector, the target audio vector, the target text vector and the target physiological vector to output a global emotion semantic vector.

[0013] Based on the above, the present application further provides an electronic device, including: Memory for storing computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the above-mentioned personnel status analysis method based on multimodal data.

[0014] The present application provides a method and device for analyzing the state of a person based on multimodal data. First, target image data and target audio data are extracted from the conversation data of the target person. Second, a first semantic mining is performed on the target image data to output a target image vector. Then, a second semantic mining is performed on the target audio data to output a target audio vector. Further, the target image vector and the target audio vector are fused to output a global emotional semantic vector. Finally, classification is performed based on the global emotional semantic vector to obtain the emotional type. Based on the above content, on the one hand, the semantic information of the two modalities of image and audio is used to analyze the emotional type, so that the semantic information used as the basis for analysis can be richer, and compared with the conventional technical solution of using a single modality for analysis, it can have higher analysis reliability. In addition, since the process of semantic mining of the image modality includes local semantic mining and global semantic mining of facial expressions, it can ensure the accuracy of the expression semantic mining while taking into account the richness of the semantics, thereby improving the semantic representation ability of the mined vector, thereby improving the problem of relatively low reliability of state analysis in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings.

[0016] Figure 1 This is a structural block diagram of the electronic device provided in an embodiment of the present application.

[0017] Figure 2 A flowchart of a personnel status analysis method based on multimodal data provided in an embodiment of the present application.

[0018] Figure 3 A schematic diagram of the depth convolution processing provided in an embodiment of the present application.

[0019] Figure 4 A schematic diagram of the target facial image structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0022] like Figure 1 As shown, an embodiment of the present application provides an electronic device, wherein the electronic device may include a memory, a processor, and a personnel status analysis device based on multimodal data.

[0023] In detail, the memory and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, the memory and the processor can be electrically connected through one or more communication buses or signal lines. The personnel status analysis device based on multimodal data includes at least one software function module stored in the memory in the form of software or firmware. The processor is used to execute the executable computer program stored in the memory, for example, the software function module and computer program included in the personnel status analysis device based on multimodal data, so as to implement the personnel status analysis method based on multimodal data provided in the embodiment of the present application.

[0024] Optionally, the memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0025] Optionally, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0026] Optionally, the personnel status analysis device based on multimodal data may include: a data extraction module, configured to extract target image data and target audio data from the target person's conversation data, wherein the target image data is formed by image acquisition during the emotional state test of the target person, and the target audio data is used to reflect the conversation content of the emotional state test; A first semantic mining module is configured to perform first semantic mining on the target image data and output a target image vector, wherein the first semantic mining includes local semantic mining and global semantic mining of facial expressions; A second semantic mining module, configured to perform second semantic mining on the target audio data and output a target audio vector; A semantic fusion module, configured to fuse the target image vector and the target audio vector to output a global emotion semantic vector; The emotion type analysis module is used to perform classification based on the global emotion semantic vector to obtain the emotion type of the target person.

[0027] It can be understood that the specific functions of the above modules can be referred to the relevant explanations below.

[0028] I understand. Figure 1 The structure shown is only for illustration, and the electronic device may also include Figure 1 More or fewer components than shown, or with Figure 1 The different configurations shown, for example, may also include a communication unit for exchanging information with other devices (such as an image acquisition device, etc.).

[0029] Combine Figure 2 , the embodiment of the present application also provides a method for analyzing the status of a person based on multimodal data that can be applied to the above electronic device. Among them, the method steps defined in the process related to the method for analyzing the status of a person based on multimodal data can be implemented by the electronic device. Figure 2 The specific process shown is explained in detail.

[0030] Step S110 , extracting target image data and target audio data from the target person's conversation data.

[0031] In an embodiment of the present application, the electronic device can extract target image data and target audio data from the target person's conversation data. The target image data is formed by image capture during the emotional state test of the target person, and the target audio data is used to reflect the content of the conversation during the emotional state test. Exemplarily, the target image data and target audio data can be formed by interacting with the target person in a face-to-face interview, video, or other manner. The target image data is formed by at least capturing an image of the target person's face, and the target audio data is used to reflect at least the content of the conversation with the target person.

[0032] Step S120: performing first semantic mining on the target image data and outputting a target image vector.

[0033] In an embodiment of the present application, after obtaining the target image data, the electronic device can perform a first semantic mining on the target image data and output a target image vector. That is, the electronic device can mine the potential semantic information in the target image data and represent it in the form of a vector, thereby obtaining a target image vector. The first semantic mining includes local semantic mining and global semantic mining of facial expressions, that is, it is necessary to pay attention not only to the local potential semantic information in the facial expression, but also to the global semantic information in the facial expression.

[0034] Step S130: perform second semantic mining on the target audio data and output a target audio vector.

[0035] In an embodiment of the present application, after obtaining the target audio data, the electronic device can perform a second semantic mining on the target audio data and output a target audio vector. That is, the potential semantic information in the target audio data can be mined and represented in the form of a vector, thereby obtaining a target audio vector.

[0036] Step S140: fusing the target image vector and the target audio vector to output a global emotion semantic vector.

[0037] In an embodiment of the present application, after extracting the target image vector and the target audio vector, the electronic device can fuse the target image vector and the target audio vector to output a global emotion semantic vector. In other words, the semantic information of the two modalities can be fused to obtain more comprehensive and reliable global emotion semantic information.

[0038] Step S150 , performing classification based on the global emotion semantic vector to obtain the emotion type of the target person.

[0039] In an embodiment of the present application, after obtaining the global emotion semantic vector, the electronic device can perform classification based on the global emotion semantic vector to obtain the emotion type of the target person, such as depression, anxiety, anger, calmness, happiness, and others.

[0040] Based on the above, on the one hand, the semantic information of both image and audio modalities is used to analyze emotion types, making the semantic information used as the basis for analysis richer. Compared with conventional technical solutions that use data from a single modality for analysis, the analysis reliability can be higher. In addition, because the semantic mining process of the image modality includes local semantic mining and global semantic mining of facial expressions, it can ensure the accuracy of expression semantic mining while also taking into account the richness of semantics, thereby improving the semantic representation ability of the mined vectors, thereby improving the relatively low reliability of state analysis in the existing technology.

[0041] First, it should be noted that in step S110, in the process of extracting the target image data and target audio data from the target person's conversation data, the extracted image data and audio data can be directly used as the target image data and target audio data, or the extracted image data and audio data can be preprocessed accordingly to obtain the target image data and target audio data. The preprocessing may include, but is not limited to, denoising and other processing. In addition, for the target image data, since the useful information is expression information, only the facial image may be required. In this way, during the preprocessing process, the facial image can also be segmented from the extracted image data as the target image data. The facial image segmentation can adopt the existing Faster RCNN or Mask RCNN, and the specific processing process will not be repeated here.

[0042] Secondly, it should be noted that for step S120, the specific method of performing the first semantic mining on the target image data is not limited and can be selected according to actual needs.

[0043] For example, in an alternative embodiment, on the one hand, the target image data can be directly convolved to achieve global semantic mining. On the other hand, the target image data can be segmented, and then convolution processing can be performed on each resulting image block to achieve local semantic mining. Finally, the convolution vectors formed by the two methods can be spliced ​​or superimposed to form the corresponding target image vector.

[0044] For example, in another alternative embodiment, in order to further improve the representation ability of the mined target image vector on the expression semantics, the above-mentioned step S120 can further include step S121, step S122 and step S123, and the specific content of each step is described as follows.

[0045] Step S121 , performing depth convolution processing on the target image data, and outputting an image depth convolution vector.

[0046] In an embodiment of the present application, the target image data may be subjected to a depth convolution process to output an image depth convolution vector. The image depth convolution vector is used to represent the facial semantic information of the target person. As previously described, in step S110, the corresponding facial image may be segmented to obtain the target image data. Thus, by performing a depth convolution process on the target image data, the depth semantic information therein may be captured, thereby obtaining an image depth convolution vector. This allows for focusing on the global semantic information of facial expressions.

[0047] Step S122 , performing key point detection on the target image data, and performing feature vector construction processing on each facial feature point obtained, to form a feature point semantic vector corresponding to each facial feature point.

[0048] In an embodiment of the present application, key point detection can also be performed on the target image data, and feature vector construction processing can be performed on each facial feature point obtained to form a feature point semantic vector corresponding to each facial feature point, thereby focusing on the local semantic information of facial expressions.

[0049] Step S123 , fusing the image depth convolution vector and the feature point semantic vector corresponding to each facial feature point to form a target image vector corresponding to the target image data.

[0050] In an embodiment of the present application, after obtaining the image depth convolution vector and the feature point semantic vector, the image depth convolution vector and the feature point semantic vector corresponding to each facial feature point can be fused to form a target image vector corresponding to the target image data. In this way, the fusion of global semantic information and local semantic information of facial expressions can be achieved, so that the semantic representation ability of the formed target image vector can be fully improved.

[0051] It can be understood that in the above-mentioned step S121, the specific manner of performing deep convolution processing on the target image data is not limited. For example, in an alternative embodiment, the inventors of this application have found through research that the semantic information of different color channels has different representation capabilities for facial expressions. Specifically, in expression recognition, the contraction and expansion of facial muscles will affect the red part of the skin. Therefore, the red channel can better help capture the subtle details in the expression. Based on this, the above-mentioned step S121 can further include step S121a, step S121b, step S121c and step S121d. The specific content of each step is described below.

[0052] Step S121a, extracting color channel data from the target image data to obtain corresponding red channel data, green channel data, and blue channel data (combined with Figure 3 ).

[0053] In the embodiment of the present application, color channel data is extracted from the target image data to obtain corresponding red channel data, green channel data, and blue channel data, as shown in the following table:

[0054] Step S121b: performing convolution on the red channel data, the green channel data, and the blue channel data respectively to form corresponding red channel vectors, green channel vectors, and blue channel vectors.

[0055] In an embodiment of the present application, after obtaining the red channel data, the green channel data and the blue channel data, the red channel data, the green channel data and the blue channel data can be convolved respectively to form corresponding red channel vectors, green channel vectors and blue channel vectors. It should be noted that, in one embodiment, the size of the convolution kernel can be 3*3, the step size can be 1, and the edges are padded, so that after the convolution, the size of the obtained channel vector can be equal to the size of the corresponding channel data. Alternatively, in other embodiments, when the edges are not padded and / or supplemented with a value greater than 1, the size of the vector after the convolution is smaller than the size of the corresponding channel data, so that the size of the vector can be restored to the same size as the corresponding channel data by upsampling such as interpolation.

[0056] Step S121c, performing two different depth mining operations inside the red channel vector, the green channel vector, and the blue channel vector, respectively, to obtain two red depth vectors, two green depth vectors, and two blue depth vectors, and correlating and aggregating the two red depth vectors to form a red aggregate vector, correlating and aggregating the two green depth vectors to form a green aggregate vector, and correlating and aggregating the two blue depth vectors to form a blue aggregate vector.

[0057] In an embodiment of the present application, after obtaining a red channel vector, a green channel vector, and a blue channel vector, two different depth mining operations can be performed within the red channel vector, the green channel vector, and the blue channel vector, respectively, to obtain two red depth vectors, two green depth vectors, and two blue depth vectors. Furthermore, the two red depth vectors can be associated and aggregated to form a red aggregate vector, the two green depth vectors can be associated and aggregated to form a green aggregate vector, and the two blue depth vectors can be associated and aggregated to form a blue aggregate vector. For example, two different pooling operations (such as mean size and maximum size) or convolution operations (such as two convolution kernels with different parameters) can be performed on the red channel vector to form two red depth vectors. By configuring the size, stride, edge padding, etc. of the pooling window and convolution kernel, the two obtained red depth vectors can be made the same size and the same size as the red channel vector. Furthermore, the depth mining of the two color channels is the same as that of the red channel. Furthermore, for associated aggregation, cross-attention processing (Cross-Attention) can be performed on one depth vector based on the other depth vector to obtain the corresponding aggregate vector.

[0058] Step S121d: Based on the preset red weight, green weight, and blue weight, a weighted sum calculation is performed on the red aggregation vector, the green aggregation vector, and the blue aggregation vector to obtain a corresponding image depth convolution vector.

[0059] In an embodiment of the present application, after obtaining the red, green, and blue aggregation vectors, a weighted summation of the red, green, and blue aggregation vectors can be performed based on preset red, green, and blue weights to obtain a corresponding image depth convolution vector: image depth convolution vector = red weight * red aggregation vector + green weight * green aggregation vector + blue weight * blue aggregation vector. The red, green, and blue weights are formed during the learning process of the corresponding neural network model on sample data, and the neural network model has the following constraints: the red weight is greater than the green weight, and the green weight is greater than the blue weight. That is, because the semantic information of the red channel has a stronger role in representing facial expressions than the green and blue channels, the corresponding weights can have a maximum value. Furthermore, considering that the semantic information of the green channel has a significant impact on details and skin color changes and cannot be ignored, while the semantic information of the blue channel has a smaller impact on facial expressions and generally only plays a certain role under special lighting conditions, the green weight can be further constrained to be greater than the blue weight.

[0060] It can be understood that in the above-mentioned step S122, the specific method of forming the feature point semantic vector corresponding to each facial feature point is not limited. For example, in an alternative embodiment, considering that the coordinate distribution relationship between facial feature points in different expressions generally has certain differences, therefore, in order to better represent facial expressions, the above-mentioned step S122 can further include step S122a, step S122b and step S122c, the specific contents of which are described below.

[0061] Step S122a: Use Dlib to perform face bounding box detection and facial feature point detection on the target image data to obtain the corresponding face bounding box and each facial feature point.

[0062] In an embodiment of the present application, Dlib can be used to perform face bounding box detection and facial feature point detection on the target image data to obtain the corresponding face bounding box and each facial feature point. Among them, the Dlib tool library is a C++ library widely used in the fields of computer vision and machine learning. It provides some powerful tools for tasks such as face detection, facial feature point positioning, object recognition, and image processing. Specifically, the Dlib tool library provides a very efficient face detector that can recognize faces in images and return the rectangular box where the face is located, that is, the face bounding box. The Dlib tool library also provides a facial landmark detector that can help identify 68 key points on the face, such as eyes, nose, mouth, etc., that is, facial feature points, such as: 1-17: facial contour (including chin, cheeks and forehead); 18-22: Landmark point of the left eyebrow; 23-27: right eyebrow landmarks; 28-36: landmark points of the nose bridge and nose tip; 37-42: Landmarks of the left eye; 43-48: Landmarks of the right eye; 49-59: Landmark points of the upper lip; 60-68: Landmark points of the lower lip.

[0063] Step S122b, for each facial feature point, based on the coordinate parameters of the facial feature point and the coordinate parameters of the center point of the face bounding box, calculate the center offset corresponding to the facial feature point, and, based on the coordinate parameters of the facial feature point and the coordinate parameters of other adjacent facial feature points, determine the corresponding adjacent feature point distance.

[0064] In an embodiment of the present application, after the facial feature points and the face bounding box are determined, for each facial feature point, the center offset corresponding to the facial feature point (i.e., the distance between the facial feature point and the center point) can be calculated based on the coordinate parameters of the facial feature point and the coordinate parameters of the center point of the face bounding box. In addition, the corresponding adjacent feature point distance (i.e., the distance between two adjacent facial feature points, such as the distance between the facial feature point corresponding to the inner corner of the left eye and the facial feature point corresponding to the inner corner of the right eye) can be determined based on the coordinate parameters of the facial feature point and the coordinate parameters of other adjacent facial feature points. In addition, it should be noted that, in some embodiments, before calculating the center offset and feature point distance, the coordinate parameters of each facial feature point may be normalized to eliminate the scale differences of different faces. For example, for the coordinates (x, y) of each facial feature point, the upper left corner (x1, y1) of the face bounding box may be subtracted from it to obtain the local coordinates relative to the face frame (x' = x - x1, y' = y - y1). Then, these coordinates may be divided by the width and height of the face frame to normalize the key point coordinates to [0, 1].

[0065] Step S122c: constructing a feature point semantic vector corresponding to each facial feature point based on the coordinate parameters, center offset, and adjacent feature point distance of each facial feature point.

[0066] In an embodiment of the present application, after obtaining the parameters related to the coordinates, a feature point semantic vector corresponding to each facial feature point can be constructed based on the coordinate parameters, center offset and adjacent feature point distance of each facial feature point, that is, the vector parameters in the feature point semantic vector are the coordinate parameters, center offset and adjacent feature point distance.

[0067] It can be understood that in the above-mentioned step S123, the specific method of fusing the image depth convolution vector and the feature point semantic vector corresponding to each facial feature point is not limited. For example, in an alternative embodiment, in order to fully integrate local semantic information and global semantic information, thereby achieving reliable representation of the semantics of facial expressions, the above-mentioned step S123 can further include step S123a and step S123b, the specific contents of which are described below.

[0068] Step S123a: For each facial feature point, concatenate the feature point semantic vector corresponding to the facial feature point and the image depth convolution vector to form a corresponding feature point concatenation vector.

[0069] In an embodiment of the present application, for each of the facial feature points, the feature point semantic vector corresponding to the facial feature point and the image depth convolution vector are spliced ​​together to form a corresponding feature point splicing vector. For example, in some embodiments, considering that the size of the feature point semantic vector can be 1*d and the size of the image depth convolution vector can be n*m, the image depth convolution vector can be first expanded to obtain a vector of size 1*(n*m), and then spliced ​​with the feature point semantic vector. Based on this, the image depth convolution vector can be preliminarily fused into the feature point semantic vector of each facial feature.

[0070] Step S123b: Fusing the feature point splicing vectors corresponding to each of the facial feature points to form a target image vector corresponding to the target image data.

[0071] In an embodiment of the present application, after obtaining the feature point stitching vector, the feature point stitching vector corresponding to each facial feature point can be fused to form a target image vector corresponding to the target image data, so that the semantic information of all feature points can be fused.

[0072] It can be understood that, in the above-mentioned step S123b, the specific method of fusing the feature point splicing vector corresponding to each of the facial feature points is not limited. For example, in an alternative embodiment, in order to achieve full fusion of semantic information, for example, the associated semantic information is captured during the fusion process so that the accuracy of the fused semantic information is higher. Based on this, the above-mentioned step S123b can further include step b1, step b2, step b3 and step b4. The specific content of each step is described as follows.

[0073] Step b1: Based on the facial feature point topology of Dlib, a graph structure is constructed for each facial feature point to form a corresponding target facial graph structure.

[0074] In the embodiment of the present application, a graph structure construction process can be performed on each facial feature point based on the facial feature point topology of Dlib to form a corresponding target facial graph structure, such as Figure 4 In the target facial graph structure, the facial feature points serve as corresponding graph nodes, and facial feature points with connection relationships have correlation relationships, such as being connected by facial muscles.

[0075] Step b2: Based on the connection relationship in the target facial graph structure, a first association semantic mining is performed on the feature point splicing vector corresponding to each facial feature point to form a feature point association vector corresponding to each facial feature point.

[0076] In an embodiment of the present application, after forming the target facial graph structure, a first association semantic mining can be performed on the feature point concatenation vector corresponding to each facial feature point based on the connectivity within the target facial graph structure, thereby forming a feature point association vector corresponding to each facial feature point. In other words, because the semantic information between connected facial feature points generally exhibits correlation (direct linkage of facial muscles), association semantic mining can be performed based on the corresponding connectivity relationships to uncover relevant semantic information.

[0077] Step b3: Based on the connection relationship in the facial graph structure, a second association semantic mining is performed on the feature point association vector corresponding to each facial feature point to form a feature point depth association vector corresponding to each facial feature point.

[0078] In an embodiment of the present application, after obtaining the feature point association vector, a second association semantic mining can be performed on the feature point association vector corresponding to each facial feature point based on the connection relationship in the facial graph structure to form a feature point deep association vector corresponding to each facial feature point. The first association semantic mining is different from the second association semantic mining. For example, the first association semantic mining is for facial feature points with a direct connection relationship, while the second association semantic mining can be for a wider range of facial feature points with a certain degree of correlation.

[0079] Step b4: determining a target image vector corresponding to the target image data based on the feature point depth association vector corresponding to each facial feature point.

[0080] In an embodiment of the present application, after obtaining the feature point depth association vector, the target image vector corresponding to the target image data can be determined based on the feature point depth association vector corresponding to each of the facial feature points. For example, the feature point depth association vectors corresponding to each of the facial feature points can be spliced ​​together to obtain the corresponding target image vector. In one embodiment, the spliced ​​result can be directly used as the target image vector. In another embodiment, the spliced ​​result can also be further processed, such as by pooling to reduce the vector size, to obtain the corresponding target image vector.

[0081] It is understandable that, in the above step b2, the specific manner of performing the first associated semantic mining on the feature point splicing vector corresponding to each facial feature point is not limited. For example, in an alternative embodiment, considering that facial feature points with a connection relationship generally have a strong correlation and reinforcement effect on the semantic representation of expression, the above step b2 may include: First, for each facial feature point, based on the connection relationship in the target facial graph structure, determine each other facial feature point that has a connection relationship with the facial feature point. For example, Figure 4 , other facial feature points having a connection relationship with facial feature point 1 include facial feature point 2 and facial feature point 3; Secondly, based on the feature point splicing vectors corresponding to each other facial feature point with a connection relationship, cross-attention processing is performed on the feature point splicing vectors corresponding to the facial feature points, and the facial attention vectors corresponding to each other facial feature point are output. For example, based on the feature point splicing vector corresponding to facial feature point 2, the feature point splicing vector corresponding to facial feature point 1 can be cross-attention processed to obtain facial attention vector 1; and based on the feature point splicing vector corresponding to facial feature point 3, the feature point splicing vector corresponding to facial feature point 1 can be cross-attention processed to obtain facial attention vector 2. In this way, the cross-attention processing can be used to effectively capture the associated semantic information between related facial feature points. Then, the feature point splicing vector corresponding to the facial feature point and the facial attention vector corresponding to each other facial feature point with a connection relationship can be summed up to obtain the feature point association vector corresponding to the facial feature point, such as the feature point splicing vector corresponding to facial feature point 1 + facial attention vector 1 + facial attention vector 2.

[0082] It is understandable that, in the above step b3, the specific manner of performing the second association semantic mining on the feature point association vector corresponding to each facial feature point is not limited. For example, in an alternative embodiment, in order to take into account that facial feature points that do not have a connection relationship also have a certain relevance for the representation of expression semantic information, based on this, the above step b3 may further include the following content: First, for each facial feature point, the facial feature point is used as a central feature point; Secondly, the facial feature points in the facial graph structure are traversed in sequence from the direction away from the central feature point to the direction close to the central feature point. For example, the central feature point can be used as the starting point, and each other facial feature point can be used as the end point. Then, the shortest travel path between the starting point and the end point is determined. In this way, during the traversal process, the facial feature points initially traversed are the facial feature points with the maximum value of the shortest travel path, such as Figure 4 Facial feature points 8 and 9 in ; Then, the current feature point association vector of the currently traversed facial feature point can be fused into the feature point association vector corresponding to the adjacent facial feature point corresponding to the currently traversed facial feature point based on the cross-attention processing (that is, based on the current feature point association vector of the currently traversed facial feature point, the feature point association vector corresponding to the adjacent facial feature point is cross-attention processed) to form the current feature point association vector of the adjacent facial feature point, wherein the adjacent facial feature point has a connection relationship with the currently traversed facial feature point, and the distance between the adjacent facial feature point and the central feature point (such as the length of the aforementioned shortest walking path) is less than the distance between the currently traversed facial feature point and the central feature point. Distance, for example, based on the current feature point association vector of facial feature point 8, the feature point association vector of facial feature point 6 can be cross-attention processed to obtain the current feature point association vector of facial feature point 6; based on the current feature point association vector of facial feature point 9, the feature point association vector of facial feature point 7 can be cross-attention processed to obtain the current feature point association vector of facial feature point 7; in addition, for facial feature point 3, since cross-attention processing will be performed based on the current feature point association vector of facial feature point 6 and the current feature point association vector of facial feature point 7 respectively, the results of the two cross-attention processing can be averaged to obtain the current feature point association vector of facial feature point 3.

[0083] Finally, after forming the current feature point association vector of the central feature point, the current feature point association vector of the central feature point is used as the corresponding feature point depth association vector.

[0084] Thirdly, it should be noted that for step S130, the target audio data is subjected to second semantic mining, and the specific method of outputting the target audio vector is not restricted and can be selected accordingly according to actual needs. For example, in an alternative embodiment, the target audio data can be converted to form a corresponding spectrogram. In this way, the spectrogram can be processed by a convolutional neural network model to obtain the corresponding target audio vector.

[0085] Fourthly, it should be noted that the specific method of outputting the global emotion semantic vector is not limited and can be selected according to actual needs.

[0086] For example, in an alternative implementation, the target image vector and the target audio vector may be concatenated or summed to obtain a corresponding global emotion semantic vector.

[0087] For example, in another alternative embodiment, in order to enrich the semantic information represented by the output global emotional semantic vector, the above-mentioned step 140 may include step S141, step S142, step S143, step S144 and step S145, and the contents of each step are as follows.

[0088] Step S141 , performing text conversion processing on the target audio data to form target text data, and obtaining target physiological data of the target person during the emotional state test.

[0089] In an embodiment of the present application, the target audio data can be converted into text (any existing speech-to-text conversion technology can be used, which is not specifically limited or described herein) to generate target text data, and target physiological data of the target person during the emotional state test can be obtained. The target physiological data includes at least heart rate data, which can be obtained, for example, by detecting a heart rate sensor.

[0090] Step S142 : performing a third semantic mining on the target text data to form a target text vector, and performing a fourth semantic mining on the target physiological data to form a target physiological vector.

[0091] In an embodiment of the present application, after obtaining the target text data and target physiological data, on the one hand, a third semantic mining can be performed on the target text data to form a target text vector. For example, the BERT model (a deep learning model based on Transformer) can be used to extract semantic features (keyword sentiment) and grammatical features (sentence complexity, interrogative sentence frequency, etc.) to enhance the accuracy of the features. On the other hand, a fourth semantic mining can be performed on the target physiological data to form a target physiological vector. For example, a CNN (convolutional neural network, the specific mining process can refer to the relevant existing technology) or a DBN (deep belief network) can be used for feature mining. In addition, the heart rate data can be first converted into a spectrogram and then convolution mining can be performed.

[0092] Step S143: Determine the confidence of the target image data based on the clarity index of the facial area and the stability index of the facial feature points in the target image data, and determine the confidence of the target audio data and the confidence of the target physiological data based on the signal-to-noise ratios of the target audio data and the target physiological data respectively, and determine the confidence of the target text data based on the dialogue coherence index represented by the target text data.

[0093] In an embodiment of the present application, the confidence of the target image data can be determined based on a clarity index (such as the SSIM index) and a facial feature point stability index (such as the Dlib tracking jitter rate, where the lower the jitter rate, the higher the stability) of the facial region in the target image data (exemplarily, there is a positive correlation between the clarity index and the confidence, and there is a positive correlation between the facial feature point stability index and the confidence). Furthermore, based on the signal-to-noise ratios of the target audio data and the target physiological data, respectively, the confidence of the target audio data and the confidence of the target physiological data can be determined (exemplarily, there is a positive correlation between the signal-to-noise ratio and the confidence). Furthermore, based on a conversation coherence index (such as topic consistency between previous and subsequent sentences) representing the target text data, the confidence of the target text data can be determined (exemplarily, there is a positive correlation between the conversation coherence index and the confidence). In natural language processing (NLP), BERT (Bidirectional Encoder Representations from Transformers) is a powerful pre-trained model that can capture contextual information in text. When the BERT model receives text input, it typically adds a [CLS] (classification) tag to the beginning of each sentence. This [CLS] tag does not represent a specific word, but rather summarizes the semantic information of the entire sentence or text. The BERT model converts each word (token) into a vector, and the [CLS] tag represents the semantic representation of the entire sentence or input text. Through its bidirectional encoding capabilities, the BERT model understands the relationship between each word in the context, thereby grasping the semantic relationship between sentences. During a conversation, BERT's [CLS] tag represents the semantic information of the entire conversation paragraph. When the topic within a conversation paragraph is consistent, the [CLS] tag representation reflects good coherence between sentences. By comparing the [CLS] tag vectors across different conversation turns, BERT can help assess whether the conversation maintains contextual consistency. If the topic changes at some point in the conversation, the probability distribution of the [CLS] tag may change significantly, reflecting incoherence in the conversation.

[0094] Step S144, based on the confidence of the target image data, the confidence of the target audio data, the confidence of the target physiological data and the confidence of the target text data, and combined with the confidence of the historical image data, the confidence of the historical audio data, the confidence of the historical physiological data and the confidence of the historical text data, determine the weight coefficient corresponding to each semantic vector.

[0095] In an embodiment of the present application, a weight coefficient corresponding to each semantic vector is determined based on the confidence level of the target image data, the confidence level of the target audio data, the confidence level of the target physiological data, and the confidence level of the target text data, in combination with the confidence levels of the historical image data, the confidence levels of the historical audio data, the confidence levels of the historical physiological data, and the confidence levels of the historical text data. For example, the weight coefficient corresponding to the target image vector can be determined based on the confidence level of the target image data and the confidence levels of the historical image data (or the weight coefficients corresponding to the historical image data). It should be noted that the person status analysis in this embodiment of the present application is performed in real time. As the test progresses, new target person conversation data is continuously generated, and the current target person conversation data becomes historical person conversation data. Therefore, to avoid misjudgments caused by transient fluctuations, the current confidence level can be combined with the confidence levels or weight coefficients at historical moments to obtain the current weight coefficient. For example, the average of the current confidence level and the weight coefficients at the previous five moments can be calculated to serve as the current weight coefficient. In addition, for the first moment, the corresponding confidence level can be directly used as the weight coefficient of the current moment. For the 2nd to 5th moments, the corresponding confidence level and the average of the weight coefficients of the previous moments can be used as the weight coefficient of the current moment.

[0096] Step S145 , based on the weight coefficient corresponding to each semantic vector, perform weighted sum calculation on the target image vector, the target audio vector, the target text vector and the target physiological vector, and output a global emotion semantic vector.

[0097] In an embodiment of the present application, after obtaining the weight coefficient corresponding to each semantic vector, the target image vector, the target audio vector, the target text vector and the target physiological vector can be weighted and summed based on the weight coefficient corresponding to each semantic vector to output a global emotional semantic vector, that is, the global emotional semantic vector = the weight coefficient corresponding to the target image vector*+the weight coefficient corresponding to the target audio vector*+the weight coefficient corresponding to the target text vector*+the weight coefficient corresponding to the target physiological vector*.

[0098] Fifthly, it should be noted that for step S150, the specific method of obtaining the emotion type of the target person is not limited and can be selected according to actual needs.

[0099] For example, in an alternative embodiment, the global emotion semantic vector can be classified and output based on an emotion classifier (e.g., an MLP model) to determine the target person's emotion type. For example, emotion types can be categorized into six categories: depression, anxiety, anger, calmness, happiness, and other (adjustable based on actual circumstances); and intensity levels can be categorized into five levels: 1 = mild, 5 = extreme. Furthermore, as previously mentioned, as testing progresses, new target person conversation data is continuously generated, and the emotion type output at the current moment can serve as a reference for subsequent testing. For example, if the emotion type is anxiety and the intensity level reaches 3, the speech rate can be reduced and deep breathing guidance can be inserted. For another example, if the emotion type is depression and the intensity level reaches 4, an active questioning mode can be used and a psychological counselor can be directed to conduct subsequent testing (i.e., a professional psychological counselor can be used for subsequent testing). For another example, if the emotion type is anger and the intensity level reaches 2, a neutral tone can be used and rhetorical questions can be avoided.

[0100] In summary, the personnel status analysis method and device based on multimodal data provided by the present application first extracts target image data and target audio data from the target personnel conversation data; secondly, performs a first semantic mining on the target image data and outputs a target image vector; then, performs a second semantic mining on the target audio data and outputs a target audio vector; further, fuses the target image vector and the target audio vector to output a global emotion semantic vector; finally, performs classification based on the global emotion semantic vector to obtain the emotion type. Based on the above content, on the one hand, the semantic information of the two modalities of image and audio is used to analyze the emotion type, so that the semantic information used as the basis for analysis can be richer, and compared with the conventional technical solution that uses a single modality for analysis, it can have higher analysis reliability. In addition, since the process of semantic mining of image modality includes local semantic mining and global semantic mining of facial expressions, it can ensure the accuracy of expression semantic mining while taking into account the richness of semantics, thereby improving the semantic representation ability of the mined vector, thereby improving the problem of relatively low reliability of status analysis in the existing technology.

[0101] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0103] If the functions are implemented in the form of software modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. It should be noted that, in this document, the terms "comprise," "include," or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0104] The foregoing description is merely a preferred embodiment of the present application and is not intended to limit the present application. Persons skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A personnel status analysis method based on multimodal data, characterized in that: include: Extracting target image data and target audio data from the target person's conversation data, wherein the target image data is formed by an image acquisition operation of the target person during the emotional state test, and the target audio data is used to reflect the conversation content of the emotional state test; Performing first semantic mining on the target image data and outputting a target image vector, wherein the first semantic mining includes local semantic mining and global semantic mining of facial expressions; Performing second semantic mining on the target audio data and outputting a target audio vector; fusing the target image vector and the target audio vector to output a global emotion semantic vector; Classification is performed based on the global emotion semantic vector to obtain the emotion type of the target person.

2. The personnel status analysis method based on multimodal data according to claim 1, characterized in that: The step of performing first semantic mining on the target image data and outputting a target image vector includes: Performing depth convolution processing on the target image data to output an image depth convolution vector, wherein the image depth convolution vector is used to represent facial semantic information of the target person; Performing key point detection on the target image data, and performing feature vector construction processing on each facial feature point obtained to form a feature point semantic vector corresponding to each facial feature point; The image depth convolution vector and the feature point semantic vector corresponding to each facial feature point are fused to form a target image vector corresponding to the target image data.

3. The personnel status analysis method based on multimodal data according to claim 2, characterized in that: The step of performing depth convolution processing on the target image data and outputting an image depth convolution vector comprises: Extracting color channel data from the target image data to obtain corresponding red channel data, green channel data, and blue channel data; Convolving the red channel data, the green channel data, and the blue channel data respectively to form corresponding red channel vectors, green channel vectors, and blue channel vectors; performing two different depth mining operations inside the red channel vector, the green channel vector, and the blue channel vector, respectively, to obtain two red depth vectors, two green depth vectors, and two blue depth vectors, and correlating and aggregating the two red depth vectors to form a red aggregate vector, correlating and aggregating the two green depth vectors to form a green aggregate vector, and correlating and aggregating the two blue depth vectors to form a blue aggregate vector; Based on the preset red weight, green weight and blue weight, a weighted summation calculation is performed on the red aggregation vector, the green aggregation vector and the blue aggregation vector to obtain a corresponding image depth convolution vector, wherein the red weight, the green weight and the blue weight are formed in the process of the corresponding neural network model learning the sample data, and the neural network model has the following constraints: the red weight is greater than the green weight, and the green weight is greater than the blue weight.

4. The personnel status analysis method based on multimodal data according to claim 2, characterized in that: The step of performing key point detection on the target image data and constructing a feature vector for each facial feature point obtained to form a feature point semantic vector corresponding to each facial feature point includes: Use Dlib to perform face bounding box detection and facial feature point detection on the target image data to obtain the corresponding face bounding box and each facial feature point; For each facial feature point, based on the coordinate parameters of the facial feature point and the coordinate parameters of the center point of the face bounding box, calculating the center offset corresponding to the facial feature point, and based on the coordinate parameters of the facial feature point and the coordinate parameters of other adjacent facial feature points, determining the corresponding adjacent feature point distance; Based on the coordinate parameters, center offset and adjacent feature point distance of each facial feature point, a feature point semantic vector corresponding to each facial feature point is constructed.

5. The personnel status analysis method based on multimodal data according to claim 2, characterized in that: The step of fusing the image depth convolution vector and the feature point semantic vector corresponding to each facial feature point to form a target image vector corresponding to the target image data includes: For each of the facial feature points, concatenating the feature point semantic vector corresponding to the facial feature point and the image depth convolution vector to form a corresponding feature point concatenation vector; The feature point splicing vectors corresponding to each of the facial feature points are fused to form a target image vector corresponding to the target image data.

6. The personnel status analysis method based on multimodal data according to claim 5, characterized in that: The step of fusing the feature point splicing vectors corresponding to each of the facial feature points to form a target image vector corresponding to the target image data includes: Based on the facial feature point topology of Dlib, a graph structure is constructed for each facial feature point to form a corresponding target facial graph structure, wherein the facial feature point serves as a corresponding graph node in the target facial graph structure; Based on the connection relationship in the target facial graph structure, performing first association semantic mining on the feature point splicing vector corresponding to each facial feature point to form a feature point association vector corresponding to each facial feature point; Based on the connection relationship in the facial graph structure, performing a second association semantic mining on the feature point association vector corresponding to each facial feature point to form a feature point deep association vector corresponding to each facial feature point, wherein the first association semantic mining is different from the second association semantic mining; Based on the feature point depth association vector corresponding to each of the facial feature points, a target image vector corresponding to the target image data is determined.

7. The personnel status analysis method based on multimodal data according to claim 6, characterized in that: The step of performing first association semantic mining on the feature point splicing vector corresponding to each facial feature point based on the connection relationship in the target facial graph structure to form a feature point association vector corresponding to each facial feature point includes: For each of the facial feature points, based on the connection relationship in the target facial graph structure, determining each other facial feature point having a connection relationship with the facial feature point; Based on the feature point splicing vectors corresponding to each other facial feature point having a connection relationship, cross-attention processing is performed on the feature point splicing vectors corresponding to the facial feature point, and a facial attention vector corresponding to each other facial feature point is output; The feature point concatenation vector corresponding to the facial feature point and the facial attention vector corresponding to each other facial feature point having a connection relationship are summed up to obtain a feature point association vector corresponding to the facial feature point.

8. The personnel status analysis method based on multimodal data according to claim 6, characterized in that: The step of performing second association semantic mining on the feature point association vector corresponding to each facial feature point based on the connection relationship in the facial graph structure to form a feature point depth association vector corresponding to each facial feature point includes: For each facial feature point, use the facial feature point as a central feature point; Traversing the facial feature points in the facial graph structure in sequence from a direction away from the central feature point to a direction close to the central feature point; fusing the current feature point association vector of the currently traversed facial feature point into the feature point association vector corresponding to the adjacent facial feature point corresponding to the currently traversed facial feature point based on cross-attention processing to form the current feature point association vector of the adjacent facial feature point, wherein the adjacent facial feature point has a connection relationship with the currently traversed facial feature point, and the distance between the adjacent facial feature point and the central feature point is less than the distance between the currently traversed facial feature point and the central feature point; After forming the current feature point association vector of the central feature point, the current feature point association vector of the central feature point is used as the corresponding feature point depth association vector.

9. The personnel status analysis method based on multimodal data according to any one of claims 1 to 8, characterized in that: The step of fusing the target image vector and the target audio vector to output a global emotion semantic vector includes: Performing text conversion processing on the target audio data to form target text data, and obtaining target physiological data of the target person during the emotional state test, wherein the target physiological data at least includes heart rate data; performing a third semantic mining on the target text data to form a target text vector, and performing a fourth semantic mining on the target physiological data to form a target physiological vector; Determining the confidence of the target image data based on a clarity index of a facial region and a stability index of facial feature points in the target image data, determining the confidence of the target audio data and the confidence of the target physiological data based on signal-to-noise ratios of the target audio data and the target physiological data, respectively, and determining the confidence of the target text data based on a dialogue coherence index represented by the target text data; Determining a weight coefficient corresponding to each semantic vector based on the confidence of the target image data, the confidence of the target audio data, the confidence of the target physiological data, and the confidence of the target text data, and in combination with the confidence of the historical image data, the confidence of the historical audio data, the confidence of the historical physiological data, and the confidence of the historical text data; Based on the weight coefficient corresponding to each semantic vector, a weighted sum calculation is performed on the target image vector, the target audio vector, the target text vector and the target physiological vector to output a global emotion semantic vector.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the personnel status analysis method based on multimodal data as described in any one of claims 1 to 9.

Images