Personnel state analysis method and device based on multi-modal data

Abstract

The personnel state analysis method and device based on multi-modal data provided in the application relate to the technical field of computers.In the application, first, target image data and target audio data are extracted from target personnel conversation data;second, first semantic mining is performed on the target image data, and a target image vector is output;then, second semantic mining is performed on the target audio data, and a target audio vector is output;further, the target image vector and the target audio vector are fused, and a global emotional semantic vector is output;finally, classification is performed based on the global emotional semantic vector, and an emotional type is obtained.Based on the above, the problem that the reliability of state analysis is relatively low in the prior art can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method and device for analyzing personnel status based on multimodal data. Background Technology

[0002] Currently, the monitoring and intervention of employees' mental health in enterprises mainly relies on the following technical solutions: 1. Traditional manual assessment: The emotional state of employees is judged manually through interviews or standardized questionnaires (such as the PHQ-9 depression scale), which relies on subjective experience; 2. Single-modal analysis: Text-based chatbots extract emotion features from a single dimension; In other words, in existing technologies, the reliability of employee status analysis is relatively low due to the unreliability of subjective experience and the limitations of textual data in representing information. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method and device for personnel status analysis based on multimodal data, so as to improve the problem of relatively low reliability of status analysis in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution: A method for analyzing personnel status based on multimodal data, comprising: Target image data and target audio data are extracted from the target person's conversation data. The target image data is formed by image acquisition during the target person's emotional state test, and the target audio data is used to reflect the conversation content of the emotional state test. Perform first semantic mining on the target image data and output the target image vector. The first semantic mining includes local semantic mining and global semantic mining of facial expressions. Perform second semantic mining on the target audio data and output the target audio vector; By fusing the target image vector and the target audio vector, a global emotion semantic vector is output. The target person's emotion type is obtained by classifying the global emotion semantic vector.

[0005] In a preferred embodiment of this application, in the aforementioned method for analyzing personnel status based on multimodal data, the step of performing first semantic mining on the target image data and outputting a target image vector includes: The target image data is subjected to depth convolution processing to output an image depth convolution vector, wherein the image depth convolution vector is used to represent the facial semantic information of the target person; Key point detection is performed on the target image data, and feature vector construction is performed on each facial feature point to form a feature point semantic vector corresponding to each facial feature point. The image depth convolution vector and the semantic vector of each facial feature point are fused to form the target image vector corresponding to the target image data.

[0006] In a preferred embodiment of this application, in the aforementioned method for analyzing personnel status based on multimodal data, the step of performing depth convolution processing on the target image data and outputting an image depth convolution vector includes: Color channel data is extracted from the target image data to obtain the corresponding red channel data, green channel data and blue channel data respectively; Convolutions are performed on the red channel data, the green channel data, and the blue channel data respectively to form corresponding red channel vectors, green channel vectors, and blue channel vectors; Two different depth mining operations are performed inside the red channel vector, the green channel vector, and the blue channel vector respectively to obtain two red depth vectors, two green depth vectors, and two blue depth vectors. The two red depth vectors are then correlated and aggregated to form a red aggregated vector, the two green depth vectors are correlated and aggregated to form a green aggregated vector, and the two blue depth vectors are correlated and aggregated to form a blue aggregated vector. Based on the preset red weights, green weights, and blue weights, the red aggregate vector, the green aggregate vector, and the blue aggregate vector are weighted and summed to obtain the corresponding image depth convolution vector. The red weights, green weights, and blue weights are formed during the learning process of the corresponding neural network model on the sample data, and the neural network model has the following constraints: the red weight is greater than the green weight, and the green weight is greater than the blue weight.

[0007] In a preferred embodiment of this application, in the aforementioned method for analyzing human state based on multimodal data, the step of performing key point detection on the target image data and constructing feature vectors for each obtained facial feature point to form a feature point semantic vector corresponding to each facial feature point includes: Dlib is used to perform face bounding box detection and facial feature point detection on the target image data to obtain the corresponding face bounding box and each facial feature point. For each facial feature point, the center offset corresponding to the facial feature point is calculated based on the coordinate parameters of the facial feature point and the coordinate parameters of the center point of the face bounding box. The distance between adjacent feature points is determined based on the coordinate parameters of the facial feature point and the coordinate parameters of other adjacent facial feature points. Based on the coordinate parameters, center offset, and distance to adjacent feature points of each facial feature point, a semantic vector corresponding to each facial feature point is constructed.

[0008] In a preferred embodiment of this application, in the aforementioned method for analyzing human state based on multimodal data, the step of fusing the image depth convolution vector and the semantic vector of each facial feature point to form the target image vector corresponding to the target image data includes: For each facial feature point, the semantic vector of the feature point corresponding to the facial feature point and the image depth convolution vector are concatenated to form the corresponding feature point concatenation vector. The feature point concatenation vectors corresponding to each of the facial feature points are fused together to form the target image vector corresponding to the target image data.

[0009] In a preferred embodiment of this application, in the aforementioned method for analyzing human state based on multimodal data, the step of fusing the feature point concatenation vectors corresponding to each facial feature point to form the target image vector corresponding to the target image data includes: Based on the facial feature point topology of Dlib, a graph structure construction process is performed on each facial feature point to form a corresponding target facial graph structure, wherein the facial feature point serves as a corresponding graph node in the target facial graph structure. Based on the connection relationships in the target facial image structure, a first association semantic mining is performed on the feature point splicing vector corresponding to each facial feature point to form a feature point association vector corresponding to each facial feature point. Based on the connection relationships in the facial graph structure, a second association semantic mining is performed on the feature point association vector corresponding to each facial feature point to form a feature point deep association vector corresponding to each facial feature point. The first association semantic mining is different from the second association semantic mining. Based on the feature point depth association vector corresponding to each of the facial feature points, the target image vector corresponding to the target image data is determined.

[0010] In a preferred embodiment of this application, in the aforementioned method for analyzing the state of individuals based on multimodal data, the step of performing a first association semantic mining on the feature point concatenation vector corresponding to each facial feature point based on the connection relationships in the target facial image structure to form a feature point association vector corresponding to each facial feature point includes: For each of the facial feature points, based on the connection relationships in the target facial image structure, each other facial feature point that has a connection relationship with that facial feature point is determined; Based on the feature point concatenation vector corresponding to each other facial feature point with a connection relationship, cross attention processing is performed on the feature point concatenation vector corresponding to the facial feature point, and the facial attention vector corresponding to each other facial feature point is output. The feature point concatenation vector corresponding to the facial feature point is summed with the facial attention vector corresponding to each other facial feature point that has a connection relationship, to obtain the feature point association vector corresponding to the facial feature point.

[0011] In a preferred embodiment of this application, in the aforementioned method for analyzing human state based on multimodal data, the step of performing second semantic mining on the feature point association vector corresponding to each facial feature point based on the connection relationships in the facial image structure to form a deep association vector for each facial feature point includes: For each of the facial feature points, the facial feature point is taken as the center feature point; The facial feature points in the facial image structure are traversed sequentially along a direction from away from the central feature point to closer to the central feature point. The current feature point association vector of the currently traversed facial feature point is fused into the feature point association vector of the adjacent facial feature point based on cross-attention processing to form the current feature point association vector of the adjacent facial feature point. The adjacent facial feature point and the currently traversed facial feature point have a connection relationship, and the distance between the adjacent facial feature point and the center feature point is less than the distance between the currently traversed facial feature point and the center feature point. After forming the current feature point association vector of the central feature point, the current feature point association vector of the central feature point is used as the corresponding feature point depth association vector.

[0012] In a preferred embodiment of this application, in the aforementioned method for analyzing human state based on multimodal data, the step of fusing the target image vector and the target audio vector to output a global emotion semantic vector includes: The target audio data is processed by text conversion to form target text data, and target physiological data of the target person during the emotional state test is obtained, wherein the target physiological data includes at least heart rate data. The target text data is subjected to third semantic mining to form a target text vector, and the target physiological data is subjected to fourth semantic mining to form a target physiological vector; Based on the clarity index and facial feature point stability index of the facial region in the target image data, the confidence level of the target image data is determined. Based on the signal-to-noise ratio of the target audio data and the target physiological data, the confidence level of the target audio data and the target physiological data are determined respectively. Based on the dialogue coherence index represented by the target text data, the confidence level of the target text data is determined. Based on the confidence levels of the target image data, the target audio data, the target physiological data, and the target text data, and combined with the confidence levels of historical image data, historical audio data, historical physiological data, and historical text data, the weight coefficients corresponding to each semantic vector are determined. Based on the weight coefficients corresponding to each semantic vector, the target image vector, the target audio vector, the target text vector, and the target physiological vector are weighted and summed to output a global emotion semantic vector.

[0013] Based on the above, this application also provides an electronic device, including: Memory, used to store computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the above-described method for analyzing personnel status based on multimodal data.

[0014] The method and apparatus for analyzing human state based on multimodal data provided in this application first extract target image data and target audio data from target human conversation data; second, perform first semantic mining on the target image data to output target image vectors; then, perform second semantic mining on the target audio data to output target audio vectors; further, fuse the target image vectors and target audio vectors to output global emotion semantic vectors; finally, classify based on the global emotion semantic vectors to obtain emotion types. Based on the above, on the one hand, it utilizes the semantic information of both image and audio modalities for emotion type analysis, making the semantic information used as the basis for analysis richer, and achieving higher analysis reliability compared to conventional techniques that use single-modal data for analysis. On the other hand, since the semantic mining process for the image modality includes both local and global semantic mining of facial expressions, it ensures both the accuracy of expression semantic mining and the richness of semantics, thereby improving the semantic representation capability of the mined vectors and addressing the problem of relatively low reliability in state analysis in existing technologies. Attached Figure Description

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0016] Figure 1 A structural block diagram of an electronic device provided in an embodiment of this application.

[0017] Figure 2 A flowchart illustrating the personnel status analysis method based on multimodal data provided in this application embodiment.

[0018] Figure 3 This is a schematic diagram of depthwise convolution processing provided in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram of the target facial image structure provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] like Figure 1 As shown in the figure, this application provides an electronic device. The electronic device may include a memory, a processor, and a personnel status analysis device based on multimodal data.

[0023] In detail, the memory and the processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, the memory and the processor can be electrically connected via one or more communication buses or signal lines. The personnel status analysis device based on multimodal data includes at least one software functional module stored in the memory in the form of software or firmware. The processor is used to execute executable computer programs stored in the memory, such as the software functional modules and computer programs included in the personnel status analysis device based on multimodal data, to implement the personnel status analysis method based on multimodal data provided in the embodiments of this application.

[0024] Optionally, the memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0025] Optionally, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0026] Optionally, the personnel status analysis device based on multimodal data may include: The data extraction module is used to extract target image data and target audio data from the target person's conversation data. The target image data is formed by image acquisition during the target person's emotional state test, and the target audio data is used to reflect the conversation content of the emotional state test. The first semantic mining module is used to perform first semantic mining on the target image data and output the target image vector. The first semantic mining includes local semantic mining and global semantic mining of facial expressions. The second semantic mining module is used to perform second semantic mining on the target audio data and output the target audio vector. The semantic fusion module is used to fuse the target image vector and the target audio vector to output a global emotion semantic vector; The emotion type analysis module is used to classify based on the global emotion semantic vector to obtain the emotion type of the target person.

[0027] It is understandable that the specific functions of the above modules can be found in the explanations below.

[0028] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown may include, for example, a communication unit for exchanging information with other devices (such as image acquisition devices).

[0029] Combination Figure 2 This application also provides a method for analyzing personnel status based on multimodal data, applicable to the aforementioned electronic device. The method steps defined in the process of this multimodal data-based personnel status analysis method can be implemented by the electronic device. The following will describe... Figure 2 The specific process shown will be explained in detail.

[0030] Step S110: Extract target image data and target audio data from the target person's conversation data.

[0031] In this embodiment, the electronic device can extract target image data and target audio data from the target person's conversation data. The target image data is generated by image capture during an emotional state test of the target person, and the target audio data reflects the conversation content during the emotional state test. For example, the target image data and target audio data can be generated through interaction with the target person via face-to-face interviews, video calls, etc. The target image data is generated by capturing at least an image of the target person's face, and the target audio data reflects at least the content of the conversation with the target person.

[0032] Step S120: Perform first semantic mining on the target image data and output the target image vector.

[0033] In this embodiment, after obtaining the target image data, the electronic device can perform a first semantic mining on the target image data and output a target image vector. That is, it can mine the latent semantic information in the target image data and represent it in vector form to obtain the target image vector. The first semantic mining includes local semantic mining and global semantic mining of facial expressions; that is, it needs to pay attention not only to the local latent semantic information in facial expressions but also to the global semantic information in facial expressions.

[0034] Step S130: Perform second semantic mining on the target audio data and output the target audio vector.

[0035] In this embodiment of the application, after obtaining the target audio data, the electronic device can perform a second semantic mining on the target audio data and output a target audio vector. That is, it can mine the potential semantic information in the target audio data and represent it in the form of a vector to obtain the target audio vector.

[0036] Step S140: Fuse the target image vector and the target audio vector to output a global emotion semantic vector.

[0037] In this embodiment, after mining the target image vector and the target audio vector, the electronic device can fuse the target image vector and the target audio vector to output a global emotion semantic vector. That is, the semantic information of the two modalities can be fused to obtain more comprehensive and reliable global emotion semantic information.

[0038] Step S150: Classify based on the global emotion semantic vector to obtain the emotion type of the target person.

[0039] In this embodiment of the application, after obtaining the global emotion semantic vector, the electronic device can classify based on the global emotion semantic vector to obtain the emotion type of the target person, such as depression, anxiety, anger, calmness, happiness, other, etc.

[0040] Based on the above, on the one hand, semantic information from both image and audio modalities is used for emotion type analysis, resulting in richer semantic information as the basis for analysis. Compared to conventional techniques that use single-modal data for analysis, this approach offers higher analytical reliability. On the other hand, since the semantic mining process for the image modality includes both local and global semantic mining of facial expressions, it ensures both the accuracy and richness of semantic information, thereby improving the semantic representation capability of the mined vectors and addressing the relatively low reliability of state analysis in existing technologies.

[0041] Firstly, regarding step S110, it should be noted that in the process of extracting target image data and target audio data from the target person's conversation data, the extracted image data and audio data can either be used directly as target image data and target audio data, or they can undergo corresponding preprocessing to obtain target image data and target audio data. Preprocessing may include, but is not limited to, noise reduction. Furthermore, regarding the target image data, since the useful information is facial expression information, only facial images are needed. Therefore, during preprocessing, facial images can be segmented from the extracted image data and used as target image data. Facial image segmentation can employ existing Faster R-CNN or Mask R-CNN methods; the specific processing steps will not be detailed here.

[0042] Secondly, it should be noted that the specific method for performing the first semantic mining on the target image data in step S120 is not limited and can be selected according to actual needs.

[0043] For example, in an alternative implementation, on the one hand, the target image data can be directly convolved to achieve global semantic mining. On the other hand, the target image data can be segmented, and then each resulting image block can be convolved to achieve local semantic mining. Finally, the convolution vectors formed by the two methods can be concatenated or superimposed to form the corresponding target image vector.

[0044] For example, in another alternative implementation, in order to further improve the ability of the mined target image vector to represent facial expression semantics, the above step S120 may further include steps S121, S122 and S123, the specific contents of each step are as follows.

[0045] Step S121: Perform depth convolution processing on the target image data and output the image depth convolution vector.

[0046] In this embodiment, the target image data can be subjected to depth convolution processing to output an image depth convolution vector. This image depth convolution vector is used to represent the facial semantic information of the target person. As mentioned earlier, in step S110, the corresponding facial image can be segmented first, thus obtaining the target image data. By performing depth convolution processing on the target image data, the depth semantic information within it can be captured, resulting in the image depth convolution vector. This allows for the focus on the global semantic information of facial expressions.

[0047] Step S122: Perform key point detection on the target image data, and construct feature vectors for each facial feature point to form a feature point semantic vector corresponding to each facial feature point.

[0048] In this embodiment of the application, key point detection can also be performed on the target image data, and feature vector construction processing can be performed on each obtained facial feature point to form a feature point semantic vector corresponding to each facial feature point, thereby achieving attention to the local semantic information of facial expressions.

[0049] Step S123: Fuse the image depth convolution vector and the feature point semantic vector corresponding to each facial feature point to form the target image vector corresponding to the target image data.

[0050] In this embodiment of the application, after obtaining the image depth convolution vector and the feature point semantic vector, the image depth convolution vector and the feature point semantic vector corresponding to each facial feature point can be fused to form the target image vector corresponding to the target image data. In this way, the fusion of global and local semantic information of facial expressions can be realized, so that the semantic representation ability of the formed target image vector can be fully improved.

[0051] It is understood that the specific method of performing deep convolution processing on the target image data in step S121 above is not limited. For example, in an alternative implementation, the inventors of this application have found that the semantic information of different color channels has different representation capabilities for facial expressions. Specifically, in expression recognition, the contraction and expansion of facial muscles affect the red part of the skin. Therefore, the red channel can better help capture the subtle details in expressions. Based on this, step S121 above can further include steps S121a, S121b, S121c and S121d, the specific contents of each step are as follows.

[0052] Step S121a: Extract color channel data from the target image data to obtain the corresponding red channel data, green channel data, and blue channel data (combined with...). Figure 3 ).

[0053] In this embodiment, color channel data is extracted from the target image data to obtain the corresponding red channel data, green channel data, and blue channel data, as shown in the table below:

[0054] Step S121b: Convolve the red channel data, the green channel data, and the blue channel data respectively to form corresponding red channel vectors, green channel vectors, and blue channel vectors.

[0055] In this embodiment, after obtaining the red channel data, the green channel data, and the blue channel data, convolution can be performed on the red channel data, the green channel data, and the blue channel data respectively to form corresponding red channel vectors, green channel vectors, and blue channel vectors. It should be noted that, in one implementation, the convolution kernel size can be 3*3, the stride can be 1, and the edges are padded. Thus, after convolution, the size of the resulting channel vector is equal to the size of the corresponding channel data. Alternatively, in other implementations, when the edges are not padded and / or padded by more than 1, the size of the vector after convolution is smaller than the size of the corresponding channel data. In this case, upsampling methods such as interpolation can be used to restore the vector size to be the same as the size of the corresponding channel data.

[0056] Step S121c involves performing two different depth mining operations within the red channel vector, the green channel vector, and the blue channel vector to obtain two red depth vectors, two green depth vectors, and two blue depth vectors. The two red depth vectors are then correlated and aggregated to form a red aggregated vector, the two green depth vectors are correlated and aggregated to form a green aggregated vector, and the two blue depth vectors are correlated and aggregated to form a blue aggregated vector.

[0057] In this embodiment, after obtaining the red channel vector, green channel vector, and blue channel vector, two different depth mining operations can be performed within each of the red, green, and blue channel vectors to obtain two red depth vectors, two green depth vectors, and two blue depth vectors. Furthermore, the two red depth vectors are correlated and aggregated to form a red aggregated vector, the two green depth vectors are correlated and aggregated to form a green aggregated vector, and the two blue depth vectors are correlated and aggregated to form a blue aggregated vector. For example, the red channel vector can be subjected to two different pooling processes (e.g., mean and maximum size) or convolution processes (e.g., different parameters for the two convolution kernels) to form two red depth vectors. By configuring the pooling window and the size, stride, and edge padding of the convolution kernels, the two resulting red depth vectors can be made to have the same size, and the same size as the red channel vector. Additionally, the depth mining of the two color channels is the same as the processing of the red channel. Furthermore, for the correlation aggregation, cross-attention processing can be performed on one depth vector to obtain the corresponding aggregated vector.

[0058] Step S121d: Based on the preset red weight, green weight, and blue weight, the red aggregate vector, the green aggregate vector, and the blue aggregate vector are weighted and summed to obtain the corresponding image depth convolution vector.

[0059] In this embodiment, after obtaining the red aggregate vector, the green aggregate vector, and the blue aggregate vector, a weighted summation can be performed on the red aggregate vector, the green aggregate vector, and the blue aggregate vector based on preset red weights, green weights, and blue weights to obtain the corresponding image depth convolution vector, i.e., image depth convolution vector = red weight * red aggregate vector + green weight * green aggregate vector + blue weight * blue aggregate vector. The red weight, green weight, and blue weight are formed during the learning process of the corresponding neural network model on the sample data, and this neural network model has the following constraints: the red weight is greater than the green weight, and the green weight is greater than the blue weight. That is, since the semantic information of the red channel has a stronger representation effect on facial expressions than the green and blue channels, the corresponding weight can have a maximum value. Furthermore, considering that the semantic information of the green channel has a significant impact on details and skin tone changes and cannot be ignored, while the semantic information of the blue channel has a smaller impact on facial expressions and may only play a certain role under special lighting conditions, the green weight can be further constrained to be greater than the blue weight.

[0060] It is understood that in step S122 above, the specific method of forming the semantic vector of each facial feature point is not limited. For example, in an alternative implementation, considering that the coordinate distribution relationship between facial feature points in different expressions generally has certain differences, in order to better represent facial expressions, step S122 above may further include steps S122a, S122b and S122c, the specific contents of which are as follows.

[0061] Step S122a: Use Dlib to perform face bounding box detection and facial feature point detection on the target image data to obtain the corresponding face bounding box and each facial feature point.

[0062] In this embodiment, Dlib can be used to perform face bounding box detection and facial feature point detection on the target image data to obtain the corresponding face bounding boxes and each facial feature point. Dlib is a C++ library widely used in computer vision and machine learning, providing powerful tools for tasks such as face detection, facial feature point localization, object recognition, and image processing. Specifically, Dlib provides a highly efficient face detector that can identify faces in an image and return the rectangular bounding box containing the face. Dlib also provides a facial landmark detector that helps identify 68 key points on the face, such as eyes, nose, and mouth, i.e., facial feature points, such as: 1-17: Facial contours (including chin, cheeks, and forehead); 18-22: Marker points on the left eyebrow; 23-27: Marker points on the right eyebrow; 28-36: Landmarks on the bridge and tip of the nose; 37-42: Landmarks for the left eye; 43-48: Landmarks for the right eye; 49-59: Marker points on the upper lip; 60-68: Marker points on the lower lip.

[0063] Step S122b: For each facial feature point, calculate the center offset corresponding to the facial feature point based on the coordinate parameters of the facial feature point and the coordinate parameters of the center point of the face bounding box, and determine the distance between adjacent feature points based on the coordinate parameters of the facial feature point and the coordinate parameters of other adjacent facial feature points.

[0064] In this embodiment of the application, after determining the facial feature points and the face bounding box, for each facial feature point, the center offset (i.e., the distance between the facial feature point and the center point) can be calculated based on the coordinate parameters of the facial feature point and the coordinate parameters of the center point of the face bounding box. Furthermore, based on the coordinate parameters of the facial feature point and the coordinate parameters of other adjacent facial feature points, the distance between adjacent feature points (i.e., the distance between two adjacent facial feature points, such as the distance between the facial feature point corresponding to the inner corner of the left eye and the facial feature point corresponding to the inner corner of the right eye) can be determined. Additionally, it should be noted that in some implementations, before calculating the center offset and feature point distance, the coordinate parameters of each facial feature point can be normalized to eliminate scale differences between different faces. For example, for the coordinates (x, y) of each facial feature point, the top left corner (x1, y1) of the face bounding box can be subtracted to obtain the local coordinates relative to the face box (x' = x - x1, y' = y - y1). Then, these coordinates can be divided by the width and height of the face box to normalize the key point coordinates to [0, 1].

[0065] Step S122c: Construct the semantic vector of each facial feature point based on the coordinate parameters, center offset, and distance to adjacent feature points of each facial feature point.

[0066] In this embodiment of the application, after obtaining the coordinate-related parameters, a feature point semantic vector corresponding to each facial feature point can be constructed based on the coordinate parameters, center offset, and distance between adjacent feature points of each facial feature point. That is, the vector parameters in the feature point semantic vector are the coordinate parameters, center offset, and distance between adjacent feature points, respectively.

[0067] It is understood that in step S123 above, the specific method of fusing the image depth convolution vector and the semantic vector of the feature point corresponding to each facial feature point is not limited. For example, in an alternative implementation, in order to fully fuse local semantic information and global semantic information to achieve a reliable representation of the semantics of facial expressions, step S123 above may further include steps S123a and S123b, the specific contents of which are as follows.

[0068] Step S123a: For each facial feature point, the semantic vector of the feature point corresponding to the facial feature point and the image depth convolution vector are concatenated to form a corresponding feature point concatenation vector.

[0069] In this embodiment, for each facial feature point, the corresponding feature point semantic vector and the image depth convolution vector are concatenated to form a corresponding feature point concatenation vector. For example, in some implementations, considering that the size of the feature point semantic vector can be 1*d and the size of the image depth convolution vector can be n*m, the image depth convolution vector can be first expanded to obtain a vector of size 1*(n*m), and then concatenated with the feature point semantic vector. Based on this, the image depth convolution vector can be initially fused into the feature point semantic vector of each facial feature.

[0070] Step S123b: The feature point splicing vectors corresponding to each of the facial feature points are fused to form the target image vector corresponding to the target image data.

[0071] In this embodiment of the application, after obtaining the feature point concatenation vector, the feature point concatenation vector corresponding to each facial feature point can be fused to form the target image vector corresponding to the target image data. In this way, the semantic information of all feature points can be fused.

[0072] It is understood that in step S123b above, the specific method of fusing the feature point concatenation vector corresponding to each facial feature point is not limited. For example, in an alternative implementation, in order to achieve full fusion of semantic information, for example, the associated semantic information is captured during the fusion process, so that the accuracy of the fused semantic information is higher. Based on this, step S123b above can further include steps b1, b2, b3 and b4, and the specific contents of each step are as follows.

[0073] Step b1: Based on the facial feature point topology of Dlib, perform graph structure construction processing on each facial feature point to form the corresponding target facial graph structure.

[0074] In this embodiment, based on the facial feature point topology of Dlib, graph structure construction processing can be performed on each facial feature point to form a corresponding target facial graph structure, such as... Figure 4 As shown in the diagram. In the target facial graph structure, the facial feature points serve as corresponding graph nodes, and the facial feature points with connectivity relationships are related, such as being connected through facial muscles.

[0075] Step b2: Based on the connection relationship in the target facial image structure, perform first association semantic mining on the feature point splicing vector corresponding to each facial feature point to form a feature point association vector corresponding to each facial feature point.

[0076] In this embodiment, after forming the target facial image structure, a first association semantic mining can be performed on the feature point concatenation vector corresponding to each facial feature point based on the connection relationships in the target facial image structure, forming a feature point association vector corresponding to each facial feature point. That is, since the semantic information between facial feature points with connection relationships is generally correlated (direct linkage of facial muscles), association semantic mining can be performed based on the corresponding connection relationships to extract the relevant semantic information.

[0077] Step b3: Based on the connection relationships in the facial graph structure, perform second association semantic mining on the feature point association vector corresponding to each facial feature point to form a feature point deep association vector corresponding to each facial feature point.

[0078] In this embodiment, after obtaining the feature point association vector, a second association semantic mining can be performed on the feature point association vector corresponding to each facial feature point based on the connection relationships in the facial graph structure, forming a feature point deep association vector corresponding to each facial feature point. The first association semantic mining differs from the second association semantic mining. For example, the first association semantic mining targets facial feature points with direct connections, while the second association semantic mining can target a broader range of facial feature points with certain correlations.

[0079] Step b4: Based on the feature point depth association vector corresponding to each of the facial feature points, determine the target image vector corresponding to the target image data.

[0080] In this embodiment, after obtaining the feature point depth correlation vector, the target image vector corresponding to the target image data can be determined based on the feature point depth correlation vector corresponding to each facial feature point. For example, the feature point depth correlation vectors corresponding to each facial feature point can be concatenated to obtain the corresponding target image vector. In one implementation, the concatenated result can be directly used as the target image vector. In another implementation, the concatenated result can be further processed, such as by pooling to reduce the vector size, thereby obtaining the corresponding target image vector.

[0081] It is understood that in step b2 above, the specific method for performing the first association semantic mining on the feature point concatenation vector corresponding to each facial feature point is not limited. For example, in an alternative implementation, considering that facial feature points with connectivity generally have a strong reinforcing effect on the semantic representation of expressions, step b2 above may include: First, for each facial feature point, based on the connectivity relationships in the target facial image structure, each other facial feature point connected to that facial feature point is determined. For example, in Figure 4 In the context, other facial feature points that are connected to facial feature point 1 include facial feature point 2 and facial feature point 3. Secondly, based on the feature point concatenation vector corresponding to each other facial feature point with a connection relationship, cross-attention processing is performed on the feature point concatenation vector corresponding to the facial feature point, and the facial attention vector corresponding to each other facial feature point is output. For example, based on the feature point concatenation vector corresponding to facial feature point 2, cross-attention processing can be performed on the feature point concatenation vector corresponding to facial feature point 1 to obtain facial attention vector 1; and based on the feature point concatenation vector corresponding to facial feature point 3, cross-attention processing can be performed on the feature point concatenation vector corresponding to facial feature point 1 to obtain facial attention vector 2. In this way, the semantic information of the association between related facial feature points can be effectively captured through cross-attention processing. Then, the feature point concatenation vector corresponding to the facial feature point and the facial attention vector corresponding to each other facial feature point with a connection relationship can be summed to obtain the feature point association vector corresponding to the facial feature point, such as the feature point concatenation vector corresponding to facial feature point 1 + facial attention vector 1 + facial attention vector 2.

[0082] It is understood that in step b3 above, the specific method for performing the second association semantic mining on the feature point association vector corresponding to each facial feature point is not limited. For example, in an alternative implementation, in order to consider that facial feature points without connection relationships also have a certain correlation with the representation of facial expression semantic information, step b3 above can further include the following: First, for each of the facial feature points, the facial feature point is taken as the center feature point; Secondly, the facial feature points in the facial image structure are traversed sequentially along a direction from away from the central feature point to closer to the central feature point. For example, the central feature point can be used as the starting point, and each other facial feature point can be used as the ending point. Then, the shortest path between the starting point and the ending point is determined. Thus, during the traversal, the facial feature points initially traversed are those whose shortest path has the maximum value, such as... Figure 4 Facial feature point 8 and facial feature point 9 in the image; Then, the current feature point association vector of the currently traversed facial feature point can be fused into the feature point association vector of the adjacent facial feature point based on cross-attention processing (i.e., cross-attention processing is applied to the feature point association vector of the adjacent facial feature point based on the current feature point association vector of the currently traversed facial feature point), forming the current feature point association vector of the adjacent facial feature point. Here, the adjacent facial feature point and the currently traversed facial feature point have a connection relationship, and the distance between the adjacent facial feature point and the center feature point (such as the length of the aforementioned shortest path) is less than the distance between the currently traversed facial feature point and the center feature point. For example, based on the current feature point association vector of facial feature point 8, cross-attention processing can be performed on the feature point association vector of facial feature point 6 to obtain the current feature point association vector of facial feature point 6; similarly, based on the current feature point association vector of facial feature point 9, cross-attention processing can be performed on the feature point association vector of facial feature point 7 to obtain the current feature point association vector of facial feature point 7; furthermore, for facial feature point 3, since cross-attention processing is performed on both the current feature point association vectors of facial feature point 6 and facial feature point 7, the average of the two cross-attention processing results can be calculated to obtain the current feature point association vector of facial feature point 3.

[0083] Finally, after forming the current feature point association vector of the central feature point, the current feature point association vector of the central feature point is used as the corresponding feature point depth association vector.

[0084] Thirdly, regarding step S130, it should be noted that the specific method of performing second semantic mining on the target audio data and outputting the target audio vector is not limited and can be selected according to actual needs. For example, in an alternative implementation, the target audio data can be converted to form a corresponding spectrogram. In this way, the spectrogram can be processed by a convolutional neural network model to obtain the corresponding target audio vector.

[0085] Fourthly, regarding step S140, it should be noted that the specific method of outputting the global sentiment semantic vector is not restricted and can be selected according to actual needs.

[0086] For example, in an alternative implementation, the target image vector and the target audio vector can be concatenated or summed to obtain the corresponding global emotion semantic vector.

[0087] For example, in another alternative implementation, in order to make the semantic information represented by the output global emotion semantic vector richer, the above step 140 may include steps S141, S142, S143, S144 and S145, the contents of each step are as follows.

[0088] Step S141: Perform text conversion processing on the target audio data to form target text data, and obtain the target physiological data of the target person during the emotional state test.

[0089] In this embodiment, the target audio data can be processed by text conversion (any existing speech-to-text conversion technology can be used, without specific limitations or descriptions here) to form target text data, and target physiological data of the target person during the emotional state test can be obtained. The target physiological data includes at least heart rate data, which can be obtained, for example, by detection using a heart rate sensor.

[0090] Step S142: Perform third semantic mining on the target text data to form a target text vector, and perform fourth semantic mining on the target physiological data to form a target physiological vector.

[0091] In this embodiment, after obtaining the target text data and target physiological data, on the one hand, third semantic mining can be performed on the target text data to form a target text vector. For example, the BERT model (a deep learning model based on Transformer) can be used to extract semantic features (keyword sentiment) and grammatical features (sentence complexity, rhetorical question frequency, etc.) to enhance feature accuracy. On the other hand, fourth semantic mining can be performed on the target physiological data to form a target physiological vector. For example, CNN (convolutional neural network, the specific mining process can refer to relevant existing technologies) or DBN (deep belief network) can be used for feature mining. Additionally, the heart rate data can be converted into a spectrogram before performing convolutional mining.

[0092] Step S143: Based on the clarity index and facial feature point stability index of the facial region in the target image data, determine the confidence level of the target image data; based on the signal-to-noise ratio of the target audio data and the target physiological data, determine the confidence level of the target audio data and the target physiological data, respectively; and based on the dialogue coherence index represented by the target text data, determine the confidence level of the target text data.

[0093] In this embodiment, the confidence level of the target image data can be determined based on the sharpness index (such as the SSIM index) and facial feature point stability index (such as Dlib tracking jitter rate, where a lower jitter rate indicates higher stability) of the facial region in the target image data (exemplarily, there is a positive correlation between the sharpness index and the confidence level, and a positive correlation between the facial feature point stability index and the confidence level). Furthermore, the confidence level of the target audio data and the target physiological data can be determined based on their respective signal-to-noise ratios (SNRs) (exemplarily, there is a positive correlation between SNRs and confidence levels). Finally, the confidence level of the target text data can be determined based on the dialogue coherence index (such as topic consistency between sentences) represented by the target text data (exemplarily, there is a positive correlation between dialogue coherence indexes and confidence levels). In Natural Language Processing (NLP), BERT (Bidirectional Encoder Representations from Transformers) is a powerful pre-trained model capable of capturing contextual information in text. When inputting text, the BERT model typically adds a [CLS] (classification) tag at the beginning of each sentence. This [CLS] tag doesn't represent a specific word, but rather summarizes the semantic information of the entire sentence or text. The BERT model converts each word (token) into a vector, and the [CLS] tag represents the semantic representation of the entire sentence or input text. Through its bidirectional encoding capabilities, the BERT model understands the relationships between words in the context, thus grasping the semantic relationships between sentences. In dialogue, BERT's [CLS] tags represent the semantic information of the entire dialogue segment. When the topic within a dialogue segment is consistent, the [CLS] tag representation will reflect good coherence between sentences. By comparing the [CLS] tag vectors from different dialogue rounds, BERT can help assess whether the dialogue maintains contextual consistency. If the topic of the dialogue changes at some point, the probability distribution of the [CLS] tags may change significantly, reflecting incoherence in the dialogue.

[0094] Step S144: Based on the confidence levels of the target image data, the target audio data, the target physiological data, and the target text data, and in conjunction with the confidence levels of historical image data, historical audio data, historical physiological data, and historical text data, determine the weight coefficients corresponding to each semantic vector.

[0095] In this embodiment, the weight coefficients corresponding to each semantic vector are determined based on the confidence levels of the target image data, the target audio data, the target physiological data, and the target text data, combined with the confidence levels of historical image data, historical audio data, historical physiological data, and historical text data. For example, the weight coefficients corresponding to the target image vectors can be determined based on the confidence levels of the target image data and the historical image data (or the weight coefficients corresponding to the historical image data). Furthermore, it should be noted that the personnel status analysis in this embodiment is performed in real time. As the test time progresses, new target personnel conversation data will be continuously generated, and the current target personnel conversation data will become historical personnel conversation data. Therefore, to avoid misjudgments due to instantaneous fluctuations, the current confidence level can be combined with the confidence level or weight coefficients from previous times to obtain the current weight coefficient. For example, the average of the current confidence level and the weight coefficients from the previous five times can be calculated as the current weight coefficient. In addition, for the first time point, the corresponding confidence level can be directly used as the weight coefficient of the current time point. For the second to fifth time points, the average of the corresponding confidence level and the weight coefficients of the previous time points can be used as the weight coefficient of the current time point.

[0096] Step S145: Based on the weight coefficients corresponding to each semantic vector, perform a weighted summation calculation on the target image vector, the target audio vector, the target text vector, and the target physiological vector to output a global emotion semantic vector.

[0097] In this embodiment of the application, after obtaining the weight coefficients corresponding to each semantic vector, the target image vector, the target audio vector, the target text vector, and the target physiological vector can be weighted and summed based on the weight coefficients corresponding to each semantic vector to output a global emotion semantic vector, that is, global emotion semantic vector = target image vector * corresponding weight coefficient + target audio vector * corresponding weight coefficient + target text vector * corresponding weight coefficient + target physiological vector * corresponding weight coefficient.

[0098] Fifthly, regarding step S150, it should be noted that the specific method for obtaining the target person's emotional type is not limited and can be selected according to actual needs.

[0099] For example, in an alternative implementation, the global emotional semantic vector can be classified and output based on an emotion classifier (an MLP model) to obtain the target person's emotion type. For instance, there can be six emotion types: depression, anxiety, anger, calmness, happiness, and others (adjustable according to actual circumstances); the intensity level can be five levels: 1 = mild, 5 = extreme. Furthermore, as mentioned earlier, new target person conversation data will be continuously generated as the testing time progresses, and the currently output emotion type can serve as a reference for subsequent tests. For example, when the emotion type is anxiety and the intensity level reaches 3, the speaking speed can be reduced and a deep breathing guidance voice can be inserted. Another example is when the emotion type is depression and the intensity level reaches 4, an active questioning mode can be used and a psychological counselor can be recommended (i.e., subsequent testing can be conducted through a professional psychological counselor). Yet another example is when the emotion type is anger and the intensity level reaches 2, a neutral tone can be used and rhetorical questions can be avoided.

[0100] In summary, the personnel state analysis method and device based on multimodal data provided in this application first extracts target image data and target audio data from target personnel conversation data; second, performs first semantic mining on the target image data to output a target image vector; then, performs second semantic mining on the target audio data to output a target audio vector; further, fuses the target image vector and target audio vector to output a global emotion semantic vector; finally, classifies based on the global emotion semantic vector to obtain the emotion type. Based on the above, on the one hand, it utilizes the semantic information of both image and audio modalities to analyze emotion types, making the semantic information used as the basis for analysis richer, and achieving higher analysis reliability compared to conventional techniques that use single-modal data for analysis. On the other hand, since the semantic mining process for the image modality includes both local and global semantic mining of facial expressions, it ensures both the accuracy of expression semantic mining and the richness of semantics, thereby improving the semantic representation capability of the mined vectors and addressing the problem of relatively low reliability in state analysis in existing technologies.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0102] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0103] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for analyzing personnel status based on multimodal data, characterized in that, include: Target image data and target audio data are extracted from the target person's conversation data. The target image data is formed by image acquisition during the target person's emotional state test, and the target audio data is used to reflect the conversation content of the emotional state test. The target image data is subjected to depthwise convolution processing to output an image depthwise convolution vector, wherein the image depthwise convolution vector is used to represent the facial semantic information of the target person; keypoint detection is performed on the target image data, and feature vector construction processing is performed on each obtained facial feature point to form a feature point semantic vector corresponding to each facial feature point; for each facial feature point, the feature point semantic vector corresponding to the facial feature point is concatenated with the image depthwise convolution vector to form a corresponding feature point concatenation vector; based on the facial feature point topology of Dlib, graph structure construction processing is performed on each facial feature point to form a corresponding target facial graph structure, wherein, in the target facial... In the graph structure, the facial feature points serve as corresponding graph nodes. Based on the connectivity in the target facial graph structure, a first association semantic mining is performed on the feature point concatenation vector corresponding to each facial feature point to form a feature point association vector corresponding to each facial feature point. Based on the connectivity in the facial graph structure, a second association semantic mining is performed on the feature point association vector corresponding to each facial feature point to form a feature point depth association vector corresponding to each facial feature point. The first association semantic mining differs from the second association semantic mining. Based on the feature point depth association vector corresponding to each facial feature point, the target image vector corresponding to the target image data is determined. Perform second semantic mining on the target audio data and output the target audio vector; By fusing the target image vector and the target audio vector, a global emotion semantic vector is output. The target person's emotion type is obtained by classifying the global emotion semantic vector.

2. The personnel status analysis method based on multimodal data according to claim 1, characterized in that, The step of performing depthwise convolution processing on the target image data and outputting an image depthwise convolution vector includes: Color channel data is extracted from the target image data to obtain the corresponding red channel data, green channel data and blue channel data respectively; Convolutions are performed on the red channel data, the green channel data, and the blue channel data respectively to form corresponding red channel vectors, green channel vectors, and blue channel vectors; Two different depth mining operations are performed inside the red channel vector, the green channel vector, and the blue channel vector respectively to obtain two red depth vectors, two green depth vectors, and two blue depth vectors. The two red depth vectors are then correlated and aggregated to form a red aggregated vector, the two green depth vectors are correlated and aggregated to form a green aggregated vector, and the two blue depth vectors are correlated and aggregated to form a blue aggregated vector. Based on the preset red weights, green weights, and blue weights, the red aggregate vector, the green aggregate vector, and the blue aggregate vector are weighted and summed to obtain the corresponding image depth convolution vector. The red weights, green weights, and blue weights are formed during the learning process of the corresponding neural network model on the sample data, and the neural network model has the following constraints: the red weight is greater than the green weight, and the green weight is greater than the blue weight.

3. The personnel status analysis method based on multimodal data according to claim 1, characterized in that, The steps of performing key point detection on the target image data and constructing feature vectors for each obtained facial feature point to form a feature point semantic vector corresponding to each facial feature point include: Dlib is used to perform face bounding box detection and facial feature point detection on the target image data to obtain the corresponding face bounding box and each facial feature point. For each facial feature point, the center offset corresponding to the facial feature point is calculated based on the coordinate parameters of the facial feature point and the coordinate parameters of the center point of the face bounding box. The distance between adjacent feature points is determined based on the coordinate parameters of the facial feature point and the coordinate parameters of other adjacent facial feature points. Based on the coordinate parameters, center offset, and distance to adjacent feature points of each facial feature point, a semantic vector corresponding to each facial feature point is constructed.

4. The personnel status analysis method based on multimodal data according to claim 1, characterized in that, The step of performing first association semantic mining on the feature point concatenation vector corresponding to each facial feature point based on the connection relationship in the target facial image structure to form the feature point association vector corresponding to each facial feature point includes: For each of the facial feature points, based on the connection relationships in the target facial image structure, each other facial feature point that has a connection relationship with that facial feature point is determined; Based on the feature point concatenation vector corresponding to each other facial feature point with a connection relationship, cross attention processing is performed on the feature point concatenation vector corresponding to the facial feature point, and the facial attention vector corresponding to each other facial feature point is output. The feature point concatenation vector corresponding to the facial feature point is summed with the facial attention vector corresponding to each other facial feature point that has a connection relationship, to obtain the feature point association vector corresponding to the facial feature point.

5. The personnel status analysis method based on multimodal data according to claim 1, characterized in that, The step of performing second association semantic mining on the feature point association vector corresponding to each facial feature point based on the connection relationship in the facial graph structure to form a feature point deep association vector corresponding to each facial feature point includes: For each of the facial feature points, the facial feature point is taken as the center feature point; The facial feature points in the facial image structure are traversed sequentially along a direction from away from the central feature point to closer to the central feature point. The current feature point association vector of the currently traversed facial feature point is fused into the feature point association vector of the adjacent facial feature point based on cross-attention processing to form the current feature point association vector of the adjacent facial feature point. The adjacent facial feature point and the currently traversed facial feature point have a connection relationship, and the distance between the adjacent facial feature point and the center feature point is less than the distance between the currently traversed facial feature point and the center feature point. After forming the current feature point association vector of the central feature point, the current feature point association vector of the central feature point is used as the corresponding feature point depth association vector.

6. The personnel status analysis method based on multimodal data according to any one of claims 1-5, characterized in that, The step of fusing the target image vector and the target audio vector to output a global emotion semantic vector includes: The target audio data is processed by text conversion to form target text data, and target physiological data of the target person during the emotional state test is obtained, wherein the target physiological data includes at least heart rate data. The target text data is subjected to third semantic mining to form a target text vector, and the target physiological data is subjected to fourth semantic mining to form a target physiological vector; Based on the clarity index and facial feature point stability index of the facial region in the target image data, the confidence level of the target image data is determined. Based on the signal-to-noise ratio of the target audio data and the target physiological data, the confidence level of the target audio data and the target physiological data are determined respectively. Based on the dialogue coherence index represented by the target text data, the confidence level of the target text data is determined. Based on the confidence levels of the target image data, the target audio data, the target physiological data, and the target text data, and combined with the confidence levels of historical image data, historical audio data, historical physiological data, and historical text data, the weight coefficients corresponding to each semantic vector are determined. Based on the weight coefficients corresponding to each semantic vector, the target image vector, the target audio vector, the target text vector, and the target physiological vector are weighted and summed to output a global emotion semantic vector.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the personnel status analysis method based on multimodal data as described in any one of claims 1-6.

Images