Audio and video acquisition processing system and method based on big data technology

Abstract

The invention relates to the field of audio and video acquisition and processing, and particularly discloses an audio and video acquisition and processing system and method based on a big data technology, and the method comprises the steps: firstly obtaining the audio data of a patient collected by voice recording equipment and the video data of the patient collected by medical monitoring equipment, and then carrying out the deep learning technology; according to the method, feature extraction and correlation analysis are carried out on the two images, and finally emotion label information of the patient is obtained through a classifier, so that the emotional state and psychological health condition of the patient are better known, personalized diagnosis and treatment planning are carried out, and the quality and effect of medical services are improved.

Description

Technical Field

[0001] The present application relates to the field of audio and video acquisition and processing, and more specifically, to an audio and video acquisition and processing system and method based on big data technology. Background Art

[0002] The audio and video acquisition and processing system is a comprehensive system that is responsible for acquiring, processing and managing audio and video data. It usually includes video acquisition equipment, audio acquisition equipment, video processing software, audio processing software and storage devices.

[0003] In the field of medical mental health, diagnosis and treatment often rely on patients’ verbal statements and doctors’ assessments. However, patients sometimes have difficulty accurately expressing their emotional states because they may lack the professional terms to accurately describe their feelings. And, despite professional training and education, doctors may not be able to fully understand patients’ emotional needs within a limited time. In addition, doctors may be influenced by their own experiences, biases, or limitations, which may lead to deviations in their understanding of patients’ emotional states.

[0004] Therefore, a system and method for audio and video acquisition and processing based on big data technology is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides an audio and video acquisition and processing system and method based on big data technology, which first obtains the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally obtains the patient's emotional label information through a classifier to better understand the patient's emotional state and mental health status, so as to carry out personalized diagnosis and treatment plans, thereby improving the quality and effect of medical services.

[0006] According to one aspect of the present application, there is provided an audio and video acquisition and processing system based on big data technology, which includes:

[0007] A patient audio and video data acquisition module, used to acquire patient audio data collected by a voice recording device and patient video data collected by a medical monitoring device;

[0008] A patient audio and video data extraction module, used to extract a patient audio feature vector and a patient facial salient feature vector from the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device;

[0009] The multimodal emotion recognition module is used to obtain the patient's emotion label information based on the patient's audio feature vector and the patient's facial salient feature vector.

[0010] In the above-mentioned audio and video acquisition and processing system based on big data technology, the patient audio and video data extraction module includes: a patient audio data feature encoding unit, which is used to feature encode the patient audio data collected by the voice recording device to obtain the patient audio feature vector; a patient video data feature encoding unit, which is used to feature encode the patient video data from the medical monitoring device to obtain the patient's facial salient feature vector.

[0011] In the above-mentioned audio and video acquisition and processing system based on big data technology, the patient audio data feature encoding unit includes: extracting a patient logarithmic Mel spectrogram, a patient cochlear spectrogram and a patient constant Q transform spectrogram from the patient audio data collected by the voice recording device; arranging the patient logarithmic Mel spectrogram, the patient cochlear spectrogram and the patient constant Q transform spectrogram into a patient multi-channel sound spectrogram; and obtaining the patient audio feature vector by passing the patient multi-channel sound spectrogram through a patient audio feature encoder using a channel attention mechanism.

[0012] In the above-mentioned audio and video acquisition and processing system based on big data technology, the patient video data feature encoding unit includes: a patient video key frame extraction subunit, used to extract multiple patient video key frames from the patient video data of the medical monitoring device; a video key frame target feature extraction subunit, used to extract target features from the multiple patient video key frames to obtain a patient facial salient feature map; a patient facial salient feature pooling subunit, used to perform maximum pooling on the patient facial salient feature map to obtain the patient facial salient feature vector.

[0013] In the above-mentioned audio and video acquisition and processing system based on big data technology, the video key frame target feature extraction subunit includes: passing the multiple patient video key frames through a patient facial target detection network to obtain multiple patient facial regions of interest; passing the multiple patient facial regions of interest through a patient facial salient target detector to obtain the patient facial salient feature map.

[0014] In the above-mentioned audio and video acquisition and processing system based on big data technology, the multimodal emotion recognition module includes: a patient emotion feature fusion unit, which is used to perform cross-modal data fusion of the patient audio feature vector and the patient facial salient feature vector based on perspective alignment projection to obtain the patient emotion classification feature vector; a patient emotion feature optimization unit, which is used to perform scattering feedback depth dependency adjustment based on class divergence space on the patient emotion classification feature vector to obtain an optimized patient emotion classification feature vector; a multimodal patient emotion label recognition unit, which is used to pass the optimized patient emotion classification feature vector through a classifier to obtain a classification result, and the classification result is used to represent the patient's emotion label information.

[0015] In the above-mentioned audio and video acquisition and processing system based on big data technology, the patient emotion feature fusion unit includes: constructing a patient audio-facial significant distance topological matrix between the patient audio feature vector and the patient facial significant feature vector; constructing a patient audio-facial significant feature value granularity association matrix between the patient audio feature vector and the patient facial significant feature vector; performing topological association modulation on the patient audio-facial significant feature value granularity association matrix based on the patient audio-facial significant distance topological matrix to obtain a patient audio-facial significant topological modulation association matrix; using the patient audio-facial significant topological modulation association matrix as a perspective co-projection space, projecting the patient audio feature vector and the patient facial significant feature vector into the perspective co-projection space to obtain a perspective modulated patient audio feature vector and a perspective modulated patient facial significant feature vector; fusing the perspective modulated patient audio feature vector and the perspective modulated patient facial significant feature vector to obtain the patient emotion classification feature vector.

[0016] In the above-mentioned audio and video acquisition and processing system based on big data technology, the patient emotion feature optimization unit includes: determining the mean of the patient emotion classification feature vector; accumulating and summing the patient emotion classification feature vectors to obtain the sum value of the patient emotion classification feature vectors; subtracting the mean of the patient emotion classification feature vector from the sum value of the patient emotion classification feature vector, dividing the result by a predetermined hyperparameter and then dividing the result by the length of the patient emotion classification feature vector to obtain a first quotient value; calculating an exponential function value with a natural constant as the base for the first quotient value to obtain a first natural exponential function value; and performing point-wise multiplication of the first natural exponential function value and the patient emotion classification feature vector to obtain an optimized patient emotion classification feature vector.

[0017] According to another aspect of the present application, a method for audio and video acquisition and processing based on big data technology is provided, which includes:

[0018] obtaining patient audio data collected by a voice recording device and patient video data collected by a medical monitoring device;

[0019] Extracting a patient audio feature vector and a patient facial salient feature vector from the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device;

[0020] Based on the patient's audio feature vector and the patient's facial salient feature vector, the patient's emotion label information is obtained.

[0021] Compared with the prior art, the present application provides an audio and video acquisition and processing system and method based on big data technology, which first obtains the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally uses a classifier to obtain the patient's emotional label information to better understand the patient's emotional state and mental health status, so as to carry out personalized diagnosis and treatment plans, thereby improving the quality and effectiveness of medical services. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 It is a block diagram of an audio and video acquisition and processing system based on big data technology according to an embodiment of the present application.

[0024] Figure 2 It is a block diagram of a patient audio and video data extraction module in an audio and video acquisition and processing system based on big data technology according to an embodiment of the present application.

[0025] Figure 3 It is a block diagram of a patient video data feature encoding unit in an audio and video acquisition and processing system based on big data technology according to an embodiment of the present application.

[0026] Figure 4 It is a block diagram of a multimodal emotion recognition module in an audio and video acquisition and processing system based on big data technology according to an embodiment of the present application.

[0027] Figure 5 This is a flowchart of an audio and video acquisition and processing method based on big data technology according to an embodiment of the present application.

[0028] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0030] Figure 1 FIG. 1 is a block diagram of an audio and video acquisition and processing system based on big data technology according to an embodiment of the present application. Figure 1 As shown, according to the audio and video acquisition and processing system 100 based on big data technology according to the embodiment of the present application, it includes: a patient audio and video data acquisition module 110, which is used to acquire the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device; a patient audio and video data extraction module 120, which is used to extract the patient audio feature vector and the patient facial significant feature vector from the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device; a multimodal emotion recognition module 130, which is used to obtain the patient's emotion label information based on the patient audio feature vector and the patient's facial significant feature vector.

[0031] In the above-mentioned audio and video acquisition and processing system 100 based on big data technology, the patient audio and video data acquisition module 110 is used to acquire the patient audio data collected by the voice recording device and the patient video data by the medical monitoring device. It should be understood that in the field of medical mental health, the patient's emotional state is crucial to diagnosis and treatment, but it may be difficult for patients to accurately describe their feelings, and doctors may not fully understand the patient's emotional needs. The doctor's personal experience and prejudice may lead to misunderstandings of the patient's emotional state. Therefore, the technical solution of the present application proposes to use a voice recording device to collect patient audio data and a medical monitoring device to obtain patient video data, and combine deep learning technology to obtain the patient's emotional label information. In this way, the patient's emotional state and mental health status can be more comprehensively understood, so as to formulate personalized diagnosis and treatment plans and improve the quality and effect of medical services. Among them, the patient audio data collected by the voice recording device can capture the patient's voice information, including speech content, intonation, emotions, etc. By analyzing the patient's voice data, the emotional information and voice features in the voice can be identified. The acquisition of such data can help medical personnel better understand the patient's status and provide more personalized and accurate medical services. On the other hand, the patient video data of the medical monitoring device provides visual information of the patient's face. These data can include the patient's facial expressions, eyes, facial muscle movements, etc. By analyzing these video data, the significant features of the patient's face can be obtained, such as changes in facial expressions, facial skin conditions, etc. This information is of great significance for diagnosing the patient's emotional state. The comprehensive use of voice data and video data can achieve multimodal emotion recognition and personalized medical services. Through the application of big data technology, the system can automatically extract audio features and video features, and combine them with emotion recognition algorithms to provide medical staff with more comprehensive patient information.

[0032] In the above-mentioned audio and video acquisition and processing system 100 based on big data technology, the patient audio and video data extraction module 120 is used to extract the patient audio feature vector and the patient facial significant feature vector from the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device. It should be understood that extracting audio feature vectors and facial significant feature vectors from the patient audio data and video data can provide more comprehensive and multi-dimensional patient information for the medical field. This information can not only help medical personnel better understand the patient's status and needs, but also provide support and guidance for personalized medical services, disease diagnosis and monitoring, etc.

[0033] Figure 2 FIG. 1 is a block diagram of a patient audio and video data extraction module in an audio and video acquisition and processing system based on big data technology according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the patient audio and video data extraction module 120 includes: a patient audio data feature encoding unit 121, which is used to feature encode the patient audio data collected by the voice recording device to obtain the patient audio feature vector; a patient video data feature encoding unit 122, which is used to feature encode the patient video data from the medical monitoring device to obtain the patient's facial significant feature vector. It should be understood that feature encoding can help extract key features from audio data. Audio data usually contains a lot of information, including sound frequency, volume, speaking speed, etc. Through the process of feature encoding, the system can extract representative features from these complex raw data, such as spectral features of speech, time domain features of sound, etc. These features can better characterize the essential characteristics of audio data.

[0034] Furthermore, feature coding can help extract key facial features from video data. The face is one of the main carriers of human emotional expression and contains rich information, such as facial expressions, eyes, wrinkles, etc. Through feature coding, the system can extract representative facial features from video data, such as facial contours, eye positions, lip movements, etc., and obtain the patient's facial salient feature vector through feature coding, which helps to extract key facial information, reduce data dimensions, and improve data expression efficiency.

[0035] In a specific embodiment of the present application, the patient audio data feature encoding unit 121 includes: extracting a patient logarithmic Mel spectrogram, a patient cochlear spectrogram and a patient constant Q transform spectrogram from the patient audio data collected by the voice recording device; arranging the patient logarithmic Mel spectrogram, the patient cochlear spectrogram and the patient constant Q transform spectrogram into a patient multi-channel spectrogram; and obtaining the patient audio feature vector by using a patient audio feature encoder of a channel attention mechanism. It should be understood that the patient logarithmic Mel spectrogram is a commonly used audio feature representation method that can better simulate the way the human auditory system perceives sound. By extracting the patient logarithmic Mel spectrogram, the spectral information in the audio data can be converted into a Mel scale to better capture the spectral characteristics of the audio signal. The patient cochlear spectrogram is a feature representation method that describes the propagation and processing process of the audio signal in the human ear. By extracting the patient cochlear spectrogram, it is possible to reflect the processing of the audio signal by the cochlea at different frequencies, which helps to understand the patient's auditory characteristics. In addition, the patient constant Q transform spectrogram is a commonly used spectrum representation method in audio signal processing, which can better capture the frequency distribution characteristics of audio signals. Extracting the patient constant Q transform spectrogram can help analyze the energy distribution of different frequency components in the audio signal. Extracting the patient logarithmic Mel spectrogram, patient cochlear spectrogram and patient constant Q transform spectrogram from the patient audio data collected by the voice recording device can help extract key audio features and provide an important data basis for auditory diagnosis, monitoring and research in the medical field.

[0036] Furthermore, the patient log-Mel spectrogram, patient cochlear spectrogram and patient constant-Q transform spectrogram represent the characterization of audio data in different feature spaces. The log-Mel spectrogram can capture the spectral characteristics of the audio signal, the cochlear spectrogram reflects the propagation and processing of the audio signal in the human ear, and the constant-Q transform spectrogram highlights the spectral resolution of the audio signal. Combining these spectrograms in different feature spaces can comprehensively consider aspects such as the spectral information, bioacoustic characteristics and spectral resolution of the audio data. Arranging these spectrograms into a multi-channel spectrogram can provide more comprehensive and rich audio feature information.

[0037] Furthermore, by using the channel attention mechanism, the model can more flexibly and intelligently choose to focus on important information of different channels when encoding the patient's multi-channel spectrogram. The channel attention mechanism can dynamically adjust the weight of each channel in the feature encoding process according to the importance of different channels, so that the model pays more attention to information that is beneficial to the current task and improves the efficiency and accuracy of feature encoding. Among them, the channel attention mechanism can help the model learn more representative and discriminative features from multi-channel spectrograms and improve the representation ability of audio features. Specifically, each layer of the patient audio feature encoder using the channel attention mechanism performs the following on the input data in the forward pass of the layer: convolution processing on the input data based on a two-dimensional convolution kernel to generate a convolution feature map; pooling processing on the convolution feature map to generate a pooling feature map; activation processing on the pooling feature map to generate an activation feature map; calculating the quotient of the eigenvalue mean of the feature matrix corresponding to each channel in the activation feature map and the sum of the eigenvalue mean of the feature matrix corresponding to all channels as the weighting coefficient of the feature matrix corresponding to each channel; and weighting the feature matrix of each channel with the weighting coefficient of each channel in the activation feature map to generate a channel attention feature map; wherein the output of the last layer of the patient audio feature encoder using the channel attention mechanism is the patient audio feature vector.

[0038] Figure 3 FIG. 1 is a block diagram of a patient video data feature encoding unit in an audio and video acquisition and processing system based on big data technology according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the patient video data feature encoding unit 122 includes: a patient video key frame extraction subunit 1221, which is used to extract multiple patient video key frames from the patient video data of the medical monitoring device; a video key frame target feature extraction subunit 1222, which is used to extract target features from the multiple patient video key frames to obtain a patient facial significant feature map; a patient facial significant feature pooling subunit 1223, which is used to perform maximum pooling on the patient facial significant feature map to obtain the patient facial significant feature vector. It should be understood that extracting multiple patient video key frames can help medical professionals obtain important video information of patients more quickly. Through key frame extraction, the most representative and information-rich frames can be screened out from a large amount of video data, reducing the time and energy cost of medical professionals when watching videos, and improving the efficiency of information acquisition. In addition, extracting multiple key frames from patient video data can also help medical professionals understand the treatment effect and disease changes of patients more comprehensively, thereby improving the efficiency of medical monitoring and diagnosis and treatment, and providing better support for the health management and treatment of patients.

[0039] Furthermore, through target feature extraction, the significant features of the patient's face can be extracted from multiple patient video key frames. These features may include information such as facial expressions, eye conditions, skin color changes, etc., which are important clues for medical professionals to diagnose and monitor diseases. By extracting the facial significant feature map, the key information of the patient's face can be intuitively displayed, which helps medical professionals to understand the changes in the patient's facial features more accurately. The face is an important information transmission and expression area of ​​the human body, and many emotions are expressed through the face. By analyzing the patient's facial significant feature map, medical professionals can observe subtle changes in the patient's face, so as to promptly detect signs of disease or monitor the development of emotions, and provide timely guidance and adjustments for treatment.

[0040] Furthermore, max pooling can help reduce the dimensionality and complexity of the data. By applying the max pooling operation on the patient's facial salient feature map, the maximum value in each feature map area can be extracted, thereby compressing the rich information in the original feature map into a more concise feature representation. In this way, the obtained patient's facial salient feature vector has a lower dimension and is easier to store, transmit and process. In addition, max pooling helps to retain important feature information. By selecting the maximum value in each feature map area as the representative feature, max pooling can effectively retain the most significant and discriminative information in the facial salient feature map. This helps to reduce information loss while improving the robustness and distinguishability of features, making the patient's facial feature vector more representative and effective.

[0041] In a specific embodiment of the present application, the video key frame target feature extraction subunit 1222 includes: passing the multiple patient video key frames through a patient facial target detection network to obtain multiple patient facial regions of interest; passing the multiple patient facial regions of interest through a patient facial salient target detector to obtain the patient facial salient feature map. It should be understood that through the patient facial target detection network, the facial regions in the multiple patient video key frames can be accurately located and identified. This means that the system can automatically identify the facial position of each patient in the video without manual intervention, thereby improving processing efficiency and accuracy. The patient facial regions of interest obtained by the detection network can help medical professionals quickly and accurately locate the patient's facial information. Specifically, the patient facial target detection network is used to process the multiple patient video key frames with the following target detection formula to obtain the multiple patient facial regions of interest;

[0042] Among them, the target detection formula is:

[0043] ROI=H(ψ det ,B)=(cls(ψ det ,B),Regr(ψdet ,B))

[0044] Among them, ψ det are multiple patient video key frames, B is the anchor frame, ROI is the facial region of interest of multiple patients, cls(ψ det ,B) indicates classification, Regr(ψ det ,B) indicates regression.

[0045] Further, through the patient face salient object detector, the patient face regions of interest can be effectively identified and located. These regions may include key parts of the face, such as eyes, mouth, nose, etc., as well as specific areas where facial signs may exist. The obtained patient face salient feature map helps to highlight the key features of the face. In the facial salient feature map, the key information of the patient's face is highlighted, allowing medical professionals to observe the details and changes of the face more clearly. Specifically, each layer of the patient face salient object detector is used to perform the following on the input data in the forward pass of the layer: convolve the input data using a first convolution kernel to obtain a first convolution feature map; convolve the first convolution feature map using a second convolution kernel to obtain a second convolution feature map, wherein the size of the first convolution kernel is larger than the size of the second convolution kernel; perform pooling on the second convolution feature map to obtain a pooled feature map; and, perform activation on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the patient face salient object detector is the patient face salient feature map, and the input of the first layer of the patient face salient object detector is the multiple patient face regions of interest.

[0046] In the above-mentioned audio and video acquisition and processing system 100 based on big data technology, the multimodal emotion recognition module 130 is used to obtain the patient's emotion label information based on the patient's audio feature vector and the patient's facial significant feature vector. It should be understood that the audio feature vector and the facial significant feature vector provide different but complementary information. The audio feature vector can capture the intonation, pitch, speech speed and other information in the patient's speech, while the facial significant feature vector reflects the patient's facial expression and emotional state. By combining these two types of information, the patient's emotional state can be more fully understood, thereby providing medical workers with more clues about the patient's inner feelings. By analyzing the patient's audio feature vector and facial significant feature vector, the extraction and classification of emotion label information can be achieved. Emotion label information can help medical workers understand the patient's emotional state, including different emotional types such as joy, anger, sorrow and happiness. This is very important for assessing the patient's mental health, adjusting treatment plans, and providing personalized care and support.

[0047] Figure 4FIG. 1 is a block diagram of a multimodal emotion recognition module in an audio and video acquisition and processing system based on big data technology according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the multimodal emotion recognition module 130 includes: a patient emotion feature fusion unit 131, which is used to perform cross-modal data fusion of the patient audio feature vector and the patient facial salient feature vector based on perspective alignment projection to obtain a patient emotion classification feature vector; a patient emotion feature optimization unit 132, which is used to perform scattering feedback depth dependency adjustment based on class divergence space on the patient emotion classification feature vector to obtain an optimized patient emotion classification feature vector; a multimodal patient emotion label recognition unit 133, which is used to pass the optimized patient emotion classification feature vector through a classifier to obtain a classification result, and the classification result is used to represent the patient's emotion label information.

[0048] It should be understood that in the technical solution of the present application, considering that the patient audio feature vector and the patient facial significant feature vector are extracted from audio data and video data respectively, they represent two different types of information: one is sound information and the other is visual information. Due to their essential differences and different extraction methods, these two feature vectors have significant feature heterogeneity. Audio feature vectors usually contain information about sound frequency, intensity and other acoustic properties, such as logarithmic Mel spectrograms, cochlear spectrograms and constant Q transform spectrograms. Facial significant feature vectors mainly capture visual information, such as changes in facial expressions, eye movements, etc., which are extracted from video data. Because they are essentially different, there is significant feature heterogeneity between the patient audio feature vector and the patient facial significant feature vector. Specifically, the patient audio feature vector and the substrate spatial structure change feature vector may exist on different manifolds, which means that they have different distribution laws in their respective spaces. Directly splicing them together may result in the final feature vector not having a clear manifold in the new space, that is, the feature manifold smoothness is insufficient. At the same time, since the two feature vectors come from different sources, they may not be well mapped to each other in the fused feature space, that is, one type of feature may not accurately reflect the feature changes of another type, resulting in increased uncertainty in feature expression. Based on this, in the technical solution of the present application, cross-modal data fusion based on perspective alignment projection is performed on the patient's audio feature vector and the patient's facial salient feature vector.

[0049] Among them, the patient audio feature vector and the patient facial salient feature vector are subjected to cross-modal data fusion based on perspective alignment projection to obtain the patient emotion classification feature vector, including: constructing a patient audio-facial salient distance topological matrix between the patient audio feature vector and the patient facial salient feature vector; constructing a patient audio-facial salient feature value granularity association matrix between the patient audio feature vector and the patient facial salient feature vector; performing topological association modulation on the patient audio-facial salient feature value granularity association matrix based on the patient audio-facial salient distance topological matrix to obtain a patient audio-facial salient topological modulation association matrix; using the patient audio-facial salient topological modulation association matrix as the perspective co-projection space, projecting the patient audio feature vector and the patient facial salient feature vector into the perspective co-projection space to obtain a perspective modulated patient audio feature vector and a perspective modulated patient facial salient feature vector; fusing the perspective modulated patient audio feature vector and the perspective modulated patient facial salient feature vector to obtain the patient emotion classification feature vector.

[0050] Specifically, the patient audio feature vector and the patient facial salient feature vector are subjected to cross-modal data fusion based on perspective alignment projection to obtain the patient emotion classification feature vector according to the following formula; wherein the formula is:

[0051] M t =D(V1,V2)

[0052]

[0053] V c =α·V1′+β·V2′

[0054] Wherein, V1 represents the patient audio feature vector, V2 represents the patient facial salient feature vector, and D(V1, V2) is the distance topological matrix between the patient audio feature vector and the patient facial salient feature vector, i.e., d i,j =d(v 1i ,v 2j ), d i,j is the eigenvalue of the position (i, j) in the distance topology matrix, d(v 1i ,v 2j ) represents the distance between the eigenvalue of the i-th position of the patient's audio feature vector and the eigenvalue of the j-th position of the patient's facial salient feature vector, and V1 and V2 are both column vectors, M t represents the distance topology matrix, T represents the transpose of the vector, Represents matrix multiplication, M v represents the eigenvalue granularity association matrix, M crepresents the topological modulation correlation matrix, V1′ represents the audio feature vector of the patient modulated by the perspective, V2′ represents the facial salient feature vector of the patient modulated by the perspective, α and β represent weighted hyperparameters, V c Represents the fused feature vector.

[0055] Based on this, in the technical solution of the present application, a cross-modal data fusion based on perspective alignment projection is performed on the patient audio feature vector and the patient facial salient feature vector, which first constructs a patient audio-facial salient distance topological matrix between the patient audio feature vector and the patient facial salient feature vector. This step utilizes the idea of ​​graph theory, taking the eigenvalues ​​of each position in the patient audio feature vector and the patient facial salient feature vector as nodes, and using a distance metric function to determine the low-dimensional embedding expression of the edges between nodes, so as to construct a feature expression that can quantify the topological correlation relationship between the patient audio feature vector and the patient facial salient feature vector. This step is crucial for understanding the spatial distribution of data points and their mutual connections, and also provides basic structural information for subsequent steps.

[0056] Next, a patient audio-facial significant feature value granularity association matrix is ​​constructed between the patient audio feature vector and the patient facial significant feature vector. This step focuses on exploring the correlation between the internal attributes of the feature vector. In a specific example, the product between the transposed vector of the patient audio feature vector and the patient facial significant feature vector is calculated to obtain the patient audio-facial significant feature value granularity association matrix. Furthermore, the patient audio-facial significant feature value granularity association matrix is ​​topologically modulated based on the patient audio-facial significant distance topological matrix to obtain the patient audio-facial significant topological modulation association matrix. That is, the feature value granularity association information of the patient audio feature vector and the patient facial significant feature vector is driven to wander on the distance topological map to construct a low-dimensional modulation perspective co-projection space domain for mapping the patient audio feature vector and the patient facial significant feature vector to a continuous high-dimensional regression space property.

[0057] Next, the patient audio-facial salient topological modulation association matrix is ​​used as the perspective co-projection space, and the patient audio feature vector and the patient facial salient feature vector are projected into the perspective co-projection space to obtain the perspective modulated patient audio feature vector and the perspective modulated patient facial salient feature vector. That is, the key to this technical solution is to find an implicit third space with modulation capability, in which as much information of the original high-dimensional data as possible is retained, so that the converted new feature vector not only reduces the dimension, but also maintains the relative position relationship between the original features.

[0058] Finally, the view-modulated patient audio feature vector and the view-modulated patient facial salient feature vector are fused to obtain the patient emotion classification feature vector. Fusion can be achieved in a variety of forms, ranging from simple arithmetic average to complex ensemble learning algorithms, depending on the application scenario and the goals pursued. The final patient emotion classification feature vector combines the main features of the two sets of input data and provides a more comprehensive data representation, which is not only conducive to improving the performance of machine learning models, but also promotes the ability of cross-modal or multi-source data analysis, thereby significantly enhancing the understanding and utilization efficiency of complex data structures.

[0059] Furthermore, the patient emotion classification feature vector is subjected to a scattering feedback depth dependency adjustment based on the class divergence space to obtain an optimized patient emotion classification feature vector, including: determining the mean of the patient emotion classification feature vector; accumulating and summing the patient emotion classification feature vectors to obtain a sum value of the patient emotion classification feature vectors; subtracting the mean of the patient emotion classification feature vector from the sum value of the patient emotion classification feature vector, dividing the sum value by a predetermined hyperparameter, and then dividing the sum value by the length of the patient emotion classification feature vector to obtain a first quotient value; calculating an exponential function value with a natural constant as the base for the first quotient to obtain a first natural exponential function value; and point-wise multiplying the first natural exponential function value by the patient emotion classification feature vector to obtain an optimized patient emotion classification feature vector.

[0060] In particular, in the technical solution of the present application, in this technical solution, the patient's emotions are classified by integrating the patient's audio data and video data. It is very important to improve the feature significance of the patient's emotion classification feature vector and avoid feature redundancy. Specifically, by improving the feature significance, the key emotional features in the patient's audio data and video data can be more accurately captured, which helps to improve the accuracy and sensitivity of emotion classification. The significant features can better distinguish different emotional states and improve the accuracy of emotion classification, thereby ensuring accurate judgment of the patient's emotional state. Improving the feature significance of the patient's emotion classification feature vector and avoiding feature redundancy can improve the performance of the emotion classification model, so that it can better distinguish different emotional states and improve the accuracy and reliability of the classification results. At the same time, by improving the feature significance, the patient's emotional state can be more accurately identified, which helps medical personnel to better understand the patient's emotional state and provide more accurate medical assistance. Based on this, the patient's emotion classification feature vector is adjusted based on the scattering feedback depth dependency of the class divergence space to obtain an optimized patient emotion classification feature vector.

[0061] More specifically, the patient emotion classification feature vector is subjected to a scattering feedback depth dependency adjustment based on a class divergence space to obtain an optimized patient emotion classification feature vector, including: flattening the patient emotion classification feature vector to obtain the patient emotion classification feature vector; and performing a scattering feedback depth dependency adjustment based on a class divergence space on the patient emotion classification feature vector using the following optimization formula to obtain the optimized patient emotion classification feature vector, wherein the optimization formula is:

[0062]

[0063] Wherein, V1 represents the patient's emotion classification feature vector, v i represents the eigenvalue of the ith position of the patient's emotion classification feature vector, represents the mean of the patient emotion classification feature vector, L represents the length of the patient emotion classification feature vector, ε represents a predetermined hyperparameter, ⊙ represents the position point multiplication, V1′ represents the optimized patient emotion classification feature vector; and, the optimized patient emotion classification feature vector is subjected to reverse feature aggregation in the feature flattening manner to obtain the optimized patient emotion classification feature vector.

[0064] Specifically, in order to improve the feature significance of the patient emotion classification feature vector and avoid feature redundancy, in the technical solution of the present application, the patient emotion classification feature vector is subjected to a scattering feedback depth dependency adjustment based on the class divergence space, which uses the global mean of the patient emotion classification feature vector as its dynamically formed deep anchor feature, and applies a coherence attention response mechanism based on the deep anchor feature to the original feature distribution, so as to perform a position scattering response deep correlation correction in the class divergence space with depth difference, thereby making the original feature distribution have a highly significant deep correlation distribution characteristic, so as to improve the feature significance of the patient emotion classification feature vector and avoid feature redundancy.

[0065] Furthermore, the use of classifiers can transform the optimized patient emotion classification feature vectors into specific emotion label information, such as joy, sadness, anger, etc., so that medical workers can more intuitively understand the patient's emotional state. Through the analysis of the classifier, abstract feature vectors can be transformed into easy-to-understand and easy-to-apply emotion labels, providing medical workers with more intuitive and practical information to help them better care for and treat patients.

[0066] In summary, the embodiment of the present application first obtains the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally uses a classifier to obtain the patient's emotional label information to better understand the patient's emotional state and mental health status, so as to carry out personalized diagnosis and treatment plans, thereby improving the quality and effectiveness of medical services.

[0067] As described above, the audio and video acquisition and processing system 100 based on big data technology according to the embodiment of the present application can be implemented in various terminal devices, such as a server deployed with an audio and video acquisition and processing control algorithm based on big data technology. In one example, the audio and video acquisition and processing system 100 based on big data technology can be integrated into the terminal device as a software module and / or a hardware module. For example, the audio and video acquisition and processing system 100 based on big data technology can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the audio and video acquisition and processing system 100 based on big data technology can also be one of the many hardware modules of the terminal device.

[0068] Alternatively, in another example, the audio and video acquisition and processing system 100 based on big data technology and the terminal device may also be separate devices, and the audio and video acquisition and processing system 100 based on big data technology may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0069] Figure 5 Flow chart of the audio and video acquisition and processing method based on big data technology according to an embodiment of the present application. Figure 5 As shown, according to the audio and video acquisition and processing method based on big data technology according to the embodiment of the present application, it includes: S110, acquiring patient audio data collected by a voice recording device and patient video data collected by a medical monitoring device; S120, extracting patient audio feature vectors and patient facial significant feature vectors from the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device; S130, obtaining the patient's emotion label information based on the patient audio feature vector and the patient's facial significant feature vector.

[0070] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned audio and video acquisition and processing method based on big data technology have been referred to above. Figures 1 to 4 The invention has been introduced in detail in the description of the audio and video acquisition and processing system based on big data technology, and therefore, its repeated description will be omitted.

[0071] Below, reference Figure 6To describe an electronic device according to an embodiment of the present application.

[0072] like Figure 6 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 14 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0073] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0074] In one embodiment, Figure 6 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the audio and video acquisition and processing methods based on big data technology described in the above embodiments.

[0075] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0076] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0077] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. An audio and video acquisition and processing system based on big data technology, characterized in that: include: A patient audio and video data acquisition module, used to acquire patient audio data collected by a voice recording device and patient video data collected by a medical monitoring device; A patient audio and video data extraction module, used to extract a patient audio feature vector and a patient facial salient feature vector from the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device; The multimodal emotion recognition module is used to obtain the patient's emotion label information based on the patient's audio feature vector and the patient's facial salient feature vector.

2. The audio and video acquisition and processing system based on big data technology according to claim 1 is characterized in that: The patient audio and video data extraction module includes: a patient audio data feature encoding unit, configured to perform feature encoding on the patient audio data collected by the voice recording device to obtain the patient audio feature vector; The patient video data feature encoding unit is used to perform feature encoding on the patient video data from the medical monitoring device to obtain a significant feature vector of the patient's face.

3. The audio and video acquisition and processing system based on big data technology according to claim 2 is characterized in that: The patient audio data feature encoding unit comprises: Extracting a patient log Mel spectrogram, a patient cochlear spectrogram, and a patient constant Q transform spectrogram from the patient audio data collected by the speech recording device; Arranging the patient's logarithmic Mel-spectrogram, the patient's cochlear spectrogram, and the patient's constant-Q transform spectrogram into a patient's multi-channel spectrogram; The patient multi-channel spectrogram is passed through a patient audio feature encoder using a channel attention mechanism to obtain the patient audio feature vector.

4. The audio and video acquisition and processing system based on big data technology according to claim 3 is characterized in that: The patient video data feature encoding unit comprises: A patient video key frame extraction subunit, used for extracting a plurality of patient video key frames from the patient video data obtained from the medical monitoring device; A video key frame target feature extraction subunit, used for extracting target features from the plurality of patient video key frames to obtain a patient facial salient feature map; The patient's facial salient feature pooling subunit is used to perform maximum pooling on the patient's facial salient feature map to obtain the patient's facial salient feature vector.

5. The audio and video acquisition and processing system based on big data technology according to claim 4 is characterized in that: The video key frame target feature extraction subunit includes: Passing the plurality of patient video key frames through a patient facial object detection network to obtain a plurality of patient facial regions of interest; The multiple patient facial regions of interest are passed through a patient facial salient object detector to obtain the patient facial salient feature map.

6. The audio and video acquisition and processing system based on big data technology according to claim 5 is characterized in that: The multimodal emotion recognition module comprises: A patient emotion feature fusion unit, used for performing cross-modal data fusion of the patient audio feature vector and the patient facial salient feature vector based on perspective alignment projection to obtain the patient emotion classification feature vector; A patient emotion feature optimization unit, used for performing a scattering feedback depth dependency adjustment on the patient emotion classification feature vector based on a class divergence space to obtain an optimized patient emotion classification feature vector; The multimodal patient emotion label recognition unit is used to pass the optimized patient emotion classification feature vector through a classifier to obtain a classification result, and the classification result is used to represent the obtained patient emotion label information.

7. The audio and video acquisition and processing system based on big data technology according to claim 6 is characterized in that: The patient emotion feature fusion unit comprises: constructing a patient audio-facial salient distance topology matrix between the patient audio feature vector and the patient facial salient feature vector; constructing a patient audio-facial salient feature value granularity association matrix between the patient audio feature vector and the patient facial salient feature vector; Performing topological correlation modulation on the patient audio-facial salient feature value granularity correlation matrix based on the patient audio-facial salient distance topological matrix to obtain a patient audio-facial salient topological modulation correlation matrix; Taking the patient audio-facial salient topological modulation association matrix as the perspective co-projection space, projecting the patient audio feature vector and the patient facial salient feature vector into the perspective co-projection space to obtain the perspective-modulated patient audio feature vector and the perspective-modulated patient facial salient feature vector; The perspective-modulated patient audio feature vector and the perspective-modulated patient facial salient feature vector are fused to obtain the patient emotion classification feature vector.

8. The audio and video acquisition and processing system based on big data technology according to claim 7 is characterized in that: The patient emotion characteristic optimization unit comprises: Determining a mean of the patient's emotion classification feature vector; Accumulating and summing the patient emotion classification feature vectors to obtain a total value of the patient emotion classification feature vectors; Subtracting the mean of the patient emotion classification feature vector from the sum of the patient emotion classification feature vectors and dividing the sum by a predetermined hyperparameter and then by the length of the patient emotion classification feature vector to obtain a first quotient value; Calculating an exponential function value with a natural constant as a base for the first quotient value to obtain a first natural exponential function value; The first natural exponential function value is multiplied by the patient emotion classification feature vector at a position point to obtain an optimized patient emotion classification feature vector.

9. A method for audio and video acquisition and processing based on big data technology, characterized in that: include: Obtaining patient audio data collected by a voice recording device and patient video data collected by a medical monitoring device; Extracting a patient audio feature vector and a patient facial salient feature vector from the patient audio data collected by the voice recording device and the patient video data collected by the medical monitoring device; Based on the patient's audio feature vector and the patient's facial salient feature vector, the patient's emotion label information is obtained.

Images