A data processing method for CCBT system based on visual-linguistic multimodal information

Abstract

The present invention discloses a CCBT system data processing method based on visual-linguistic multimodal information, and relates to the technical field of depression detection. The CCBT system data processing method based on visual-linguistic multimodal information includes the following steps: data acquisition; audio effectiveness evaluation; video effectiveness evaluation; optimization effectiveness evaluation. The present invention obtains an audio effectiveness evaluation value through the acquired voice features, determines whether to perform audio optimization based on the audio effectiveness evaluation value, then obtains a video effectiveness evaluation value based on the acquired video features, determines whether to perform video optimization based on the video effectiveness evaluation value, and finally obtains an optimization effectiveness evaluation value based on the test data and the test data obtained after the test optimization, determines whether to perform feedback based on the optimization effectiveness evaluation value, thereby improving the accuracy of depression diagnosis data processing in the CCBT system, and solving the problem of low accuracy of depression diagnosis data processing in the CCBT system in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of depression detection, and in particular to a CCBT system data processing method based on visual-linguistic multimodal information. Background Art

[0002] Computerized Cognitive-Behavioral Therapy (CCBT) is a new treatment method that combines computer technology and cognitive behavioral therapy. Cognitive behavioral therapy is a psychotherapy method that aims to improve the patient's mood and psychological state by changing their thinking and behavior patterns. Depression is a common psychological disorder, especially among adolescents and young adults. The diagnosis of depression usually relies on clinical assessment and self-reported symptoms. However, since the symptoms of depression are diverse and often coexist with other psychological disorders, the accuracy of diagnosis may be limited. The CCBT system can automatically recognize and analyze the patient's visual and verbal information, thereby providing more accurate treatment recommendations.

[0003] Existing technologies combine visual (such as facial expressions, postures, etc.) and language (such as voice intonation, speech content, etc.) data to conduct in-depth analysis of the patient's emotional state and behavior patterns to diagnose and treat depression.

[0004] For example, the invention patent with publication number: CN114898861A discloses a multimodal depression detection method and system based on a full attention mechanism, including: collecting a user's medical consultation video, where the medical consultation video is recorded by a doctor while asking the user questions about the Hamilton Depression Rating Scale; obtaining visual features based on the medical consultation video; extracting audio data from the medical consultation video to obtain voice features; transcribing and annotating the audio data to obtain text data and obtaining text features; and using a pre-built cross-modal transformer model to predict the user's depression level based on the visual features, voice features, and text features.

[0005] For example, the invention patent with announcement number: CN115249539B announces a method for constructing a multimodal small-sample depression prediction model, including: a multimodal fusion network and a small-sample learning module based on GCN; the multimodal fusion network combines three feature vectors: real-time audio information of interviewed subjects in a depression dataset, video acquisition information formed by an open-face face recognition framework, and text information transcribed from real-time audio, to form a fixed-length modal fusion representation through a fusion network; the small-sample learning module based on GCN includes two parts: node information update and iterative calculation of the adjacency matrix.

[0006] However, in the process of implementing the technical solution of the embodiments of the present invention, it is found that the above technology has at least the following technical problems:

[0007] In the prior art, due to the diversity of external environmental conditions, especially factors such as lighting changes, the effectiveness of the micro-expression image dataset of patients with depression is poor, which in turn increases the difficulty of feature extraction, and there is a problem of low accuracy in depression diagnosis data processing in the CCBT system. Summary of the invention

[0008] The purpose of the present invention is to provide a CCBT system data processing method based on visual-linguistic multimodal information, which solves the problem of low accuracy in processing depression diagnosis data in CCBT systems in the prior art and improves the accuracy in processing depression diagnosis data in CCBT systems.

[0009] A CCBT system data processing method based on visual-linguistic multimodal information comprises the following steps:

[0010] S1, obtaining initial test data of the patient to be monitored under emotion induction and stress test, and processing the initial test data to obtain test data;

[0011] S2, obtaining an audio effectiveness evaluation value according to the acquired speech features, and determining whether to perform audio optimization based on the audio effectiveness evaluation value, wherein the audio effectiveness evaluation value is used to evaluate the quality of the audio;

[0012] S3, if audio optimization is to be performed, a video effectiveness evaluation value is obtained according to the acquired video features, and whether to perform video optimization is determined based on the video effectiveness evaluation value, wherein the video effectiveness evaluation value is used to evaluate the quality of the video;

[0013] S4: If test optimization is performed, an optimization effective evaluation value is obtained based on the test data obtained after the test optimization and the test data, and whether to provide feedback is determined based on the optimization effective evaluation value. The optimization effective evaluation value is used to evaluate the effectiveness of the test optimization measures, and the test optimization includes audio optimization and video optimization.

[0014] Furthermore, the test data includes voice features and video features; the voice features include signal-to-noise ratio, total harmonic distortion and noise level; the video features include resolution, frame rate, bit rate and color difference value; the color difference value is obtained by comparing the color value obtained by measuring the color in the video with a color analyzer with the standard color value.

[0015] Furthermore, the specific method for obtaining the audio effective evaluation value based on the acquired speech features is as follows: obtaining a signal-to-noise ratio deviation based on the relative relationship between the signal-to-noise ratio and a preset signal-to-noise ratio in a database, and the signal-to-noise ratio deviation is obtained by processing the ratio operation of the signal-to-noise ratio and the preset signal-to-noise ratio in the database; obtaining a total harmonic distortion deviation based on the relative relationship between the total harmonic distortion and the preset total harmonic distortion in the database, and the total harmonic distortion deviation is obtained by processing the ratio operation of the total harmonic distortion and the preset total harmonic distortion in the database; obtaining a noise level deviation based on the relative relationship between the noise level and the preset noise level in the database, and the noise level deviation is obtained by processing the ratio operation of the noise level and the preset noise level in the database; and processing the signal-to-noise ratio deviation, the total harmonic distortion deviation and the noise level deviation to obtain the audio effective evaluation value.

[0016] Furthermore, the specific process of determining whether to perform audio optimization based on the audio effectiveness evaluation value is as follows: determining whether the audio effectiveness evaluation value is less than a preset audio effectiveness threshold in a database: if the audio effectiveness evaluation value is less than the preset audio effectiveness threshold in the database, audio optimization is performed; if the audio effectiveness evaluation value is not less than the preset audio effectiveness threshold in the database, audio optimization is not performed; the audio optimization includes noise reduction processing, dynamic range compression and audio enhancement.

[0017] Furthermore, the specific process of obtaining the video effective evaluation value based on the acquired video features is as follows: obtaining the resolution deviation based on the relative relationship between the resolution and the preset resolution in the database; obtaining the frame rate deviation based on the relative relationship between the frame rate and the preset frame rate in the database; obtaining the color difference value deviation based on the relative relationship between the color difference value level and the preset color difference value in the database; if the color difference value is less than the preset color difference value in the database, processing the resolution deviation, frame rate deviation and bit rate deviation to obtain the video effective evaluation value; if the color difference value is not less than the preset color difference value in the database, processing the resolution deviation, frame rate deviation, bit rate deviation and color difference value deviation to obtain the video effective evaluation value.

[0018] Furthermore, the specific process of obtaining the optimized effective evaluation value based on the test data obtained after the test optimization and the test data is as follows: obtaining the optimized audio effective evaluation value based on the optimized voice feature and the preset voice feature data, and the optimized audio effective evaluation value is obtained by processing the ratio operation of the optimized voice feature and the preset voice feature data; obtaining the audio optimization coefficient based on the relative relationship between the optimized audio effective evaluation value and the audio effective evaluation value, and the audio optimization coefficient is used to evaluate the effectiveness of the audio optimization; obtaining the optimized video effective evaluation value based on the optimized video feature and the preset video feature data, and the optimized video effective evaluation value is obtained by processing the ratio operation of the optimized video feature and the preset video feature data; obtaining the video optimization coefficient based on the relative relationship between the optimized video effective evaluation value and the video effective evaluation value, and the video optimization coefficient is used to evaluate the effectiveness of the video optimization; processing the audio optimization coefficient and the video optimization coefficient to obtain the optimized effective evaluation value.

[0019] Furthermore, the specific limiting expression of the optimized effective evaluation value is as follows:

[0020] ;

[0021] Where m represents the number of test optimization times, m=1,2,…,M, M represents the total number of test optimization times, YHY m represents the optimization effective evaluation value of the mth test optimization, YINH m represents the audio optimization coefficient of the mth test optimization, SPYH m Indicates the video optimization coefficient of the mth test optimization.

[0022] The present invention has the following technical effects or advantages:

[0023] 1. Determine whether to perform audio optimization by using the audio effective evaluation value obtained through the acquired speech features, then obtain the video effective evaluation value based on the acquired video features, determine whether to perform video optimization based on the video effective evaluation value, finally obtain the optimized effective evaluation value based on the test data and the test data obtained after the test optimization, determine whether to provide feedback based on the optimized effective evaluation value, thereby improving the quality of the test data, and further achieving an improvement in the accuracy of depression diagnosis data processing in the CCBT system, effectively solving the problem of low accuracy of depression diagnosis data processing in the CCBT system in the prior art.

[0024] 2. The signal-to-noise ratio deviation is obtained through the relative relationship between the signal-to-noise ratio and the preset signal-to-noise ratio in the database, and then the total harmonic distortion deviation is obtained according to the relative relationship between the total harmonic distortion and the preset total harmonic distortion in the database. Then, the noise level deviation is obtained according to the relative relationship between the noise level and the preset noise level in the database. Finally, the signal-to-noise ratio deviation, total harmonic distortion deviation and noise level deviation are processed to obtain the audio effective evaluation value, thereby quantitatively evaluating the quality of the audio signal, thereby achieving the improvement of audio quality in the process of depression diagnosis.

[0025] 3. The audio optimization coefficient is obtained through the relative relationship between the optimized audio effective evaluation value and the audio effective evaluation value, and then the video optimization coefficient is obtained according to the relative relationship between the optimized video effective evaluation value and the video effective evaluation value. Finally, the audio optimization coefficient and the video optimization coefficient are processed to obtain the optimized effective evaluation value, thereby quantitatively evaluating the effectiveness of the test optimization and achieving the improvement of the quality of video and audio in the process of depression diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a CCBT system data processing method based on visual-linguistic multimodal information provided by an embodiment of the present invention;

[0027] Figure 2 A schematic diagram showing the change of the effective audio evaluation value with the signal-to-noise ratio provided by an embodiment of the present invention;

[0028] Figure 3 A schematic diagram showing how the effective audio evaluation value varies with the total harmonic distortion provided by an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of how the effective audio evaluation value varies with the noise level provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The embodiment of the present invention solves the problem of low accuracy in processing depression diagnosis data in the CCBT system in the prior art by providing a CCBT system data processing method based on visual-linguistic multimodal information. An audio effective evaluation value is obtained by acquiring speech features, and whether to perform audio optimization is determined based on the audio effective evaluation value. Then, a video effective evaluation value is obtained according to the acquired video features, and whether to perform video optimization is determined based on the video effective evaluation value. Finally, an optimized effective evaluation value is obtained according to the test data and the test data obtained after the test optimization, and whether to perform feedback is determined based on the optimized effective evaluation value, thereby improving the accuracy of depression diagnosis data processing in the CCBT system.

[0031] The technical solution in the embodiment of the present invention is to solve the problem of low accuracy in processing depression diagnosis data in the above CCBT system. The overall idea is as follows:

[0032] Whether to perform audio optimization is determined by the obtained audio effective evaluation value, and then whether to perform video optimization is determined based on the obtained video effective evaluation value. Finally, whether to provide feedback is determined based on the obtained test data and the optimized effective evaluation value obtained from the test data, thereby achieving the effect of improving the accuracy of depression diagnosis data processing in the CCBT system.

[0033] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0034] like Figure 1 As shown, it is a flowchart of a CCBT system data processing method based on visual-language multimodal information provided by an embodiment of the present invention, and the method includes the following steps: S1, data acquisition: obtaining the initial test data of the patient to be monitored under emotion induction and stress test, and processing based on the initial test data to obtain test data; S2, audio effectiveness evaluation: obtaining an audio effectiveness evaluation value according to the acquired voice features, judging whether to perform audio optimization based on the audio effectiveness evaluation value, and the audio effectiveness evaluation value is used to evaluate the quality of the audio; S3, video effectiveness evaluation: if audio optimization is performed, obtaining a video effectiveness evaluation value according to the acquired video features, judging whether to perform video optimization based on the video effectiveness evaluation value, and the video effectiveness evaluation value is used to evaluate the quality of the video; S4, optimization effectiveness evaluation: if test optimization is performed, obtaining an optimization effectiveness evaluation value according to the test data obtained after the test optimization and the test data, judging whether to perform feedback based on the optimization effectiveness evaluation value, and the optimization effectiveness evaluation value is used to evaluate the effectiveness of the test optimization measures, and the test optimization includes audio optimization and video optimization.

[0035] It should be added that the test data includes voice features and video features; voice features include signal-to-noise ratio, total harmonic distortion and noise level; video features include resolution, frame rate, bit rate and color difference value; the color difference value is obtained by measuring the color in the video with a color analyzer, recording the color value of each pixel, and then comparing the measured color value with the standard color value.

[0036] In this embodiment, the emotion induction and stress test refers to a test conducted on patients with depression, which is used to induce emotional reactions or stress reactions through different situations or tasks, observe changes in the emotions and psychological states of patients with depression, and thus evaluate the manifestations of depression; evaluate and diagnose depression by comprehensively analyzing visual and linguistic multimodal data (such as audio and video), and continuously optimize the quality of test data, thereby improving the quality of test data, thereby achieving an improvement in the accuracy of depression diagnosis data processing in the CCBT system.

[0037] The signal-to-noise ratio is obtained by calculating the ratio of the signal power and the noise power obtained by power spectrum analysis of the audio. The higher the signal-to-noise ratio, the clearer the signal in the audio and the less noise.

[0038] Total harmonic distortion measures the amplitude of the fundamental and harmonic components in the voice signal and calculates the ratio of the harmonic components to the fundamental components. The smaller the total harmonic distortion, the lower the distortion of the audio signal and the better the sound quality.

[0039] The noise level is measured by performing spectral analysis on the voice signal and measuring the amplitude of the noise component; the resolution, frame rate and bit rate are obtained through video processing software; the lower the noise level, the less noise there is in the audio signal and the higher the purity.

[0040] Directly read the video resolution, frame rate and bit rate through video processing software (such as MediaInfo); the higher the resolution, the clearer the video image and the better the detail performance; the higher the frame rate, the smoother the video looks and the better the dynamic performance; the higher the bit rate, the more details and color information can be included, and the better the video quality.

[0041] The smaller the color difference value, the closer the video color is to the standard color and the higher the color restoration degree.

[0042] Furthermore, a specific method for obtaining an audio effective evaluation value based on the acquired speech features is as follows: obtaining preset speech feature data from a preset database, the preset speech feature data including a preset signal-to-noise ratio, a preset total harmonic distortion, and a preset noise level; obtaining a signal-to-noise ratio deviation (i.e., the value of the signal-to-noise ratio in the method for obtaining the audio effective evaluation value) based on a relative relationship between the signal-to-noise ratio and the preset signal-to-noise ratio in the database; ), the signal-to-noise ratio deviation is obtained by processing the ratio of the signal-to-noise ratio and the preset signal-to-noise ratio in the database; the total harmonic distortion deviation is obtained according to the relative relationship between the total harmonic distortion and the preset total harmonic distortion in the database (that is, the total harmonic distortion deviation in the method of obtaining the effective audio evaluation value). ), the total harmonic distortion deviation is obtained by processing the ratio of the total harmonic distortion and the preset total harmonic distortion in the database; the noise level deviation is obtained according to the relative relationship between the noise level and the preset noise level in the database (i.e., the method for obtaining the audio effective evaluation value ), the noise level deviation is obtained by processing the ratio of the noise level and the preset noise level in the database; the signal-to-noise ratio deviation, the total harmonic distortion deviation and the noise level deviation are processed to obtain the audio effective evaluation value.

[0043] The method for obtaining the effective audio evaluation value is as follows:

[0044] ;

[0045] Where i represents the number of depression diagnoses, i=1,2,…,T, T represents the total number of depression diagnoses, YIN i represents the effective audio evaluation value of the i-th depression diagnosis, SNR i represents the signal-to-noise ratio of the ith depression diagnosis, THD i represents the total harmonic distortion of the ith depression diagnosis, ZDP i represents the noise level of the i-th depression diagnosis, SNR0 represents the preset signal-to-noise ratio, THD0 represents the preset total harmonic distortion, and ZDP0 represents the preset noise level.

[0046] The audio effective evaluation value in this algorithm involves processing multiple independent variables (signal-to-noise ratio, noise level and total harmonic distortion), and there is a mutual influence relationship between these independent variables; when the signal-to-noise ratio decreases, the noise increases, which may mask the details in the original audio signal, thereby increasing the perception of total harmonic distortion and increasing the total harmonic distortion; the higher the noise level, the lower the signal-to-noise ratio and the worse the sound quality; the higher the signal-to-noise ratio, the lower the total harmonic distortion and the lower the noise level usually lead to a higher audio effective evaluation value, and the larger the audio effective evaluation value, the higher the audio quality.

[0047] For example, the preset signal-to-noise ratio is 20dB, the preset total harmonic distortion is 0.1%, and the preset noise level is 0.1dB. Figure 2 As shown, it is a schematic diagram of the change of the audio effective evaluation value provided by an embodiment of the present invention with the change of the signal-to-noise ratio. When the total harmonic distortion is 0.1% and the noise level is 0.1dB, the audio effective evaluation value increases with the increase of the signal-to-noise ratio.

[0048] like Figure 3 As shown, it is a schematic diagram of the change of the audio effective evaluation value provided by an embodiment of the present invention with the change of the total harmonic distortion. When the signal-to-noise ratio is 20dB and the noise level is 0.1dB, the audio effective evaluation value decreases with the increase of the total harmonic distortion.

[0049] like Figure 4 As shown, it is a schematic diagram of the change of the audio effective evaluation value provided by an embodiment of the present invention with the change of the noise level. When the signal-to-noise ratio is 20dB and the total harmonic distortion is 0.1%, the audio effective evaluation value decreases as the noise level increases.

[0050] The preset signal-to-noise ratio is represented by the average value of the signal-to-noise ratio in the historical time period; the preset total harmonic distortion is represented by the average value of the total harmonic distortion in the historical time period; the preset noise level is represented by the average value of the noise level in the historical time period; through the above steps, the quality of the audio signal is quantitatively evaluated, thereby achieving improvement in audio quality.

[0051] Furthermore, the specific process of determining whether to perform audio optimization based on the audio effectiveness evaluation value is as follows: determine whether the audio effectiveness evaluation value is less than a preset audio effectiveness threshold in the database: if the audio effectiveness evaluation value is less than the preset audio effectiveness threshold in the database, perform audio optimization; if the audio effectiveness evaluation value is not less than the preset audio effectiveness threshold in the database, do not perform audio optimization; audio optimization includes noise reduction processing, dynamic range compression and audio enhancement.

[0052] In this embodiment, the preset audio validity threshold is represented by the maximum value of the audio validity evaluation value in the historical time period; the noise reduction processing separates speech and noise by utilizing a convolutional neural network, and utilizes adaptive noise reduction technology to automatically adjust the noise reduction parameters according to the ambient noise.

[0053] Dynamic range compression is achieved by determining whether the speech signal is less than a preset compression threshold. If so, the signal is amplified, otherwise the signal is compressed. Amplification is achieved by increasing the amplitude of the signal; compression is achieved by reducing the amplitude of the signal. The preset compression threshold is set to -20dB.

[0054] Audio enhancement removes the echo by utilizing adaptive echo cancellation technology to analyze the input signal and the echo path, generate a waveform that is the inverse of the echo, and subtract it from the audio signal.

[0055] By performing audio optimization, the quality of the audio signal during the diagnosis of depression is improved; noise reduction processing reduces background noise during the diagnosis of depression, making the voice clearer; dynamic range compression balances the strength of the audio signal, making the audio more balanced during the diagnosis of depression; audio enhancement further improves the clarity and audibility of the audio recorded during the diagnosis of depression by removing echoes.

[0056] Furthermore, the specific process of obtaining the effective evaluation value of the video according to the obtained video features is as follows: obtaining preset video feature data from a preset database, the preset video feature data including a preset resolution, a preset frame rate, a preset bit rate and a preset color difference value; obtaining a resolution deviation according to the relative relationship between the resolution and the preset resolution in the database, the resolution deviation being obtained by processing the ratio operation of the resolution and the preset resolution in the database; obtaining a frame rate deviation according to the relative relationship between the frame rate and the preset frame rate in the database, the frame rate deviation being obtained by processing the ratio operation of the frame rate and the preset frame rate in the database; obtaining a color difference value according to the relative relationship between the bit rate level and the preset bit rate in the database Bit rate deviation, the bit rate deviation is obtained by processing the ratio operation of the bit rate and the preset bit rate in the database; the color difference value deviation is obtained according to the relative relationship between the color difference value level and the preset color difference value in the database, and the color difference value deviation is obtained by processing the ratio operation of the color difference value and the preset color difference value in the database; if the color difference value is less than the preset color difference value in the database, the resolution deviation, frame rate deviation and bit rate deviation are processed to obtain the video effective evaluation value; if the color difference value is not less than the preset color difference value in the database, the resolution deviation, frame rate deviation, bit rate deviation and color difference value deviation are processed to obtain the video effective evaluation value.

[0057] The specific limiting expression of the effective evaluation value of the video is:

[0058] ;

[0059] ;

[0060] Where i represents the number of depression diagnoses, i=1,2,…,T, T represents the total number of depression diagnoses, SPY i represents the effective evaluation value of the video for the i-th depression diagnosis, LV i The first coefficient representing the ith depression diagnosis, FEN i represents the resolution of the ith depression diagnosis, ZEN i represents the frame rate of the ith depression diagnosis, MAL i represents the code rate of the ith depression diagnosis, SEC i represents the color difference value of the i-th depression diagnosis, FEN0 represents the preset resolution, ZEN0 represents the preset frame rate, MAL0 represents the preset bit rate, and SEC0 represents the preset color difference value.

[0061] In this embodiment, the video effective evaluation value involves processing multiple independent variables (resolution, frame rate, bit rate and color difference value), and there is a mutual influence relationship between these independent variables; an increase in resolution will increase the size of the video file, resulting in an increase in the bit rate; there is a certain balance between resolution and frame rate, and while maintaining a certain bit rate, increasing the resolution may reduce the frame rate; while maintaining a certain video quality, reducing the bit rate can reduce the size of the video file, but may reduce the resolution; a video with a higher resolution can display more color details, thereby reducing color distortion, and thus improving color accuracy; a higher frame rate usually means a higher bit rate because more frames are transmitted per second; an increase in frame rate usually does not directly affect the color difference value, but in a video with a high frame rate, if the bit rate is insufficient, it may cause color information to be lost or distorted, thereby increasing the color difference value; higher resolution, higher frame rate, higher bit rate and lower color difference value usually result in a higher video effective evaluation value.

[0062] The preset resolution is set to 1920x1080; the preset frame rate is set to 30 frames per second; the preset bit rate is set to 4000kbps; the color difference value is set to 3; by comparing the video features with the preset video feature data in the preset database, and calculating a video effective evaluation value based on the deviation between the video features and the preset video feature data in the preset database, the quality of the video in the depression diagnosis process is quantitatively evaluated, thereby achieving the improvement of the video quality in the depression diagnosis process.

[0063] Furthermore, the specific process of determining whether to perform video optimization based on the video effectiveness evaluation value is as follows: determine whether the video effectiveness evaluation value is less than a preset video effectiveness threshold in the database: if the video effectiveness evaluation value is less than the preset video effectiveness threshold, perform video optimization; if the video effectiveness evaluation value is not less than the preset video effectiveness threshold, do not perform video optimization; video optimization includes improving resolution, repairing frame rate, calibrating color space and denoising.

[0064] In this embodiment, the preset video validity threshold is represented by the maximum value of the video validity evaluation value in the historical time period; the resolution is improved by super-resolution reconstruction using a convolutional neural network model; the frame rate is repaired by obtaining the number of frames that need to be differencing or deleted through the frame rate of the current video and the preset frame rate, and the video is resampled on the time axis to balance the frame number; denoising is performed by using Gaussian filtering to remove static noise in the video; through the above steps, the video quality is improved in the process of depression diagnosis, thereby improving the accuracy of depression diagnosis.

[0065] Among them, the specific steps of calibrating the color space are: determine whether the video effective evaluation value before calibrating the color space is less than the preset video effective threshold in the database: if the video effective evaluation value is less than the preset video effective threshold, calibrate the color space; if the video effective evaluation value is not less than the preset video effective threshold, do not calibrate the color space; calibrating the color space includes white balance correction, color enhancement, color gamut conversion and chromatic aberration correction.

[0066] Specifically, white balance correction obtains color information in the video frame and calculates the average color value, and adjusts the average color value according to the standard white balance (such as D65 lighting conditions); color enhancement is achieved by adjusting the pixel value to a preset pixel value range through a linear stretching algorithm; the preset pixel value range is 0-255; color gamut conversion is achieved by converting the video to a standard color gamut (such as Rec.709 or Rec.2020) through a color gamut conversion matrix; chromatic aberration correction is achieved by using an edge detection algorithm to identify a preset area in the video, and a color adjustment tool (such as Magic Bullet Colorista) corrects the deviation of the preset area; the preset area represents the pixel point in the video whose color difference value is greater than the preset color difference value. By calibrating the color space, the color performance of the video is improved, thereby ensuring the color consistency of the video on different devices and platforms.

[0067] Further, the specific process of obtaining the optimized effective evaluation value based on the test data obtained after the test optimization and the test data is as follows: according to the optimized voice features and the preset voice feature data, the optimized audio effective evaluation value (that is, the YIN in the specific restriction expression of the optimized effective evaluation value) is obtained. i ′ ), the optimized audio effective evaluation value is obtained by processing the ratio operation of the optimized voice feature and the preset voice feature data; the audio optimization coefficient (that is, the YINH in the specific restriction expression of the optimized effective evaluation value) is obtained according to the relative relationship between the optimized audio effective evaluation value and the audio effective evaluation value. i ), the audio optimization coefficient is used to evaluate the effectiveness of audio optimization; the optimized video effective evaluation value (i.e., SPY in the specific restriction expression of the optimized effective evaluation value) is obtained according to the optimized video features and the preset video feature data. i ′ ), the optimized video effective evaluation value is obtained by processing the ratio operation of the optimized video features and the preset video feature data; the video optimization coefficient (i.e., SPYH in the specific restriction expression of the optimized effective evaluation value) is obtained according to the relative relationship between the optimized video effective evaluation value and the video effective evaluation value. i ), the video optimization coefficient is used to evaluate the effectiveness of video optimization; the audio optimization coefficient and the video optimization coefficient are processed to obtain the optimization effective evaluation value.

[0068] Among them, the specific restriction expression for optimizing the effective evaluation value is as follows:

[0069] ;

[0070] ;

[0071] ;

[0072] Where m represents the number of test optimization times, m=1,2,…,M, M represents the total number of test optimization times (theoretically, the number of test optimization times is not greater than the number of depression diagnoses), YHY m represents the optimization effective evaluation value of the mth test optimization, YINH m represents the audio optimization coefficient of the mth test optimization, SPYH m Indicates the video optimization coefficient of the mth test optimization, YIN m represents the effective audio evaluation value of the mth test optimization (referring to the effective audio evaluation value corresponding to the case where test optimization is required after the diagnosis of depression), SPY m The effective evaluation value of the video for the mth test optimization refers to the effective evaluation value of the video when test optimization is required after the diagnosis of depression), YIN m ′ represents the effective evaluation value of the optimized audio after the mth test optimization (referring to the effective evaluation value of the audio obtained after the test optimization is required after the diagnosis of depression), SPY m ′ The optimized video effective evaluation value of the mth test optimization refers to the video effective evaluation value obtained after the video optimization is required when the test optimization is required after the diagnosis of depression), YIN0 represents the preset audio effective threshold, and SPY0 represents the preset video effective threshold.

[0073] In this embodiment, taking the total number of depression diagnoses as 1 as an example, the statistical table of changes in the optimized effective evaluation values ​​is shown in Table 1:

[0074] Table 1 Statistics of changes in optimization effective evaluation values

[0075]

[0076] It can be seen from the first and second groups of data in the table that when the video optimization coefficient increases, the optimization effective evaluation value increases; it can be seen from the second and third groups of data that when the audio optimization coefficient increases, the optimization effective evaluation value increases; therefore, the video optimization coefficient and the audio optimization coefficient are positively correlated with the optimization effective evaluation value.

[0077] The optimization effective evaluation value involves processing multiple independent variables (audio optimization coefficient and video optimization coefficient), and there is a mutual influence relationship between these independent variables; when processing resources are limited, there may be a competitive relationship between the audio optimization coefficient and the video optimization coefficient; for example, in the diagnosis of depression, if more resources are allocated to audio processing and the audio optimization coefficient increases accordingly, then the resources for video processing may be reduced accordingly, and the video optimization coefficient decreases accordingly; as the audio optimization coefficient increases, the audio quality will be improved, thereby improving the user's auditory experience, and thus increasing the optimization effective evaluation value; as the video optimization coefficient increases, the video quality will be improved, thereby improving the user's visual experience, and thus increasing the optimization effective evaluation value. The larger the optimization effective evaluation value, the more effective the test optimization measures are.

[0078] Through the above steps, the effectiveness of test optimization in the depression diagnosis process was quantitatively evaluated, thereby improving the quality of video and audio recorded during the depression diagnosis process.

[0079] Furthermore, the specific process of judging whether to provide feedback based on the optimization effective evaluation value is as follows: judging whether the optimization effective evaluation value is less than the preset optimization effective threshold in the database, if the optimization effective evaluation value is less than the preset optimization effective threshold in the database, adjusting the acquisition device, otherwise not adjusting the acquisition device, the acquisition device includes a camera and a microphone; judging whether the optimization effective evaluation value obtained after adjusting the acquisition device is less than the preset optimization effective threshold in the database, if the optimization effective evaluation value is less than the preset optimization effective threshold, providing feedback; if the optimization effective evaluation value is not less than the preset optimization effective threshold, not providing feedback; adjusting the acquisition device includes equipment upgrades and acquisition technology improvements; acquisition technology improvements are achieved by increasing redundant acquisition.

[0080] In this embodiment, the preset optimization effective threshold is represented by the maximum value of the optimization effective evaluation value in the historical time period; equipment upgrade represents the replacement of a clearer camera or a microphone with clearer sound quality. For example, the original camera resolution is 720p, which can be replaced with a 1080p camera or the original dynamic microphone can be upgraded to a condenser microphone; redundant acquisition represents the addition of an additional set of acquisition equipment on the basis of the original acquisition equipment to ensure that when a certain acquisition equipment fails, the required data can still be obtained through other acquisition equipment; through the above steps, the optimization effect of the test optimization measures is quantitatively evaluated, and the acquisition equipment is automatically adjusted or feedback is provided according to the evaluation results, thereby improving the quality of test data in the process of depression diagnosis and reducing errors or failures caused by acquisition equipment problems.

[0081] To summarize, the embodiment of the present invention determines whether to perform audio optimization by using the audio effective evaluation value obtained through the acquired speech features, then obtains the video effective evaluation value based on the acquired video features, determines whether to perform video optimization based on the video effective evaluation value, and finally obtains the optimized effective evaluation value based on the test data and the test data obtained after the test optimization, and determines whether to perform feedback based on the optimized effective evaluation value, thereby improving the quality of the test data, and further achieving an improvement in the accuracy of depression diagnosis data processing in the CCBT system, effectively solving the problem of low accuracy of depression diagnosis data processing in the CCBT system in the prior art.

[0082] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0086] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0087] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A CCBT system data processing method based on visual-linguistic multimodal information, characterized in that: The following steps are involved: S1, obtaining initial test data of the patient to be monitored under emotion induction and stress test, and processing the initial test data to obtain test data; S2, obtaining an audio effectiveness evaluation value according to the acquired speech features, and determining whether to perform audio optimization based on the audio effectiveness evaluation value, wherein the audio effectiveness evaluation value is used to evaluate the quality of the audio; S3, if audio optimization is performed, obtaining a video effectiveness evaluation value according to the acquired video features, and determining whether to perform video optimization based on the video effectiveness evaluation value, wherein the video effectiveness evaluation value is used to evaluate the quality of the video; S4, if test optimization is performed, then an optimization effective evaluation value is obtained according to the test data obtained after the test optimization, and whether to provide feedback is determined based on the optimization effective evaluation value, wherein the optimization effective evaluation value is used to evaluate the effectiveness of the test optimization measures, and the test optimization includes audio optimization and video optimization; The test data includes voice features and video features; The speech characteristics include signal-to-noise ratio, total harmonic distortion and noise level; The video features include resolution, frame rate, bit rate and color difference value; The color difference value is obtained by comparing a color value obtained by measuring the color in the video with a standard color value using a color analyzer; The specific method for obtaining the audio effective evaluation value according to the acquired speech features is as follows: Obtaining a signal-to-noise ratio deviation according to a relative relationship between the signal-to-noise ratio and a preset signal-to-noise ratio in a database, wherein the signal-to-noise ratio deviation is obtained by processing a ratio operation of the signal-to-noise ratio and the preset signal-to-noise ratio in the database; Obtaining a total harmonic distortion deviation according to a relative relationship between the total harmonic distortion and a preset total harmonic distortion in a database, wherein the total harmonic distortion deviation is obtained by processing a ratio operation of the total harmonic distortion and the preset total harmonic distortion in the database; Obtaining a noise level deviation according to a relative relationship between the noise level and a preset noise level in a database, wherein the noise level deviation is obtained by processing a ratio operation of the noise level and the preset noise level in the database; The signal-to-noise ratio deviation, the total harmonic distortion deviation and the noise level deviation are processed to obtain an audio effective evaluation value; The specific process of obtaining the effective evaluation value of the video according to the acquired video features is as follows: The resolution deviation is obtained according to the relative relationship between the resolution and the preset resolution in the database; Obtaining a frame rate deviation according to a relative relationship between the frame rate and a preset frame rate in a database; Obtaining a color difference value deviation according to a relative relationship between the color difference value level and a preset color difference value in a database; If the color difference value is less than the preset color difference value in the database, the resolution deviation, frame rate deviation and bit rate deviation are processed to obtain a video effective evaluation value; If the color difference value is not less than the preset color difference value in the database, the resolution deviation, frame rate deviation, bit rate deviation and color difference value deviation are processed to obtain a video effective evaluation value; The specific process of analyzing the test data obtained after test optimization to obtain the optimized effective evaluation value is as follows: Obtaining an optimized audio effective evaluation value according to the optimized voice feature and the preset voice feature data, wherein the optimized audio effective evaluation value is obtained by processing the ratio operation of the optimized voice feature and the preset voice feature data; Obtaining an audio optimization coefficient according to a relative relationship between the optimized audio effective evaluation value and the audio effective evaluation value, wherein the audio optimization coefficient is used to evaluate the effectiveness of audio optimization; Obtaining an optimized video effective evaluation value according to the optimized video features and the preset video feature data, wherein the optimized video effective evaluation value is obtained by processing the ratio operation of the optimized video features and the preset video feature data; Obtaining a video optimization coefficient according to the relative relationship between the optimized video effective evaluation value and the video effective evaluation value, wherein the video optimization coefficient is used to evaluate the effectiveness of video optimization; The audio optimization coefficient and the video optimization coefficient are processed to obtain an optimized effective evaluation value.

2. A CCBT system data processing method based on visual-linguistic multimodal information as claimed in claim 1, characterized in that: The specific process of determining whether to perform audio optimization based on the audio effective evaluation value is as follows: Determine whether the audio validity evaluation value is less than the preset audio validity threshold in the database: If the audio validity evaluation value is less than the preset audio validity threshold in the database, audio optimization is performed; If the audio validity evaluation value is not less than the preset audio validity threshold in the database, no audio optimization is performed; The audio optimization includes noise reduction, dynamic range compression and audio enhancement.

3. A CCBT system data processing method based on visual-linguistic multimodal information as claimed in claim 1, characterized in that: The specific process of determining whether to perform video optimization based on the video effective evaluation value is as follows: Determine whether the video validity evaluation value is less than the preset video validity threshold in the database: If the video validity evaluation value is less than the preset video validity threshold, the video is optimized; If the video validity evaluation value is not less than the preset video validity threshold, no video optimization is performed; The video optimization includes increasing resolution, repairing frame rate, calibrating color space and denoising.

4. A CCBT system data processing method based on visual-linguistic multimodal information as claimed in claim 3, characterized in that: The specific steps of calibrating the color space are: Determine whether the video validity evaluation value before calibrating the color space is less than the preset video validity threshold in the database: If the video validity evaluation value is less than the preset video validity threshold, the color space is calibrated; If the video validity evaluation value is not less than the preset video validity threshold, the color space is not calibrated; The calibrated color space includes white balance correction, color enhancement, color gamut conversion and color difference correction.

5. The CCBT system data processing method based on visual-linguistic multimodal information as claimed in claim 1, characterized in that: The specific limiting expression of the optimized effective evaluation value is as follows: ； Where m represents the number of test optimization times, m=1,2,…,M, M represents the total number of test optimization times, YHY m represents the optimization effective evaluation value of the mth test optimization, YINH m represents the audio optimization coefficient of the mth test optimization, SPYH m Indicates the video optimization coefficient of the mth test optimization.

6. A CCBT system data processing method based on visual-linguistic multimodal information as claimed in claim 1, characterized in that: The specific process of judging whether to provide feedback based on the optimization effective evaluation value is as follows: Determine whether the optimization effective evaluation value is less than a preset optimization effective threshold in the database, if the optimization effective evaluation value is less than the preset optimization effective threshold in the database, adjust the acquisition device, otherwise do not adjust the acquisition device, the acquisition device includes a camera and a microphone; Determine whether the optimization effective evaluation value obtained after adjusting the acquisition device is less than the preset optimization effective threshold in the database. If the optimization effective evaluation value is less than the preset optimization effective threshold, feedback is given; If the optimization effectiveness evaluation value is not less than the preset optimization effectiveness threshold, no feedback is given.

Images