User state auxiliary detection method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses a user state auxiliary detection method and device, equipment, a medium and a product. The method comprises the following steps: acquiring an initial state detection result of a detection end on a user, and displaying a plurality of corresponding state tags; displaying a plurality of recommendation scales corresponding to the target state tag at the detection end; in response to a selection operation of the detection end on the target recommendation scale, sending the target recommendation scale to the user end; obtaining a filling operation of the user side on the target recommendation scale and determining a detection score and a conclusion; determining a detection problem table at the detection end according to the detection score and conclusion; determining a state detection degree of the user and a reference state detection result in response to a filling operation of the detection end on the detection question table; and in response to a determination operation of the detection end on the state detection degree and the reference state detection result, generating a final state detection result of the user. According to the method, scale recommendation can be carried out according to real-time state detection conditions, and a detection end is helped to standardize a user state detection process.

Description

Technical Field

[0001] The present invention relates to the field of computer-aided detection technology, and in particular to a user status auxiliary detection method, device, equipment, medium and product. Background Art

[0002] In daily life and work, unstable user status may cause certain adverse effects, such as threatening life or causing property loss, etc. Therefore, it is very important to detect the user status.

[0003] In the prior art, the user status detection is usually based on professional technicians such as psychologists to make judgments based on professional questions and answers to users. However, this method relies on the technical level of professionals and cannot form a standardized detection process and method. As a result, the user status detection is uneven, and the abnormal status of the user cannot be discovered in time, which in turn causes adverse effects. Summary of the invention

[0004] The present invention provides a user status auxiliary detection method, device, equipment, medium and product to standardize user status detection and improve the accuracy of user status detection.

[0005] According to one aspect of the present invention, a user status auxiliary detection method is provided, the method comprising:

[0006] Obtaining preliminary status detection results of the user by the detection end, and displaying multiple status tags corresponding to the preliminary status detection results on the detection end;

[0007] In response to a triggering operation of the detection end on the target state tag, displaying a plurality of recommendation scales corresponding to the target state tag on the detection end;

[0008] In response to the detection end selecting the target recommendation scale, sending the target recommendation scale to the user end;

[0009] Obtain the user's filling operation on the target recommendation scale, and determine the detection score and conclusion of the target recommendation scale;

[0010] Displaying the test scores and conclusions of the target recommendation scale at the test end, and determining the test question table based on the test scores and conclusions;

[0011] Displaying a test question table at the test end, and determining the user's state test degree and reference state test results in response to the test end filling out the test question table;

[0012] The state detection degree and the reference state detection result are displayed at the detection end, and in response to the detection end's determination operation on the state detection degree and the reference state detection result, a final state detection result of the user is generated.

[0013] According to another aspect of the present invention, there is provided a user status auxiliary detection device, the device comprising:

[0014] A status label display module, used to obtain the preliminary status detection result of the detection end on the user, and display multiple status labels corresponding to the preliminary status detection result on the detection end;

[0015] A recommended meter display module, configured to display a plurality of recommended meter displays corresponding to the target state tag on the detection end in response to a trigger operation of the detection end on the target state tag;

[0016] A recommended quantity table sending module, configured to send the target recommended quantity table to the user terminal in response to the detection terminal selecting the target recommended quantity table;

[0017] The detection score and conclusion determination module is used to obtain the filling operation of the target recommendation scale by the user end and determine the detection score and conclusion of the target recommendation scale;

[0018] A detection problem table determination module is used to display the detection scores and conclusions of the target recommendation scale on the detection end, and determine the detection problem table based on the detection scores and conclusions;

[0019] A reference state detection result determination module, used to display a detection question table on the detection end, and determine the state detection degree of the user and the reference state detection result in response to the filling operation of the detection question table by the detection end;

[0020] The final state detection result generating module is used to display the state detection degree and the reference state detection result on the detection end, and generate the user's final state detection result in response to the detection end's determination operation on the state detection degree and the reference state detection result.

[0021] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the user status auxiliary detection method described in any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the user status auxiliary detection method described in any embodiment of the present invention when executed.

[0026] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method for auxiliary detection of user status according to any embodiment of the present invention is implemented.

[0027] The technical solution of the embodiment of the present invention obtains the preliminary state detection result of the user by the detection end, and displays multiple state tags corresponding to the preliminary state detection result at the detection end; in response to the trigger operation of the detection end on the target state tag, displays multiple recommendation scales corresponding to the target state tag at the detection end; in response to the selection operation of the detection end on the target recommendation scale, sends the target recommendation scale to the user end; obtains the filling operation of the user end on the target recommendation scale, and determines the detection score and conclusion of the target recommendation scale; displays the detection score and conclusion of the target recommendation scale at the detection end, and determines the detection problem table according to the detection score and conclusion; displays the detection problem table at the detection end, and determines the user's state detection degree and reference state detection result in response to the filling operation of the detection end on the detection problem table; displays the state detection degree and reference state detection result at the detection end, and generates the user's final state detection result in response to the determination operation of the state detection degree and reference state detection result by the detection end, thereby solving the auxiliary detection problem of the user's state. The method can recommend scales according to the real-time state detection situation through the interaction between the detection end and the user end, help the detection end standardize the user state detection process, and provide the accuracy of user state detection.

[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flow chart of a user status auxiliary detection method provided according to an embodiment of the present invention;

[0031] Figure 2 is a structural schematic diagram of a user status auxiliary detection device provided according to an embodiment of the present invention;

[0032] Figure 3It is a structural schematic diagram of an electronic device for implementing the user status auxiliary detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Figure 1 This is a flowchart of a user status auxiliary detection method provided according to an embodiment of the present invention. This embodiment can be applied to auxiliary detection of abnormal user status. The method can be executed by a user status auxiliary detection device. The user status auxiliary detection device can be implemented in the form of hardware and / or software. The user status auxiliary detection device can be configured in an electronic device such as a smart phone or a computer.

[0036] In the embodiment of the present invention, the personal information is obtained after authorization by the user, and the method of obtaining it is reasonable and legal.

[0037] like Figure 1 As shown, the method includes:

[0038] Step 110: Obtain the preliminary status detection result of the user by the detection terminal, and display multiple status tags corresponding to the preliminary status detection result on the detection terminal.

[0039] Among them, the detection end can be an end that detects the user status. For example, the detection end can be a doctor, a friend of the user, or the user himself. The user end can be the user himself who performs the user status detection. The detector can perform a preliminary status detection on the user to form a preliminary status detection result. The preliminary status detection can be a simple inquiry and observation to determine whether the user status is abnormal. If the preliminary status detection result is that the user status is abnormal, multiple status tags corresponding to the preliminary status detection result are displayed on the detection end.

[0040] Status labels include, but are not limited to, "pregnancy," "postpartum," "insomnia," "anxiety," and "family history."

[0041] Step 120 : In response to the detection end triggering the target state tag, a plurality of recommendation scales corresponding to the target state tag are displayed on the detection end.

[0042] The tester can select from various status tags according to the user's current status stage and determine the target status tag. The target status tag can be one or more. Each status tag corresponds to a different recommendation scale. When there are multiple target status tags, the recommendation scales corresponding to each target status tag can be displayed.

[0043] The recommended scale is used to assist in detecting the user's status. The recommended scale can be filled in by the user himself. In an embodiment of the present invention, the recommended scale includes but is not limited to: Self-Rating Anxiety Scale (SAS), Depression-Anxiety-Stress Scale (DASS-21), Anxiety and Depression Scale (HADS), Generalized Anxiety Scale (GAD-7), State-Trait Anxiety Inventory (STAI-Form Y), Health Questionnaire (PHQ-9), Self-Rating Depression Scale (Zung SDS), Hamilton Depression Rating Scale (HAMD), Pittsburgh Sleep Scale, and Edinburgh Depression Scale.

[0044] Exemplarily, the recommended scales corresponding to the status label "pregnancy" may include PHQ-9, Zung SDS, Edinburgh Depression Scale, STAI-Form Y, and GAD-7. The recommended scales corresponding to the status label "postpartum" may include PHQ-9, STAI-Form Y, Zung SDS, Edinburgh Depression Scale, and Pittsburgh Sleep Scale. The recommended scales corresponding to the status label "insomnia" may include PHQ-9 and Pittsburgh Sleep Scale. The recommended scales corresponding to the status label "anxiety" may include SAS, DASS-21, PHQ-9, HADS, and GAD-7. The recommended scales corresponding to the status label "family history" may include PHQ-9, Pittsburgh Sleep Scale, and GAD-7.

[0045] Step 130: In response to the detection end's selection operation on the target recommendation table, the target recommendation table is sent to the user end.

[0046] The device or system that executes the method provided by the embodiment of the present invention can obtain the selection operation instruction of the detection end after determining and displaying multiple recommendation scales to further filter out the target state label that is more in line with the user state detection. Alternatively, the detection end can also directly use the recommendation scale determined by the device or system as the target recommendation scale. After determining the target recommendation scale, the target recommendation scale can be sent to the user end to perform a self-state assessment on the user.

[0047] Step 140: Obtain the filling operation of the target recommendation scale by the user, and determine the detection score and conclusion of the target recommendation scale.

[0048] The user can fill in the target recommendation scale to reflect his current status truthfully. The device or system can score according to the user's filling in the target recommendation scale, generate a detection score for the target recommendation scale, and determine the conclusion based on the detection score.

[0049] Among them, scoring the completion of the target recommendation scale can be to assign different scores to the options of each question in the target recommendation scale, or to perform semantic analysis on the content of each question, and classify the semantic analysis results into categories, with different categories corresponding to different scores. Then, the scores of all questions in the target recommendation scale are summarized to obtain the test score. Different test score ranges can be matched to different types of conclusions.

[0050] Step 150: Display the detection score and conclusion of the target recommendation scale at the detection end, and determine the detection question table based on the detection score and conclusion.

[0051] The detection score and conclusion may be mapped to the detection question table. For example, when the detection score and conclusion indicate that the user state is anxious, the detection question table may be determined to be an anxiety question bank. When the detection score and conclusion indicate that the user state is mildly depressed, the detection question table may be determined to be a mild depression question bank. When the detection score and conclusion indicate that the user state is moderately depressed, the detection question table may be determined to be a moderate depression question bank. When the detection score and conclusion indicate that the user state is severely depressed, the detection question table may be determined to be a severe depression question bank. The detection question table may be composed of different types of questions. The detection question table is used for the detection end to answer according to the user state or to fill in through Q&A interaction with the user. The detection question table may be statically generated or dynamically generated. For example, the questions in the detection question table may be adjusted in real time according to the user state. Exemplarily, the detection question table may be an ICD-10 including 3 core symptoms, 7 additional symptoms, hazard conditions, abnormal state duration, and abnormal state symptoms.

[0052] Step 160: Display a test question table at the test end, and determine the user's state test degree and reference state test results in response to the test end filling out the test question table.

[0053] The tester can make auxiliary judgments based on the user's current situation in terms of abnormality and impact, and fill in the test question table. The device or system can evaluate the results of filling in the test question table to obtain the user's state detection degree and reference state detection results. The user's state detection degree and reference state detection results can be displayed in the form of numerical values, conclusions, and charts.

[0054] The user's state detection degree may include, but is not limited to, the degree of abnormal state symptoms, the degree of depression, the degree of mental impairment, and the degree of ability impairment. Exemplarily, in the detection problem table, when the core symptoms are >= 2 and the additional symptoms are >= 2, the degree of abnormal state symptoms is determined to be mild; in the detection problem table, when the core symptoms are >= 2 and the additional symptoms are >= 3, the degree of abnormal state symptoms is determined to be moderate; in the detection problem table, when the core symptoms are = 3 and the additional symptoms are >= 4, the degree of abnormal state symptoms is determined to be severe; the rest of the cases are normal user status. In the detection problem table, when the abnormal state duration is within two weeks, the degree of depression is mild. In the detection problem table, when the abnormal state duration is more than two weeks, the degree of depression is severe. In the detection problem table, when the abnormal state symptoms are sometimes, the degree of mental impairment is yes. In the detection problem table, when the abnormal state symptoms are no, the degree of mental impairment is no. In the detection problem table, when the hazard situation is mildly impaired, the degree of ability impairment is mildly impaired. In the detection problem table, when the hazard situation is moderately impaired, the degree of ability impairment is moderately impaired. When the hazard situation in the test question table is severe impairment, the degree of impairment of ability is severe impairment.

[0055] The reference state detection result can be a reference conclusion obtained by integrating the various degrees in the state detection degree. The reference state detection result includes but is not limited to the abnormal state of the user being schizophrenia tendency, bipolar disorder, anxiety state, post-traumatic stress state, and abnormal state caused by physical illness, etc. In the reference state detection result, each abnormal state can be described in detail, so that the tester can make a determination operation based on the description of the situation.

[0056] Step 170: Display the state detection degree and the reference state detection result at the detection end, and generate the user's final state detection result in response to the detection end's determination operation on the state detection degree and the reference state detection result.

[0057] The tester can determine the situation based on the status detection degree given by the device or system and the reference status detection results, and make further judgments based on his own cognition. The user's final status detection result can be in the form of an electronic report. The user's final status detection result includes but is not limited to the tester's determination of the status detection degree and the reference status detection result, and the tester's additional detection results. In the electronic report, an editable area can be set to facilitate the tester to fill in personalized suggestions and tester information.

[0058] The technical solution of this embodiment obtains the preliminary state detection result of the user at the detection end, and displays a plurality of state tags corresponding to the preliminary state detection result at the detection end; in response to the trigger operation of the detection end on the target state tag, displays a plurality of recommendation scales corresponding to the target state tag at the detection end; in response to the selection operation of the detection end on the target recommendation scale, sends the target recommendation scale to the user end; obtains the filling operation of the target recommendation scale at the user end, and determines the detection score and conclusion of the target recommendation scale; displays the detection score and conclusion of the target recommendation scale at the detection end, and determines the detection question table according to the detection score and conclusion; The end displays a detection question table, and in response to the detection end's filling operation on the detection question table, determines the user's state detection degree and a reference state detection result; the state detection degree and the reference state detection result are displayed on the detection end, and in response to the detection end's determination operation on the state detection degree and the reference state detection result, generates the user's final state detection result, thereby solving the problem of auxiliary detection of the user's state. The method can make a scale recommendation based on the real-time state detection situation through the interaction between the detection end and the user end, thereby helping the detection end to standardize the user state detection process, improve the accuracy of user state detection, and reduce the detection level requirements for the detector.

[0059] On the basis of the above implementation, optionally, in response to the triggering operation of the detection end on the target status label, multiple recommendation scales corresponding to the target status label are displayed on the detection end, including: calculating the target association strength between the exposure factor and the user status; obtaining the user's basic data and physical examination data at the detection end, and performing semantic analysis on the basic data and physical examination data to determine the corresponding target exposure factors; determining the risk assessment coefficient of the user status based on the target exposure factors and the association strength; and determining multiple recommendation scales in the initial recommendation scale set corresponding to the target status label based on the risk assessment coefficient.

[0060] The target association strength between the exposure factor and the user status may be a mapping table pre-generated based on the work experience of those skilled in the art. Alternatively, the target association strength between the exposure factor and the user status may be a mapping table obtained by analyzing historical data. The target association strength may be an odds ratio (OR) value.

[0061] The OR value is a statistic used to describe the ratio of two ratios in human health research. The OR value is mainly used in human health control studies to analyze the relationship between a certain factor and abnormal health conditions. The OR value represents the ratio of the ratio of exposure to non-exposure in the abnormal health group to the ratio of exposure to non-exposure in the control group. The OR value reflects the strength of the association between a certain factor and health effects. In human health research, the OR value is often used to analyze the impact of a certain lifestyle or environmental factor on human health.

[0062] In the embodiment of the present invention, the OR value can be used to analyze the influence of each factor on the user status. Specifically, when the OR value is lower than 1, the addition of the corresponding exposure factor will reduce the risk of abnormal user status; when the OR value is higher than 1, the addition of the corresponding exposure factor will increase the risk of abnormal user status.

[0063] In an optional implementation of an embodiment of the present invention, the target association strength between the exposure factor and the user status is calculated, including: obtaining an experience value of the association strength between the exposure factor and the user status, historical user data, and the corresponding historical user status; inputting the historical user data and the corresponding historical user status into a preset neural network model, and training the model based on the experience value of the association strength to obtain the target association strength between the exposure factor and the user status.

[0064] Among them, the empirical value of the association strength between the exposure factor and the user status can be an OR value. Exposure factors include but are not limited to age, gender, education level, family status, genetic factors, biochemical factors, hormonal changes, changes in brain structure and function, and social environmental factors. Historical user data can be basic data and physical examination data of the user. Historical user data can be data related to exposure factors. Historical user status can be a historical status detection result. The neural network model can be used to train and verify the model based on the empirical value of the association strength, historical user data, and the corresponding historical user status to improve the accuracy and reliability of the model. Thus, the target association strength between the exposure factor and the user status is obtained based on the preset neural network model.

[0065] Exemplarily, the target association strength between the exposure factor and the depression state in the user state may include but is not limited to: when the number of pregnancies is greater than or equal to 3 times, the target association strength is 1.306; when there is a miscarriage, the target association strength is 1.06; when the education level is high, the target association strength is 0.66; when the living conditions are poor, the target association strength is 1.819; when being bullied, the target association strength is 2.96; when the family monthly income is low, the target association strength is 0.706; when the relationship between husband and wife is not good, the target association strength is 2.064; when the work is stressful, the target association strength is 2.213 ; When the diet tends to be Western fast food, the target association strength is 1.31; when the diet is nutritionally balanced, the target association strength is 0.43; when there is a habit of exercise, the target association strength is 0.67; when there is sleep problem, the target association strength is 5.17; when not smoking, the target association strength is 0.78; when systolic blood pressure / diastolic blood pressure is <120 / 80mmHg, the target association strength is 1.79; when c-reactive protein I>24.00mg / L, the target association strength is 1.98; when having hypertension, the target association strength is 5.28; when having an easy-going personality, the target association strength is 0.447.

[0066] Exemplarily, the target association strength between the exposure factor and the anxiety state in the user state may include but is not limited to: when the user is a female, the target association strength is 0.669; when there is miscarriage, the target association strength is 3.042; when the education level is high, the target association strength is 0.962; when the growth environment is not good, the target association strength is 12.28; when the number of friends is relatively large, the target association strength is 0.673; when the environmental adaptability is good, the target association strength is 1.023; when the relationship between husband and wife is not good, the target association strength is 2.75; when the physical condition is not good, the target association strength is 1.395; when the work and study burden is heavy, the target association strength is 1.762; when there is chronic kidney disease, the target association strength is 1.1; when there is coronary heart disease, the target association strength is 1.83; when there is diabetes, the target association strength is 2.

[0067] The user's basic data may include but is not limited to: age, gender, education level, family income, marital status, family status, whether or not they have children, past medical history and chronic disease history, immune indicators, CT values, support system status, and self-confidence level, etc. Physical examination data may include but is not limited to: genetic factors (such as medical history), biochemical factors (such as thyroid function tests), hormonal changes (such as hormone levels), changes in brain structure and function (such as CT scan results), psychological factors, social environmental factors, life events and stress, physical health, sleep problems, substance use (such as alcohol or drug testing), trauma and violent experiences, occupational factors, culture and beliefs, environmental factors, and personal history (such as early illness or malnutrition).

[0068] By analyzing the user's basic data and physical examination data through the semantic analysis module, the specific basic data and physical examination data can be classified into specific exposure factors. Each exposure factor can be weighted with the corresponding OR value to obtain the risk assessment coefficient of the user status. According to the risk assessment coefficient of the user status, multiple recommended scales can be further screened out from the initial recommended scale set corresponding to the target status label to improve the reliability of the recommended scale determination.

[0069] Exemplarily, when the risk assessment coefficients of depression and anxiety are both greater than 2, the determined recommended scale may include a scale with severe anxiety and depression problems in the initial recommended scale set corresponding to the target state label, and the number of scale questions is about 15, so as to avoid the user's abnormal state being serious and difficult to complete the test; at this time, the user's abnormal state can be additionally marked with a serious mark. When at least one of the risk assessment coefficients of depression and anxiety is greater than 1.01, and both coefficients are not greater than 2, the determined recommended scale may include a scale with moderate anxiety and depression problems in the initial recommended scale set corresponding to the target state label, and the number of scale questions is about 25, so as to accurately detect the user's state; at this time, the user's state can be additionally marked with a more serious mark. When the risk assessment coefficients of depression and anxiety are both lower than 1.01, the recommended scale can be empty, and the test question table is directly recommended for the tester to ask questions. If the recommended scale is empty, the test score and conclusion can be anxiety, and then the test question table is determined to be an anxiety question bank.

[0070] The technical solution of the embodiment of the present invention determines the risk assessment coefficient of the user status by the target correlation strength between the exposure factor and the user status, and then optimizes the screening recommendation scale, which can further improve the accuracy and efficiency of the user status assessment. Under the premise of ensuring the normal user status detection, the user can be targeted and reliably detected, and the detection deviation caused by human factors can be reduced by optimizing the recommended scale. Moreover, by limiting the number of questions in the recommended scale, the time for users to fill in the scale can be reduced, the user's compliance can be improved, and the reliability of detection can be guaranteed.

[0071] In an optional implementation of an embodiment of the present invention, a detection problem table is determined based on the detection score and conclusion, including: collecting call data between the detection end and the user end and performing semantic analysis to obtain the user's first auxiliary detection state result; collecting the user's facial expression and performing computer vision analysis to obtain the user's second auxiliary detection state result; determining the target auxiliary detection state result based on the first auxiliary detection state result and the second auxiliary detection state result; determining the detection problem table based on the target auxiliary detection state result, the detection score and the conclusion.

[0072] Among them, the semantic analysis of the call data can be to extract key information from the call data, and perform processing including but not limited to named entity recognition, relationship extraction, sentiment analysis and intent recognition to parse the call data to obtain the first auxiliary detection status result.

[0073] For example, named entity recognition can be used to identify symptoms, drugs, and disease names mentioned in call data. Relationship extraction can be used to determine the association between named entities, such as the relationship between symptoms, drugs, and disease names and user status. Sentiment analysis can be used to determine the severity of a user's abnormal state through emotional state. Intent recognition can be used to identify intent in call data to better guide call content.

[0074] In an embodiment of the present invention, call data includes but is not limited to voice, text, and video. Therefore, the processing of call data involves the fusion of multimodal data. Specifically, a multimodal deep learning model can be constructed to simultaneously process data such as voice, text, and video. Among them, the multimodal deep learning model may include a modal encoder, a multimodal fusion layer, and a classifier. The modal encoder encodes the data of each modality respectively. The multimodal fusion layer fuses the encoded information of each modality together. The classifier performs the final user state classification on the fused information. The modal encoder includes a text encoder, a speech encoder, and a visual encoder. The text encoder uses a convolutional neural network (CNN) or a recurrent neural network (RNN) to extract text features and generate a text embedding vector. The speech encoder uses CNN or RNN to extract audio signal features and generate a speech feature vector. The visual encoder uses CNN or RNN to extract facial expression features in the video and generate a facial expression feature vector. The fusion strategy of the multimodal fusion layer can be diverse, such as splicing, weighted summation, or interactive fusion. For example, an attention mechanism can be used to assign weights to each modality so that the model can decide which modalities are more important according to the specific situation. When introducing the attention mechanism, the attention mechanism can be introduced in each modal encoder and between modal encoders. The attention mechanism in each modal encoder facilitates the extraction and analysis of key data in the current modal data. The attention mechanism between modal encoders facilitates the model to better understand the relationship between different modalities. The classifier may include a fully connected layer and an output layer. The output layer may use an activation function such as a softmax function to predict the probability distribution of different categories of user abnormal states. That is, the first auxiliary detection state result is the probability that the user's abnormal state belongs to different categories. When training the model, a loss function that can balance positive and negative samples may be used to avoid user state detection errors; a focal loss function may also be used to reduce the weight of easily classified samples so that the model pays more attention to difficult-to-classify samples.

[0075] In an embodiment of the present invention, the computer vision analysis of the user's facial expression can be to identify the user's emotional state and capture facial features to classify the user's state. The computer vision analysis of the user's facial expression can be based on Ekman's basic emotion theory. Ekman's basic emotion theory includes six basic emotions: happiness, sadness, anger, fear, surprise and disgust. When performing computer vision analysis, the Facial Action Coding System (FACS) can be used to identify, analyze and encode subtle changes in facial expressions. FACS divides facial muscles into several action units (AUs), each of which has a specific expression meaning. That is, the second auxiliary detection state result can reflect the user's emotional state.

[0076] In computer vision analysis, you can collect user facial features, extract facial expression features through CNN, and use labeled data sets to train classifiers, such as support vector machines (SVM), random forests, or deep learning models. Use pre-trained models to fine-tune to fit specific data sets. Based on the trained model, define mapping rules to map facial expression features to specific user emotional states. Set thresholds for each emotion category, and when the facial features reach a certain standard, the emotional state is considered to be activated.

[0077] A complete emotional expression process can be divided into the following stages: 1) Triggering, external stimuli or internal feelings trigger emotional reactions, such as hearing a joke; 2) Preparation, facial muscles begin to prepare to make corresponding expressions, such as facial muscles preparing to smile; 3) Manifestation, facial muscles move to form specific expressions, such as the corners of the mouth rising and laugh lines around the eyes; 4) Maintenance, the expression is maintained for a period of time, such as a smile lasting from a few seconds to tens of seconds; 5) Recovery, the expression gradually disappears and returns to a neutral state, such as the expression gradually returns to a calm state.

[0078] By using the second auxiliary detection status result when confirming the detection question list, the detection question list can be updated according to the real-time user emotional state reaction, so as to dynamically and accurately detect the user state. For example, in the current call, the user shows significant sadness or anger, and the relevant topics in the detection can be adjusted according to the second auxiliary detection status result.

[0079] In an embodiment of the present invention, the target auxiliary detection state result may be an integration of the first auxiliary detection state result and the second auxiliary detection state result. That is, the target auxiliary detection state result may include the probability that the user state belongs to each state classification and the user's emotional state. When determining the detection question table, the target auxiliary detection state result, the detection score and the conclusion may be combined to comprehensively adjust the detection question table to improve the personalization of the detection and the detection accuracy.

[0080] In an optional implementation of an embodiment of the present invention, a target auxiliary detection state result is determined based on the first auxiliary detection state result and the second auxiliary detection state result, including: determining the auxiliary detection state result at the current moment and the auxiliary detection state change trend based on the first auxiliary detection state results at multiple moments and the corresponding second auxiliary detection state results at multiple moments; and using the auxiliary detection state result at the current moment and the auxiliary detection state change trend as the target auxiliary detection state result.

[0081] Among them, the first auxiliary detection state result includes the probability of each state to which the user belongs. As time changes, the probability of each category is dynamically adjusted. The probability change can be determined by fitting the function to the probability change, and the next probability change trend can be determined to determine the next user state. The second auxiliary detection state result can reflect the user's emotional state in real time. The user's next emotional state can be determined by the changes in the emotional state at multiple moments. The auxiliary detection state change trend can be generated by the next user state and emotional state. When determining the detection question table, the auxiliary detection state change trend is used as one of the reference bases, and the dynamic adjustment of the detection question can be realized, so that the detector knows whether to continue to communicate the current topic or switch to the next topic, and the detection question type of the next topic can be clarified, thereby personalizing and adjusting the user state detection in real time, and improving the reliability and accuracy of the user state detection.

[0082] In an optional implementation of an embodiment of the present invention, in response to the detection end's determination operation on the state detection degree and the reference state detection result, a final state detection result of the user is generated, including: obtaining the user's physical examination data at the detection end, and determining the first reference coefficient of the user in each alternative state based on the physiological indicators related to the user's state in the physical examination data; determining the second reference coefficient of the user in each alternative state based on the detection score and conclusion of the target recommendation scale; determining the third reference coefficient of the user in each alternative state based on the state detection degree and the reference state detection result; determining the fourth reference coefficient of the user in each alternative state based on the target auxiliary detection state result; determining the target reference coefficient of the user in each alternative state based on the first reference coefficient, the second reference coefficient, the third reference coefficient and the fourth reference coefficient of the user in each alternative state; in response to the detection end's determination operation on the state detection degree and the reference state detection result, combined with the target reference coefficient of the user in each alternative state, the final state detection result of the user is generated.

[0083] Among them, the determination method of the first reference coefficient can be the same as the risk assessment coefficient of the user state. Alternatively, a mapping table can be constructed to classify and map the physiological indicators in the physical examination data to determine the first reference coefficient. The second reference coefficient can be to determine the user state based on the conclusion, and determine the coefficient value of the corresponding state based on the detection score. The third reference coefficient can be to determine the user state based on the reference state detection result, and determine the coefficient value of the corresponding state based on the state detection degree. The fourth reference coefficient can be to determine the user state and the corresponding coefficient value based on the user state probability of the first auxiliary detection state result in the target auxiliary detection state result. The target reference coefficient can be obtained by performing a weighted sum operation on the first reference coefficient, the second reference coefficient, the third reference coefficient and the fourth reference coefficient under each alternative state. When generating the final state detection result of the user, adding the target reference coefficient can improve the comprehensiveness of the user state determination, making its detection result more reliable and more confident.

[0084] On the basis of the above implementation, optionally, when designing the user interface, intuitiveness and ease of use can be considered so that the interface can effectively display key information to improve detection efficiency. For example, a circular dashboard is placed at the top center of the detector's user interface, and different colors are used to represent different emotional states, such as green for calm and red for anger. A vertical timeline is placed on the left side of the interface to show the changing trend of the user's emotional state over time. On the right side can be a bar chart showing the distribution of each emotion category. A quick navigation bar can be set at the bottom, including several main function buttons such as "Overview", "Details", and "History". There is a small panel in the upper left corner of the interface that can display the user's basic information. There is a gear icon in the upper right corner, which can expand the custom settings menu after clicking.

[0085] Figure 2 is a schematic diagram of the structure of a user status auxiliary detection device provided according to an embodiment of the present invention. Figure 2 As shown, the device includes: a state label display module 210, a recommended scale display module 220, a recommended scale sending module 230, a detection score and conclusion determination module 240, a detection problem table determination module 250, a reference state detection result determination module 260, and a final state detection result generation module 270. Among them:

[0086] The status label display module 210 is used to obtain the preliminary status detection result of the user by the detection terminal, and display a plurality of status labels corresponding to the preliminary status detection result on the detection terminal;

[0087] A recommended meter display module 220, configured to display a plurality of recommended meter displays corresponding to the target state tag at the detection end in response to a trigger operation of the detection end on the target state tag;

[0088] The recommended quantity table sending module 230 is used to send the target recommended quantity table to the user terminal in response to the detection terminal selecting operation on the target recommended quantity table;

[0089] The detection score and conclusion determination module 240 is used to obtain the filling operation of the target recommendation scale by the user end and determine the detection score and conclusion of the target recommendation scale;

[0090] A detection problem table determination module 250 is used to display the detection score and conclusion of the target recommendation scale at the detection end, and determine the detection problem table according to the detection score and conclusion;

[0091] A reference state detection result determination module 260 is used to display a detection question table on the detection end, and determine the state detection degree of the user and the reference state detection result in response to the filling operation of the detection question table on the detection end;

[0092] The final state detection result generating module 270 is used to display the state detection degree and the reference state detection result at the detection end, and generate the user's final state detection result in response to the detection end's determination operation on the state detection degree and the reference state detection result.

[0093] Optionally, the recommended scale display module 220 includes:

[0094] A target association strength calculation unit, used to calculate the target association strength between the exposure factor and the user state;

[0095] A target exposure factor determination unit is used to obtain the user's basic data and physical examination data at the detection end, and perform semantic analysis on the basic data and physical examination data to determine the corresponding target exposure factors;

[0096] A risk assessment coefficient determination unit, used to determine the risk assessment coefficient of the user status according to the target exposure factor and the correlation strength;

[0097] The recommended scale determination unit is used to determine multiple recommended scales in an initial recommended scale set corresponding to the target state label according to the risk assessment coefficient.

[0098] Optionally, a target association strength calculation unit includes:

[0099] A historical data acquisition subunit, used to obtain the empirical value of the association strength between the exposure factor and the user status, historical user data, and the corresponding historical user status;

[0100] The target association strength determination subunit is used to input historical user data and corresponding historical user status into a preset neural network model, perform model training based on the association strength experience value, and obtain the target association strength between the exposure factor and the user status.

[0101] Optionally, the detection problem table determination module 250 includes:

[0102] A first auxiliary detection status result determination unit, used to collect call data between the detection end and the user end and perform semantic analysis to obtain a first auxiliary detection status result of the user;

[0103] A second auxiliary detection state result determination unit, used for collecting the user's facial expression and performing computer vision analysis to obtain the user's second auxiliary detection state result;

[0104] a target auxiliary detection state result determining unit, configured to determine a target auxiliary detection state result according to the first auxiliary detection state result and the second auxiliary detection state result;

[0105] The detection problem table determination unit is used to determine the detection problem table according to the target auxiliary detection state results, detection scores and conclusions.

[0106] Optionally, the target auxiliary detection state result determination unit includes:

[0107] A result and trend determination subunit, used to determine the auxiliary detection state result at the current moment and the auxiliary detection state change trend according to the first auxiliary detection state results at multiple moments and the corresponding second auxiliary detection state results at multiple moments;

[0108] The target auxiliary detection state result determination subunit is used to use the auxiliary detection state result at the current moment and the auxiliary detection state change trend as the target auxiliary detection state result.

[0109] Optionally, the final state detection result generating module 270 includes:

[0110] A first reference coefficient determination subunit, used to obtain the user's physical examination data at the detection end, and determine the user's first reference coefficient in each candidate state according to the physiological indicators related to the user's state in the physical examination data;

[0111] A second reference coefficient determination subunit, used to determine the second reference coefficient of the user in each alternative state according to the detection score and conclusion of the target recommendation scale;

[0112] A third reference coefficient determination subunit, configured to determine a third reference coefficient of the user in each candidate state according to the state detection degree and the reference state detection result;

[0113] A fourth reference coefficient determination subunit, configured to determine a fourth reference coefficient of the user in each candidate state according to the target auxiliary detection state result;

[0114] a target reference coefficient determination subunit, configured to determine a target reference coefficient of the user in each alternative state according to the first reference coefficient, the second reference coefficient, the third reference coefficient and the fourth reference coefficient of the user in each alternative state;

[0115] The final state detection result generating subunit is used to generate the user's final state detection result in response to the detection end's determination operation on the state detection degree and the reference state detection result, combined with the user's target reference coefficient in each alternative state.

[0116] The user status auxiliary detection device provided in the embodiment of the present invention can execute the user status auxiliary detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0117] In the technical solution of the embodiment of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information (such as basic data, physical examination data, facial information, voice information, etc.) are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0118] Figure 3 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0119] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0120] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0121] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the user status auxiliary detection method.

[0122] In some embodiments, the user status auxiliary detection method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the user status auxiliary detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the user status auxiliary detection method in any other appropriate manner (for example, by means of firmware).

[0123] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0125] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0128] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0129] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0130] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A user status auxiliary detection method, characterized in that: include: Obtaining preliminary status detection results of the user by the detection end, and displaying multiple status tags corresponding to the preliminary status detection results on the detection end; In response to a triggering operation of the detection end on the target state tag, displaying a plurality of recommendation scales corresponding to the target state tag on the detection end; In response to the detection end selecting the target recommendation scale, sending the target recommendation scale to the user end; Obtain the user's filling operation on the target recommendation scale, and determine the detection score and conclusion of the target recommendation scale; Displaying the test scores and conclusions of the target recommendation scale at the test end, and determining the test question table based on the test scores and conclusions; Displaying a test question table at the test end, and determining the user's state test degree and reference state test results in response to the test end filling out the test question table; The state detection degree and the reference state detection result are displayed at the detection end, and in response to the detection end's determination operation on the state detection degree and the reference state detection result, a final state detection result of the user is generated.

2. The method according to claim 1, characterized in that In response to the triggering operation of the detection end on the target state tag, a plurality of recommendation scales corresponding to the target state tag are displayed on the detection end, including: Calculate the target association strength between the exposure factor and the user state; The user's basic data and physical examination data are obtained at the detection end, and semantic analysis is performed on the basic data and physical examination data to determine the corresponding target exposure factors; Determining a risk assessment coefficient of a user state according to the target exposure factor and the association strength; According to the risk assessment coefficient, a plurality of recommended scales are determined in an initial recommended scale set corresponding to the target state label.

3. The method according to claim 2, characterized in that Calculate the target association strength between exposure factors and user status, including: Obtaining the empirical value of the correlation strength between the exposure factor and the user status, historical user data, and the corresponding historical user status; The historical user data and the corresponding historical user status are input into the preset neural network model, and the model is trained based on the empirical value of the association strength to obtain the target association strength between the exposure factor and the user status.

4. The method according to claim 1, characterized in that According to the test scores and conclusions, a test question table is determined, including: Collect call data between the detection end and the user end and perform semantic analysis to obtain the user's first auxiliary detection status result; Collect the user's facial expression and perform computer vision analysis to obtain the user's second auxiliary detection status result; Determine a target auxiliary detection state result according to the first auxiliary detection state result and the second auxiliary detection state result; A detection problem table is determined based on the target auxiliary detection status result, the detection score and the conclusion.

5. The method according to claim 4, characterized in that Determining the target auxiliary detection state result according to the first auxiliary detection state result and the second auxiliary detection state result includes: Determine the auxiliary detection state result at the current moment and the auxiliary detection state change trend according to the first auxiliary detection state results at multiple moments and the corresponding second auxiliary detection state results at multiple moments; The auxiliary detection state result at the current moment and the auxiliary detection state change trend are used as the target auxiliary detection state result.

6. The method according to claim 4, characterized in that In response to the detection end determining the state detection degree and the reference state detection result, generating a final state detection result of the user, including: Acquire the user's physical examination data at the detection end, and determine the first reference coefficient of the user in each alternative state according to the physiological indicators related to the user's state in the physical examination data; Determine the second reference coefficient of the user in each alternative state according to the detection score and conclusion of the target recommendation scale; Determining a third reference coefficient of the user in each candidate state according to the state detection degree and the reference state detection result; Determining a fourth reference coefficient of the user in each alternative state according to the target auxiliary detection state result; Determining a target reference coefficient of the user in each alternative state according to the first reference coefficient, the second reference coefficient, the third reference coefficient, and the fourth reference coefficient of the user in each alternative state; In response to the detection end's determination operation on the state detection degree and the reference state detection result, combined with the user's target reference coefficient in each candidate state, a final state detection result of the user is generated.

7. A user status auxiliary detection device, characterized in that: include: A status label display module, used to obtain the preliminary status detection result of the detection end on the user, and display multiple status labels corresponding to the preliminary status detection result on the detection end; A recommended meter display module, configured to display a plurality of recommended meter displays corresponding to the target state tag on the detection end in response to a trigger operation of the detection end on the target state tag; A recommended quantity table sending module, configured to send the target recommended quantity table to the user terminal in response to the detection terminal selecting the target recommended quantity table; The detection score and conclusion determination module is used to obtain the filling operation of the target recommendation scale by the user end and determine the detection score and conclusion of the target recommendation scale; A detection problem table determination module is used to display the detection scores and conclusions of the target recommendation scale on the detection end, and determine the detection problem table based on the detection scores and conclusions; A reference state detection result determination module, used to display a detection question table on the detection end, and determine the state detection degree of the user and the reference state detection result in response to the filling operation of the detection question table by the detection end; The final state detection result generating module is used to display the state detection degree and the reference state detection result on the detection end, and generate the user's final state detection result in response to the detection end's determination operation on the state detection degree and the reference state detection result.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the user status auxiliary detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the user status auxiliary detection method according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the user status auxiliary detection method according to any one of claims 1 to 6.

Images