Auxiliary method, system and equipment based on artificial intelligence, medium and product

Abstract

The invention relates to an auxiliary method, system and device based on artificial intelligence, a medium and a product. The method comprises the steps that multi-modal detection data, collected by an intelligent device, of a user is acquired; wherein the multi-modal detection data comprises at least one of the following modals: audio data, video data and sensor data; based on the multi-modal detection data, determining state information of the user for the target state; wherein the target state at least comprises an emotional state, a body state and an environment state; processing the state information through a large model to obtain decision-making interaction information; obtaining feedback information of the user for the decision-making interaction information, determining an intimacy value of the feedback information, and adjusting an acquisition strategy of the feedback information according to the intimacy value; wherein the intimacy value is used for indicating the satisfaction degree of the user on the feedback information. The personal habits of the user can be accurately obtained, the pertinence of the auxiliary method based on artificial intelligence to the user is improved, and the experience feeling of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an auxiliary method, system, device, medium and product based on artificial intelligence. Background Art

[0002] With the development of artificial intelligence, AI-based personal assistants are gradually entering people's lives. Personal assistants enhance the user experience by providing effective support and seamless interaction. They are usually used to provide information and conduct assessments to patients, and can form unique and presentable images and short videos, which can help shorten patients' recovery process and improve their mental health.

[0003] In the related art, the personal assistant system relies too much on the accuracy of input information, lacks recognition of spatial environmental information, and cannot accurately obtain the user's current information; and the artificial intelligence model used by the personal assistant system in the related art is general, cannot penetrate into the user's life, and is not very targeted to the user. Summary of the invention

[0004] The present disclosure provides an auxiliary method, system, device, medium and product based on artificial intelligence.

[0005] According to a first aspect of the present disclosure, there is provided an artificial intelligence-based assistance method, the method comprising:

[0006] Acquire multimodal detection data of the user collected by the smart device; wherein the multimodal detection data includes at least one of the following modalities: audio data, video data, and sensor data;

[0007] Based on the multimodal detection data, determine the state information of the user for a target state; wherein the target state includes at least: an emotional state, a physical state, and an environmental state;

[0008] Processing the state information through a large model to obtain decision-making interaction information;

[0009] Obtain feedback information from the user regarding the decision-making interaction information, determine an intimacy value of the feedback information, and adjust a collection strategy for the feedback information according to the intimacy value; wherein the intimacy value is used to indicate the user's satisfaction with the feedback information.

[0010] Further, determining the state information of the user with respect to the target state based on the multimodal detection data includes:

[0011] Determine the characteristic information of the user based on the multimodal detection data; the characteristic information includes at least one of the following: facial expression characteristics, body movement characteristics, emotion characteristics, sensor value characteristics;

[0012] Determining the degree of association between the target state and each of the feature information;

[0013] Performing weighted summation on the multiple association degrees to obtain a state value of each target state;

[0014] Based on the state value of each of the target states, state information of the user with respect to the target state is determined.

[0015] Further, determining the intimacy value of the feedback information includes:

[0016] Determining the degree of time correlation between the state information and the feedback information based on a difference between a timestamp of the state information and a timestamp of the feedback information collected in each collection period;

[0017] Determining a feedback result based on the feedback information collected in each collection cycle; wherein the feedback result reflects the user's satisfaction with the decision-making interaction information;

[0018] The intimacy value is determined based on the degree of association and the feedback result.

[0019] Furthermore, adjusting the collection strategy of feedback information according to the intimacy value includes:

[0020] If the intimacy value calculated for the first number of consecutive collection cycles increases, reducing the collection frequency of the feedback information;

[0021] If the intimacy value calculated for a second number of consecutive collection cycles decreases, the collection frequency of the feedback information is increased.

[0022] Further, the determining the intimacy value based on the association degree and the feedback result includes:

[0023] Determining a first calculation coefficient of the feedback result;

[0024] The intimacy value is obtained by performing a product sum operation on the feedback result calculated in each collection cycle and the correlation degree according to the first calculation coefficient.

[0025] Further, the determining of the first calculation coefficient of the feedback result includes:

[0026] Determine the average of the intimacy values ​​calculated for the third consecutive collection period in the past moment;

[0027] If the average value is greater than a preset threshold, reducing the second calculation coefficient of the feedback result by a first preset value to obtain the first calculation coefficient;

[0028] If the average value is less than a preset threshold, the second calculation coefficient of the feedback result is increased by a second preset value to obtain the first calculation coefficient; wherein the second calculation coefficient is the previous calculation coefficient of the feedback result used to calculate intimacy.

[0029] According to a second aspect of the present disclosure, an auxiliary system based on artificial intelligence is provided, including an artificial intelligence decision module, an interactive control module, and a device sensing module, wherein:

[0030] The device sensing module is used to obtain multimodal detection data;

[0031] The data processing module is used to obtain multimodal detection data of the user collected by the smart device; wherein the multimodal detection data includes at least one of the following modes: audio data, video data, and sensor data; based on the multimodal detection data, determine the state information of the user for the target state; wherein the target state includes at least: emotional state, physical state, and environmental state; and obtain feedback information, determine the intimacy value of the feedback information, and adjust the collection strategy of the feedback information according to the intimacy value; wherein the intimacy value is used to indicate the user's satisfaction with the feedback information;

[0032] The artificial intelligence decision module is used to process the state information through a large model to obtain decision-making interaction information;

[0033] The interaction control module is used to determine the feedback information of the user for the decision-making interaction information based on the decision-making interaction information, and feed back the feedback information to the data processing module.

[0034] Furthermore, the system further comprises: an authorization and authentication module and a communication module, wherein:

[0035] The communication module is used for communication connection between various modules in the artificial intelligence-based auxiliary system;

[0036] The authorization and authentication module is used to authenticate the identity of the user in response to the user's access operation to the artificial intelligence decision-making module.

[0037] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0038] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0039] According to a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the method of the present disclosure is implemented.

[0040] The embodiments of the present disclosure provide an auxiliary method, system, device, medium and product based on artificial intelligence. In the embodiments of the present application, first, multimodal detection data of the user collected by the smart device is obtained; wherein the multimodal detection data includes at least one of the following modalities: audio data, video data, sensor data; then, based on the multimodal detection data, the user's state information for the target state is determined; wherein the target state includes at least: emotional state, physical state, environmental state; then, the state information is processed by a large model to obtain decision-making interaction information; finally, feedback information of the user for the decision-making interaction information is obtained, and the intimacy value of the feedback information is determined, and the collection strategy of the feedback information is adjusted according to the intimacy value; wherein the intimacy value is used to indicate the user's satisfaction with the feedback information.

[0041] From the above description, it can be seen that the status information for the user's target state is determined based on multimodal detection data. The multimodal detection data may include audio data, video data, and sensor data. The comprehensive multimodal detection data enables the user's current information and environment to be highly integrated, ensuring the comprehensiveness and accuracy of the user's status information. After obtaining the user's status information, the status information can be processed by the big model to obtain decision-making interaction information, and the intimacy value of the feedback information can be determined by obtaining the user's feedback information on the decision-making interaction information. Finally, the collection strategy of the feedback information can be adjusted according to the intimacy value. The adjustment of the collection strategy can train the big model more accurately, so that the big model can go deeper into the user's life, more accurately obtain the user's personal habits, improve the pertinence of the auxiliary method based on artificial intelligence to the user, and further enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0043] Figure 1 A flowchart of an artificial intelligence-based assistance method provided for an exemplary embodiment of the present disclosure;

[0044] Figure 2 A schematic diagram of a feedback information interaction page provided for an exemplary embodiment of the present disclosure;

[0045] Figure 3 A flowchart of an assistance method based on artificial intelligence provided by another exemplary embodiment of the present disclosure;

[0046] Figure 4 A schematic block diagram of the structure of an auxiliary system based on artificial intelligence provided by an exemplary embodiment of the present disclosure;

[0047] Figure 5 A schematic block diagram of the structure of an artificial intelligence-based auxiliary system provided for another exemplary embodiment of the present disclosure;

[0048] Figure 6 A schematic block diagram of the structure of an artificial intelligence-based auxiliary system provided for another exemplary embodiment of the present disclosure;

[0049] Figure 7 A schematic diagram of module interaction provided for an exemplary embodiment of the present disclosure;

[0050] Figure 8 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0051] Fig. 9 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure;

[0052] Fig.10 A structural block diagram of a computer program product provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0054] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0055] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0056] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0057] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0058] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0059] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0060] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0061] In one embodiment, Figure 1 As shown, an artificial intelligence-based auxiliary method is provided, comprising the following steps:

[0062] Step 101: Acquire multimodal detection data of a user collected by a smart device.

[0063] Here, the multimodal detection data of the user collected by the smart device can be obtained through the server, wherein the multimodal detection data includes at least one of the following modalities: audio data, video data, and sensor data.

[0064] In a possible embodiment, the server obtains the multimodal detection data of the user collected by the device sensor module, that is, obtains the original data of the user in the private space, which includes various types of smart devices. For example, the smart device can be a smart device under the whole house smart or wearable terminal, including mobile phones, tablets, watches, bracelets, thermometers and hygrometers, sports watches, smart doorbells, smart cameras, Extended Reality (Extended Reality, XR) devices, humanoid robots, sweeping robots, smart home appliances, etc. It should be noted that the specific types of smart devices are not limited here. The multimodal detection data of the user collected by the smart device includes: audio data, video data, and sensor data. For example, audio data can be a recording of the user recorded by the Internet of Things device; video data can be a video taken by a smart camera; sensor data can be the body parameters such as weight and body fat of the user extracted by a smart body fat scale, or the health parameters such as heart rate, body temperature, exercise data, sleep status of the user extracted by a smart watch, or the air quality, water quality, brightness and other indexes of the user's living environment extracted by the environmental monitoring smart device. It should be noted that multimodal detection data may also include raw data obtained by various IoT devices, which is not limited here.

[0065] Step 102: Determine the user's state information for the target state based on the multimodal detection data.

[0066] Here, after acquiring the multimodal detection data of the user collected by the smart device, the server can determine the user's state information for the target state based on the multimodal detection data, wherein the target state includes at least: emotional state, physical state, and environmental state.

[0067] In a possible embodiment, determining the state information of the user with respect to the target state based on the multimodal detection data includes the following steps:

[0068] Determining characteristic information of the user based on the multimodal detection data;

[0069] Determine the degree of association between the target state and each feature information;

[0070] Perform weighted summation on multiple association degrees to obtain the state value of each target state;

[0071] Based on the status value of each target status, the status information of the user with respect to the target status is determined.

[0072] Specifically, first, after obtaining the multimodal detection data of the user collected by the smart device, the multimodal detection data is first cleaned. For example, for video data in the multimodal detection data, a video processing library, such as OpenCV (cross-platform computer vision library), can be used to remove blurry, jittery or irrelevant parts of the video; for audio data in the multimodal detection data, an audio processing library, such as librosa, can be used to remove background noise or irrelevant audio segments in the audio data. After completing the data cleaning, the integrity of the data can be checked. For missing or abnormal data points, interpolation or elimination can be performed. For example, if the data of the temperature sensor obtained in a certain period of time is abnormal, the data points before and after the period can be used for linear interpolation to fill the missing values. It should be noted that the processing method of the multimodal detection data is not limited here.

[0073] Then, the user's characteristic information is determined based on the multimodal detection data. For example, for video data, a deep learning model, such as a pre-trained convolutional neural network (CNN) model, can be used to extract the user's facial expressions and body movement features, and output the encoding vectors of the facial expressions and body movements; for audio data, a speech recognition algorithm can be used to extract the speech content, and a sentiment analysis algorithm, such as a dictionary-based method or a deep learning model, can be used to extract sentiment features; for sensor data, numerical values ​​can be directly extracted as sensor numerical features. It should be noted that other methods can also be used to extract characteristic information, which is not limited here, and is subject to what can be implemented.

[0074] After determining the characteristic information of the user based on the multimodal detection data, determine the degree of association between the target state and each characteristic information. Here, the degree of association between different characteristic information and the state information of the user for the target state can be analyzed by the server. Exemplarily, correlation analysis, such as the Pearson correlation coefficient, or machine learning models, such as decision trees and random forests, can be used to determine the degree of association, and weights can be set according to the degree of association. Weight mapping can be performed using weighted average or a method based on a machine learning model, and a weighted sum is performed on multiple degrees of association to obtain a state value of each target state. Exemplarily, the characteristic information of the user is determined as a face based on the multimodal detection data. The characteristic values ​​of facial expression information, body movement information, emotion information, and sensor information are 80, 90, -110, and 50, respectively. According to the degree of correlation between each characteristic information and the user's emotional state, the weights of the four characteristic information corresponding to the emotional state are determined to be 0.3, 0.3, 0.3, and 0.1, respectively. The state value of the emotional state is calculated to be 80*0.3+90*0.3-110*0.3+50*0.1=23. Similarly, according to the degree of correlation between each characteristic information and the physical state and environmental state, the weight of each characteristic information corresponding to the physical state and environmental state is determined, and then the state value of the physical state and environmental state is calculated based on the weight and the characteristic information.

[0075] After weighted summation of multiple association degrees is performed to obtain the state value of each target state, the state information of the user for the target state is determined based on the state value of each target state, wherein the state value of each target state can reflect the state information of the target state, and the state value of the target state can be graded according to actual conditions. For example, when the state value of the emotional state is 90-100, the state information of the emotional state is excellent, when the state value of the emotional state is 90-60, the state information of the emotional state is good, when the state value of the emotional state is 60-30, the state information of the emotional state is sad, and when the state value of the emotional state is less than 30, the state information of the emotional state is painful. The calculated state value of the emotional state is 23, and the corresponding state information of the emotional state is painful. Similarly, the state values ​​of the physical state and the environmental state are divided to obtain the state information of the physical state and the environmental state. It should be noted that the above division of the state values ​​is only exemplary, and the state values ​​can be divided into different dimensions and more detailed according to actual conditions. The specific division method of the state values ​​is not limited here.

[0076] Step 103, the state information is processed by a large model to obtain decision-making interaction information.

[0077] Here, after determining the user's state information for the target state based on the multimodal detection data, the state information can be processed through a large model to obtain decision-making interaction information.

[0078] In a possible embodiment, the big model is a core general multimodal big model, and the published mature general version can be privately deployed on a private server. The server can be a high-performance PC or a humanoid mobile robot. At a certain moment, after obtaining the user's current emotional state, physical state, and environmental state based on the video data, audio data, and sensor data in the multimodal detection data, the preprocessed multimodal detection data and the corresponding state information are stored in a common file format, such as CSV, NumPy array or TensorFlow / PyTorch dataset object for loading for big model training to determine decision-making interaction information.

[0079] The decision-making interactive information can be divided into multiple types: emergency response, such as monitoring and inquiry of abnormal indicators of users' sudden physical condition, medication reminders, alarms, etc.; emotional care, such as active care when monitoring users' low mood, which can simulate the initiation of real-life video voice, etc.; dispute intervention, active intervention when monitoring users' quarrels, such as playing warm scenes of getting along with each other according to the quarreling objects, etc.; life habit assistant, intelligent device control according to user habits, such as automatically opening curtains when users walk to the window in the morning; life suggestion, reminding users to sleep in time according to the user's sleep quality data, detection reminders to give bedtime suggestions, etc.; space environment suggestion, for example, when a physically disabled user detects that it is difficult to pass through a certain place in the room, relevant suggestions are provided in combination with space layout, storage, etc., which can be provided in the form of pictures or videos, or when monitoring a certain place with poor air quality, it is recommended to plant green plants there, etc.; anti-fraud, timely reminding users when suspected fraud calls are detected. It should be noted that the above is only an exemplary display of decision-making interactive information, and there can be more types of decision-making interactive information in actual application, which are not limited here.

[0080] Furthermore, the following exemplifies a specific application scenario in which the user's status information is processed by the big model to obtain decision-making interaction information: when the user is alone and the current emotional state is sad, the big model has learned through training how to best relieve the user's sadness. If the method is to video chat with family members, but the video is not connected, the big model is trained through historical video data, and the real voices, expressions, and actions of virtual family members are used. Through smart screen terminals such as speakers, AI family images are used to simulate video calls with users, and IoT audio devices are controlled to play soothing music to relieve the user's sadness.

[0081] When the smart camera detects that the user has fallen at home, the video data is pre-processed and corresponds to the user's physical condition as poor. The big model decides that it needs to perform overall physical monitoring through the capabilities reported by each device, such as initiating heart rate detection on a smart watch, and voice inquiries about the current status through a smart speaker or its own integrated voice interaction device. Then the big model determines the actions to be taken based on the health data fusion after pre-processing of the collected data. For example, if the user faints and there is no one around, call family members or 110 to report the current situation by controlling smart call devices. If the user faints but there are people around, call them by voice and search and play the emergency treatment audio and video of the corresponding scene through the smart screen. Then, the big model analyzes the historically processed video body shape and other data of the user at the corresponding location here, and identifies whether there is a problem with the home design and placement here based on general capabilities. It shows users reasonable suggestions through voice or pictures, and suggests that family members change the layout, install carpets, etc., and continuously analyzes and optimizes through such events to achieve a perfect match between users and home environments.

[0082] When the smart camera and audio interaction terminal detect that the user is arguing with others, and the user's current emotional state is angry and physical condition is poor, the smart camera or audio interaction terminal records audio and video data. The big model recognizes the normal scene through the pre-processed audio and video data, and recognizes the relationship between the two people through preliminary training, and associates and extracts the warm scenes of the two people in the local storage. Then, the big model plays the extracted photos and videos through the audio and video interaction terminal, plays soothing and sensational music, controls the lights to warm light, etc. to construct a warm indoor scene, so as to ease the quarrel;

[0083] When the smart camera and smart watch detect that the user is tired and the environment is dry, the big model reminds the user to take a rest or adjust the ambient temperature and humidity to the most suitable point through a humidifier; the big model detects through the smart watch that the user's physical condition (heart rate and breathing) in a certain place in the room is obviously better than other places, and the voice suggests that the user can go to a designated place to rest for a while when the physical condition or mood is not good; the user is looking for something at home and can't find it, the big model recognizes the object through historical video pictures, or uses a mobile robot with a camera to take pictures in the room for identification, circle and display it on the smart screen, and give the user subsequent storage suggestions.

[0084] Step 104, obtaining user feedback information regarding the decision-making interaction information, determining an intimacy value of the feedback information, and adjusting a collection strategy for the feedback information according to the intimacy value.

[0085] Here, the state information is processed by a large model to obtain decision-making interaction information, and then the user's feedback information on the decision-making interaction information is obtained, and the intimacy value of the feedback information is determined, and the feedback information collection strategy is adjusted according to the intimacy value, where the intimacy value is used to indicate the user's satisfaction with the feedback information.

[0086] In a possible embodiment, determining the intimacy value of feedback information includes the following steps:

[0087] Determine the time correlation degree between the state information and the feedback information based on the difference between the timestamp of the state information and the timestamp of the feedback information collected in each collection cycle;

[0088] Determine the feedback result based on the feedback information collected in each collection cycle;

[0089] Based on the degree of association and feedback results, the intimacy value is determined.

[0090] Specifically, the feedback result is determined based on the feedback information collected in each collection cycle. The feedback information can be obtained by periodically asking the user through the interactive terminal. For example, Figure 2 As shown, Figure 2 A schematic diagram of a feedback information interaction page is exemplarily shown. The interaction page can be projected in the user's current environment through a projection device, and the user can make a selection by voice. Each option corresponds to a feedback information, and the feedback information corresponds to the user's feedback result, wherein the feedback result reflects the user's satisfaction with the decision-making interaction information. For example, the "very satisfied" state corresponds to a user feedback result of 100, the "relatively satisfied" state corresponds to a user feedback result of 60, the "unsatisfied" state corresponds to a user feedback result of -20, and the "very unsatisfied" state corresponds to a user feedback result of -50. The feedback result can also be obtained through the multimodal detection data of the user collected by the smart device. The modal detection data calculates the state value of the emotional state, and takes the state value of the emotional state as the feedback result. The state values ​​of the emotional state, physical state, and environmental state can also be weighted and summed to determine the feedback result. For example, the state values ​​of the current user's emotional state, physical state, and environmental state are -60, 30, and 80, respectively. The weights of the emotional state, physical state, and environmental state for the feedback results are 0.5, 0.5, and 0.1, respectively. The feedback result is -60*0.5+30*0.5+80*0.1=-7. The weight can be customized by the user or determined by big data. It should be noted that the specific method for determining the feedback result is not limited here.

[0091] While determining the feedback result based on the feedback information collected in each collection cycle, the time correlation degree between the state information and the feedback information is determined based on the difference between the timestamp of the state information and the timestamp of the feedback information collected in each collection cycle, and the intimacy value is determined based on the correlation degree and the feedback result. For example, the timestamp of the user's state information for the target state is 9:15, the timestamp of the feedback information collected in the first collection cycle is 9:16, and the timestamp of the feedback information collected in the second collection cycle is 10:15. The interval time between the feedback information collected in the first collection cycle and the state information of the user for the target state is short, so the correlation degree between the feedback information collected in the first collection cycle and the state information of the user for the target state is large, and the weight assigned to the first collection cycle is 0.95. The correlation degree between the feedback information collected in the second collection cycle and the state information of the user for the target state is small, and the weight assigned to the second collection cycle is 0.5. Based on the weight and feedback result of each collection cycle, the intimacy value Q is determined, and the calculation formula is:

[0092]

[0093] Among them, i is the period, the initial value of Q is Qc, which can be set to 0, Ai is the weight of the collection period i, and Ji is the feedback result of the collection period i.

[0094] Exemplarily, the initial value Qc of Q is 0, in the first collection cycle, A1 is 0.95, J1 is 100, in the second collection cycle, A2 is 0.5, J2 is 60, then the intimacy value Q=0.95*100+0.5*60=125 is calculated, at the end of each collection cycle, the intimacy value is determined based on the degree of association and feedback results, and the collection strategy of feedback information is adjusted according to the intimacy value. In order to avoid large fluctuations in the intimacy value Q, the collection frequency of feedback information is dynamically adjusted to enable the large model to be trained to be closer to user habits.

[0095] In one embodiment, adjusting the feedback information collection strategy according to the intimacy value includes the following steps:

[0096] If the intimacy value calculated for the first number of consecutive collection cycles increases, reducing the collection frequency of the feedback information;

[0097] If the intimacy value calculated for a second number of consecutive collection cycles decreases, the collection frequency of the feedback information is increased.

[0098] Specifically, after obtaining the user's feedback information on decision-making interaction information and determining the intimacy value of the feedback information, if the intimacy value calculated for the first number of consecutive collection cycles increases, the collection frequency of the feedback information is reduced; if the intimacy value calculated for the second number of consecutive collection cycles decreases, the collection frequency of the feedback information is increased. Exemplarily, the first number is three collection cycles, and the sampling frequency is once every 30 minutes. If the intimacy value Q of the first collection cycle is 20, the intimacy value Q of the second collection cycle is 26, and the intimacy value Q of the third collection cycle is 29, then if the intimacy value calculated for three consecutive collection cycles increases, the sampling frequency is reduced from once every 30 minutes to once every 20 minutes. Similarly, if the intimacy value calculated for the second number of consecutive collection cycles decreases, the collection frequency of the feedback information is increased. It should be noted that the first number and the second number and the reduced or increased frequency can be user-defined settings or calculated based on big data, and are not limited here.

[0099] The embodiments of the present disclosure provide an auxiliary method, system, device, medium and product based on artificial intelligence. In the embodiments of the present application, first, multimodal detection data of the user collected by the smart device is obtained; wherein the multimodal detection data includes at least one of the following modalities: audio data, video data, sensor data; then, based on the multimodal detection data, the user's state information for the target state is determined; wherein the target state includes at least: emotional state, physical state, environmental state; then, the state information is processed by a large model to obtain decision-making interaction information; finally, feedback information of the user for the decision-making interaction information is obtained, and the intimacy value of the feedback information is determined, and the collection strategy of the feedback information is adjusted according to the intimacy value; wherein the intimacy value is used to indicate the user's satisfaction with the feedback information.

[0100] From the above description, it can be seen that the status information for the user's target state is determined based on multimodal detection data. The multimodal detection data may include audio data, video data, and sensor data. The comprehensive multimodal detection data enables the user's current information and environment to be highly integrated, ensuring the comprehensiveness and accuracy of the user's status information. After obtaining the user's status information, the status information can be processed by the big model to obtain decision-making interaction information, and the intimacy value of the feedback information can be determined by obtaining the user's feedback information on the decision-making interaction information. Finally, the collection strategy of the feedback information can be adjusted according to the intimacy value. The adjustment of the collection strategy can train the big model more accurately, so that the big model can go deeper into the user's life, more accurately obtain the user's personal habits, improve the pertinence of the auxiliary method based on artificial intelligence to the user, and further enhance the user experience.

[0101] In one embodiment, Figure 3As shown, in step 101, the intimacy value is determined based on the association degree and the feedback result, and the following steps are also included:

[0102] Step 301, determining a first calculation coefficient of a feedback result.

[0103] Here, after determining the intimacy value Q based on the weight and feedback result of each acquisition cycle, the first calculation coefficient of the feedback result can be determined.

[0104] In one embodiment, determining a first calculation coefficient of a feedback result includes the following steps:

[0105] Determine the average of the intimacy values ​​calculated for the third consecutive collection period in the past moment;

[0106] If the average value is greater than the preset threshold, the second calculation coefficient of the feedback result is reduced by the first preset value to obtain the first calculation coefficient;

[0107] If the average value is less than the preset threshold, the second calculation coefficient of the feedback result is increased by a second preset value to obtain the first calculation coefficient.

[0108] Specifically, after determining the intimacy value Q based on the weight and feedback result of each collection cycle, the average value of the intimacy values ​​calculated for the third consecutive collection cycles in the past is determined. Exemplarily, the third number is four, and the intimacy values ​​of four consecutive collection cycles in the past are determined to be 50, 65, 80, and 45, respectively. The average value of the intimacy values ​​of the four consecutive collection cycles in the past is 60, and the preset threshold value of the average value of the intimacy value is 50. If the average value of the intimacy values ​​of the four consecutive collection cycles in the past is greater than the preset threshold, the second calculation coefficient of the feedback result is reduced by the first preset value to obtain the first calculation coefficient, wherein the second calculation coefficient is the previous calculation coefficient of the feedback result used to calculate intimacy. If the second calculation coefficient is 1 and the first preset value is 0.2, the first calculation coefficient is determined to be 0.8. Similarly, if the average value is less than the preset threshold, the second calculation coefficient of the feedback result is increased by the second preset value to obtain the first calculation coefficient. It should be noted that the situation where the average value is less than the preset threshold is not repeated here.

[0109] Step 302: Perform product sum operation on the feedback results and the correlation degree calculated in each collection cycle according to the first calculation coefficient to obtain an intimacy value.

[0110] Here, after determining the first calculation coefficient of the feedback result, the feedback result and the correlation degree calculated in each collection period are multiplied and summed according to the first calculation coefficient to obtain the intimacy value.

[0111] In a possible embodiment, after determining the first calculation coefficient of the feedback result, the feedback result and the correlation degree calculated in each collection period are multiplied and summed according to the first calculation coefficient to obtain the intimacy value Q, and the calculation formula is:

[0112]

[0113] Among them, Ai is the weight of collection cycle i, Ji is the feedback result of collection cycle i, and Bi is the first calculation coefficient of collection cycle i.

[0114] For example, if A1 is 0.95, J1 is 100, and Bi is 0.9 in the first acquisition cycle, and A2 is 0.5, J2 is 60, and Bi is 0.5 in the second acquisition cycle, then the intimacy value Q is calculated to be 0.95*100*0.9+0.5*60*0.5=100.5.

[0115] In this embodiment, first, the first calculation coefficient of the feedback result is determined, and then the feedback result and the degree of association calculated in each collection period are multiplied and summed according to the first calculation coefficient to obtain the intimacy value. It can be seen from the above description that the intimacy value is obtained by multiplying the feedback result and the degree of association calculated in each collection period according to the first calculation coefficient. When the average value of the intimacy value is greater than the preset threshold, the second calculation coefficient is appropriately reduced to the first calculation coefficient, which can reduce the rate of change of the intimacy value. On the contrary, when the average value of the intimacy value is less than the preset threshold, the second calculation coefficient is appropriately increased to the first calculation coefficient, which can speed up the rate of change of the intimacy value, reduce or increase the rate of change of the intimacy value, and make the change of the intimacy value more stable. The server can more accurately adjust the collection frequency of feedback information according to the increase or decrease of the intimacy value, so that the large model can learn more accurately, improve the pertinence of the auxiliary method based on artificial intelligence to users, and make the trained large model closer to the user's personal habits, and improve the user's experience.

[0116] In one embodiment, Figure 4 As shown, an auxiliary system based on artificial intelligence is provided, including an artificial intelligence decision module, an interactive control module, and an equipment sensing module, wherein:

[0117] Device sensor module 401, used to obtain multimodal detection data;

[0118] The data processing module 402 is used to obtain multimodal detection data of the user collected by the smart device; wherein the multimodal detection data includes at least one of the following modes: audio data, video data, and sensor data; based on the multimodal detection data, determine the state information of the user for the target state; wherein the target state includes at least: emotional state, physical state, and environmental state; and obtain feedback information, determine the intimacy value of the feedback information, and adjust the collection strategy of the feedback information according to the intimacy value; wherein the intimacy value is used to indicate the user's satisfaction with the feedback information;

[0119] The artificial intelligence decision module 403 is used to process the state information through a large model to obtain decision-making interaction information;

[0120] The interaction control module 404 is used to determine the user's feedback information for the decision-making interaction information based on the decision-making interaction information, and feed the feedback information back to the data processing module.

[0121] Specifically, Figure 5 As shown, the device sensor module 401 is used to obtain multimodal detection data, wherein the device sensor module 401 includes a variety of smart devices, and the various smart devices can be smart devices under the whole house intelligence or wearable terminals, including mobile phones, tablets, watches, bracelets, thermometers and hygrometers, sports watches, smart doorbells, smart cameras, XR devices, humanoid robots, sweeping robots, smart home appliances, etc.

[0122] The data processing module 402 is used to obtain the multimodal detection data of the user collected by the smart device, as well as to obtain feedback information, determine the intimacy value of the feedback information, and adjust the collection strategy of the feedback information according to the intimacy value. The data processing module 402 includes a data collection unit, a local storage unit, and a data preprocessing unit. The data collection unit is used to obtain the multimodal detection data of the user collected by the smart device, as well as to obtain the feedback information sent by the interactive control module 404, and send the feedback information to the data preprocessing unit to determine the intimacy value of the feedback information, and adjust the collection strategy of the feedback information according to the intimacy value. The local storage unit is used to store the multimodal detection data of the user, and the data preprocessing unit is used to perform data cleaning and data feature extraction on the multimodal detection data to determine the user's status information for the target state.

[0123] The artificial intelligence decision module 403 is used to process the state information through the big model to obtain decision-making interactive information. Among them, the artificial intelligence decision module 403 adopts the core general multimodal big model, and privatizes the published mature general version and deploys it on a private server. The server can be a high-performance PC or a humanoid mobile robot, which is not limited here. After the general multimodal big model is deployed, it is only connected to the home intranet. Basic information of personnel, personal living habits, physical condition and experience can be input for initialization. Information input can be achieved through template input, voice inquiry, etc. The data generated by the model inquiry can be stored in the local storage unit and encrypted and authorized. It should be noted that the big model can be upgraded and replaced. After upgrading or replacing the big model, the retained data needs to be retrained for the general model. For example, the user replaces the general big model A with the general big model B, and the general ability is upgraded. After the data with personal labels is imported for training, an auxiliary system based on artificial intelligence with the same ability can be trained.

[0124] The interactive control module 404 is used to determine the user's feedback information for the decision-making interactive information based on the decision-making interactive information, and feed back the feedback information to the data processing module, wherein the decision-making interactive information includes emergency handling, such as monitoring and inquiry of abnormal indicators of the user's sudden physical condition, medication reminders, alarms, etc.; emotional care, such as active care when the user is monitored to be depressed, which can simulate the initiation of real-life video voice, etc.; dispute intervention, when the user is monitored to quarrel, actively intervene, such as playing warm scenes of getting along with each other according to the quarreling objects, etc.; life habit assistant, smart device control according to user habits, such as automatically opening curtains when the user walks to the window in the morning; life suggestion, reminding the user to sleep in time according to the user's sleep quality data, detection reminders to give bedtime suggestions, etc.; space environment suggestion, for example, when a physically disabled user detects that it is difficult to pass through a certain place in the room, relevant suggestions are provided in combination with space layout and storage, which can be provided in the form of pictures or videos, or when the air quality is poor in a certain place, it is recommended to plant green plants there, etc.; anti-fraud, when a suspected fraud call is detected, the user is reminded in time.

[0125] In this embodiment, the artificial intelligence decision-making module adopts a core universal multimodal big model, and privatizes the published mature general version on a private server, thereby improving the security of the artificial intelligence-based auxiliary system. After deployment, the universal multimodal big model is only connected to the home intranet, which protects the user's personal privacy. In addition, the core universal multimodal big model used by the artificial intelligence decision-making module can be upgraded and replaced, and the data of the big model can be reused, independent of the specific big model, thereby improving the applicability and flexibility of the artificial intelligence-based auxiliary system.

[0126] Among them, the implementation methods and beneficial effects of the steps executed by the data processing module, the interactive control module and the device sensing module can be found in the relevant embodiments applied to the artificial intelligence-based auxiliary method above, and will not be repeated here.

[0127] In one embodiment, Figure 6 As shown, the artificial intelligence-based auxiliary system also includes: an authorization and authentication module and a communication module, wherein:

[0128] Communication module 601, used for communication connection between various modules in the auxiliary system based on artificial intelligence;

[0129] The authorization and authentication module 602 is used to authenticate the identity of the user in response to the user's access operation to the artificial intelligence decision-making module.

[0130] Specifically, Figure 6 As shown, the communication module 601 is used for communication connection between various modules in the auxiliary system based on artificial intelligence. For example, the communication module 601 can be a general local area network such as wifi, ZigBee, etc. The gateway corresponding to the communication module 601 also installs a software development kit (SDK) for intelligentization. The authorization and authentication module 602 is used to authenticate the user's identity in response to the user's access operation to the artificial intelligence decision module 403. Figure 7 The module interaction diagram is shown as an example. Figure 7 As shown, the smart devices under the device sensor module 401 communicate with the artificial intelligence decision-making module 403 through the smart gateway, among which the core function of the authorization authentication module 602 is to make permission correspondence between the general multimodal large model and all smart terminals participating in the training data collection, device discovery, device permission control, personnel permission control, user access control, etc. Each smart device and gateway can use the installed SDK for communication and interaction.

[0131] For permission correspondence and device discovery, personnel who log in to the authorization authentication module and activate the AI ​​big model need to keep the information consistent before they can operate. The logged-in personnel can bind the discovered smart devices to generate device identification IDs, and the binding relationship can be many-to-many. After the smart device is bound, the device sensing module 401 will report the device capabilities, which can include: what information the device can detect, the corresponding call interface; what actions the device can complete, the corresponding call interface and what impact it has on the environment; what applications the device contains, and the description of the effect achieved by the application; if it is a wearable device, who the device corresponds to; what data the device can process, what effect it can achieve, and the corresponding call interface, so as to accept the generative effect of AI; what interactions the device can complete with the user, and the corresponding call interface. The authorization authentication module 602 can perform access control list (ACL) control on the reported device capabilities, which includes user access control. This control is also effective for the big model corresponding to the activated personnel, that is, after granting permissions to a certain logged-in user, the big model activated by the corresponding user can obtain the data collected by the corresponding authorized smart device for training, and can also control the smart device. The SDK involved can realize functions such as LAN registration, login authentication, capability collection and reporting, collected data reporting to designated storage, and device control implemented by general large model control.

[0132] In this embodiment, smart devices under the whole-house smart or wearable terminals can communicate with the artificial intelligence decision-making module through the smart gateway by installing SDK or app, and perform capability reporting, mutual authentication, and interactive authority control through the authorization authentication module, thereby ensuring the security and stability of the artificial intelligence auxiliary system and further enhancing the user experience.

[0133] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0134] Figure 8 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the electronic device 800 includes at least one processor 801 and a memory 802 coupled to the processor 801 , and the processor 801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0135] The processor 801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 801 or by instructions in the form of software. The processor 801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 801 reads the information in the memory 802 and completes the steps of the method in combination with its hardware.

[0136] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Fig. 9 The computer system 900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can execute various functions, including the functions described above. Fig. 9 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0137] Computer system 900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0138] like Fig. 9As shown, the computer system 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the computer system 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0139] A plurality of components in the computer system 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 may be any type of device capable of inputting information to the computer system 900, and the input unit 906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 may include, but is not limited to, a disk, an optical disk. The communication unit 909 allows the computer system 900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0140] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 902 and / or the communication unit 909. In some embodiments, the computing unit 901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0141] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0142] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0144] Fig.10 A computer program product is provided for an exemplary embodiment of the present disclosure. The computer program product 1000 includes a computer program 1001. When the computer program 1001 is executed by a processor, the method disclosed in the embodiment of the present disclosure is implemented.

[0145] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0146] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0148] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0149] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0150] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An artificial intelligence-based auxiliary method, characterized in that: The method comprises: Acquire multimodal detection data of the user collected by the smart device; wherein the multimodal detection data includes at least one of the following modalities: audio data, video data, and sensor data; Based on the multimodal detection data, determine the state information of the user for a target state; wherein the target state includes at least: an emotional state, a physical state, and an environmental state; Processing the state information through a large model to obtain decision-making interaction information; Obtain feedback information from the user regarding the decision-making interaction information, determine an intimacy value of the feedback information, and adjust a collection strategy for the feedback information according to the intimacy value; wherein the intimacy value is used to indicate the user's satisfaction with the feedback information.

2. The method according to claim 1, characterized in that The determining, based on the multimodal detection data, the state information of the user with respect to the target state includes: Determine the characteristic information of the user based on the multimodal detection data; the characteristic information includes at least one of the following: facial expression characteristics, body movement characteristics, emotion characteristics, sensor value characteristics; Determining the degree of association between the target state and each of the feature information; Performing weighted summation on the multiple association degrees to obtain a state value of each target state; Based on the state value of each of the target states, state information of the user with respect to the target state is determined.

3. The method according to claim 1, characterized in that The determining the intimacy value of the feedback information includes: Determining the degree of time correlation between the state information and the feedback information based on a difference between a timestamp of the state information and a timestamp of the feedback information collected in each collection period; Determining a feedback result based on the feedback information collected in each collection cycle; wherein the feedback result reflects the user's satisfaction with the decision-making interaction information; The intimacy value is determined based on the degree of association and the feedback result.

4. The method according to claim 1 or 3, characterized in that: The step of adjusting the feedback information collection strategy according to the intimacy value includes: If the intimacy value calculated for the first number of consecutive collection cycles increases, reducing the collection frequency of the feedback information; If the intimacy value calculated for a second number of consecutive collection cycles decreases, the collection frequency of the feedback information is increased.

5. The method according to claim 3, characterized in that: The determining the intimacy value based on the association degree and the feedback result includes: Determining a first calculation coefficient of the feedback result; The intimacy value is obtained by performing a product sum operation on the feedback result calculated in each collection cycle and the correlation degree according to the first calculation coefficient.

6. The method according to claim 5, characterized in that The determining of the first calculation coefficient of the feedback result includes: Determine the average of the intimacy values ​​calculated for the third consecutive collection period in the past moment; If the average value is greater than a preset threshold, reducing the second calculation coefficient of the feedback result by a first preset value to obtain the first calculation coefficient; If the average value is less than a preset threshold, the second calculation coefficient of the feedback result is increased by a second preset value to obtain the first calculation coefficient; wherein the second calculation coefficient is the previous calculation coefficient of the feedback result used to calculate intimacy.

7. An artificial intelligence-based assistance system, characterized in that: It includes data processing module, artificial intelligence decision-making module, interactive control module and equipment sensing module, among which: The device sensing module is used to obtain multimodal detection data; The data processing module is used to obtain multimodal detection data of the user collected by the smart device; wherein the multimodal detection data includes at least one of the following modes: audio data, video data, and sensor data; based on the multimodal detection data, determine the state information of the user for the target state; wherein the target state includes at least: emotional state, physical state, and environmental state; and obtain feedback information, determine the intimacy value of the feedback information, and adjust the collection strategy of the feedback information according to the intimacy value; wherein the intimacy value is used to indicate the user's satisfaction with the feedback information; The artificial intelligence decision module is used to process the state information through a large model to obtain decision-making interaction information; The interaction control module is used to determine the feedback information of the user for the decision-making interaction information based on the decision-making interaction information, and feed back the feedback information to the data processing module.

8. The system according to claim 7, characterized in that The system also includes: an authorization and authentication module and a communication module, wherein: The communication module is used for communication connection between various modules in the artificial intelligence-based auxiliary system; The authorization and authentication module is used to authenticate the identity of the user in response to the user's access operation to the artificial intelligence decision-making module.

9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 6.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images