Intelligent interaction method and system based on AI human-sensing large model, terminal and medium
Through the AI human-sensing model, collect and analyze the data of multiple AI perception modules of smart terminals, build scenarios and generate interactive instructions, solving the problem that independent operating modules cannot interact in a comprehensive manner, and improving the active service experience of smart terminals.
Patent Information
- Application Number
- CN202411149662.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-08
AI Technical Summary
Currently, the AI perception modules equipped with smart terminals run independently, and cannot comprehensively utilize multiple submodules to provide more intelligent active service interaction, resulting in poor experience of active service interaction.
The AI human-sensing model is used to collect and analyze the behavioral data of multiple AI perception modules, build AI perception scenarios, and generate corresponding intelligent interaction instructions to realize comprehensive interaction between smart terminals and users.
It improves the active service interaction experience of smart terminals, and provides more intelligent user interaction by comprehensively analyzing the data of each submodule, enhancing the user experience.
Smart Images

Figure CN118915916A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of terminal interaction technology. More specifically, the present disclosure relates to an intelligent interaction method, system, terminal and medium based on an AI human perception big model. Background Art
[0002] Early intelligent terminals (such as intelligent TVs) all require users to issue an interactive operation instruction before the intelligent terminal will respond to the interactive operation instruction. This method of interaction between intelligent terminals and users is a user-triggered interaction method.
[0003] In recent years, smart terminals have incorporated AI perception modules, which include multiple sub-modules (such as millimeter wave modules, camera modules, far-field voice modules, and light sensing modules), so that smart terminals can also interact with users in a certain proactive service manner. For example, a millimeter wave module or a camera module is used to determine the user's viewing distance, and the user is reminded of the viewing distance or a better sound experience is provided. However, each sub-module in the AI perception module currently carried by smart terminals operates independently and does not interfere with each other, so that each sub-module can only provide independent services to users, and it is impossible to combine multiple sub-modules at the same time to comprehensively provide users with more intelligent proactive service interactions, resulting in a poor proactive service interaction experience for current smart terminals. Summary of the invention
[0004] The purpose of this disclosure is to provide an intelligent interaction method, system, terminal and medium based on an AI human perception model, which can improve the proactive service interaction experience of intelligent terminals. This disclosure is mainly achieved through the following technical solutions:
[0005] In a first aspect of the present disclosure, an intelligent interaction method based on an AI human perception model is provided, comprising:
[0006] Use multiple submodules of the AI perception module in the smart terminal to collect behavioral data;
[0007] The first AI human perception model of the smart terminal is used to analyze and process the behavior data to construct a first AI perception scene;
[0008] Based on the first AI perception scenario, a first intelligent interaction instruction corresponding to the first AI perception scenario is generated and executed to respond to the behavior data to achieve interaction between the smart terminal and the user.
[0009] According to an embodiment of the present disclosure, the step of using multiple submodules of the AI perception module in the smart terminal to collect behavior data includes:
[0010] Perform one or more of the following steps:
[0011] The millimeter wave module of the AI perception module is used to collect the first installation information, the first environment space information and the user usage information of the behavior data;
[0012] The camera module of the AI perception module is used to collect user face images, usage scene images and user behavior images of the behavior data;
[0013] The far-field voice module of the AI perception module is used to collect user voice information and environmental noise information of the behavior data;
[0014] The light sensing module of the AI perception module is used to collect the ambient light information of the behavior data.
[0015] According to an embodiment of the present disclosure, the step of using the first AI human perception model of the smart terminal to analyze and process the behavior data to construct a first AI perception scene includes:
[0016] Classify and process the behavior data to obtain at least one piece of data to be processed;
[0017] Inferring all the data to be processed by using the first AI human perception model to obtain an inference element corresponding to each data to be processed;
[0018] The first AI perception scenario is constructed based on all the reasoning elements.
[0019] According to an embodiment of the present disclosure, before the step of using multiple submodules of the AI perception module in the smart terminal to collect behavior data, the intelligent interaction method based on the AI human perception model further includes:
[0020] Detecting the installation environment information and the operating environment information of the smart terminal;
[0021] The operating state of the smart terminal is adjusted according to the installation environment information and the operating environment information.
[0022] According to an embodiment of the present disclosure, the step of detecting the installation environment information and the operating environment information of the smart terminal includes:
[0023] Perform one or more of the following steps:
[0024] The millimeter wave module of the AI perception module is used to detect the second installation information and the second environment space information of the installation environment information;
[0025] Using the light sensing module of the AI perception module to detect the operating environment light information of the operating environment information;
[0026] The millimeter wave module of the AI perception module is used to detect the viewing position information of the operating environment information.
[0027] According to an embodiment of the present disclosure, the step of adjusting the operating state of the smart terminal according to the installation environment information and the operating environment information includes:
[0028] adjusting the volume of the smart terminal based on the second installation information and the second environment space information;
[0029] Adjusting the backlight brightness of the smart terminal based on the operating environment light information;
[0030] The display brightness and sound playing angle of the smart terminal are adjusted based on the viewing position information.
[0031] According to an embodiment of the present disclosure, before the step of generating and executing a first intelligent interaction instruction corresponding to the first AI perception scenario based on the first AI perception scenario, the intelligent interaction method based on the AI human perception big model further includes:
[0032] Detecting whether the first AI perception scene matches the smart terminal;
[0033] In a case where the first AI perception scenario matches the smart terminal, generating and executing a first smart interaction instruction corresponding to the first AI perception scenario based on the first AI perception scenario;
[0034] When the first AI perception scenario does not match the smart terminal, the multimodal reasoning strategy of the first AI human perception big model is adjusted to obtain a second AI human perception big model; the second AI human perception big model is used to analyze and process the behavior data to construct a second AI perception scenario; based on the second AI perception scenario, a second intelligent interaction instruction corresponding to the second AI perception scenario is generated and executed.
[0035] The second aspect of the present disclosure provides an intelligent interaction system based on an AI human perception model, comprising:
[0036] A behavior data collection module, used to collect behavior data using multiple submodules of the AI perception module in the smart terminal;
[0037] A first AI perception scene output module, used to analyze and process the behavior data using the first AI human perception model of the smart terminal to construct a first AI perception scene;
[0038] The first intelligent interaction instruction execution module is used to generate and execute a first intelligent interaction instruction corresponding to the first AI perception scenario based on the first AI perception scenario, so as to respond to the behavior data and realize the interaction between the smart terminal and the user.
[0039] According to the third aspect of the present disclosure, a smart terminal is provided, which includes a memory, a processor and an AI perception module, wherein the AI perception module includes a millimeter wave module, a camera module, a far-field voice module and a light sensing module, the memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to execute the steps of the intelligent interaction method based on the AI human perception model provided in the first aspect of the present disclosure, and the millimeter wave module, the camera module, the far-field voice module and the light sensing module are all connected to the processor.
[0040] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute the steps of the intelligent interaction method based on the AI human sense model provided in the first aspect of the present disclosure.
[0041] The beneficial effects of the present disclosure include at least:
[0042] The present disclosure designs a first AI human perception model, and uses the first AI human perception model to analyze and process the behavioral data collected by all submodules of the AI perception module of the smart terminal, and construct a first AI perception scenario. Therefore, the present disclosure can achieve the purpose of comprehensive analysis of the behavioral data collected by each submodule in the AI perception module; then, based on the first AI perception scenario, the present disclosure generates and executes the first intelligent interaction instruction corresponding to the first AI perception scenario to respond to the behavioral data and realize the interaction between the smart terminal and the user. Based on the first AI perception scenario, the present disclosure can provide users with more intelligent proactive service interaction. Therefore, compared with the prior art, the present disclosure can improve the proactive service interaction experience of smart terminals. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the present disclosure or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.
[0044] Figure 1 A flowchart of the intelligent interaction method based on the AI human sense big model disclosed in some embodiments;
[0045] Figure 2 A flowchart of the intelligent interaction method based on the AI human sense model disclosed in some embodiments;
[0046] Figure 3 A schematic diagram of a method of an intelligent interaction method based on an AI human perception model in some embodiments of the present disclosure;
[0047] Figure 4 Schematic diagram of the method of the intelligent interaction method based on the AI human perception model in other embodiments of the present disclosure;
[0048] Figure 5 It is a structural block diagram of the intelligent interaction system based on the AI human sense model disclosed in some embodiments;
[0049] Figure 6 The figure is a functional block diagram of the intelligent terminal of the present invention in some embodiments. DETAILED DESCRIPTION
[0050] In order to make the above-mentioned purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present disclosure, so the present disclosure is not limited by the specific embodiments disclosed below.
[0051] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0052] The terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in the present disclosure as "exemplary" or "for example" should not be interpreted as being more preferred or advantageous over other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete way.
[0053] The terms "comprises," "includes," or any other variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements explicitly listed but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.
[0054] The AI recorded in this article can be understood as artificial intelligence (Artificial Intelligence).
[0055] Unless otherwise defined, all technical and scientific terms used in the specification of the present disclosure have the same meaning as those commonly understood by those skilled in the art of the present disclosure. The terms used in the specification of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The term "and / or" used in the specification of the present disclosure includes any and all combinations of one or more related listed items.
[0056] The specific embodiments of the present disclosure are further described below in conjunction with the accompanying drawings.
[0057] <Intelligent interaction method based on AI human perception model>
[0058] like Figure 1 and Figure 2 As shown, Figure 1 A flowchart of an intelligent interaction method based on an AI human perception model provided in the first aspect of the present disclosure, Figure 2 The figure is a flowchart of the intelligent interaction method based on the AI human perception model. Figure 1 and Figure 2 , the intelligent interaction method based on the AI human perception model includes:
[0059] S1. Use multiple sub-modules of the AI perception module in the smart terminal to collect behavioral data.
[0060] In the present disclosure, the smart terminal includes but is not limited to smart TVs, wearable devices, smart phones and notebooks.
[0061] The use of the AI perception module can improve the accuracy of AI perception services.
[0062] The multiple submodules include at least a millimeter wave module, a camera module, a far-field voice module, and a light sensing module. In other embodiments, the multiple submodules also include a temperature sensor, a humidity sensor, and the like.
[0063] It should be understood that the AI perception module has different AI human perception capabilities depending on the hardware (i.e., submodule) that supports AI perception. Different hardware determines different AI human perception data collection, behavior analysis, and human interaction reasoning, so that the final experience accuracy of AI human perception interaction changes with the change of hardware.
[0064] The behavior data includes at least first installation information, first environment space information, user usage information, user face image, usage scene image, user behavior image, user voice information, environment noise information and environment light information.
[0065] The behavior data is stored locally in the smart terminal or in a cloud server. In other implementations, it can also be stored in an AI perception big data platform, whose database can be locally or in a cloud server.
[0066] S2. Use the first AI human perception model of the smart terminal to analyze and process the behavior data to construct a first AI perception scenario.
[0067] The first AI human perception model includes a human perception environment analysis model, a human perception scene analysis model, a human perception interaction analysis model, and a human perception service model. In other embodiments, the first AI human perception model may also include other models, which are determined by those skilled in the art according to actual needs.
[0068] S3. Based on the first AI perception scenario, generate and execute a first intelligent interaction instruction corresponding to the first AI perception scenario to respond to the behavior data and realize interaction between the smart terminal and the user.
[0069] The first intelligent interaction instruction may contain only one control instruction or multiple control instructions at the same time. The first intelligent interaction instruction may be used to adjust the parameters of the hardware in the intelligent terminal, adjust the parameters of the system in the intelligent terminal, adjust the response parameters of the application in the intelligent terminal, adjust the response parameters of the service in the intelligent terminal, and adjust the response instructions of the electronic device wirelessly connected to the intelligent terminal.
[0070] Exemplarily, when the first AI perception scenario is "the user gets up to answer the phone with the lights off", the first intelligent interaction instruction is to lower the volume of the smart terminal and / or pause the playback screen of the smart terminal, turn on the indoor lights, and increase the display brightness and ambient light brightness of the smart terminal; when the first AI perception scenario is "multiple users chatting scenario", the first intelligent interaction instruction is to lower the volume of the smart terminal or turn off the smart terminal. In other embodiments, the first intelligent interaction instruction can also be to play a preset sound, display a preset image or video, open a preset application, and start a hardware device wirelessly connected to the smart terminal. The wireless connection described in this article can be understood as an IOT Internet of Things connection.
[0071] It should be understood that when multiple control instructions in the first intelligent interaction instruction belong to the "or" situation, the control instruction that is used most times by the user is selected according to the historical records of the intelligent terminal.
[0072] For each time the first AI perception scene and the first intelligent interaction instruction used by the smart terminal, the smart terminal stores them locally on the smart terminal or in a cloud server.
[0073] Through the above technical solution, the present disclosure adopts the first AI human perception model to analyze and process the behavioral data collected by all submodules of the AI perception module of the smart terminal, and construct the first AI perception scene. Therefore, the present disclosure can achieve the purpose of comprehensive analysis of the behavioral data collected by each submodule in the AI perception module; then, based on the first AI perception scene, the present disclosure generates and executes the first intelligent interaction instruction corresponding to the first AI perception scene to respond to the behavioral data and realize the interaction between the smart terminal and the user. Based on the first AI perception scene, the present disclosure can provide users with more intelligent proactive service interaction. Therefore, compared with the prior art, the present disclosure can improve the proactive service interaction experience of smart terminals.
[0074] The S1 step is described in detail below.
[0075] The S1 step includes:
[0076] Execute one or more of the following steps S11, S12, S13 and S14:
[0077] S11. Use the millimeter wave module of the AI perception module to collect the first installation information, the first environmental space information and the user usage information of the behavior data.
[0078] Specifically, the first installation information is the installation method of the smart terminal. The installation method is wall-mounted installation or base-mounted installation. In other embodiments, the installation method is not limited to the above two methods, and can be set by those skilled in the art according to actual needs. The collection of the first installation information can be used to adjust the output angle of the sound of the smart terminal.
[0079] The first environmental space information includes the size of the environmental space in which the smart terminal operates and information about items around the smart terminal. The size of the environmental space can be understood as the size of the indoor space; the item information includes but is not limited to fans, tables, chairs, refrigerators, cabinets, and sofas.
[0080] The user usage information includes whether there is someone near the smart terminal, the number of people near the smart terminal, the specific positions of the people near the smart terminal relative to the smart terminal, posture information of the user using the smart terminal, and user usage scenario information, wherein the posture information includes but is not limited to sitting upright, lying down, lightly sleeping, and already asleep; the usage scenario information includes but is not limited to personal usage scenarios, multi-person usage scenarios, chatting usage scenarios, elderly usage scenarios, young children usage scenarios, user usage scenarios with enhanced sound requirements, and user usage scenarios with special requirements for sound recognition bands.
[0081] S12. Use the camera module of the AI perception module to collect user face images, usage scene images and user behavior images of the behavior data.
[0082] Specifically, the user behavior images include images of users chatting, images of users making phone calls, images of users working, and images of users in meetings. The user behavior images can be used for gesture, posture, lip movement, and other visual recognition.
[0083] S13. Use the far-field voice module of the AI perception module to collect the user voice information and environmental noise information of the behavior data. In other embodiments, the far-field voice module is also used to process the user voice information and environmental noise information. Exemplarily, the user voice information is analyzed to obtain the user's voiceprint information. The voiceprint information can be used to filter out the first AI perception scene corresponding to the current user from the historical records of the local or cloud server to establish a more accurate AI human perception model.
[0084] The user voice information includes the user's spoken voice and the ringtone of the user's mobile phone. In other implementations, the specific content of the user voice information can be determined by those skilled in the art according to actual needs.
[0085] The environmental noise information includes the volume of the noise, the direction of the noise propagation, and whether the noise affects the viewing experience of the current smart terminal. In other implementations, the specific content of the environmental noise information can be determined by those skilled in the art according to actual needs.
[0086] S14. Use the light sensing module of the AI perception module to collect ambient light information of the behavior data.
[0087] It should also be understood that the same behavior data can be collected by multiple submodules. Exemplarily, if you want to identify which user is using the smart terminal, you can use the camera module and the far-field voice module to collect data together. Specifically, the camera module collects the user's face image for identification, and the far-field voice module collects the user's voice information and identifies it through the user's voiceprint; if you want to identify the user's state of using the smart terminal, you can use the camera module and the millimeter wave module to collect data together. Specifically, the camera module collects the user's behavior image for identification, and the millimeter wave module collects the user's distance, orientation and posture information from the smart terminal for identification; if you want to identify whether there is noise when the smart terminal is running, you can use the far-field voice module, the millimeter wave module and the camera module to collect data together. Specifically, the far-field voice module collects the environmental noise information for identification, the millimeter wave module collects the equipment that may produce noise in the use (or viewing) environment for identification, and the camera module collects the use scene image and locates the noise equipment for identification.
[0088] The S2 step is described in detail below.
[0089] The S2 step includes:
[0090] S21. Classify and process the behavior data to obtain at least one piece of data to be processed.
[0091] In the present disclosure, the at least one data to be processed includes first data to be processed, second data to be processed, third data to be processed and fourth data to be processed.
[0092] The present disclosure classifies the first environmental space information, "whether there is anyone near the smart terminal" in the user usage information, "the number of people near the smart terminal" in the user usage information, "the specific position of the people near the smart terminal at the smart terminal" in the user usage information, the environmental noise information and the environmental light information as the first data to be processed;
[0093] classifying the “user usage scenario information” and the usage scenario image in the user usage information as the second data to be processed;
[0094] Classifying the user voice information as the third data to be processed;
[0095] The “posture information of the user using the smart terminal” in the user usage information, the user face image and the user behavior image are classified as the fourth data to be processed.
[0096] In other implementations, the classification method is not limited to the above examples, and can be determined by those skilled in the art according to actual needs. The number of the at least one data to be processed is also not limited to the aforementioned four, and the specific number and content of the at least one data to be processed can be determined by those skilled in the art according to actual needs.
[0097] S22. Infer all the data to be processed through the first AI human perception model to obtain an inference element corresponding to each data to be processed.
[0098] Furthermore, the step S22 includes:
[0099] S221: Infer the first to-be-processed data through the human-sensing environment analysis model of the first AI human-sensing large model to obtain a first inference element. The human-sensing environment analysis model is used to analyze behavioral data related to the environment in which the user is located to provide a more suitable user experience.
[0100] S222: Infer the second data to be processed by the human-sensing scene analysis model of the first AI human-sensing model to obtain a second inference element. The human-sensing scene analysis model is used to analyze behavioral data related to the user's usage scenario to determine the most appropriate interaction method.
[0101] S223: Inferring the third data to be processed by the human-sensing interaction analysis model of the first AI human-sensing large model to obtain a third inference element. The human-sensing interaction analysis model is used to analyze behavioral data related to the interaction between the user and the smart terminal to optimize the interaction experience.
[0102] S224: Infer the fourth data to be processed by the human perception service model of the first AI human perception model to obtain a fourth inference element. The human perception service model is used to analyze behavior data related to the user's behavior and preferences to provide personalized services.
[0103] S23. Construct the first AI perception scenario based on all the reasoning elements.
[0104] In the present disclosure, the first AI perception scene is constructed based on the first reasoning element, the second reasoning element, the third reasoning element and the fourth reasoning element.
[0105] Based on the above steps S21-S23, it can be implemented according to the following example:
[0106] Example 1: When the light sensing module collects the ambient light information that the light is turned on, the camera module collects the user behavior image that the user stands up, and the far-field voice module collects the user voice information that the user's mobile phone ringtone, the first data to be processed is "the light is turned on", the fourth data to be processed is "the image of the user standing up", and the third data to be processed is "the user's mobile phone ringtone". Since the second data to be processed has no matching information, the second data to be processed is empty at this time. Then, the human-sensing environment analysis model infers the first data to be processed, and the first inference element obtained is to increase the display brightness and ambient light brightness of the smart terminal; the human-sensing interaction analysis model infers the third data to be processed, and the third inference element obtained is to reduce the volume of the smart terminal; the human-sensing service model infers the fourth data to be processed, and the fourth inference element obtained is to reduce the volume of the smart terminal, turn off the smart terminal, turn on the indoor lights, or turn off the indoor lights; since the second data to be processed is empty, the second inference element is empty. Next, based on the first reasoning element, the second reasoning element, the third reasoning element and the fourth reasoning element, the first AI perception scenario is constructed as "a scenario in which a user stands up to answer a phone call with the lights turned off."
[0107] Example 2: When the millimeter wave module collects the user usage information that the number of people near the smart terminal is 3, the camera module collects the user behavior image as an image of multiple users in a conversation state, or the camera module collects the user behavior image as an image of multiple users' lip movements, and the far-field voice module collects the user voice information as sound information of multiple people talking, the first data to be processed is "the number of people near the smart terminal is 3 people", the second data to be processed is "chatting usage scenario", the third data to be processed is "sound information of multiple people talking", and the fourth data to be processed is "image of multiple users in a conversation state" or "image of multiple users' lip movements". Next, the human-sensing environment analysis model infers the first data to be processed, and the first inference element obtained is to lower the volume of the smart terminal or turn off the smart terminal; the human-sensing scene analysis model infers the second data to be processed, and the second inference element obtained is to lower the volume of the smart terminal; the human-sensing interaction analysis model infers the third data to be processed, and the third inference element obtained is to lower the volume of the smart terminal or turn off the smart terminal; the human-sensing service model infers the fourth data to be processed, and the fourth inference element obtained is to lower the volume of the smart terminal or turn off the smart terminal. Next, based on the first inference element, the second inference element, the third inference element and the fourth inference element, the first AI perception scene is constructed as a "multiple users chatting scene".
[0108] It should be understood that the first AI perception scenario is not limited to the above two example scenarios, and those skilled in the art can set it according to actual needs.
[0109] It should also be understood that in the early stage of the operation of the AI human perception model, due to the incompleteness of the data, there will be a low degree of matching. Therefore, multiple first AI perception scene interactions will be constructed at the same time for users to choose. At this time, users can choose the most appropriate scene for interaction according to their needs. Of course, by default, the first AI perception scene with the highest matching degree is selected. With the in-depth use of users and the improvement of data integrity, the constructed second AI perception scene will gradually meet the expectations of users.
[0110] In other implementations, if the smart terminal is in a single-person interaction situation, the user can interact with the smart terminal through voice or other interaction methods.
[0111] In some implementations, before the step S1, the intelligent interaction method based on the AI human perception model further includes:
[0112] S4. Detect the installation environment information and the operating environment information of the smart terminal.
[0113] Furthermore, the step S4 comprises:
[0114] Execute one or more of the following steps S41, S42 and S43:
[0115] S41, using the millimeter wave module of the AI perception module to detect second installation information and second environmental space information of the installation environment information;
[0116] The content of the second installation information is the same as that of the first installation information.
[0117] The second environmental space information is the size of the environmental space in which the smart terminal operates (ie, the size of the indoor space).
[0118] S42, using the light sensing module of the AI perception module to detect the operating environment light information of the operating environment information.
[0119] The operating environment light information refers to information about the influence of the light around the smart terminal on the display effect of the smart terminal.
[0120] S43, using the millimeter wave module of the AI perception module to detect the viewing position information of the operating environment information.
[0121] The viewing position information refers to the viewing position of the user.
[0122] S5. Adjust the operating state of the smart terminal according to the installation environment information and the operating environment information.
[0123] The S5 step includes:
[0124] S51. Adjust the volume of the smart terminal based on the second installation information and the second environment space information.
[0125] In the present disclosure, the smart terminal adjusts the volume parameter to the most suitable value.
[0126] S52: Adjust the backlight brightness of the smart terminal based on the operating environment light information.
[0127] In the present disclosure, the backlight brightness of the smart terminal is adjusted to an optimal state.
[0128] S53: Adjust the display brightness and sound playing angle of the smart terminal based on the viewing position information.
[0129] In the present disclosure, the smart terminal adjusts the display brightness to an optimal brightness and the sound angle to an optimal angle according to the viewing position of the user.
[0130] The steps S4 and S5 are the initialization steps of the smart terminal, which can also be understood as the initialization of each sub-module of the AI perception module.
[0131] In some implementations, before the step S3, the intelligent interaction method based on the AI human perception model further includes:
[0132] S6. Detect whether the first AI perception scene matches the smart terminal.
[0133] Due to the different training data of the AI human perception model, the different hardware and software configurations of each smart terminal, and the different multimodal reasoning strategies of the AI human perception model, the first reasoning element, the second reasoning element, the third reasoning element, and the fourth reasoning element derived from the reasoning analysis will change with the changes in the hardware and software configurations of the AI human perception model and the smart terminal, so that the constructed first AI perception scene can generate different first smart interaction instructions. Regardless of which instruction, it should match the smart terminal, that is, the smart terminal can execute the first smart interaction instruction.
[0134] In order to achieve the above purpose, the present disclosure can set multiple preset scenes in the smart terminal, and when the first AI perception scene is any of the preset scenes, the first AI perception scene matches the smart terminal. In this way, it can be avoided that the first AI perception scene does not match the smart terminal, resulting in the subsequent inability of the smart terminal to execute the first smart interaction instruction.
[0135] S7. When the first AI perception scenario matches the smart terminal, based on the first AI perception scenario, generate and execute a first smart interaction instruction corresponding to the first AI perception scenario.
[0136] S8. When the first AI perception scenario does not match the smart terminal, adjust the multimodal reasoning strategy of the first AI human perception big model to obtain a second AI human perception big model; use the second AI human perception big model to analyze and process the behavior data to construct a second AI perception scenario; based on the second AI perception scenario, generate and execute a second intelligent interaction instruction corresponding to the second AI perception scenario.
[0137] When the first AI perception scene does not belong to any of the preset scenes, the first AI perception scene does not match the smart terminal.
[0138] The multimodal reasoning strategy can refer to the above-mentioned "Example 1" and "Example 2". The adjustment of the multimodal reasoning strategy can be understood as adjusting the correspondence between the first reasoning element and the first data to be processed, the correspondence between the second reasoning element and the second data to be processed, the correspondence between the third reasoning element and the third data to be processed, or the correspondence between the fourth reasoning element and the fourth data to be processed. Exemplarily, when the fourth data to be processed is "an image of a user standing up" and the fourth reasoning element is to turn on the indoor lights, the corresponding first intelligent interaction instruction is the instruction to turn on the indoor lights. If the smart terminal does not establish a wireless connection with the indoor lights (establish an Internet of Things connection), the smart terminal cannot execute the instruction to turn on the indoor lights. This is the case where the first AI perception scene does not match the smart terminal. At this time, the multimodal reasoning strategy can be adjusted so that the fourth reasoning element obtained by the modified human perception service model reasoning is to turn off the smart terminal or lower the display brightness of the smart terminal. At this point, the first AI human perception model is modified into the second AI human perception model, so that the second intelligent interaction instruction that can be successfully executed by the smart terminal can be generated. In this example, the second intelligent interaction instruction is to turn off the intelligent terminal or lower the display brightness of the intelligent terminal.
[0139] In some embodiments, after the S3 step, the intelligent interaction method based on the AI human perception model further includes: when the camera module of the AI perception module collects the user facial image as an emotional image that the user is dissatisfied with or the far-field voice module collects the user voice information as dissatisfied with the current first AI perception scene, outputting a prompt message, wherein the prompt message is used to prompt the user whether to switch to another first AI perception scene for interaction.
[0140] The intelligent interaction method based on the AI human perception big model provided by the present disclosure is suitable for application scenarios of smart TVs.
[0141] Specifically, the intelligent interaction method based on the AI human perception model includes:
[0142] S101, using multiple sub-modules of the AI perception module in the smart TV to collect behavior data.
[0143] like Figure 3 and Figure 4 As shown in FIG. 1 , the AI perception module includes a camera module, a light sensing module, a millimeter wave module, and a far-field voice module. Of course, the AI perception module may also include other intelligent modules. The AI perception module as a whole can be used to realize distance perception, human perception, environment perception, user recognition, scene recognition, environment recognition, behavior recognition, and other recognition (refer to FIG. 1 ). Figure 4 ).
[0144] S102: Use the first AI human perception model of the smart TV to analyze and process the behavior data to construct a first AI perception scene.
[0145] Figure 3 The “AI human perception big model” is the first AI human perception big model, which is used to realize AI perception behavior. Specific AI perception behaviors include but are not limited to interactive behavior, environmental perception, intelligent perception and other perceptions.
[0146] S103. Based on the first AI perception scenario, generate and execute a first intelligent interaction instruction corresponding to the first AI perception scenario to respond to the behavior data and realize interaction between the smart TV and the user.
[0147] The purpose of the first intelligent interaction instruction can refer to Figure 3 and Figure 4 , used to adjust or control the sound, image, system, application, service, AIOT (device connected to the Internet of Things of the smart TV) and others of the smart TV.
[0148] In some embodiments, the step S101 includes:
[0149] Execute one or more of the following steps S1011, S1012, S1013 and S1014: S1011, using the millimeter wave module of the AI perception module to collect first installation information, first environment space information and user usage information of the behavior data;
[0150] S1012, using the camera module of the AI perception module to collect the user face image, usage scene image and user behavior image of the behavior data;
[0151] S1013, using the far-field voice module of the AI perception module to collect user voice information and environmental noise information of the behavior data;
[0152] S1014. Use the light sensing module of the AI perception module to collect ambient light information of the behavior data.
[0153] In some embodiments, the step S102 includes:
[0154] S1021: Classify the behavior data to obtain at least one piece of data to be processed, wherein the at least one piece of data to be processed includes first piece of data to be processed, second piece of data to be processed, third piece of data to be processed, and fourth piece of data to be processed.
[0155] S1022. Infer all the data to be processed through the first AI human perception model to obtain an inference element corresponding to each data to be processed.
[0156] Specifically, the step S1022 includes:
[0157] S10221. Infer the first data to be processed through the human perception environment analysis model of the first AI human perception model to obtain a first inference element.
[0158] S10222. Infer the second data to be processed through the human perception scene analysis model of the first AI human perception model to obtain a second inference element.
[0159] S10223. Infer the third data to be processed through the human-sensing interaction analysis model of the first AI human-sensing large model to obtain a third inference element.
[0160] S10224. Infer the fourth data to be processed through the human perception service model of the first AI human perception model to obtain a fourth reasoning element.
[0161] S1023. Construct the first AI perception scene based on all the reasoning elements.
[0162] More specifically, the first AI perception scene is constructed based on the first reasoning element, the second reasoning element, the third reasoning element and the fourth reasoning element.
[0163] It should be understood that the first AI perception scenario can be one selected from a plurality of scenarios (that is, one selected from scene 1, scene 2, scene 3...scene N), and each scenario can be set by technical personnel in this field according to actual needs.
[0164] In some implementations, before step S101, the intelligent interaction method based on the AI human perception model further includes:
[0165] S104, detecting the installation environment information and the operating environment information of the smart TV;
[0166] S105: Adjust the operating state of the smart TV according to the installation environment information and the operating environment information.
[0167] In some embodiments, the step S104 includes:
[0168] Execute one or more of the following steps S1041, S1042, and S1043:
[0169] S1041. Use the millimeter wave module of the AI perception module to detect second installation information and second environmental space information of the installation environment information;
[0170] S1042, using the light sensing module of the AI perception module to detect operating environment light information of the operating environment information;
[0171] S1043, using the millimeter wave module of the AI perception module to detect the viewing position information of the operating environment information.
[0172] In some embodiments, the step S105 includes:
[0173] S1051, adjusting the volume of the smart TV based on the second installation information and the second environment space information;
[0174] S1052, adjusting the backlight brightness of the smart TV based on the operating environment light information;
[0175] S1053: adjusting the display brightness and sound playing angle of the smart TV based on the viewing position information.
[0176] In some implementations, before step S103, the intelligent interaction method based on the AI human perception model further includes:
[0177] S106, detecting whether the first AI perception scene matches the smart TV;
[0178] S107: When the first AI perception scene matches the smart TV, based on the first AI perception scene, generate and execute a first smart interaction instruction corresponding to the first AI perception scene;
[0179] S108. When the first AI perception scene does not match the smart TV, adjust the multimodal reasoning strategy of the first AI human perception big model to obtain a second AI human perception big model; use the second AI human perception big model to analyze and process the behavior data to construct a second AI perception scene; based on the second AI perception scene, generate and execute a second intelligent interaction instruction corresponding to the second AI perception scene.
[0180] In some embodiments, reference Figure 4 Before the step S101, the intelligent interaction method based on the AI human perception model further includes:
[0181] Turning on the smart TV;
[0182] Loading the smart TV operating system;
[0183] Load the AI perception module.
[0184] <Intelligent interactive system based on AI human perception model>
[0185] like Figure 5 As shown, it is a structural block diagram of an intelligent interaction system based on an AI human perception model provided by the second aspect of the present disclosure. Figure 5 In the embodiment, the intelligent interaction system 200 based on the AI human perception model includes:
[0186] The behavior data collection module 201 is used to collect behavior data using multiple submodules of the AI perception module in the smart terminal;
[0187] The first AI perception scene output module 202 is used to analyze and process the behavior data using the first AI human perception model of the smart terminal to construct a first AI perception scene;
[0188] The first intelligent interaction instruction execution module 203 is used to generate and execute a first intelligent interaction instruction corresponding to the first AI perception scenario based on the first AI perception scenario, so as to respond to the behavior data and realize the interaction between the smart terminal and the user.
[0189] <Smart Terminal>
[0190] like Figure 6 As shown, it is a principle block diagram of the smart terminal provided by the third aspect of the present disclosure, the smart terminal 300 includes a memory 301, a processor 302 and an AI perception module 303, the AI perception module 303 includes a millimeter wave module 3031, a camera module 3032, a far-field voice module 3033 and a light sensing module 3034, the memory 301 is used to store computer programs, the processor 302 is used to call and run the computer program stored in the memory 301, and execute the steps of the intelligent interaction method based on the AI human perception large model provided by the first aspect of the present disclosure, the millimeter wave module 3031, the camera module 3032, the far-field voice module 3033 and the light sensing module 3034 are all connected to the processor 302 through the system bus 304.
[0191] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the smart terminal to which the scheme of the present disclosure is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.
[0192] In some embodiments, the AI perception module further includes a temperature sensor and a humidity sensor, and both the temperature sensor and the humidity sensor are connected to the processor. It should be understood that the AI perception module may also include other hardware modules, which may be determined by those skilled in the art according to actual needs.
[0193] <Computer Readable Storage Medium>
[0194] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute the steps of the intelligent interaction method based on the AI human sense model provided in the first aspect of the present disclosure.
[0195] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0196] The technical features of the above embodiments can be combined without changing the basic principles of the present disclosure. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0197] The above embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the patent protection scope of the present disclosure shall be subject to the attached claims.
Claims
1. An intelligent interaction method based on an AI human perception model, applied to an intelligent terminal, characterized in that: include: Use multiple submodules of the AI perception module in the smart terminal to collect behavioral data; The first AI human perception model of the smart terminal is used to analyze and process the behavior data to construct a first AI perception scene; Based on the first AI perception scenario, a first intelligent interaction instruction corresponding to the first AI perception scenario is generated and executed to respond to the behavior data to achieve interaction between the smart terminal and the user.
2. The intelligent interaction method based on the AI human perception model according to claim 1 is characterized in that: The step of using multiple submodules of the AI perception module in the smart terminal to collect behavior data includes: Perform one or more of the following steps: The millimeter wave module of the AI perception module is used to collect the first installation information, the first environment space information and the user usage information of the behavior data; The camera module of the AI perception module is used to collect user face images, usage scene images and user behavior images of the behavior data; The far-field voice module of the AI perception module is used to collect user voice information and environmental noise information of the behavior data; The light sensing module of the AI perception module is used to collect the ambient light information of the behavior data.
3. The intelligent interaction method based on the AI human perception model according to claim 1 is characterized in that: The step of using the first AI human perception model of the smart terminal to analyze and process the behavior data to construct a first AI perception scene includes: Classify and process the behavior data to obtain at least one piece of data to be processed; Inferring all the data to be processed by using the first AI human perception model to obtain an inference element corresponding to each data to be processed; The first AI perception scenario is constructed based on all the reasoning elements.
4. The intelligent interaction method based on AI human perception model according to claim 1 is characterized in that: Before the step of using multiple submodules of the AI perception module in the smart terminal to collect behavior data, the intelligent interaction method based on the AI human perception model also includes: Detecting the installation environment information and the operating environment information of the smart terminal; The operating state of the smart terminal is adjusted according to the installation environment information and the operating environment information.
5. The intelligent interaction method based on the AI human perception model according to claim 4 is characterized in that: The step of detecting the installation environment information and the operating environment information of the smart terminal includes: Perform one or more of the following steps: The millimeter wave module of the AI perception module is used to detect the second installation information and the second environment space information of the installation environment information; Using the light sensing module of the AI perception module to detect the operating environment light information of the operating environment information; The millimeter wave module of the AI perception module is used to detect the viewing position information of the operating environment information.
6. The intelligent interaction method based on the AI human perception model according to claim 5 is characterized in that: The step of adjusting the operating state of the smart terminal according to the installation environment information and the operating environment information comprises: adjusting the volume of the smart terminal based on the second installation information and the second environment space information; Adjusting the backlight brightness of the smart terminal based on the operating environment light information; The display brightness and sound playing angle of the smart terminal are adjusted based on the viewing position information.
7. The intelligent interaction method based on AI human perception model according to claim 1 is characterized in that: Before the step of generating and executing a first intelligent interaction instruction corresponding to the first AI perception scenario based on the first AI perception scenario, the intelligent interaction method based on the AI human perception big model further includes: Detecting whether the first AI perception scene matches the smart terminal; In a case where the first AI perception scenario matches the smart terminal, generating and executing a first smart interaction instruction corresponding to the first AI perception scenario based on the first AI perception scenario; When the first AI perception scenario does not match the smart terminal, the multimodal reasoning strategy of the first AI human perception big model is adjusted to obtain a second AI human perception big model; the second AI human perception big model is used to analyze and process the behavior data to construct a second AI perception scenario; based on the second AI perception scenario, a second intelligent interaction instruction corresponding to the second AI perception scenario is generated and executed.
8. An intelligent interactive system based on AI human perception model, characterized in that: include: A behavior data collection module, used to collect behavior data using multiple submodules of the AI perception module in the smart terminal; A first AI perception scene output module, used to analyze and process the behavior data using the first AI human perception model of the smart terminal to construct a first AI perception scene; The first intelligent interaction instruction execution module is used to generate and execute a first intelligent interaction instruction corresponding to the first AI perception scenario based on the first AI perception scenario, so as to respond to the behavior data and realize the interaction between the smart terminal and the user.
9. An intelligent terminal, characterized in that: include: A memory, a processor and an AI perception module, the AI perception module includes a millimeter wave module, a camera module, a far-field voice module and a light sensing module, the memory is used to store computer programs, the processor is used to call and run the computer programs stored in the memory, and execute the steps of the intelligent interaction method based on the AI human perception model as described in any one of claims 1 to 7 above, the millimeter wave module, the camera module, the far-field voice module and the light sensing module are all connected to the processor.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, which enables a computer to execute the steps of the intelligent interaction method based on the AI human sense model as described in any one of claims 1 to 7 above.
Citation Information
Cited By
Smart home scene understanding and interaction method and system based on multi-modal fusion
CN119398159A
