Human-computer interaction method, device, equipment and medium

Abstract

The invention provides a man-machine interaction method and device, equipment and a medium, and the method comprises the steps: obtaining end-side data of a user side, the end-side data comprising at least one of user input data and end-side collection data; based on the end-side large model, generating interaction information corresponding to the end-side data; the end side large model is a large language model arranged at a local device where the user side is located; and executing the interaction information. According to the man-machine interaction method, the man-machine interaction device, the man-machine interaction equipment and the medium provided by the invention, the end-side large model is deployed locally, so that the inference and decision-making capabilities of the user side are realized, and even under the condition of no network, efficient operation can be realized, and continuous interaction experience is ensured. Meanwhile, due to local end side large model processing, data is prevented from being uploaded to the cloud, and the privacy risk is remarkably reduced, so that personalized data processing interaction tasks can be flexibly applied, interaction responses better meeting user requirements are generated, and the limitation that a traditional method depends on preset rules and is insufficient in flexibility is overcome.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a human-computer interaction method, device, equipment and medium. Background Art

[0002] Human-computer interaction technology has evolved from command lines to multimodal interaction, and has now become an important technology in the fields of vehicle systems, smart homes, etc. With the popularization of large models and deep learning, human-computer interaction is developing in a more natural and intelligent direction, improving user experience through voice, images and other methods, and bringing new possibilities for personalized services.

[0003] Nowadays, human-computer interaction technology mostly relies on cloud processing, and user data needs to be uploaded to the server and analyzed by a large model. This method will limit the interactive function in an environment with no network or weak network, and there is also the risk of user privacy leakage. In addition, personalized recommendations are mostly based on preset rules, lacking flexibility and intelligence, and it is difficult to meet users' needs for more natural interaction. Summary of the invention

[0004] The present invention provides a human-computer interaction method, device, equipment and medium, which are used to solve the defects of poor interaction ability and high privacy security risk in a non-network environment in the related technology.

[0005] The present invention provides a human-computer interaction method, comprising the following steps: Acquire terminal side data of the user terminal, where the terminal side data includes at least one of user input data and terminal side collected data; Based on the large model on the end side, generating interaction information corresponding to the end side data; the large model on the end side is a large language model set at the local device where the user end is located; The interactive information is executed.

[0006] According to a human-computer interaction method provided by the present invention, the generating of interaction information corresponding to the terminal-side data based on the terminal-side big model includes: Determining an interaction intent corresponding to the terminal-side data; Based on the end-side big model, interaction information corresponding to the interaction intention and the end-side data is generated.

[0007] According to a human-computer interaction method provided by the present invention, the determining the interaction intention corresponding to the terminal-side data includes: In a case where the terminal-side data includes the user input data, the interaction intent corresponding to the terminal-side data is determined based on a previous intention classification model.

[0008] According to a human-computer interaction method provided by the present invention, the determining the interaction intention corresponding to the terminal-side data includes: In a case where the terminal side data does not include the user input data, determining whether to trigger an active interaction scenario based on the terminal side data; In the case where it is determined that an active interaction scenario is triggered, an interaction intention corresponding to the terminal side data is determined based on the triggered active interaction scenario.

[0009] According to a human-computer interaction method provided by the present invention, generating interaction information corresponding to the interaction intention and the terminal-side data based on the terminal-side big model includes: Determining a prompt corresponding to the interaction intention; The prompt and the end-side data are input into the end-side large model to obtain the interaction information output by the end-side large model.

[0010] According to a human-computer interaction method provided by the present invention, the generating of interaction information corresponding to the terminal-side data based on the terminal-side big model includes: Searching for historical interaction information matching the end-side data from a self-learning cache of the end-side large model, wherein the self-learning cache stores the historical interaction information of the end-side large model; In the case where historical interaction information matching the terminal-side data is found, the historical interaction information matching the terminal-side data is used as the interaction information corresponding to the terminal-side data; If no historical interaction information matching the end-side data is found, interaction information corresponding to the end-side data is generated based on the end-side big model.

[0011] According to a human-computer interaction method provided by the present invention, searching for historical interaction information matching the end-side data from the self-learning cache of the end-side large model includes: Performing interactive function classification on the terminal side data to obtain an interactive function type corresponding to the terminal side data; In a case where the interaction function type belongs to a candidate interaction function type of the end-side large model, historical interaction information matching the end-side data is searched from the self-learning cache of the end-side large model.

[0012] The present invention also provides a human-computer interaction device, comprising the following modules: A terminal side data acquisition unit, which acquires terminal side data of a user terminal, wherein the terminal side data includes at least one of user input data and terminal side collected data; An interaction information generating unit generates interaction information corresponding to the end-side data based on an end-side large model; the end-side large model is a large language model set at a local device where the user end is located; An execution unit is used to execute the interaction information.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the human-computer interaction methods described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the human-computer interaction method as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the human-computer interaction method described above is implemented.

[0016] The human-computer interaction method, device, equipment and medium provided by the present invention realize the user-side self-contained reasoning and decision-making capabilities by locally deploying a large end-side model, and can operate efficiently even without a network, ensuring a continuous interactive experience. At the same time, since the local end-side large model processing avoids uploading data to the cloud, the privacy risk is significantly reduced, so it is possible to flexibly apply personalized data processing interaction tasks to generate interactive responses that better meet user needs, overcoming the limitations of traditional methods that rely on preset rules and lack flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the human-computer interaction method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the graphic and text understanding and interaction process in the vehicle-machine scenario provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the application flow of the active interaction scenario and the passive interaction scenario provided by the present invention in the vehicle system.

[0021] Figure 4 It is a flow chart of the local self-learning cache mechanism provided by the present invention.

[0022] Figure 5 It is a flow chart of the cloud self-learning cache update mechanism provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the testing device provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The development of human-computer interaction technology has undergone tremendous changes in the past few decades, from the initial command line interface to the graphical user interface, and then to today's multimodal interactive interface. With the advancement of artificial intelligence technology, especially the widespread application of large models and deep learning, human-computer interaction is no longer limited to simple voice commands or touch operations, but can comprehensively utilize multiple input methods such as voice, images, gestures, etc. to achieve a more natural and intelligent interactive experience. The development of this technology not only improves the ease of use of equipment, but also plays an important role in the fields of vehicle systems, smart homes, medical care, etc., bringing more convenient and personalized services to users.

[0027] However, existing human-computer interaction technologies mainly rely on cloud processing, that is, user data needs to be uploaded to the server and understood and analyzed by a large model. This approach has significant disadvantages: when the device is in an environment with no network or weak network, the reliability of the interaction will be greatly reduced, or even unable to be used normally; at the same time, uploading the user's personalized data to the cloud also brings privacy and security risks. In addition, existing personalized recommendations and responses are mostly based on preset rules and fixed logic, resulting in insufficient flexibility of the system and unable to adapt to the user's growing demand for intelligent and natural interaction.

[0028] In view of the above problems, an embodiment of the present invention provides a human-computer interaction method. Figure 1 is a flow chart of the human-computer interaction method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110: Acquire terminal-side data of the user terminal, where the terminal-side data includes at least one of user input data and terminal-side collected data.

[0029] Specifically, end-side data refers to data collected locally on user devices and does not rely on cloud servers. The acquisition of end-side data mainly includes two types: user input data and end-side collected data. The combination of the two can provide a richer source of information for human-computer interaction.

[0030] User input data is information provided by users through active interaction, including but not limited to voice commands, text input, gesture operations, etc. This data is usually collected in real time through the input interface of the device. For example, when a user issues a voice command, the microphone captures the audio signal and converts it into text data through the voice recognition module; in the text input scenario, the user can enter text information through the touch screen or keyboard. In addition, the user's gesture operations can also be captured by sensors such as cameras or touchpads and converted into processable data. The characteristics of this type of data are that it is actively provided by the user, the information content is relatively clear, and it is easy to understand and process accurately.

[0031] The data collected on the end side is the environment and / or status data automatically collected by the device through built-in sensors or interfaces. These data do not require active input from the user, but are automatically obtained during daily use. For example, in a car-mounted scenario, the device can obtain the vehicle's operating status through the interface, such as vehicle speed, gear position, air conditioning working status, sunroof and door opening status, etc. At the same time, environmental information such as temperature, humidity, and air quality in the car can also be collected from sensors. In more complex cases, the device may automatically detect the driver's status (such as fatigue, emotional changes) and the distribution of passengers in the car (such as whether there are child passengers) through cameras or infrared sensors. These data help the device understand the current environment and usage status, and provide important contextual information for subsequent human-computer interaction.

[0032] By collecting user input data and data collected on the device, the large model on the device can more comprehensively understand user needs and current usage scenarios during the interaction process. For example, when a user turns on the air conditioner through voice request, the device can automatically adjust the temperature and mode of the air conditioner based on the temperature and air quality information in the car, thereby achieving smarter interaction. This not only improves the efficiency and personalization of interaction, but also continues to provide high-quality services without a network.

[0033] Step 120: Generate interaction information corresponding to the terminal data based on the terminal large model; the terminal large model is a large language model set at the local device where the user terminal is located.

[0034] Specifically, the large model on the end is a large language model that is deployed on the local device after cutting the large model on the cloud. The large model on the end has certain reasoning and processing capabilities, and can perform complex calculations and decisions independently on the local device without communicating with the cloud and using the computing resources of the large model on the cloud.

[0035] By analyzing and processing the device-side data, the device-side big model can generate corresponding interactive information. This process involves a comprehensive understanding and reasoning of the device-side data, including the analysis of user input data and the comprehensive understanding of the device-side collected data. The device-side big model can use multiple data sources, such as user input data such as voice commands and gesture operations, and device-side collected data such as images collected by cameras and environmental data from temperature sensors, to generate appropriate responses in combination with the current context information. These interactive information may appear as voice replies, text prompts, operation instructions, or contextualized suggestions. For example, when a user issues a command to "turn on the air conditioner," the model will generate an operation instruction to adjust the air conditioner temperature in combination with the current temperature in the car. The generation process of interactive information can not only consider the user's direct needs, but also combine the environmental information perceived by the device to achieve more intelligent interaction.

[0036] The key advantage of the large model on the end is local deployment, which means that all reasoning and decision-making are completed on the user's local device, that is, the user end, without relying on network connection. This localized processing method ensures the real-time and continuity of interaction in an environment with no network or weak network. At the same time, local data processing avoids uploading user data to the cloud, significantly improves privacy protection, and reduces the risk of data leakage. Although the large model on the end is tailored from the large model on the cloud to adapt to the computing resource limitations of the local device, its design retains the core reasoning capabilities. Therefore, this deployment method not only improves processing efficiency, but also provides users with a more personalized and private interactive experience.

[0037] Step 130: execute the interaction information.

[0038] This step involves the specific execution of the interaction information. The process of executing the interaction information may include various forms of operations, depending on the type of interaction information and the function of the device.

[0039] If the interactive information is a device control command, such as turning on the air conditioner or adjusting the volume, the user end will directly complete the corresponding operation according to the instruction. For example, when the user requests to turn on the air conditioner, the user end will combine the current environmental data (such as the temperature and air quality in the car) to turn on the air conditioner and automatically adjust it to the appropriate temperature and wind speed to ensure a comfortable in-car environment.

[0040] For text prompts and voice replies, the execution process involves presenting the generated interactive content to the user in an appropriate manner. If the interactive information is a voice reply, the user end will use text-to-speech technology to broadcast the reply content in voice; if it is a text prompt, the corresponding information will be displayed on display devices such as display screens, such as safety reminders, navigation instructions, etc. In addition, in the in-vehicle scenario, the user end can also provide visual prompts through the dashboard or in-vehicle display to enhance the user's interactive experience.

[0041] By executing the interactive information, the user end converts the generated interactive information into actual operational behavior, realizing a complete closed loop from data analysis to specific execution, thereby providing users with a smoother and more natural interactive experience.

[0042] The human-computer interaction method provided by the embodiment of the present invention realizes the user's own reasoning and decision-making capabilities by locally deploying a large model on the end side, and can run efficiently even without a network, ensuring a continuous interactive experience. At the same time, since the local large model processing on the end side avoids uploading data to the cloud, the privacy risk is significantly reduced, so it can flexibly apply personalized data processing interaction tasks to generate interactive responses that better meet user needs, overcoming the limitations of traditional methods that rely on preset rules and lack flexibility.

[0043] Based on the above embodiment, in step 120, generating the interaction information corresponding to the terminal side data based on the terminal side big model includes: Determining an interaction intent corresponding to the terminal-side data; Based on the end-side big model, interaction information corresponding to the interaction intention and the end-side data is generated.

[0044] Interaction intent refers to the goal or operation requirement that the user wants to achieve in a certain interaction scenario, such as turning on the air conditioner, adjusting the seat, checking the weather, etc. Interaction intent represents the user's real needs and is the basis for generating specific interaction responses. Only by accurately identifying the user's interaction intent can we further generate interaction information that meets the user's expectations.

[0045] After the interaction intention is determined, the end-side big model will generate interaction information corresponding to the intention and end-side data. Interaction information is a specific implementation of the interaction intention, which aims to meet user needs and complete specific operations. For example, after determining that the user's intention is to "increase the temperature in the car", the model will combine the current temperature in the car, the working status of the air conditioner, and the external environmental conditions to generate corresponding operation suggestions or instructions, such as "increase the air conditioner temperature to 22 degrees" or "turn on the seat heating."

[0046] There is a close connection between interaction intent and interaction information. Interaction intent determines the content and form of interaction information. Interaction intent clarifies the user's needs or goals in a specific scenario, while interaction information is the specific response or operation steps generated based on this goal. Through this intention-based interaction information generation method, the user end can more accurately meet the user's actual needs and provide intelligent and personalized interactive services in changing scenarios.

[0047] Based on the above embodiment, determining the interaction intent corresponding to the terminal-side data includes: In a case where the terminal-side data includes the user input data, the interaction intent corresponding to the terminal-side data is determined based on a previous intention classification model.

[0048] Specifically, when the client-side data contains user input data, the pre-intention classification model is used for processing. This is because user input data (such as voice commands, text input, gesture operations, etc.) directly reflects the user's needs and intentions and is the most critical source of information for determining interaction intentions. When it is detected that the user actively provides input, the pre-intention classification model is used to identify the intention, which can more accurately analyze the user's real needs.

[0049] The pre-intention classification model is a model specifically used for intent classification. It can deeply analyze user input data and automatically classify it into specific intent categories. The model uses natural language processing technology and pre-trained semantic analysis algorithms to train for different interaction scenarios, enabling it to efficiently and accurately identify multiple types of user intent. By analyzing the input content, the model quickly determines the operation that the user wants the user end to perform, providing an accurate basis for the subsequent generation of interaction information.

[0050] The categories of intent classification can cover a variety of interaction needs, including but not limited to the user's desire to control the user's end operations through interaction, such as turning on the air conditioner, turning up the volume, etc.; requests involving information queries, such as asking about the weather or finding a nearby gas station; requests related to navigation and driving instructions, such as navigating to a specific location or displaying prompts for the next intersection; and reminders for the current user end or environmental status, such as checking the oil level or prompting that the temperature in the car is too high. In addition, it may also involve personalized recommendations, providing suggestions based on user habits or current situations, such as playing the user's preferred music or suggesting a rest after a long drive. It is understandable that in an interaction process, the user's intention may include more than one.

[0051] Considering that the performance of the large model on the end is weaker than that of the large model on the cloud, the introduction of the pre-intention classification model before the large model on the end can identify the user's intention in advance and optimize the task processing flow, thereby reducing the computational burden of the large model on the end. By identifying these categories, the pre-intention classification model enables the user end to quickly understand the user's needs and provide an accurate basis for the subsequent generation of interactive information. In this way, whether in a complex multi-round dialogue or a simple single-round instruction, the user end can make corresponding intelligent responses based on the user's input.

[0052] Based on the above embodiment, determining the interaction intent corresponding to the terminal-side data includes: In a case where the terminal side data does not include the user input data, determining whether to trigger an active interaction scenario based on the terminal side data; In the case where it is determined that an active interaction scenario is triggered, an interaction intention corresponding to the terminal side data is determined based on the triggered active interaction scenario.

[0053] Specifically, when the user does not actively input data, it is necessary to analyze the automatically collected data on the end side to determine whether to trigger an active interaction scenario. An active interaction scenario refers to an interaction behavior automatically initiated by the user end based on changes in the environment or state. The triggering of this scenario does not rely on the user's active input, but the user end determines whether interaction is required based on preset conditions or rules. Active interaction can provide intelligent reminders or optimization suggestions without the user's awareness to improve the user experience and security.

[0054] The difference between active interaction scenarios and passive interaction scenarios lies mainly in the triggering method and initiative of the interaction. Passive interaction scenarios are interactive processes initiated by users. Users express their needs clearly by inputting voice commands, text or gestures, and the user end generates corresponding responses based on the user's input. At this time, the content and direction of the interaction are actively controlled by the user, and the task of the user end is to identify the user's intention and give appropriate feedback.

[0055] Relatively speaking, active interaction scenarios do not require active user operations. Instead, the user side determines whether interaction is required based on the collected end-side data. After confirming that interaction is required, the interaction intention in the current scenario is determined based on the collected end-side data related to the current interaction scenario. For example, when the vehicle speed exceeds the safety threshold and the weather conditions are bad, the user side may automatically trigger a safety reminder and recommend reducing the speed; when the driver is detected to have been driving for a long time and showing signs of fatigue, the user side may actively recommend taking a rest at the nearest service area and ask if music needs to be played to refresh or windows opened for ventilation. The interaction triggering and interaction intention of these scenarios all rely on the analysis of end-side data. After the interaction intention is determined, the corresponding interaction information is generated by calling the end-side big model and displayed to the user in the form of voice, image or text.

[0056] Based on the above embodiment, generating the interaction information corresponding to the interaction intention and the terminal side data based on the terminal side big model includes: Determining a prompt corresponding to the interaction intention; The prompt and the end-side data are input into the end-side large model to obtain the interaction information output by the end-side large model.

[0057] After clarifying the user's interaction intention, prompts are generated or dynamically generated based on the preset prompt library. Prompts are short messages used to guide the end-side big model to understand the interaction intention during the interaction process. Prompts usually contain the classification label of the interaction and the environment and device status information provided by the end-side collected data. For example, when the user says that the weather is too hot, the interaction intention obtained by analysis is that the user needs a more comfortable environment. The prompt may be: {Classification: bad weather, status: [temperature: 37, car window: open, air conditioning: close, xxx], as a voice assistant, please help me give the user some suggestions based on the current vehicle status}. The prompts will not be displayed directly to the user, but will be input into the end-side big model as part of the internal processing. It helps the end-side big model to more accurately understand the user's current needs or status and generate appropriate interaction information accordingly.

[0058] After receiving the prompt and the end-side data, the end-side big model not only analyzes the user intent reflected in the prompt, but also conducts in-depth analysis of the end-side data. By understanding these data, the end-side big model can identify the current device status, environmental conditions, etc., and thus generate a response that is more in line with the scenario. For example, when it is detected that the temperature in the car is high and the air conditioner is not turned on, combined with the prompt, the end-side big model may deduce the interactive information "The weather is a bit hot, let me help you close the window and turn on the air conditioner", and broadcast this interactive information to the user. This multi-level analysis enables the big model to provide more accurate and dynamic feedback when generating interactive information, ensuring the intelligence and adaptability of the interaction.

[0059] Based on the above embodiment, the step of generating interaction information corresponding to the end-side data based on the end-side big model includes: Searching for historical interaction information matching the end-side data from a self-learning cache of the end-side large model, wherein the self-learning cache stores the historical interaction information of the end-side large model; In the case where historical interaction information matching the terminal-side data is found, the historical interaction information matching the terminal-side data is used as the interaction information corresponding to the terminal-side data; If no historical interaction information matching the end-side data is found, interaction information corresponding to the end-side data is generated based on the end-side big model.

[0060] Specifically, when the end-side big model generates interaction information, it first uses its internal self-learning cache to generate interaction information. The self-learning cache is an important knowledge base inside the end-side big model, which stores all historical interaction information previously generated by the end-side big model. When new end-side data and prompts are input into the end-side big model, the model first searches in the self-learning cache to try to find historical interaction information that matches the current end-side data.

[0061] The storage content of the self-learning cache includes various types of historical interaction information, which may involve historical interaction information generated for different users, different device states, different environmental conditions, etc. The storage form of this information is also diverse, which may be text, numbers, images, or any other data form that can be understood and processed by the large model on the end. Through such a design, the self-learning cache can support the model's rapid response and accurate decision-making when dealing with various complex situations.

[0062] If the self-learning cache finds historical interaction information that matches the current end-side data, the end-side large model will directly use this information as a response to the current data. This approach can significantly improve processing efficiency because the model can directly use the existing knowledge base without complex reasoning and calculation. If the self-learning cache does not find matching historical interaction information, the end-side large model will start the reasoning mechanism and generate new interaction information based on the current end-side data.

[0063] The self-learning mechanism of the large model on the end can quickly respond to different situations and provide users with an accurate and efficient interactive experience. At the same time, when the match is successful, the large model on the end generates interactive information based on the experience of previous successful interactions, ensuring the accuracy and consistency of the interactive information.

[0064] Based on the above embodiment, searching the self-learning cache of the large model on the end side for historical interaction information matching the end side data includes: Performing interactive function classification on the terminal side data to obtain an interactive function type corresponding to the terminal side data; In a case where the interaction function type belongs to a candidate interaction function type of the end-side large model, historical interaction information matching the end-side data is searched from the self-learning cache of the end-side large model.

[0065] Specifically, the process of matching the end-side data with the historical interaction information in the self-learning cache is targeted, that is, the input end-side data will first be classified by interaction function. Interaction function classification refers to judging the interaction function type to which the end-side data belongs based on the understanding and analysis of the end-side data. This type is usually closely related to the nature of the data and the needs of the user. For example, if the end-side data is about weather conditions, then its interaction function type may be "weather query"; if the data is about the device status, then the interaction function type may be "device control". Through such classification, the needs contained in the current end-side data can be more clearly understood, so that relevant information can be found in the self-learning cache in a targeted manner.

[0066] After determining the interaction function type corresponding to the end-side data, it is also necessary to determine whether the interaction function type belongs to the candidate interaction function type of the end-side large model. Candidate interaction function types refer to a series of possible interaction function types pre-defined or learned by the end-side large model, which represent the range of end-side data that the end-side large model can process and understand. If the interaction function type of the end-side data belongs to the candidate interaction function type, the end-side large model will continue to search for historical interaction information matching this data from the self-learning cache.

[0067] By classifying the interaction functions of the end-side data and limiting the search scope to the candidate interaction function types, it is possible to avoid blind searches in the self-learning cache, significantly improve search efficiency and accuracy, and find historical interaction information that matches the current end-side data more quickly. This not only improves the response speed of the large end-side model, but also reduces unnecessary computing resource consumption.

[0068] Based on the above embodiments, Figure 2 This is a schematic diagram of the graphic and text understanding and interaction process in the vehicle-mounted scenario provided by the present invention. Figure 2 As shown in the figure, when the vehicle starts, the shooting function is triggered first, and the camera captures the image of the current environment or specific objects. This step is used to obtain visual input from the user and the outside world as the basis for subsequent processing. Subsequently, the acquired image is understood and analyzed by the vehicle's large model, that is, the end-side large model. The model can identify objects and scenes in the image, and can even infer some contextual information, such as whether there are obstacles in front of the vehicle and the complexity of the current driving environment. By analyzing the image, the vehicle system will determine whether the image data meets specific interaction conditions.

[0069] The interaction conditions can be a combination of multiple situations, such as whether the user issues a specific instruction, or whether the image captured by the camera contains elements that trigger the interaction, such as traffic signs, pedestrians, weather changes, etc. When the interaction conditions are met, corresponding personalized interaction recommendations will be generated based on the results of image understanding. These recommendations can include prompting the driver to perform certain operations, adjusting the temperature in the car, or providing navigation suggestions based on the driving environment.

[0070] After the interactive recommendation is generated, the result will be broadcast to the driver in the form of voice. The content of the voice broadcast may involve operation suggestions, environmental feedback, or other information that helps the driver make decisions. This voice feedback not only improves the naturalness of human-computer interaction, but also reduces the driver's dependence on the screen or physical buttons during driving, thereby improving driving safety and convenience.

[0071] Based on the above embodiments, Figure 3FIG. 1 is a schematic diagram of the application flow of the active interaction scenario and the passive interaction scenario in the vehicle system provided by the present invention. Figure 3 As shown in the figure, the active scenario does not require the user to make a voice input, but is automatically triggered based on the status of the vehicle and driving environment. In this process, the driver and passenger status information is first collected through the Occupant Monitoring System (OMS) and the Driver Monitoring System (DMS). The OMS can monitor the emotions and postures of the passengers, while the DMS monitors the driver's fatigue, distraction and other states. If the driver is detected to be tired, distracted or driving for a long time, the vehicle system will automatically determine whether these data meet the trigger conditions, such as driving for too long or the driver's inattention. In addition, the vehicle will also combine external environmental data (such as weather changes and complex road conditions) to determine whether corresponding suggestions need to be generated. These automatically triggered recommendations may include reminding the driver to take a rest, suggesting to slow down or adjust the driving route. These operations are fully automated, without any instructions from the user, and are automatically generated based on the current vehicle and environmental data to ensure that users receive instant and safe driving suggestions.

[0072] Passive scenarios require users to actively issue voice input. In this case, users interact with the system through the voice input module and may issue commands such as "navigate to a certain location", "check the weather" or "adjust the temperature in the car". The user's voice input is first processed by the voice recognition module and converted into text or structured data. Then, in the skill domain module, the user's voice input interaction intention is determined and a prompt is generated. The next step is to determine whether the prompt is within the recommended range of the large model on the end side, that is, the recommended domain.

[0073] Whether it is an active scenario or a passive scenario, all information will be integrated through the "information merging" module to combine the monitoring data of the OMS / DMS system with the environmental information inside and outside the car. If the conditions are met, a scenario recommendation will be generated, such as optimizing the route, adjusting the in-vehicle equipment, or providing safety tips. Finally, the generated recommendations are fed back to the user through voice broadcast, so that the driver can obtain interactive information without manual operation or viewing the screen, ensuring safety and convenience during driving. In this way, the entire interactive process realizes a closed loop from multi-source data collection to voice feedback, ensuring that whether it is a scenario under active monitoring or a scenario under user active request, it can provide the driver with timely and intelligent suggestions and operations.

[0074] Based on the above embodiments, Figure 4 Schematic diagram of the local self-learning cache mechanism provided by the present invention. Figure 4As shown in the figure, the process starts from the pre-intervention stage, which is responsible for handling simple tasks that do not require the intervention of large models, such as directly opening an application or performing navigation. If the pre-intervention is successful, the interactive task will be processed quickly and ended directly. If the pre-intervention fails, the interactive task enters the operation stage of the self-learning cache.

[0075] Next, determine whether there is a historical record in the cache that matches the current task. If a matching record is found, the task enters the filtering mechanism for judgment; if no matching content is found, the task directly enters the large model reasoning stage and is further processed by the large model. The semantic complexity of the rewritten text and the original text is judged through the RC4 filtering mechanism. If the rewritten text is similar to the original text in form, but the semantics are more complex (conforms to the RC4 standard), the task will be regarded as a complex operation, and the process will not end immediately. Instead, the cached data will be used to assist the large model reasoning to generate personalized interaction information, and the corresponding records will be deleted from the self-learning cache. If the rewritten text does not have complex semantic changes and does not meet the RC4 standard, the task is considered to be relatively simple, and it ends directly and executes the results in the cache without calling the large model.

[0076] After the rewritten text is generated by the large model inference, it is necessary to determine whether it meets the cache mechanism. The cache mechanism will determine the following conditions: the conversation corpus does not contain some context or nouns in the blacklist configuration, such as "this, that, also like this, also, also"; each word of the rewritten text can be completely found in the original text; the rewritten text is completely consistent with the original text semantics (in compliance with the RC0 standard). If it meets the cache mechanism, the cache will be updated and the new interaction information will be added to the cache for quick matching and use in subsequent tasks.

[0077] Based on the above embodiments, Figure 5 It is a flow chart of the cloud-based self-learning cache update mechanism provided by the present invention. As shown in the figure, the process starts from the cloud and is processed by AIKIT (intelligent tool or platform in the cloud). After AIKIT processing, the current interactive information is judged according to the rules of the cache mechanism to determine whether it meets the requirements of the self-learning cache mechanism. If it is judged that the interactive information meets some conditions of the cache mechanism, that is, the conversation material does not contain some context or nouns referred to by the blacklist configuration, such as "this, that, also like this, also, also"; each word of the rewritten text can be completely found in the original text, and the interactive information will be added to the self-learning cache so that subsequent similar tasks can be processed faster. If the interactive information does not meet the requirements of the cache mechanism, the interactive information will not be stored in the cache, and the interactive task process ends directly without performing the cache operation. Finally, the process ends through the cloud and returns the interactive information.

[0078] The human-computer interaction device provided by the present invention is described below. The human-computer interaction device described below and the human-computer interaction method described above can be referred to each other. Figure 6 It is a structural schematic diagram of the testing device provided by the present invention, such as Figure 6 As shown, the device comprises: The terminal side data acquisition unit 610 is used to acquire terminal side data of the user terminal, where the terminal side data includes at least one of user input data and terminal side collected data.

[0079] The interaction information generating unit 620 is used to generate interaction information corresponding to the terminal data based on the terminal large model; the terminal large model is a large language model set at the local device where the user terminal is located.

[0080] The execution unit 630 is configured to execute the interaction information.

[0081] The human-computer interaction device provided by the embodiment of the present invention realizes the user-side self-contained reasoning and decision-making capabilities by locally deploying a large end-side model. It can run efficiently even without a network, ensuring a continuous interactive experience. At the same time, local processing avoids uploading data to the cloud, significantly reducing privacy risks. Combining multi-dimensional data sources to generate personalized responses that better meet user needs also overcomes the limitations of traditional methods that rely on preset rules and lack flexibility.

[0082] Based on any of the foregoing embodiments, the interaction information generating unit is specifically used to: Determining an interaction intent corresponding to the terminal-side data; Based on the end-side big model, interaction information corresponding to the interaction intention and the end-side data is generated.

[0083] In a case where the terminal-side data includes the user input data, the interaction intent corresponding to the terminal-side data is determined based on a previous intention classification model.

[0084] In a case where the terminal side data does not include the user input data, determining whether to trigger an active interaction scenario based on the terminal side data; In the case where it is determined that an active interaction scenario is triggered, an interaction intention corresponding to the terminal side data is determined based on the triggered active interaction scenario.

[0085] Determining a prompt corresponding to the interaction intention; The prompt and the end-side data are input into the end-side large model to obtain the interaction information output by the end-side large model.

[0086] Searching for historical interaction information matching the end-side data from a self-learning cache of the end-side large model, wherein the self-learning cache stores the historical interaction information of the end-side large model; In the case where historical interaction information matching the terminal-side data is found, the historical interaction information matching the terminal-side data is used as the interaction information corresponding to the terminal-side data; If no historical interaction information matching the end-side data is found, interaction information corresponding to the end-side data is generated based on the end-side big model.

[0087] Performing interactive function classification on the terminal side data to obtain an interactive function type corresponding to the terminal side data; In a case where the interaction function type belongs to a candidate interaction function type of the end-side large model, historical interaction information matching the end-side data is searched from the self-learning cache of the end-side large model.

[0088] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the human-computer interaction method, which includes: Acquire terminal side data of the user terminal, where the terminal side data includes at least one of user input data and terminal side collected data; Based on the large model on the end side, generating interaction information corresponding to the end side data; the large model on the end side is a large language model set at the local device where the user end is located; The interactive information is executed.

[0089] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0090] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the human-computer interaction method provided by the above methods, the method comprising: Acquire terminal side data of the user terminal, where the terminal side data includes at least one of user input data and terminal side collected data; Based on the large model on the end side, generating interaction information corresponding to the end side data; the large model on the end side is a large language model set at the local device where the user end is located; The interactive information is executed.

[0091] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the human-computer interaction method provided by the above methods is implemented, and the method includes: Acquire terminal side data of the user terminal, where the terminal side data includes at least one of user input data and terminal side collected data; Based on the large model on the end side, generating interaction information corresponding to the end side data; the large model on the end side is a large language model set at the local device where the user end is located; The interactive information is executed.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human-computer interaction method, characterized in that: include: Acquire terminal side data of the user terminal, where the terminal side data includes at least one of user input data and terminal side collected data; Based on the large model on the end side, generating interaction information corresponding to the end side data; the large model on the end side is a large language model set at the local device where the user end is located; The interactive information is executed.

2. The human-computer interaction method according to claim 1, characterized in that: The generating, based on the end-side big model, interaction information corresponding to the end-side data includes: Determining an interaction intent corresponding to the terminal-side data; Based on the end-side big model, interaction information corresponding to the interaction intention and the end-side data is generated.

3. The human-computer interaction method according to claim 2, characterized in that: The determining the interaction intent corresponding to the terminal-side data includes: In a case where the terminal-side data includes the user input data, the interaction intent corresponding to the terminal-side data is determined based on a previous intention classification model.

4. The human-computer interaction method according to claim 2, characterized in that: The determining the interaction intent corresponding to the terminal-side data includes: In a case where the terminal side data does not include the user input data, determining whether to trigger an active interaction scenario based on the terminal side data; In the case where it is determined that an active interaction scenario is triggered, an interaction intention corresponding to the terminal side data is determined based on the triggered active interaction scenario.

5. The human-computer interaction method according to claim 2, characterized in that: The generating, based on the device-side big model, interaction information corresponding to the interaction intention and the device-side data includes: Determining a prompt corresponding to the interaction intention; The prompt and the end-side data are input into the end-side large model to obtain the interaction information output by the end-side large model.

6. The human-computer interaction method according to any one of claims 1 to 5, characterized in that: The generating, based on the end-side big model, interaction information corresponding to the end-side data includes: Searching for historical interaction information matching the end-side data from a self-learning cache of the end-side large model, wherein the self-learning cache stores the historical interaction information of the end-side large model; In the case where historical interaction information matching the terminal-side data is found, the historical interaction information matching the terminal-side data is used as the interaction information corresponding to the terminal-side data; If no historical interaction information matching the end-side data is found, interaction information corresponding to the end-side data is generated based on the end-side big model.

7. The human-computer interaction method according to claim 6, characterized in that: The searching, from the self-learning cache of the large model on the end side, for historical interaction information matching the end side data includes: Performing interactive function classification on the terminal side data to obtain an interactive function type corresponding to the terminal side data; In a case where the interaction function type belongs to a candidate interaction function type of the end-side large model, historical interaction information matching the end-side data is searched from the self-learning cache of the end-side large model.

8. A human-computer interaction device, characterized in that: include: A terminal side data acquisition unit, used to acquire terminal side data of a user terminal, wherein the terminal side data includes at least one of user input data and terminal side collected data; An interaction information generating unit, configured to generate interaction information corresponding to the terminal data based on a terminal-side large model; the terminal-side large model is a large language model set at a local device where the user terminal is located; An execution unit is used to execute the interaction information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the human-computer interaction method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the human-computer interaction method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the human-computer interaction method according to any one of claims 1 to 7 is implemented.

Images