A personalized voice interaction method, device, MR device and storage medium

Abstract

The present invention discloses a personalized voice interaction method, device, MR device and storage medium, which relates to the field of human-computer interaction. The method includes: responding to the current voice command received from the user, identifying the operation intention in the current voice command; querying the behavior habit label matching the operation intention from the label library; retrieving the resources corresponding to the behavior habit label, and controlling the operation state of the MR device to switch to the device habit state corresponding to the behavior habit label; the device habit state is based on the user's multiple historical voice commands, and the user's historical behavior data and the operation state of the MR device within the preset time before and after each historical voice command is issued. The historical behavior data at least includes the user's operation data on the MR device. The present invention solves the technical problem of the difficulty in performing targeted voice interaction with the user in the prior art.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a personalized voice interaction method, device, MR equipment and storage medium. Background Art

[0002] With the continuous advancement of artificial intelligence technology, especially the development of large model technology, machine learning models have made remarkable achievements in processing natural language tasks. These models can not only understand complex grammatical structures, but also perform a certain degree of logical reasoning, greatly improving the quality of human-computer interaction.

[0003] Even so, command processing based on personalized user intent remains a pressing problem. Although large model technology has significantly improved the understanding ability of voice interaction systems, it still faces challenges in processing personalized intent commands. Users have different language habits, emotional expressions, and background knowledge, resulting in the same command having multiple meanings. For example, "play music" may refer to different types of tracks or artists. In addition, contextual understanding and situational awareness are also key, such as "open the window" means different things under different weather conditions. In order to provide smarter services, the system needs to continuously learn user preferences and build detailed user portraits to improve the recognition accuracy and execution effect of personalized commands.

[0004] Existing intent recognition methods either purely understand the user's current instructions or purely label the user's instructions, making it difficult to conduct targeted voice interaction with the user. Summary of the invention

[0005] The embodiments of the present application provide a personalized voice interaction method, apparatus, MR device and storage medium to solve the current technical problem that it is difficult to perform targeted voice interaction with users.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a personalized voice interaction method, the method comprising:

[0008] In response to receiving a current voice instruction from a user, identifying an operation intention in the current voice instruction;

[0009] Querying a behavior habit tag matching the operation intention from a preset tag library; retrieving a resource corresponding to the behavior habit tag, and controlling the operation state of the MR device to switch to the device habit state corresponding to the behavior habit tag;

[0010] The device habit state is generated based on multiple historical voice commands of the user, as well as the historical behavior data of the user and the operating state of the MR device within a preset time before and after each historical voice command is issued, and the historical behavior data at least includes the user's operation data on the MR device.

[0011] In combination with the first aspect, in a possible design manner, the method further includes:

[0012] Acquire historical behavior data of the user in different operation scenarios and the operating status of the MR device;

[0013] Extracting common behavior information of the user from the historical behavior data of the user in each of the operation scenarios;

[0014] Extracting common status information of the MR device from the operating status of the MR device in each of the operating scenarios;

[0015] generating a device habit state of the MR device in the operation scenario by combining the behavior commonality information and the state commonality information;

[0016] The operation scenario is associated with the device habit state to obtain the behavior habit label, wherein the resource corresponding to the behavior habit label refers to the resource provided by the MR device to the user for use in the operation scenario.

[0017] In combination with the first aspect, in a possible design, the operating state is used to characterize the hardware state and software state of the MR device; and the controlling the operating state of the MR device to switch to the device habit state corresponding to the behavior habit tag includes:

[0018] Based on the device habit state corresponding to the behavior habit tag, the hardware module and software application in the MR device are turned on, turned off or set.

[0019] In combination with the first aspect, in a possible design, the resource includes a primary resource, and the primary resource includes a plurality of secondary resources;

[0020] The behavior habit tags include the primary behavior habit tags corresponding to the primary resources and the secondary behavior habit tags corresponding to the secondary resources;

[0021] The step of searching a behavior habit tag matching the operation intention from a preset tag library; retrieving a resource corresponding to the behavior habit tag, and controlling the operation state of the MR device to switch to the device habit state corresponding to the behavior habit tag includes:

[0022] Matching the operation intention with a secondary behavior habit tag in a preset tag library to obtain a first matching result; if the first matching result indicates that the operation intention matches the secondary behavior habit tag, retrieving resources corresponding to the secondary behavior habit tag, and controlling the operation state of the MR device to switch to the device habit state corresponding to the secondary behavior habit tag;

[0023] If the first matching result indicates that the operation intention does not match the secondary behavior habit label, the operation intention is matched with the primary behavior label in the preset label library to obtain a second matching result. If the second matching result indicates that the operation intention matches the primary behavior habit label, the resources corresponding to the primary behavior habit label are retrieved, and the operating state of the MR device is controlled to switch to the device habit state corresponding to the primary behavior habit label.

[0024] In combination with the first aspect, in a possible design method, querying a behavior habit label matching the operation intention from a preset label library; retrieving a resource corresponding to the behavior habit label, and controlling the operation state of the MR device to switch to the device habit state corresponding to the behavior habit label, further includes:

[0025] If the second matching result indicates that the operation intention does not match the first-level behavior habit label, then the device habit status corresponding to the initialization label is called, or the device habit status corresponding to other behavior habit labels whose similarity with the first-level behavior habit label is greater than a preset threshold is called, and the other behavior habit labels and the initialization label are labels of the same level as the first-level behavior habit label in the label library.

[0026] In combination with the first aspect, in a possible design mode, the operation scenarios include: a photo-taking scenario, a movie-playing scenario, a music-playing scenario, and a variety-playing scenario;

[0027] The first-level behavior habit tag is used to schedule the following resources: camera resource scheduling (such as photography and video recording), film and television resource scheduling (such as movies, TV series, variety shows and short videos), device management scheduling (such as volume, brightness, power on and off, and cache cleaning), music and copyright scheduling (music applications), and social resource scheduling (social applications).

[0028] The first-level behavior habit label can be subdivided into multiple second-level behavior habit labels. Taking movie resource scheduling as an example, its second-level behavior habit label can be classified by film and television type, such as horror, suspense, romance, comedy, animation, action, science fiction, etc. Or it can also be classified by the leading actor, such as director, male lead, female lead, male No. 1, male No. 2, female No. 1, female No. 2, or director, screenwriter, leading actor, supporting actor, etc. Or it can be subdivided by movie tags, such as movie ratings, award-winning types, etc.

[0029] Continuing with the example of movie resource scheduling, when the user says I want to watch a movie starring a certain actor, the MR device recognizes that the user's first intention is "film resource scheduling: movie" and the second intention is "starring: a certain actor". When matching, the second intention will be used as a keyword to search in the tag library. If it is found, the movie starring the actor will be opened. If the second intention is not matched, the first intention "movie" will be matched. If the first intention "movie" is matched, the movie list or movie category will be displayed, allowing the user to browse the movie list or search in the movie category by themselves.

[0030] In combination with the first aspect, in a possible design, the historical behavior data further includes usage time information of the user using the MR device;

[0031] The extracting the user's behavior commonality information from the historical behavior data of the user in each of the operation scenarios; extracting the MR device's state commonality information from the operation state of the MR device in each of the operation scenarios; and generating the device habitual state of the MR device in the operation scenario by combining the behavior commonality information and the state commonality information include:

[0032] Extracting common operation information of the user from the operation information generated by the user operating the MR device in each of the operation scenarios;

[0033] Extracting common usage time information of the user from usage time information of the MR device by the user in each of the operation scenarios;

[0034] Extracting common status information of the MR device from the hardware status and / or software status of the MR device in each of the operation scenarios;

[0035] The device habitual state of the MR device in the operation scenario is generated by combining the operation commonality information, the duration commonality information and the state commonality information.

[0036] In combination with the first aspect, in a possible design manner, the opening, closing or setting operation of the hardware module and the software application in the MR device based on the device habit state corresponding to the behavior habit tag includes:

[0037] Based on the device habit status corresponding to the behavior habit tag, the user's body posture information in the current operation scenario, and the worn command input device, the hardware modules and software applications in the MR device are turned on, off, or set.

[0038] In a second aspect, an embodiment of the present application provides a personalized voice interaction device, the device comprising:

[0039] A semantic recognition module, configured to recognize the operation intention in the current voice instruction in response to the received current voice instruction of the user;

[0040] A tag matching module is used to search for a behavior habit tag that matches the operation intention from a preset tag library;

[0041] A device control module, used to retrieve the resource corresponding to the behavior habit tag, and control the operation state of the MR device to switch to the device habit state corresponding to the behavior habit tag;

[0042] The device habit state is generated based on multiple historical voice commands of the user, as well as the historical behavior data of the user and the operating state of the MR device within a preset time before and after each historical voice command is issued, and the historical behavior data at least includes the user's operation data on the MR device.

[0043] In a third aspect, an embodiment of the present application provides an MR device, comprising at least one processor, a memory, a camera module, and a display module, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the processor to execute the method of the first aspect and its possible design methods.

[0044] In a fourth aspect, an embodiment of the present application provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the method of the first aspect and its possible design manner when running.

[0045] Compared with the prior art, the embodiments of the present application provide a personalized voice interaction method, device, MR device and storage medium, wherein the method is applied to the MR device, the MR device obtains the historical voice instructions sent by the user, performs voice recognition processing on the historical voice instructions, and performs behavioral habit labeling on the user's behavior before and after the historical voice instructions to obtain the user's behavioral habit label. When receiving the user's current voice instruction, the MR device identifies the operation intention in the current voice instruction, queries the behavioral habit label that matches the operation intention, retrieves the resource corresponding to the behavioral habit label to respond to the user, and controls the operation state of the MR device to switch to the device habit state corresponding to the behavioral habit label, in this way, the setting of the MR device conforms to the user's habits. This adjustment does not require manual intervention by the user, thereby improving the user experience. In addition, by constructing a user portrait through behavioral habit labels, it is also possible to avoid misunderstanding of the user's voice instructions and improve the recognition accuracy and execution effect of personalized voice instructions.

[0046] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A diagram showing an application environment of a personalized voice interaction method provided by an embodiment of the present application is shown;

[0049] Figure 2 A flowchart of a personalized voice interaction method provided by an embodiment of the present application is shown;

[0050] Figure 3 A flowchart of another personalized voice interaction method provided by an embodiment of the present application is shown;

[0051] Figure 4 A flowchart of a hierarchical scheduling label provided in an embodiment of the present application is shown;

[0052] Figure 5 A hardware structure diagram of a personalized voice interaction device provided in an embodiment of the present application is shown;

[0053] Figure 6 A hardware structure block diagram of MR glasses provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] The development of technology has brought about tremendous changes in human-computer interaction. Users have increased their demands for intelligent and personalized voice interaction. How to maximize the use of voice interaction to improve the user interaction experience has become a current research hotspot. The solution of this application can be applied in human-computer interaction scenarios, such as human-computer interaction in terminals and human-computer interaction scenarios in vehicle-mounted systems. Figure 1 FIG. 1 shows an application environment diagram of a personalized voice interaction method provided by an embodiment of the present application, such as Figure 1As shown, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can obtain the user's historical behavior data in different operation scenarios and the operating status of the MR device; extract the user's common behavior information from the user's historical behavior data in each operation scenario; extract the MR device's state common information from the MR device's operating status in each operation scenario; combine the behavior common information and the state common information to generate the device habit state of the MR device in the operation scenario; associate the operation scenario with the device habit state to obtain a behavior habit label. Alternatively, the above-mentioned label construction process is put into the server 104 for execution, that is, the server obtains the user's historical behavior data in different operation scenarios and the operating status of the MR device; extracts the user's common behavior information from the user's historical behavior data in each operation scenario; extracts the MR device's state common information from the MR device's operating status in each operation scenario; combines the behavior common information and the state common information to generate the device habit state of the MR device in the operation scenario; associates the operation scenario with the device habit state to obtain a behavior habit label.

[0057] The terminals may include smart phones, smart home appliances, tablet computers, virtual reality headsets (VR headsets), augmented reality glasses (AR glasses), electronic display screens, and mixed reality (MR) devices, etc. MR devices may include MR glasses, MR helmets, MR cameras, etc. The vehicle-mounted system may include vehicle-mounted chips, vehicle-mounted devices (such as vehicle computers, vehicle-mounted computers, sensors with voice recognition functions, etc.), etc.

[0058] Figure 2 A flowchart of a personalized voice interaction method provided by an embodiment of the present application is shown. Figure 2 The method shown can be applied to the process in which a user interacts with an MR device using voice, and the MR device can be MR glasses. The method includes step S201 and step S202.

[0059] Step S201: In response to receiving a current voice instruction from a user, identifying an operation intention in the current voice instruction.

[0060] Step S202: Query the behavior habit label that matches the operation intention from the preset label library; retrieve the resources corresponding to the behavior habit label, and control the operation state of the MR device to switch to the device habit state corresponding to the behavior habit label. The device habit state is generated based on the user's multiple historical voice commands, and the user's historical behavior data and the operation state of the MR device within a preset time before and after each historical voice command is issued. The historical behavior data at least includes the user's operation data on the MR device.

[0061] In this method, the MR device obtains the historical voice commands sent by the user, performs voice recognition processing on the historical voice commands, and labels the user's behavior before and after the historical voice commands to obtain the user's behavior habit label. When receiving the user's current voice command, the MR device identifies the operation intention in the current voice command, queries the behavior habit label that matches the operation intention, retrieves the resource corresponding to the behavior habit label to respond to the user, and controls the operation state of the MR device to switch to the device habit state corresponding to the behavior habit label, so that the setting of the MR device conforms to the user's habits.

[0062] In some of the embodiments, the MR device will record some label behaviors during initialization. When the user performs behavioral operations multiple times, these labels will be updated. In this way, the MR device continuously updates the user's behavioral habit labels and realizes personalized voice interaction with the user based on the behavioral habit labels.

[0063] The following describes various ways in which users can interact with MR glasses.

[0064] The first interaction method:

[0065] When the user issues an instruction, the MR glasses will identify the user's operation intention and match the operation intention with the pre-generated behavior habit label. If a behavior habit label is matched, the MR glasses will not only dispatch the resources corresponding to the behavior habit label (such as opening the application), but also control the running state to switch to the device habit state corresponding to the behavior habit label. For example, when the user issues an instruction to watch a variety show, the MR glasses will open the variety show for the user. In addition, when the MR glasses play the variety show, they will call the MR glasses setting habits corresponding to the user's habit label and adjust some corresponding playback parameters, such as opening the video barrage software, adjusting the barrage to medium density, 5 lines and normal font. Messages not in the current software are not pushed, but social application message push is turned on. Because the user watches this variety show for more than 30 minutes each time, the current power is not enough to support it, and the user will be prompted to charge in advance to ensure the quality of the viewing. At the same time, the hardware parameters of the MR glasses are controlled, such as adjusting the volume to 55% and the brightness to 65%. Turn off other hardware that is not related to weakening video playback, and give priority to providing video playback effects, such as microphone reception, turning off the gyroscope, turning on GPU hardware video acceleration, and adjusting the unity engine rendering mode. For another example, the behavior habit label indicates that the user likes voice control while watching variety shows, and the MR glasses can control the shooting module 103, the gesture control module, and other interactive modules except the microphone to be turned off.

[0066] The second interaction method:

[0067] When the user issues an instruction, MR will determine the current operating state of the smart device, which is used to characterize the hardware state and software state of the smart device, and determine the user's body posture at this time through the shooting module 103 and the IMU module 106 to determine whether to turn on the shooting module 103 and the IMU module (or Inertial Measurement Unit) 106 in real time. For example, when the user is in a sitting or lying state, the shooting module 103 is turned off. When the threshold parameter is exceeded, it is determined that the user may be moving, and the shooting module 103 is turned on to display the real interface for the user.

[0068] The third interaction method:

[0069] After the user issues an instruction, it is detected whether the user is wearing an instruction input device, and the instruction input device that meets the preset requirements among the user's multiple instruction input devices is turned off in combination with the user's behavioral habit tags. The preset requirements include turning off instruction input devices that are used less frequently, or turning off instruction input devices with lower sensitivity. For example, if it is identified that the user is wearing a Bluetooth ray ring, and it is determined based on the user's behavioral habit tags that the user is accustomed to voice interaction, the Bluetooth ray ring that the user does not often use is turned off. For another example, if it is identified that the user is wearing a neural ray wristband and a Bluetooth ray ring at the same time, the two devices may emit multiple rays at the same time, affecting the actual operation. The MR glasses will intelligently turn off the Bluetooth ray ring that the user does not often use, and replace it with a more sensitive neural ray wristband.

[0070] It is worth mentioning that the above three interaction methods can be combined in multiple ways to realize the intelligent response of MR glasses to user voice commands. As an example, the above three interaction methods are combined. When the user issues a command, the MR glasses open, close or set the hardware modules and software applications in the MR glasses based on the device habit state corresponding to the behavior habit label (corresponding to the first interaction method), the user's body posture information in the current operation scenario (corresponding to the second interaction method) and the command input device worn (corresponding to the third interaction method).

[0071] During the interaction between the user and the MR glasses, the MR glasses will also synchronize according to the user's instructions and adjust to the MR glasses labels that the user is accustomed to:

[0072] The user's operations on the MR glasses will be stored locally, and the large model will regularly analyze the association between the label and the device habit state to analyze the reasons for such user operations. For example, the user will always listen to and watch a certain TV series for a long time without any instructions, and will not take off the MR glasses even if the battery is running out or the TV series is over. The label determines that the user will fall asleep while watching this TV series. Therefore, when the user watches it for more than 10:30 at night, the low battery and excessive eye use reminders will be displayed silently in text state, without playing voice and interrupting the video. The cause analysis of such labels is used for user behavior habit analysis and local end-side intention prediction.

[0073] The behavior habit labels involved in the embodiments of the present application are derived from a series of user behavior data and the operating status of the MR glasses when the user interacts with the MR glasses in voice in each operation scenario, and the device habit status is extracted from it, and the operation scenario is associated with the device habit status to obtain the behavior habit label and store it locally. Then when the user issues a similar or identical voice command again, even if the user's voice command is not clear enough, the MR glasses can still query the behavior habit label that matches the voice command from the label library, and retrieve the resources corresponding to the behavior habit label for the user. After the MR glasses record the user's behavior habit label, the hardware settings and software settings of the MR glasses can be automatically adjusted to match the user's habits. This adjustment does not require manual intervention by the user, thereby improving the user experience. In addition, by constructing a user portrait through behavioral habit labels, it is also possible to avoid misunderstandings of the user's voice commands, thereby improving the recognition accuracy and execution effect of personalized voice commands.

[0074] Combine the following Figure 3 The method provided in the embodiments of the present application is described. Figure 3 A flowchart of another personalized voice interaction method provided in an embodiment of the present application is shown. Figure 3 As shown, the method includes steps S301 to S308.

[0075] Step S301: Obtain historical behavior data of the user in different operation scenarios and the operating status of the MR glasses.

[0076] The operation scenarios include: photo shooting, movie playing, music playing, and variety show playing. The following takes the variety show playing scenario as an example to illustrate this step.

[0077] The terminal is a smart device equipped with voice control function (such as MR glasses). The user issues a voice command "I want to watch ×× variety show". The MR glasses play ×× variety show for the user and record the user's behavior data (i.e. historical behavior data) in the variety show playback scenario. At the same time, it records the operating status of the MR glasses, such as the on / off status and parameter settings of the hardware module, and the on / off status and application settings of the software application.

[0078] As an example, in the variety show playback scene, the MR glasses record that the user has turned on the video barrage, adjusted the barrage to medium density, 5 lines and normal font, and the user has turned on the push-free message of non-current software and the message push of social applications. The MR glasses also record the continuous length of time the user watches this variety show each time. And also record the hardware parameters of the MR glasses, such as the volume is 55%, the brightness is 65%, the gyroscope is turned off, the GPU hardware video acceleration is turned on, and the unity engine is currently in variety show rendering mode.

[0079] Step S302: extracting common behavior information of users from historical behavior data of users in each operation scenario.

[0080] Continuing with the example of the operation scenario being the variety show playback scenario, in some embodiments, the historical behavior data corresponding to the user's multiple identical operations are used as the user's common behavior information. For example, if the user turns on the barrage to watch variety shows multiple times, the user's common behavior information includes: turning on the video barrage. In other embodiments, if the user's operations are different each time, the user's common behavior information is analyzed from multiple historical behavior data. If the user watches the variety show three times, and adjusts the video barrage font once each time, the first time the font size is adjusted to a normal font, the second time the font size is adjusted to a large font, and the third time the font size is adjusted to a small font. Then the MR glasses can analyze the correlation between the variety shows watched by the user and the user's adjustment of the barrage font size. If the analysis shows that the purpose of the user adjusting the barrage font size is to prevent the barrage from being blocked, the user's common behavior information includes: the barrage font size is adaptively adjusted based on the purpose of anti-blocking.

[0081] In some embodiments, the historical behavior data also includes information on the duration of use of the MR device by the user, and step S302 further includes: extracting common operation information of the user from the operation information generated by the user operating the MR device in each operation scenario; extracting common duration information of the user from the usage duration information of the MR device in each operation scenario.

[0082] This embodiment extracts the user's usage time habits, that is, how long the user usually uses the device, by analyzing the continuous usage time of the user's multiple uses of the MR glasses. This information helps to predict the power consumption of the device. For example, if the user usually uses the device for a long time, and the device power is not enough to support the expected usage time, the MR glasses can actively display a prompt message to remind the user that the battery is low. In this way, the user can get a low battery warning in advance during use, thereby avoiding sudden interruptions caused by exhaustion of power. At the same time, this intelligent reminder mechanism can also enhance the intelligent management of MR glasses, be more in line with the user's usage habits, and reduce the inconvenience caused by power problems.

[0083] Step S303: extracting common status information of the MR glasses from the operating status of the MR glasses in each operating scenario.

[0084] The operating status includes the hardware status and / or software status. The hardware status refers to the on state, off state and parameter settings of the hardware module of the MR glasses. The software status refers to the on state, off state and parameter settings of the software application of the MR glasses.

[0085] This step further includes: extracting common status information of the MR device from the hardware status and / or software status of the MR device in each operation scenario.

[0086] Step S304: Generate a device habit state of the MR glasses in the operation scenario by combining the behavior commonality information and the state commonality information.

[0087] Regarding steps S302 to S304, in this embodiment, a large model can be used to identify similar behavioral features from a large amount of historical behavior data of the user, and generate a device habit state. Specifically, a large model refers to an artificial intelligence model trained based on a large amount of data and a deep learning algorithm. It can understand and analyze patterns in the data in multiple dimensions, and make inferences and predictions based on these patterns. For example, the LLM large language model (Large Language Model) under the Transformer architecture and the BERT (Bidirectional Encoder Representations from Transformers) pre-trained intent recognition small model. The large model is used to analyze multiple historical behavior data of the user, identify the potential commonalities of multiple operation data, and generate the user's habitual settings for MR glasses in each operation scenario, that is, generate the device habit state.

[0088] Specifically, this step includes: inputting multiple historical behavior data into the preset big model for feature extraction to obtain the feature information corresponding to each historical behavior data; extracting the common information of multiple feature information to obtain the user's device habit status in each operation scenario. Through the judgment and generation of user portraits by the big model, it is possible to better assist voice recognition based on the user's historical operations and adopt intelligent response logic to provide users with application response logic that is closer to their operating habits, thereby improving user experience. Based on the user behavior labels sorted out after the big model judgment and intervention, after voice interaction, relevant resources are retrieved through the labels to better realize human-computer interaction.

[0089] Step S305: associate the operation scenario with the device habit state to obtain a behavior habit label.

[0090] Exemplarily, the MR glasses can generate a "variety show playing preference" tag for the user by associating the variety show playing scene with the device habit state of the MR glasses when the user watches the variety show. The variety show resources corresponding to the "variety show playing preference" are called, and the hardware modules and software applications of the MR glasses are turned on, off or set to adapt to the user's habits when watching variety shows. Alternatively, the MR glasses can generate a "photographing preference" tag for the user by associating the photo taking scene with the device habit state of the MR glasses when the user takes photos (such as commonly used photo taking parameters). The camera application corresponding to the "photographing preference" is called, and the hardware modules and software applications of the MR glasses are turned on, off or set to adapt to the user's habits when taking photos.

[0091] In some embodiments, these tags are first-level tags, and second-level tags can be further subdivided under the first-level tags. Different levels of tags represent different levels of classification. The first-level tag is the highest-level tag in the tag library, and the second-level tag is a subdivided tag under the first-level tag, which is used to further refine the content in the first-level tag. In addition, third-level tags and fourth-level tags can be used to further distinguish. For example, but not limitation, behavioral habit tags include first-level behavioral habit tags corresponding to first-level resources, and second-level behavioral habit tags corresponding to second-level resources. First-level behavioral habit tags such as "variety show playback preferences", "movie playback preferences", "music playback preferences", "photography preferences", etc. Among them, the "movie playback preferences" tag can be subdivided into second-level behavioral habit tags such as "horror movie playback preferences", "comedy playback preferences", and "suspense playback preferences". When matching the operation intention and the tag, the tag with the lowest level will be matched first. If it is not matched, it will be matched upward one level at a time, such as matching the operation intention and the second-level behavioral habit tag first, and then matching the operation intention and the first-level behavioral habit tag to obtain a more accurate matching result.

[0092] The above steps S301 to S305 illustrate the process of constructing behavior habit labels. It is understandable that the behavior habit labels can be pre-constructed and stored in the label library, or they can be constructed after receiving the user's current voice command, that is, the labels are generated in real time every time the user issues a voice command. This application does not limit this. After constructing the behavior habit labels, the MR glasses execute step S306, as described below.

[0093] Step S306: In response to receiving the current voice instruction of the user, identifying the operation intention in the current voice instruction.

[0094] Specifically, the current voice command refers to the command or request that the user sends to the MR glasses through voice at the current moment. These contents are usually in the form of natural speech, so the MR glasses understand the operational intention expressed by the current voice command by extracting semantic information. Extracting semantic information is not only about recognizing the text in the voice (i.e., voice recognition), but also about understanding the meaning of these words in context. For example, when the user says "Let's watch a movie", the voice is first converted into text, and then the operational intention is extracted. The operational intention is that the user wants to watch a movie.

[0095] In this step, the semantic information in the current voice command can be extracted through the big model. Although the big model technology significantly improves the big model's ability to recognize voice, the same voice may refer to different meanings for different users. In order to provide a more intelligent service, after step S306, the MR glasses combine behavioral habit tags to achieve intelligent response. For details, please refer to the description of step S307.

[0096] Step S307: Query a behavior habit tag that matches the operation intention from a preset tag library.

[0097] In this step, the tag library is pre-built. After the operation intention is identified, the MR glasses match the operation intention with each behavioral habit tag in the tag library. If a match is found, the method of responding to the user is determined based on the matched behavioral habit tag.

[0098] As an example, whether there is a match can be determined by the degree of similarity between the operation intention and the existing behavioral habit tags in the tag library. The similarity threshold is used to determine whether the match is high enough to trigger subsequent operations. If the match is greater than the similarity threshold, it means that the user's voice command meets a behavioral habit tag, and the MR glasses will execute the relevant response. If the match is less than or equal to the similarity threshold, the MR glasses may refuse to perform the operation or provide other responses.

[0099] Step S308, retrieve the resources corresponding to the behavior habit tag, and control the operation state of the MR glasses to switch to the device habit state corresponding to the behavior habit tag.

[0100] Specifically, this step opens, closes or sets the hardware modules and software applications in the MR device based on the device habit status corresponding to the behavior habit tag.

[0101] Specifically, resources include: camera resources, movie resources, music resources, and variety show resources. For example, in the photo taking scenario, the resource is the camera resource, and in the movie playing scenario, the resource is the movie resource.

[0102] In this step, tags can be divided into primary tags and secondary tags. For an introduction to tags, please refer to the introduction to S305 above. The following mainly describes how to schedule resources corresponding to tags and device habit states in the step.

[0103] Please refer to Figure 4 , Figure 4 A flowchart of a hierarchical scheduling label provided in an embodiment of the present application is shown. Steps S307 to S308 may further include:

[0104] Step S401, match the operation intention with the secondary behavior habit tag in the preset tag library to obtain a first matching result. If the first matching result indicates that the operation intention matches the secondary behavior habit tag, retrieve the resources corresponding to the secondary behavior habit tag, and control the operating state of the MR glasses to switch to the device habit state corresponding to the secondary behavior habit tag.

[0105] Step S402: If the first matching result indicates that the operation intention does not match the secondary behavior habit label, the operation intention is matched with the primary behavior label in the preset label library to obtain a second matching result. If the second matching result indicates that the operation intention matches the primary behavior habit label, the resources corresponding to the primary behavior habit label are retrieved, and the operating state of the MR glasses is controlled to switch to the device habit state corresponding to the primary behavior habit label.

[0106] Step S403: If the second matching result indicates that the operation intention does not match the first-level behavior habit label, then call the device habit state corresponding to the initialization label, or call the device habit state corresponding to other behavior habit labels whose similarity with the first-level behavior habit label is greater than a preset threshold, and the other behavior habit labels and the initialization label are labels of the same level as the first-level behavior habit label in the label library.

[0107] Exemplarily, the first-level behavior habit tag is used to schedule the following resources: camera resource scheduling (such as photography and video recording), film and television resource scheduling (such as movies, TV series, variety shows and short videos), device management scheduling (such as volume, brightness, power on and off, and cache clearing), music and copyright scheduling (music applications), and social resource scheduling (social applications).

[0108] The first-level behavior habit label can be subdivided into multiple second-level behavior habit labels. Taking movie resource scheduling as an example, its second-level behavior habit label can be classified by film and television type, such as horror, suspense, romance, comedy, animation, action, science fiction, etc. Or it can also be classified by the leading actor, such as director, male lead, female lead, male No. 1, male No. 2, female No. 1, female No. 2, or director, screenwriter, leading actor, supporting actor, etc. Or it can be subdivided by movie tags, such as movie ratings, award-winning types, etc.

[0109] Continuing with the example of movie resource scheduling, when the user says I want to watch a movie starring a certain actor, the MR device recognizes that the user's first intention is "film resource scheduling: movie" and the second intention is "starring: a certain actor". When matching, the second intention will be used as a keyword to search in the tag library. If it is found, the movie starring the actor will be opened. If the second intention is not matched, the first intention "movie" will be matched. If the first intention "movie" is matched, the movie list or movie category will be displayed, allowing the user to browse the movie list or search in the movie category by themselves.

[0110] As another example, the MR glasses preset initialization tags, and the MR glasses will match the tags generated based on the user's historical behavior data as the highest priority tags. When these tags are not matched, the MR glasses will search for the initialization tags preset by the MR glasses.

[0111] Through the above steps S301 to S308, the MR glasses record the user's historical voice commands and corresponding operations in each voice interaction with the user, and analyze the user's operations through a large model to identify the user's behavioral commonalities (or usage habits of MR glasses, i.e., device habit status), and generate behavioral habit labels. MR glasses parse the intention of the current voice command and understand the user's needs. MR glasses match the generated user behavior habit labels according to the operation intention of the current voice command. If the matching degree is higher than the similarity threshold, it means that the label matching is successful, and the MR glasses provide personalized responses based on the labels. Personalized response specifically means that after the MR glasses record the user's behavior habit labels, the hardware settings and software settings of the MR glasses can be automatically adjusted to meet the user's habits. This adjustment does not require manual intervention by the user, thereby improving the user experience. In addition, by building a user portrait through behavioral habit labels, it is also possible to avoid misunderstandings of the user's voice commands and improve the recognition accuracy and execution effect of personalized voice commands.

[0112] In some embodiments, the historical behavior data also includes the continuous usage time of the MR glasses by the user. The MR device extracts the common information of multiple continuous usage times of the MR glasses by the user to obtain the device habitual usage time of the MR glasses by the user; when the remaining power of the MR glasses is lower than the estimated consumption time of the device habitual usage time, a prompt message is displayed after the resource corresponding to the behavior habit tag is called to indicate that the remaining power is insufficient.

[0113] In some embodiments, the MR glasses also control the hardware modules and software applications to turn on, turn off or set operations according to the body posture. For example, the MR glasses obtain the body posture recognized by the image data collected by the shooting module; based on the category of the body posture, the MR glasses are controlled to respond to the action corresponding to the body posture.

[0114] This embodiment recognizes the user's body posture through the image data collected by the shooting module, in this way the MR glasses can intelligently perform operations that match these postures. For example, when the user is in a lying or sitting posture, the shooting module is turned off to protect the user's privacy. Alternatively, when the user makes a specific gesture or body movement, the MR glasses can recognize the head / hand posture and respond automatically, such as in gesture control, the user raises his hand to trigger brightness or volume adjustment.

[0115] This embodiment collects the user's body posture to respond to the corresponding action, so that the MR glasses can interact with the user more intelligently, improving the convenience and intuitiveness of operation. In addition, this interaction method based on body posture can improve the user experience, reduce the dependence on traditional input methods such as touch and voice, make the operation more natural and smooth, especially in watching videos, sports, games or scenes where you need to free your hands, it can provide a better experience.

[0116] In some of the embodiments, the above-mentioned body posture category-based control of the MR glasses to respond to actions corresponding to the body posture includes: when the body posture category indicates that the user is in a sitting or lying state, controlling the MR glasses to turn off the shooting module, wherein the IMU of the MR glasses is provided with a threshold parameter for controlling the shooting module to turn on when the posture parameter exceeds the threshold parameter.

[0117] This embodiment automatically turns off the shooting module when it recognizes that the user is in a sitting or lying state, which saves power and reduces unnecessary privacy interference. At the same time, combined with the IMU threshold parameters, it ensures that the shooting module is turned on in time when the user's posture changes, thereby improving the intelligence of the device.

[0118] In some of the embodiments, after matching the tag, the MR glasses will not directly complete the current round of work, but will record the operating status of the MR glasses and compare it with the previous operating status of the same type in the form of a tag to determine the common behavioral habits of the user when watching film and television resources multiple times, and finally generate an operating status tag for the MR glasses.

[0119] This embodiment extracts the user's device usage habits, so that when the user issues a voice command, the MR glasses can intelligently adjust the device status based on these habits. For example, when the user is watching film and television resources, the MR device can know that the user prefers a certain volume or brightness setting, or likes to open certain specific applications, and provide a response. This personalized service can make the device more in line with user habits and improve the convenience and comfort of operation. For another example, when the user does not need high-performance operation, the system may automatically reduce the operating frequency of certain hardware, thereby saving power and extending the battery life of the device.

[0120] This embodiment combines historical voice commands with the operating status of MR glasses to extract the user's device usage habits, and can intelligently optimize the status and operation of the device. By matching the operation intention and behavior habit labels, MR glasses can automatically adjust the hardware and software settings without too much intervention, save power consumption, and improve the accuracy of voice recognition. This not only enhances the intelligent voice interaction level of MR glasses, but also provides users with a more convenient and personalized user experience.

[0121] The method provided in the embodiment of the present application is described below with a specific example. The user interacts with the MR glasses through voice, and the MR glasses generate a behavior habit label for the user and store it in the behavior label library. When the user starts to ask questions, the local model (large model) of the MR glasses will match different operation intentions with the behavior label library, identify the user's intentions, and determine the workflow mode corresponding to the label. Specifically, when the user issues the instruction "I want to watch ××", the user's behavior habit label includes variety shows; it is determined that the instruction meets the label of "variety show playback preferences", and the MR glasses play variety shows for the user, and adjust the settings of the MR glasses to meet the user's variety show playback preferences.

[0122] A personalized voice interaction method provided in an embodiment of the present application has been described in detail above. A personalized voice interaction device is introduced below.

[0123] Please refer to Figure 5 , Figure 5 FIG. 1 shows a hardware structure diagram of a personalized voice interaction device provided in an embodiment of the present application. Figure 5 As shown, the device comprises:

[0124] The semantic recognition module 501 is used to recognize the operation intention in the current voice instruction in response to the current voice instruction received from the user;

[0125] The tag matching module 502 is used to search for a behavior habit tag matching the operation intention from a preset tag library;

[0126] The device control module 503 is used to retrieve the resources corresponding to the behavior habit tag and control the operation state of the MR device to switch to the device habit state corresponding to the behavior habit tag;

[0127] The device habit state is generated based on the user's multiple historical voice commands, as well as the user's historical behavior data and the operating state of the MR device within a preset time before and after each historical voice command is issued. The historical behavior data at least includes the user's operation data on the MR device.

[0128] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0129] The present application also provides an MR device, such as Figure 6 As shown, the MR glasses include a processor 601 , a memory 602 , a shooting module 603 , a display module 604 , an interaction module 605 , an IMU module 606 , a gyroscope 607 , a geomagnetic sensor 608 and a wireless communication module 609 .

[0130] The memory 602 can be used to store computer programs, such as software programs and modules of application software. The processor 601 executes various functional applications and data processing by running the computer programs stored in the memory 602, such as implementing the method provided in the embodiment of the present application.

[0131] The shooting module 603 is mainly responsible for shooting the image information of the real environment and obtaining the user's gesture information, etc. The shooting module can be a camera.

[0132] The display module 604 is used to display virtual images and augmented reality contents, and includes a micro-projection system and optical elements.

[0133] The interactive module 605 includes a microphone, an eye tracking sensor, a finger ring, a wristband, etc.

[0134] The wireless communication module 609 includes: 4G / 5G full network access, WiFi, Bluetooth, etc.

[0135] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only and does not limit the structure of the MR glasses. Figure 6 More or fewer components as shown, or with Figure 6 Different configurations are shown.

[0136] The following is a brief introduction to the application scenarios of MR glasses.

[0137] In the voice interaction scenario of MR glasses, users can control MR glasses by voice, for example, users send voice commands to the digital human in MR glasses to control MR glasses. In some possible examples, users can use voice to open applications such as movies, games, photos, and music, adjust the volume, turn on or off subtitles, turn on or off Bluetooth, etc.

[0138] In addition, in combination with the method provided in the above embodiment, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the personalized voice interaction methods in the above embodiment is implemented.

[0139] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0140] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0141] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0142] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A personalized voice interaction method, characterized in that: Applied to MR equipment, the method comprises: In response to receiving a current voice instruction from a user, identifying an operation intention in the current voice instruction; Querying a behavior habit tag matching the operation intention from a preset tag library; retrieving a resource corresponding to the behavior habit tag, and controlling the operation state of the MR device to switch to the device habit state corresponding to the behavior habit tag; The device habit state is generated based on a plurality of historical voice commands of the user, and historical behavior data of the user and the operating state of the MR device within a preset time before and after each of the historical voice commands is issued, wherein the historical behavior data at least includes operation data of the user on the MR device; The method further comprises: Acquire historical behavior data of the user in different operation scenarios and the operating status of the MR device; Extracting common behavior information of the user from the historical behavior data of the user in each of the operation scenarios; Extracting common status information of the MR device from the operating status of the MR device in each of the operating scenarios; generating a device habit state of the MR device in the operation scenario by combining the behavior commonality information and the state commonality information; The operation scenario is associated with the device habit state to obtain the behavior habit label.

2. The personalized voice interaction method according to claim 1, characterized in that: The operating status is used to characterize the hardware status and software status of the MR device; The controlling the operating state of the MR device to switch to the device habit state corresponding to the behavior habit tag includes: Based on the device habit state corresponding to the behavior habit tag, the hardware module and software application in the MR device are turned on, turned off or set.

3. The personalized voice interaction method according to claim 1, characterized in that: The resources include primary resources, and the primary resources include multiple secondary resources; The behavior habit tags include the primary behavior habit tags corresponding to the primary resources and the secondary behavior habit tags corresponding to the secondary resources; The step of searching a preset tag library for a behavior habit tag that matches the operation intention; Retrieving the resource corresponding to the behavior habit tag, and controlling the operation state of the MR device to switch to the device habit state corresponding to the behavior habit tag, including: Matching the operation intention with a secondary behavior habit tag in a preset tag library to obtain a first matching result; if the first matching result indicates that the operation intention matches the secondary behavior habit tag, retrieving resources corresponding to the secondary behavior habit tag, and controlling the operation state of the MR device to switch to the device habit state corresponding to the secondary behavior habit tag; If the first matching result indicates that the operation intention does not match the secondary behavior habit label, the operation intention is matched with the primary behavior label in the preset label library to obtain a second matching result. If the second matching result indicates that the operation intention matches the primary behavior habit label, the resources corresponding to the primary behavior habit label are retrieved, and the operating state of the MR device is controlled to switch to the device habit state corresponding to the primary behavior habit label.

4. The personalized voice interaction method according to claim 3, characterized in that: The step of searching a behavior habit tag matching the operation intention from a preset tag library; retrieving a resource corresponding to the behavior habit tag, and controlling the operation state of the MR device to switch to the device habit state corresponding to the behavior habit tag, further includes: If the second matching result indicates that the operation intention does not match the first-level behavior habit label, then the device habit status corresponding to the initialization label is called, or the device habit status corresponding to other behavior habit labels whose similarity with the first-level behavior habit label is greater than a preset threshold is called, and the other behavior habit labels and the initialization label are labels of the same level as the first-level behavior habit label in the label library.

5. The personalized voice interaction method according to claim 1, characterized in that: The historical behavior data also includes usage time information of the user using the MR device; extracting common behavior information of the user from the historical behavior data of the user in each of the operation scenarios; Extracting common status information of the MR device from the operating status of the MR device in each of the operating scenarios; Generating a device habit state of the MR device in the operation scenario in combination with the behavior commonality information and the state commonality information includes: Extracting common operation information of the user from the operation information generated by the user operating the MR device in each of the operation scenarios; Extracting common usage time information of the user from usage time information of the MR device by the user in each of the operation scenarios; Extracting common status information of the MR device from the hardware status and / or software status of the MR device in each of the operation scenarios; The device habitual state of the MR device in the operation scenario is generated by combining the operation commonality information, the duration commonality information and the state commonality information.

6. The personalized voice interaction method according to claim 2, characterized in that: The opening, closing or setting operation of the hardware module and the software application in the MR device based on the device habit state corresponding to the behavior habit tag includes: Based on the device habit status corresponding to the behavior habit tag, the user's body posture information in the current operation scenario, and the worn command input device, the hardware modules and software applications in the MR device are turned on, off, or set.

7. A personalized voice interaction device, characterized in that: The device comprises: A semantic recognition module, configured to recognize the operation intention in the current voice instruction in response to the received current voice instruction of the user; A tag matching module is used to search for a behavior habit tag that matches the operation intention from a preset tag library; A device control module, used to retrieve the resource corresponding to the behavior habit tag, and control the operation state of the MR device to switch to the device habit state corresponding to the behavior habit tag; The device habit state is generated based on a plurality of historical voice commands of the user, and historical behavior data of the user within a preset time before and after each of the historical voice commands is issued, and an operating state of the MR device, wherein the historical behavior data at least includes operation data of the user on the MR device; The device also includes: A historical data processing module, used to obtain the historical behavior data of the user in different operation scenarios and the operating status of the MR device; The historical data processing module is further used to extract the user's common behavior information from the user's historical behavior data in each of the operation scenarios; The historical data processing module is further used to extract common status information of the MR device from the operating status of the MR device in each of the operating scenarios; The historical data processing module is further used to generate a device habit state of the MR device in the operation scenario by combining the behavior commonality information and the state commonality information; The historical data processing module is also used to associate the operation scenario with the device habit state to obtain the behavior habit label.

8. An MR device, characterized in that: It includes at least one processor, a memory, a camera module and a display module. The at least one processor is coupled to the memory and is used to read and execute instructions in the processor to execute the personalized voice interaction method as described in any one of claims 1 to claim 6.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the personalized voice interaction method according to any one of claims 1 to 6 when running.

Images