Touch screen and voice fusion interaction method and device, computer equipment and readable storage medium

Abstract

The invention discloses a touch screen and voice fusion interaction method and device, computer equipment and a readable storage medium, and the method comprises the steps: firstly obtaining a target demand instruction of touch screen or voice of a user, sending the target demand instruction to a central control service for processing to obtain a processed instruction, then sending the processed instruction to a multi-mode interaction engine to obtain a user demand feedback result, and sending the user demand feedback result to the central control service; and finally, the central control service processes the result to obtain and display a final demand feedback result. According to the method, two interaction modes are fused, and the flexibility, accuracy and convenience of interaction are improved so as to adapt to user requirements in different scenes.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interaction, and in particular to a touch screen and voice fusion interaction method, device, computer equipment and readable storage medium. Background Art

[0002] With the widespread use of smart devices, user interaction methods are constantly evolving. Touch screen technology provides an intuitive operating interface, is suitable for visual operation and is widely used in various devices. The accuracy of voice technology has been greatly improved in recent years, and it can easily realize human-computer interaction in many scenarios. However, a single interaction method has limitations. For example, in some scenarios, touch screen operation may be affected by the environment or the user's physical condition (such as poor eyesight or both hands are occupied), and voice recognition is interfered by factors such as noise. Summary of the invention

[0003] The object of the present invention is to provide a touch screen and voice fusion interaction method, device, computer equipment and readable storage medium.

[0004] In a first aspect, an embodiment of the present invention provides a touch screen and voice fusion interaction method, including:

[0005] Obtaining a target demand instruction input by a target user, wherein the target demand instruction is a touch screen instruction or a voice instruction;

[0006] Sending the target demand instruction to the central control service for processing to obtain the processed target demand instruction;

[0007] Sending the processed target demand instruction to the multimodal interaction engine for processing to obtain a user demand feedback result;

[0008] The user demand feedback result is processed by the central control service to obtain the final demand feedback result and display it.

[0009] In a possible implementation manner, when the target demand instruction is the voice instruction, sending the target demand instruction to the central control service for processing to obtain the processed target demand instruction includes:

[0010] The central control service calls the ASR service to convert the voice command to obtain a text command corresponding to the voice command.

[0011] In a possible implementation, sending the processed target demand instruction to a multimodal interaction engine for processing to obtain a user demand feedback result includes:

[0012] Extracting information and understanding semantics of the text instruction through the multimodal interaction engine to obtain key information of the text instruction;

[0013] The multimodal interaction engine calls a pre-trained dialogue model to process the key information of the text instruction to obtain a text feedback result corresponding to the text instruction.

[0014] In a possible implementation, the processing of the user demand feedback result by the central control service to obtain and display the final demand feedback result includes:

[0015] The central control service calls the TTS service and the TTSA service to convert the text feedback result to obtain the corresponding voice feedback result and the virtual human feedback result;

[0016] Perform streaming voice broadcast based on the voice feedback result, and control the virtual human to execute the virtual human feedback result.

[0017] In a possible implementation, the method further includes:

[0018] The target demand instruction is synchronized to the Redis database in real time for caching, and an expiration time and index strategy for the target demand instruction are configured.

[0019] In a possible implementation, the method further includes:

[0020] Obtaining the historical behavior and user profile of the target user;

[0021] The historical behavior and the user portrait are used as optimization data for the conversation model.

[0022] In a possible implementation, the target demand instruction includes a health monitoring instruction, and the final demand feedback result includes a health feedback result.

[0023] In a second aspect, an embodiment of the present invention provides a touch screen and voice fusion interaction device, including:

[0024] An acquisition module is used to acquire a target demand instruction input by a target user, wherein the target demand instruction is a touch screen instruction or a voice instruction;

[0025] The interactive module is used to send the target demand instruction to the central control service for processing to obtain the processed target demand instruction; send the processed target demand instruction to the multimodal interaction engine for processing to obtain the user demand feedback result; process the user demand feedback result through the central control service to obtain the final demand feedback result and display it.

[0026] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the method described in the first aspect.

[0027] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method described in the first aspect.

[0028] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a touch screen and voice fusion interaction method, device, computer equipment and readable storage medium disclosed in the present invention, by obtaining the target demand instruction of the user's touch screen or voice, sending it to the central control service for processing to obtain the processed instruction, and then sending it to the multimodal interaction engine to obtain the user demand feedback result, and finally the central control service processes the result to obtain the final demand feedback result and display it. This method integrates two interaction modes, improves the flexibility, accuracy and convenience of interaction, and adapts to user needs in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0030] Figure 1 A schematic diagram of the steps of the touch screen and voice fusion interaction method provided by an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a framework for touch screen and voice fusion interaction provided by an embodiment of the present invention;

[0032] Figure 3 A schematic block diagram of the structure of a touch screen and voice fusion interaction device provided in an embodiment of the present invention;

[0033] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0035] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0036] In order to solve the technical problems in the aforementioned background technology, Figure 1 The present invention provides a flow chart of a touch screen and voice fusion interaction method according to an embodiment of the present invention. The touch screen and voice fusion interaction method is introduced in detail below.

[0037] Step S201, obtaining a target demand instruction input by a target user, wherein the target demand instruction is a touch screen instruction or a voice instruction;

[0038] Step S202, sending the target demand instruction to the central control service for processing to obtain a processed target demand instruction;

[0039] Step S203, sending the processed target demand instruction to the multimodal interaction engine for processing to obtain a user demand feedback result;

[0040] Step S204: Process the user demand feedback result through the central control service to obtain the final demand feedback result and display it.

[0041] In an embodiment of the present invention, illustratively, in the scenario of smart elderly care, the room where the elderly live is equipped with a smart device terminal with a touch screen function, such as a smart bedside terminal. The terminal screen displays icons of various service options, which clearly indicate the functions, such as "make a dining appointment", "make a physical therapy appointment", "check blood pressure", etc. When the elderly want to check their blood pressure data, they directly touch the "check blood pressure" icon on the screen with their fingers. At this time, the server receives this touch screen instruction, which is a clear operation signal, indicating that the elderly want to obtain blood pressure data.

[0042] Touch screen interactive devices are also set up in the waiting area of ​​medical institutions. Patients can use the touch screen to select and view the queue status of different departments, doctor information, etc. For example, if a patient wants to view the queue status of the internal medicine department, he clicks the "Internal Medicine Queue Query" button on the touch screen device, and the server will receive this touch screen instruction. This is a touch screen operation for specific information query.

[0043] In the smart home scenario, the smart central control touch screen device in the home displays the control options of home devices, such as lighting control, air conditioning temperature adjustment, etc. When a family member wants to adjust the brightness of the living room light, he slides the light brightness adjustment slider on the touch screen device, and the server obtains the touch screen instruction to adjust the light brightness.

[0044] Still in the smart elderly care scenario, an elderly person with poor eyesight wants to make a reservation for catering services. He speaks to the microphone of the smart device: "I want to make a reservation for lunch today, and I want a light meal." The server receives this voice command, which contains the elderly person's needs, that is, to make a reservation for lunch and to have a light meal.

[0045] In the scenario of an in-vehicle intelligent system, the driver wants to check the road conditions at the destination while driving. He uses voice commands to say, "Check the road conditions to the city center." The server receives this voice command, and since the driver needs to control the steering wheel with both hands, voice commands become a convenient and safe way of interaction.

[0046] In a home entertainment scenario, the user wants to play a specific piece of music, so he says to the smart speaker: "Play Song A by singer A." The server receives this voice command, which is a clear voice command for a music playback request.

[0047] For the touch screen blood pressure query command in the smart elderly care scenario, the server sends the touch screen command to the central control service. The central control service will first verify the legitimacy of the command, such as checking whether the touch screen operation complies with the device's operating specifications (such as whether the touch is performed in the correct blood pressure query function area). Then, the central control service will format the command and convert it into an internally recognizable format. For example, the touch screen command "check blood pressure" is converted into a specific code or data structure, such as {"operation":"check_blood_pressure","source":"touch_screen"}, for subsequent processing.

[0048] After the instruction to query the internal medicine queue on the touch screen in the waiting area of ​​the medical institution is sent to the central control service, the central control service will analyze the instruction. It may interact with the hospital's information system to obtain real-time data on the current internal medicine queue. If it is found that there may be some ambiguity in the touch screen instruction, such as the existence of multiple internal medicine clinics, the central control service will further clarify whether the query is the overall internal medicine queue or the queue of a specific internal medicine clinic, and then supplement or modify the original touch screen instruction to obtain the processed instruction.

[0049] After the touch screen instruction for adjusting the brightness of the lights in the smart home reaches the central control service, the central control service will determine whether the instruction is compatible with the current state of the device. For example, if the lighting device is undergoing maintenance or is in a special mode where the brightness cannot be adjusted, the central control service will process the instruction accordingly, such as returning an error message to the server to inform the user that the operation cannot be performed; if it can be performed, the central control service will convert the touch screen instruction into a control instruction for a specific lighting device, such as converting the position of the brightness adjustment slider into a specific brightness value (such as converting from 50% brightness on the touch screen to a specific brightness value instruction of 150 lumens).

[0050] In the smart elderly care scenario, after the voice command of the elderly to make a lunch reservation is sent to the central control service, the central control service must first parse the voice command. Since the voice command is in natural language form, the central control service needs to call the ASR (speech to text) service to convert the voice command "I want to make a reservation for lunch today, and I want a light set meal." into the text form "I want to make a reservation for lunch today, and I want a light set meal." Then, the central control service will conduct a preliminary analysis of the text and extract key information, such as the reservation time is "today noon" and the meal requirement is "light set meal". It may also process some vague statements. For example, if the elderly do not specify which restaurant to make a reservation, the central control service will determine the restaurant based on the elderly's history or default settings, and obtain the processed voice command related information, such as {"operation":"lunch_reservation","time":"todaynoon","meal_type":"light_package","restaurant":"default_restaurant"}.

[0051] After the voice command to query the road conditions in the in-vehicle intelligent system reaches the central control service, the central control service calls the ASR service to convert the voice command into text and then processes the text. It will determine the specific range of the destination "city center" and may interact with the in-vehicle map system to obtain more accurate geographic location information. At the same time, the central control service will adjust the processing method of this voice command according to the current driving status of the vehicle (such as whether it is driving at high speed, whether it is close to the destination, etc.). If the vehicle is about to reach the destination, the central control service may remind the user whether he really wants to inquire about the road conditions at the destination, or directly provide parking information around the destination, etc., so as to obtain the relevant content of the processed voice command.

[0052] When the voice command to play music in the home entertainment scenario reaches the central control service, the central control service calls the ASR service to convert it into text and analyzes the text. It searches the music library for "Singer A's Song a". If there are multiple versions, the central control service determines the specific music resource to be played based on the user's preferences (such as the version previously heard, the high-quality version, etc.). At the same time, the central control service also checks the audio playback status of the current device (such as whether other music is playing, volume settings, etc.), and adjusts the original voice command to obtain the processed instruction information for playing music, such as {"operation":"play_music","music_name":"Singer A Song a","version":"preferred_version","audio_state":"check_and_adjust"}.

[0053] In the smart elderly care scenario, the instruction to query blood pressure data after being processed by the central control service is sent to the multimodal interaction engine. The multimodal interaction engine will first query the health data repository of the elderly according to this instruction. Assume that this health data repository contains various health indicator data of the elderly, such as blood pressure, blood sugar, heart rate, etc. The multimodal interaction engine will search in the blood pressure data section and find the elderly’s most recent blood pressure measurement value, such as 130 / 80mmHg. Then, the multimodal interaction engine will generate user demand feedback results based on this data. It may analyze this blood pressure data to determine whether it is within the normal range (assuming that for the elderly, this blood pressure value is within the normal range), and then generate a feedback result "Your blood pressure is 130 / 80mmHg, which is within the normal range. Please continue to maintain a healthy lifestyle."

[0054] In the waiting area of ​​a medical institution, after the processed instruction to query the queue status of the internal medicine department reaches the multimodal interaction engine, the multimodal interaction engine will interact with the hospital's queue management system. This queue management system stores the patient queue information of each department, including patient name, registration time, expected consultation time, etc. The multimodal interaction engine will query the queue data of the internal medicine department. For example, it will find that there are 5 patients in line ahead, and the average waiting time is about 30 minutes. Then, the multimodal interaction engine will generate feedback results based on this information, "There are currently 5 patients in line in the internal medicine department. The expected waiting time is about 30 minutes. Please wait patiently."

[0055] In the smart home scenario, the processed light brightness adjustment instruction is sent to the multimodal interaction engine. The multimodal interaction engine interacts with the control logic of the smart home device. It generates feedback results based on the current brightness adjustment instruction (such as setting the light brightness to 150 lumens) and the ambient light sensor data of the room (assuming that the room is currently dark). For example, it may determine whether this brightness setting is appropriate, and if so, generate feedback results "The brightness of the living room light has been adjusted to 150 lumens to meet your lighting needs." If it is not appropriate, it may recommend a more appropriate brightness value and generate feedback results "Based on the current light conditions in the room, it is recommended to adjust the brightness to 200 lumens. Do you want to adjust it?"

[0056] In the smart elderly care scenario, the lunch reservation instruction processed by the central control service is sent to the multimodal interaction engine. The multimodal interaction engine interacts with the catering management system of the elderly care institution. This catering management system contains data such as menu information, restaurant capacity, and reservation status. The multimodal interaction engine will check whether there is a light meal set available on the menu for lunch today, and check the restaurant's reservation status, assuming that the restaurant has enough seats and has a light meal set. Then, the multimodal interaction engine will generate feedback results based on this information, "You have successfully reserved a light meal set for lunch today. Please go to the restaurant on time for dinner."

[0057] In the in-vehicle intelligent system, after the processed road condition query instruction reaches the multimodal interaction engine, the multimodal interaction engine will interact with the traffic information platform. This traffic information platform collects road condition information from various sections of the road, such as traffic flow, accident conditions, road construction, etc. The multimodal interaction engine will query the road conditions to the city center. Suppose it finds that a section of the road is under construction, causing traffic congestion. Then, the multimodal interaction engine will generate feedback results based on this information, "A section of the road to the city center is under construction, and traffic congestion may occur. Please plan your route in advance."

[0058] In the home entertainment scenario, the processed music playback command is sent to the multimodal interaction engine. The multimodal interaction engine interacts with the music playback platform. This music playback platform contains information such as the music resource library, playback history, and user favorites. The multimodal interaction engine will find the playback resource of singer A's "Song A" according to the command, and set the appropriate playback mode (such as single loop, random play, etc.) according to the user's playback history, and then generate a feedback result "Singer A's "Song A" is playing for you, and it has been set to single loop mode according to your playback habits."

[0059] In the smart elderly care scenario, the blood pressure data feedback result generated by the multimodal interaction engine, "Your blood pressure is 130 / 80 mmHg, which is within the normal range. Please continue to maintain a healthy lifestyle." is sent back to the central control service. The central control service will perform the final processing on this feedback result. It may convert this result into a format suitable for display on the smart bedside terminal screen, for example, adjusting the font size, color, etc., to facilitate the elderly to view. Then, the central control service will send this final demand feedback result to the smart bedside terminal for display, and the elderly can see the blood pressure data and related prompt information on the terminal screen.

[0060] In the waiting area of ​​a medical institution, the multimodal interaction engine generates a feedback result about the queue situation of the internal medicine department, "There are currently 5 patients waiting in line in the internal medicine department. The estimated waiting time is about 30 minutes. Please wait patiently." The central control service will process this result, such as adding information such as the hospital logo and queue number. Then, the central control service will send this final demand feedback result to the touch screen device in the waiting area for display, and patients can see the detailed information of their queue in the internal medicine department on the touch screen device.

[0061] In the smart home scenario, the central control service will process the feedback results of light brightness adjustment. If the multimodal interaction engine recommends adjusting the brightness value, the central control service will convert this suggestion into an actionable instruction form, such as displaying a button on the screen that says "Adjust brightness to 200 lumens". The central control service will then send this final demand feedback result to the smart home central control touch screen device for display, and the user can choose whether to further adjust the light brightness based on this display result.

[0062] In the smart elderly care scenario, the feedback result of the lunch reservation, "A light meal for lunch today has been successfully reserved for you. Please go to the restaurant on time for your meal." is received by the central control service. The central control service will call the TTS (text-to-speech) service and the TTSA (text-to-speech service related to virtual people, if applicable) service to convert this feedback result. Suppose the TTS service is used to convert this text into voice, "A light meal for lunch today has been successfully reserved for you. Please go to the restaurant on time for your meal." Then, the central control service will broadcast this voice feedback result in streaming voice and play it to the elderly through the speakers of the smart device. If a virtual human service is involved, the central control service will control the virtual human to convey this information with appropriate expressions and actions, such as the virtual human smiling and saying, "A light meal for lunch today has been successfully reserved for you. Please go to the restaurant on time for your meal."

[0063] In the in-vehicle intelligent system, the feedback result of the traffic query "A section of the road to the city center is under construction, and traffic congestion may occur. Please plan your route in advance." is received by the central control service. The central control service will call the TTS service to convert this text into voice "A section of the road to the city center is under construction, and traffic congestion may occur. Please plan your route in advance." Then, the central control service will stream this voice feedback result through the in-vehicle audio system so that the driver can hear the traffic information and make corresponding decisions.

[0064] In the home entertainment scenario, the feedback result of music playback "Song A by singer A is playing for you, and it has been set to single loop mode according to your playback habits." is received by the central control service. The central control service will call the TTS service to convert this text into voice "Song A by singer A is playing for you, and it has been set to single loop mode according to your playback habits." Then, the central control service will stream this voice feedback result through the smart speaker. At the same time, if there is a related visual interface, the central control service will also display this feedback result on the screen in an appropriate form (such as song information, playback mode icon, etc.).

[0065] In the embodiment of the present invention, when the target demand instruction is the voice instruction, sending the target demand instruction to the central control service for processing to obtain the processed target demand instruction can be implemented through the following example.

[0066] The central control service calls the ASR service to convert the voice command to obtain a text command corresponding to the voice command.

[0067] In the embodiment of the present invention, for example, in a room of a smart elderly care community, an elderly person with mild visual impairment wants to check his activity schedule for the day. He says to the smart device in the room: "I want to know what activities are there today." After receiving this voice command, the server sends it to the central control service. The central control service immediately calls the ASR service to process this voice command.

[0068] The ASR service first collects voice signals. It captures the voice of the elderly through the microphone on the smart device and converts it into electrical signals. These electrical signals contain the original information of the voice, such as the frequency and amplitude of the sound, which is the basis for subsequent processing.

[0069] Next, pre-processing is performed. Since there may be some background noise in the environment of the nursing home, such as light conversations from other rooms or the hum of equipment operation, the ASR service will remove these background noises through filtering and other methods to make the voice signal clearer. At the same time, the amplitude, frequency and other characteristics of the voice signal are standardized to ensure that subsequent processing can be carried out in a unified manner. Then, endpoint detection is performed to accurately determine the starting and ending points of the voice signal and remove the silent parts, such as the short pauses that may exist before and after the elderly speak, which can reduce the amount of calculation.

[0070] Then feature extraction is performed to convert the preprocessed speech signal into a form that can be processed by the computer, and extract feature vectors that reflect the characteristics of the speech. These feature vectors contain key information about the speech, such as the characteristics of phonemes.

[0071] In terms of acoustic modeling, the ASR service uses deep neural network modeling. It is trained through a large amount of pre-labeled speech data to learn the acoustic features and patterns of speech. At the same time, in language modeling, based on the method of statistical language model, the probability of adjacent words in the text is calculated to evaluate the language rationality of the text. For example, in the elderly care scenario, words such as "activities", "health lectures", and "rehabilitation training" often appear, and the adjacent probability relationship between these words is learned and analyzed in language modeling. In addition, recurrent neural networks (RNNs) and their variants are used to capture long-distance dependencies in texts, such as longer sentence structure relationships such as "Today's activities are health lectures in the morning and rehabilitation training in the afternoon."

[0072] Finally, through decoding and recognition, the outputs of the acoustic model and the language model are combined, and the most likely text sequence is found as the recognition result through a specific decoding algorithm. After this series of processing, the original voice command "I want to know what activities are there today." is successfully converted into the corresponding text command. After receiving this text command, the central control service can further analyze and process it, for example, identifying that this is a request to query the day's activity schedule, preparing for subsequent interaction with the activity management system of the retirement community.

[0073] In a smart home environment, the owner is in the living room, holding something in his hands, and wants to control the smart TV in the living room to turn on a specific channel. He says to the smart voice assistant: "Turn on the TV and switch to the sports channel." The server receives this voice command and sends it to the central control service, which calls the ASR service for processing.

[0074] During the ASR service's voice signal collection process, the microphone of the intelligent voice assistant accurately captures the owner's voice and converts it into an electrical signal. In the preprocessing stage, since there may be some environmental noise in the living room, such as the sound of air conditioning or traffic outside the window, the ASR service will remove these noises to make the voice signal clear. At the same time, the voice signal is standardized and endpoint detected.

[0075] During feature extraction, a vector representing the characteristics of the voice command is extracted. In acoustic modeling, the acoustic modeling method commonly used in traditional ASR systems is combined with a deep neural network to adapt to different speech situations. In language modeling, common words in smart home scenarios, such as "TV", "channel", "turn on", "turn off", etc., are considered to calculate the adjacent probability relationship between them. For example, "turn on" and "TV" often appear together, and this relationship is studied in detail in language modeling.

[0076] Through decoding and recognition, the ASR service converts this voice command into a text command "Turn on the TV and switch to the sports channel." After the central control service obtains this text command, it can interact with the control system of the smart TV, find relevant information about the sports channel and send control commands to turn on the TV and switch to the sports channel.

[0077] When the driver is driving the car, he wants to check the location of a nearby gas station, so he says to the in-vehicle intelligent system: "Find nearby gas stations." The server receives this voice command and transmits it to the central control service, which calls the ASR service for processing.

[0078] The ASR service collects voice signals. The microphones here are usually optimized for the acoustic environment in the car to accurately capture the driver's voice. During the preprocessing process, background noise such as engine noise and wind noise that may exist in the car will be removed, and the voice signal will be standardized and endpoint detected.

[0079] The feature extraction process extracts the key feature vectors of speech. In acoustic modeling, an acoustic model suitable for the vehicle environment is used, taking into account special circumstances such as acoustic reflections in the vehicle. In terms of language modeling, probability analysis and long-distance relationship capture are performed on common words in vehicle scenarios, such as "gas station", "nearby", and "navigation".

[0080] After decoding and recognition, the voice command is converted into a text command "Find nearby gas stations." After the central control service receives this text command, it can interact with the vehicle navigation system to query the location information of nearby gas stations, and then provide the driver with navigation instructions or related gas station information.

[0081] In an embodiment of the present invention, the processed target demand instruction is sent to a multimodal interaction engine for processing to obtain a user demand feedback result, which can be implemented through the following example.

[0082] Extracting information and understanding semantics of the text instruction through the multimodal interaction engine to obtain key information of the text instruction;

[0083] The multimodal interaction engine calls a pre-trained dialogue model to process the key information of the text instruction to obtain a text feedback result corresponding to the text instruction.

[0084] In the embodiment of the present invention, for example, in a smart elderly care scenario, assume that the voice command issued by the elderly is converted into a text command "I want to make an appointment for physical therapy service tomorrow morning" by the ASR service. After receiving this text command, the multimodal interaction engine begins to extract information and understand semantics.

[0085] Information extraction:

[0086] Entity recognition: The multimodal interaction engine first recognizes the entities in it, such as "I" (here represents the elder himself), "tomorrow morning" (this is the time entity), and "physical therapy service" (this is the service type entity). These entities are the key elements to understand the entire instruction.

[0087] Relationship extraction: The engine analyzes that there is an appointment relationship between "me" and "physiotherapy services", and this appointment relationship is associated with the time "tomorrow morning". The extraction of this relationship helps to clarify the specific intention of the instruction.

[0088] Semantic understanding:

[0089] Intent recognition: By analyzing the entire text instruction, the multimodal interaction engine understands that the main intention of the elderly is to make an appointment for physical therapy services at a specific time (tomorrow morning). It not only recognizes individual words, but also understands the semantics of the entire sentence. For example, it can understand that "physical therapy services" are different from ordinary medical services and are a type of rehabilitation physical therapy services.

[0090] Contextual understanding: Considering the scenario of smart elderly care, the multimodal interaction engine will combine contextual information such as the elderly’s living environment, past appointment habits, and the service arrangements of the elderly care institution. For example, if the elderly care institution has a specific physiotherapist on duty tomorrow morning, or there is a personalized physiotherapy package for the elderly, these contextual information will be incorporated into the understanding of this text instruction. Through such information extraction and semantic understanding, the multimodal interaction engine obtains the key information of the text instruction, namely the appointment subject (elderly), appointment time (tomorrow morning), appointment service (physiotherapy service) and the relationship between them (appointment relationship).

[0091] Suppose the text instruction is "Dim the bedroom lights."

[0092] Information extraction:

[0093] Entity recognition: The multimodal interaction engine recognizes "bedroom" (this is a spatial entity that specifies the location of the operation), "light" (the operation object entity), and "darker" (this is a description entity for adjusting the light status).

[0094] Relationship extraction: Analyze the existence of an operation relationship of "dimming the bedroom lights".

[0095] Semantic understanding:

[0096] Intent recognition: It is clear that this is an intention to adjust the brightness of the bedroom light to a lower level. The multimodal interaction engine knows that this is not to turn off the light, but to fine-tune the brightness.

[0097] Contextual understanding: In the context of smart homes, it will consider the current bedroom light sensor data (if any), the dimming range of the lighting equipment, and other information. For example, if the bedroom is currently very dark, it may understand this instruction based on some preset rules, such as not dimming the lights too much to ensure safety. The key information obtained is the operation location (bedroom), the operation object (light), and the operation content (dimming).

[0098] Based on the key information obtained previously (appointment subject, appointment time, appointment service), the multimodal interaction engine calls the pre-trained dialogue model for processing.

[0099] This dialogue model is trained on a large amount of data related to elderly care services, including information such as the service items, schedules, and staff arrangements of elderly care institutions.

[0100] The multimodal interaction engine inputs key information into the dialogue model, which first checks the arrangement of physiotherapy services in nursing homes. Assume that the physiotherapy services in nursing homes are divided into different types, such as massage therapy, rehabilitation therapy, etc., and different types of physiotherapy services have different appointment rules and available time periods.

[0101] The dialogue model searches for the type of physiotherapy service and available time slots suitable for the elderly based on tomorrow morning's time. If available, it generates text feedback results, such as "You have been successfully booked for rehabilitation physiotherapy service at 10 a.m. tomorrow. Please prepare in advance." If not, it may provide alternative options based on actual conditions, such as "Tomorrow morning's physiotherapy service is full. Would you consider the physiotherapy service at 2 p.m.?"

[0102] For the key information obtained previously (operation location, operation object, operation content), the multimodal interaction engine calls the pre-trained smart home dialogue model.

[0103] This dialogue model contains information such as control rules and device status of various smart devices in the home.

[0104] The multimodal interaction engine inputs the key information of dimming the bedroom light into the dialogue model. The dialogue model first checks the current status of the bedroom light device, such as the current brightness value, dimming capability, etc. Assume that the current brightness of the bedroom light is 80% and the dimming range is 10% to 100%.

[0105] Based on the request of "dimming a little", the dialogue model calculates an appropriate brightness value, such as 60%, according to the preset algorithm. Then it generates a text feedback result, "The bedroom light brightness has been adjusted to 60%." If the lighting equipment is faulty or the brightness cannot be adjusted as required, the dialogue model may generate a text feedback result such as "The bedroom light equipment is faulty and the brightness cannot be adjusted. Please contact maintenance personnel."

[0106] In the embodiment of the present invention, the user demand feedback result is processed by the central control service to obtain the final demand feedback result and display it, which can be implemented through the following examples.

[0107] The central control service calls the TTS service and the TTSA service to convert the text feedback result to obtain the corresponding voice feedback result and the virtual human feedback result;

[0108] Perform streaming voice broadcast based on the voice feedback result, and control the virtual human to execute the virtual human feedback result.

[0109] In the embodiment of the present invention, for example, in the smart elderly care scenario, it is assumed that the text feedback result generated by the multimodal interaction engine is "You have successfully made an appointment for rehabilitation therapy service at 10 am tomorrow. Please prepare in advance."

[0110] Calling TTS service:

[0111] After receiving the text feedback result, the central control service first calls the TTS service. The TTS service starts text preprocessing. In this example, it normalizes the input text, for example, converts the number "10" to "ten" to make the voice broadcast more natural. Then it performs word segmentation and part-of-speech tagging, dividing the sentence into words and phrases such as "has been made for you", "successfully booked", "tomorrow morning", "ten", "points", "of", "rehabilitation and physiotherapy services", "please", "in advance", "be prepared", and annotates the part-of-speech.

[0112] Next, language analysis is performed, including grammatical and semantic analysis. The TTS service understands the grammatical structure of the sentence and determines the role of each word in the sentence. For example, "has been prepared for you" is a structure that indicates the object of an action, and "successfully booked" is a predicate verb phrase, etc. Then phoneme conversion is performed, and the phoneme dictionary is searched to decompose each word into phonemes, such as the phoneme combination corresponding to "rehabilitation". Then the prosodic markers are determined, and the stress, rhythm, and intonation changes of the syllables are determined based on the semantics and grammatical structure of the sentence. For this sentence, "rehabilitation therapy services" may be the stressed part because it is the key information, while the "please prepare in advance" at the end of the sentence will adopt a gentler, suggestive tone.

[0113] In the acoustic modeling and waveform synthesis stage, based on the phoneme sequence, the TTS service uses the acoustic model to generate the corresponding acoustic feature parameters, which describe the spectral and temporal characteristics of the speech. Then the speech waveform is generated through waveform synthesis technology. If the splicing synthesis method is used here, pre-recorded speech fragments (such as words, syllables or phonemes) will be used to splice into a complete speech output. For example, the word "rehabilitation" may be composed of pre-recorded syllables of "kang" and "fu". If it is parameter synthesis, a mathematical model is used to simulate the human vocal system and generate a speech waveform based on the acoustic feature parameters. Finally, after speech post-processing, sound enhancement is performed, such as adjusting the volume to a level suitable for the hearing of the elderly, and smoothing (if it is a TTS system based on unit splicing), the speech feedback result is obtained.

[0114] Call TTSA service:

[0115] If the retirement community uses virtual human services, the central control service will call the TTSA service to generate virtual human feedback results. The TTSA service will also process the text feedback results, but will consider the virtual human's expression more. For example, while generating the voice, the virtual human's expression, movement and other related information will be determined. For the message of successful appointment, the virtual human may show a smiling expression, lean forward slightly, and make welcoming and prompting gestures. And the voice tone will be more vivid to match the image and style of the virtual human.

[0116] Calling TTS service:

[0117] After the central control service calls the TTS service, the text preprocessing stage will normalize the sentence, such as processing the pronunciation of the number "60". Word segmentation and part-of-speech tagging divide the sentence into parts such as "has been adjusted to" "bedroom", "light", "brightness", "sixty percent", etc.

[0118] In language analysis, the grammatical structure and semantic relationship are determined, for example, "bedroom" is an attributive that modifies "light". When performing phoneme conversion and prosodic marking, "bedroom light" may have a certain emphasis because it is the object of the operation, and "sixty percent" at the end of the sentence will also have a clear intonation.

[0119] In the acoustic modeling and waveform synthesis stage, the speech waveform is generated according to the corresponding technical process, and the speech feedback result is obtained after speech post-processing.

[0120] Call TTSA service:

[0121] If the smart home system is equipped with a virtual human interface, the TTSA service will generate virtual human feedback results based on the text feedback results. The virtual human may point in the direction of the bedroom with an expression on his face to confirm the completion of the operation, and the voice will be broadcast in a concise and clear tone.

[0122] Streaming voice broadcast:

[0123] The central control service sends the voice feedback results to the smart device (such as the voice broadcast device in the elderly's room) for streaming voice broadcast. Due to the use of streaming voice broadcast technology, voice data can be played while it is generated, without waiting for the entire voice file to be generated. This can reduce delays, and the elderly can hear the message of successful appointment in time, such as "You have successfully made an appointment for rehabilitation and physiotherapy services at 10 am tomorrow. Please prepare in advance." This voice will be played in a clear and natural tone with a moderate volume, which is convenient for the elderly to listen.

[0124] Virtual human execution feedback results:

[0125] If the elderly care community has a virtual human service, the central control service will control the virtual human to execute the virtual human feedback results on the screen of the elderly's smart device. The virtual human will appear on the screen with a smiling expression and a welcoming gesture, and repeat the message of successful appointment in a vivid voice. After the broadcast is completed, the virtual human may stay for a while, waiting for the elderly to further operate or ask questions, providing a more humane interactive experience.

[0126] Streaming voice broadcast:

[0127] The central control service sends the generated voice feedback results to the voice devices of the smart home (such as smart speakers) for streaming voice broadcast. For example, the voice "The bedroom light brightness has been adjusted to 60%." will be played clearly in the room, letting the user know that the light brightness has been adjusted as required.

[0128] Virtual human execution feedback results:

[0129] If there is a virtual human interface, the virtual human will point to the direction of the bedroom on the screen of the smart home central control device, with an expression on his face confirming that the operation is completed, and at the same time broadcast voice content, giving the user an intuitive operation feedback and enhancing the user's interactive experience with the smart home system.

[0130] In the embodiments of the present invention, the following implementation modes are also provided.

[0131] The target demand instruction is synchronized to the Redis database in real time for caching, and an expiration time and index strategy for the target demand instruction are configured.

[0132] In an embodiment of the present invention, for example, in a smart elderly care community, an elder uses a touch screen device to query his or her medication plan. When the elder clicks the "view medication plan" icon on the touch screen device, the server receives the touch screen instruction. The server immediately synchronizes the touch screen instruction to the Redis database in real time for caching.

[0133] Cache operation details:

[0134] When caching this touchscreen instruction, we first need to determine how to store it in the Redis database. Redis is a high-performance key-value database. For this touchscreen instruction, the type of touchscreen operation (such as "medication plan query") can be used as the key, and some metadata related to this operation (such as the timestamp of the query, the room number of the elderly, etc.) can be used as the value. For example, the key is "medication_plan_query" and the value is "{'query_time': '2023-08-1510:00:00', 'room_number': '101'}".

[0135] Expiration time configuration:

[0136] Considering that medication plans may be updated frequently, a relatively short expiration time is configured for the touch screen command cache of "Medication Plan Query", such as 10 minutes. The reason for this setting is that if the elderly person queries the medication plan again within 10 minutes, it may be because he has doubts about the results of the previous query or did not see it clearly. The system can directly obtain the information related to the touch screen command from the cache without reprocessing the entire query process, thereby improving the system's response speed. At the same time, the 10-minute expiration time also ensures that if the medication plan is updated after 10 minutes, the system will re-obtain the latest medication plan data the next time it is queried.

[0137] Indexing strategy:

[0138] In order to facilitate quick retrieval of the touch screen command cache, an index strategy based on operation type and time is adopted. For example, you can create an index classified by operation type, so that when the system needs to find cache related to medication plan query, it can quickly locate all cache items of type "medication_plan_query". In addition, you can also create an index in chronological order so that you can efficiently clean up expired cache or analyze touch screen command operations within a specific time period.

[0139] For example, an elder says through voice command: "Check today's activity schedule." After receiving this voice command, the server synchronizes the voice command to the Redis database in real time for caching before sending it to the central control service for processing.

[0140] Cache operation details:

[0141] In the Redis database, the hash value of the voice command (calculated by a specific algorithm and used to uniquely identify the voice command) is used as the key, for example, the key is "voice_command_hash_12345" (here 12345 is an assumed hash value), and the value is the text content of the voice command "Query today's event schedule." and some additional information, such as the timestamp of receiving the voice command, the identity of the elder, etc., for example, the value is "{'command': 'Query today's event schedule.', 'receive_time': '2023-08-1510:30:00', 'elderly_id': 'E001'}".

[0142] Expiration time configuration:

[0143] Since the schedule of activities is relatively fixed within a day, but there may be temporary adjustments, a longer expiration time, such as 24 hours, is configured for the voice command cache of "query today's schedule of activities". In this way, if the elderly query today's schedule of activities multiple times within a day, the system can directly obtain relevant information from the cache to reduce processing time. In addition, the 24-hour expiration time also ensures that if the schedule of activities is updated the next day, the old instructions in the cache will not affect the new query results.

[0144] Indexing strategy:

[0145] For the cache of voice commands, an indexing strategy based on the identity of the elder and the content of the command is adopted. Create an index classified by the identity of the elder, so that when the system needs to find the voice command cache of a certain elder, it can be quickly located. At the same time, create an index based on the keywords of the command content. For example, for the command "Query today's event schedule", keywords such as "event schedule" and "today" can be used as index items, so that when you need to find the voice command cache related to the event schedule, you can quickly retrieve it.

[0146] In a smart home environment, the user slides the slider on the smart central control touch screen device to adjust the temperature of the living room air conditioner. After the server receives this touch screen command, it synchronizes it to the Redis database for caching.

[0147] Cache operation details:

[0148] In Redis, the description of the touch screen operation (such as "living room air conditioning temperature adjustment") is used as the key, and the value is information related to this operation, such as "{'adjustment_value': '22℃', 'adjustment_time': '2023-08-1511:00:00'}", where "adjustment_value" represents the adjusted temperature value, and "adjustment_time" represents the time of the adjustment operation.

[0149] Expiration time configuration:

[0150] Considering that the air conditioner temperature may not change frequently after adjustment, and the user may check the previous adjustment status in a short time, configure a moderate expiration time, such as 30 minutes. In this way, within 30 minutes, if the user wants to check the previous temperature adjustment status of the living room air conditioner, the system can directly obtain the information from the cache to improve the response speed. At the same time, if the operating status of the air conditioner changes after 30 minutes (such as automatic energy-saving mode to adjust the temperature, etc.), the old instructions in the cache will not interfere with the new operation.

[0151] Indexing strategy:

[0152] Use an indexing strategy based on device location (living room) and operation type (air conditioning temperature adjustment). Create an index classified by device location so that when the system needs to find the touch screen command cache related to the living room device, it can quickly locate it. At the same time, create an index based on operation type to facilitate efficient retrieval when looking for all caches related to air conditioning temperature adjustment.

[0153] The home user says through voice command: "Turn on the smart light in the living room." After the server receives this voice command, it caches the voice command in the Redis database before sending it to the central control service.

[0154] Cache operation details:

[0155] In the Redis database, the unique identifier of the voice command (such as a specific code generated by the voice command content) is used as the key. For example, the key is "voice_light_command_67890" (here 67890 is an assumed code), and the value is "{'command': 'Turn on the smart light in the living room.', 'issue_time': '2023-08-1511:30:00'}", where "issue_time" indicates the time when the voice command was issued.

[0156] Expiration time configuration:

[0157] For the voice command cache of "turn on the smart light in the living room", since the light status is relatively stable, a longer expiration time is configured, such as 12 hours. Within 12 hours, if the user queries the status of the smart light in the living room again or performs operations related to the light, the system can obtain the information from the cache. In addition, the 12-hour expiration time also takes into account that the light is unlikely to be frequently changed by other factors (such as automatic timed shutdown, etc.) during this period.

[0158] Indexing strategy:

[0159] Use an indexing strategy based on device type (smart light) and operation (open operation). Create an index classified by device type to facilitate finding all voice command caches related to smart lights. At the same time, create an index based on operation type so that the system can quickly locate all voice command caches related to open operations when it needs to find them.

[0160] In the embodiment of the present invention, the embodiment of the present invention also provides the following implementation manner.

[0161] Obtaining the historical behavior and user profile of the target user;

[0162] The historical behavior and the user portrait are used as optimization data for the conversation model.

[0163] In an embodiment of the present invention, illustratively, in a smart elderly care community, the server continuously records various interactive behavior data of each elder, thereby constructing historical behavior records and user portraits.

[0164] Examples of historical behavior:

[0165] For an elder named Grandpa Li, the server records his interaction history with the smart elderly care system in the past month. For example, he checks his blood pressure data twice a week, often makes appointments for physiotherapy services on Thursday afternoons, prefers light food, and likes to listen to opera programs between 7 and 9 p.m. These specific operations, such as checking blood pressure, making appointments for physiotherapy, choosing diet, and listening to programs, all constitute Grandpa Li's historical behavior data.

[0166] User portrait construction example:

[0167] Based on these historical behaviors, the server constructs a user profile for Grandpa Li. In terms of health, because he often pays attention to blood pressure data and undergoes physical therapy regularly, the system marks him as an elder who pays attention to health management. In terms of dietary preferences, "light food" is recorded in the user profile as an important dietary feature of him. In terms of entertainment, "opera lover" becomes his entertainment label for a specific time period in the evening. In addition, his living habits information, such as waking up time, rest time, etc., is also recorded. This information is combined to form a comprehensive user profile, for example: {"name":"Grandpa Li","health_conscious":true,"diet_preference":"light food","entertainment":"opera lover","wake_up_time":"07:00","rest_time":"21:00"}.

[0168] Optimization for health management:

[0169] When Grandpa Li queries health-related information through voice or touch screen commands, the dialogue model will use his historical behavior and user profile for optimization. For example, when Grandpa Li queries his blood pressure data again, the dialogue model will respond more quickly and provide more detailed relevant information based on his previous historical behavior of frequently querying blood pressure, such as comparison with the last blood pressure data, analysis of recent blood pressure fluctuations, etc. If he asks about the impact of diet on blood pressure, since the user profile records his preference for light food, the dialogue model can provide targeted professional advice on the relationship between light diet and blood pressure control, rather than giving general dietary advice.

[0170] Optimization for therapy bookings:

[0171] When Grandpa Li makes an appointment for a physiotherapy service, the dialogue model will give priority to recommending Thursday afternoon physiotherapy sessions, based on his history of frequently making appointments for physiotherapy on Thursday afternoons, even if he does not specify a time. In addition, because of his user profile characteristics of focusing on health management, the dialogue model can proactively provide some health tips related to physiotherapy after the appointment is successful, such as precautions before and after physiotherapy.

[0172] Optimized for entertainment needs:

[0173] In the evening, if Grandpa Li issues a voice command to seek entertainment content, the dialogue model will prioritize the recommendation of opera programs, such as the latest opera performance videos or opera radio channels, based on his user profile as an opera lover. If he indicates that he has heard these recommendations, the dialogue model can recommend opera programs of similar styles or other works of opera actors based on his entertainment history, such as the types of opera he has heard before.

[0174] In a smart home environment, the server records the historical behaviors of home users to build user profiles.

[0175] Examples of historical behavior:

[0176] Take a family of three as an example. Dad often turns on the home theater to watch sports events on weekend nights. Mom likes to adjust the brightness of the living room lights in the afternoon to create a comfortable reading environment. The child uses the smart speaker to listen to stories at around 9 o'clock every night. These different operating behaviors, including device usage time and usage scenarios, are all historical behavior data of family users.

[0177] User portrait construction example:

[0178] Based on these historical behaviors, we build user profiles for family users. For dad, his user profile features include "sports enthusiast" and "weekend home theater user". Mom's user profile has labels such as "reading enthusiast" and "pay attention to the comfort of living room lighting". The child's user profile is "story listener" and "use smart speakers around 9 pm". The user profile of the whole family may be: {"family_members":["Dad","Mom","Child"],"Dad":{"hobby":"Sports","usage":"Weekend home theater"},"Mom":{"hobby":"Reading","usage":"Adjust living room lighting in the afternoon"},"Child":{"hobby":"Listen to stories","usage":"Smart speaker at 9 pm"}}.

[0179] Optimization for dads:

[0180] When Dad issues a voice command, the dialogue model is optimized based on his user profile. For example, if he says "turn on the TV" on a weekend night, the dialogue model will directly turn on the TV and switch to the sports channel he often watches based on his historical behavior of watching sports events, instead of displaying the default channel. If he asks if there are any new sports events recommended, the dialogue model will recommend related football events or other similar sports events based on the types of sports he has watched before (such as football).

[0181] Optimization for mothers:

[0182] When the mother adjusts the lights, the dialogue model provides more personalized suggestions when she adjusts the light brightness based on her historical behavior of paying attention to the comfort of the living room lighting. For example, if she dims the lights in the afternoon, the dialogue model may remind her whether she needs to turn on the reading light for better reading light. If she issues a voice command to query information related to the lights, the dialogue model will provide a lighting setting solution that better meets her needs based on her previous historical data on adjusting the lights.

[0183] Optimized for kids:

[0184] At around 9 p.m., if the child issues a voice command, the dialogue model will prioritize the recommendation of stories suitable for his age based on his user profile. If the child expresses dislike for a certain type of story, the dialogue model will analyze his preferences based on the types of stories he has heard before and historical behavior data, and then recommend other stories he may like.

[0185] In an embodiment of the present invention, the target demand instruction includes a health monitoring instruction, and the final demand feedback result includes a health feedback result.

[0186] In the embodiment of the present invention, for example, in a smart elderly care community, the elderly often need to perform health monitoring. For example, a grandmother Zhang who suffers from high blood pressure has her room equipped with smart health monitoring equipment, such as a blood pressure monitor, a blood glucose meter, etc., which are connected to the smart system of the elderly care community.

[0187] When Grandma Zhang uses a sphygmomanometer to measure her blood pressure, the sphygmomanometer sends the measurement data to the server. This process can be seen as the server receiving a health monitoring instruction, which contains the key information of Grandma Zhang's blood pressure measurement value.

[0188] After receiving the health monitoring command, the server will first verify and format the command. It will check the integrity of the data to ensure that the received blood pressure value is accurate. For example, if the received data is formatted incorrectly or missing, the server will send a request to the blood pressure monitor device to resend the correct data.

[0189] The server sends this health monitoring instruction to the central control service. The central control service will search for Grandma Zhang's health record based on her identity. Grandma Zhang's health record records her basic health information, such as age, medical history, and past blood pressure data.

[0190] The central control service will compare the blood pressure value measured this time with her historical blood pressure data. For example, the average blood pressure of Grandma Zhang in the past week was 130 / 80mmHg, and the blood pressure value measured this time was 140 / 90mmHg. The central control service will analyze the changes in this blood pressure value to determine whether it exceeds the normal fluctuation range.

[0191] The central control service sends information including blood pressure values ​​and related analysis results to the multimodal interaction engine, which considers more factors to generate health feedback results.

[0192] The multimodal interaction engine may analyze the cause of the increase in blood pressure based on Grandma Zhang's activities on that day. For example, if the community organized an exciting event that day, it may cause a temporary increase in blood pressure. It also considers environmental factors, such as the temperature and humidity of the day, which have a potential impact on blood pressure.

[0193] Based on these comprehensive analyses, the multimodal interaction engine generates health feedback results. If the blood pressure rise is judged to be temporary, the following health feedback results may be generated: "Grandma Zhang, your blood pressure value measured this time is 140 / 90 mmHg, which is slightly higher than your recent average value. This may be due to the excitement of the community activities today. Please take a rest first, and we will measure your blood pressure again in one hour."

[0194] The multimodal interaction engine sends the health feedback results back to the central control service, which will perform the final processing on the results.

[0195] The central control service may convert this result into a format suitable for display on the smart terminal in Grandma Zhang's room. For example, it may adjust the font size and color, add voice prompts and other functions, so that Grandma Zhang can view and listen to it conveniently.

[0196] The central control service then sends the final health feedback results to the smart terminal in Grandma Zhang's room for display. Grandma Zhang can see the blood pressure value, blood pressure change analysis and corresponding suggestions on the terminal screen, and can also hear voice prompts.

[0197] In the home environment, young Mr. Li pays attention to his health. He uses a smart bracelet to monitor his heart rate, sleep and other health data.

[0198] When Mr. Li sleeps at night, the smart bracelet continuously monitors his sleep data. The next morning, the smart bracelet sends Mr. Li's sleep data to the server, which is equivalent to the server receiving a health monitoring instruction. This instruction contains Mr. Li's sleep duration, sleep depth, heart rate changes and other sleep-related health data.

[0199] After receiving this instruction, the server will conduct a preliminary check and sorting of the data. For example, it will check whether there are any data transmission errors and convert data in different formats into a format that is convenient for subsequent processing.

[0200] The server sends the health monitoring instructions to the central control service. The central control service will look up Mr. Li’s previously set health goals and preferences based on his user account. For example, Mr. Li hopes that his sleep time can reach 7-8 hours, and the deep sleep ratio can reach more than 20%.

[0201] The central control service compares the sleep monitoring data with Mr. Li's health goals. If Mr. Li's sleep duration is 6.5 hours and the deep sleep ratio is 15%, the central control service will analyze the gap between these data and the goals.

[0202] The central control service sends information including sleep data and comparative analysis results to the multimodal interaction engine, which considers more life factors to generate health feedback results.

[0203] The multimodal interaction engine may combine Mr. Li’s work pressure the day before, his dinner diet, and his exercise before bedtime to analyze the reasons for poor sleep quality. For example, if Mr. Li was under a lot of work pressure the day before and had a late dinner, these factors may affect his sleep.

[0204] Based on these analyses, the multimodal interaction engine generates health feedback results: "Mr. Li, you slept for 6.5 hours last night, and the deep sleep ratio was 15%, which did not meet your set health goals. This may be because you were under a lot of work pressure the day before and had dinner too late. It is recommended that you reduce work pressure appropriately today, have dinner earlier, and do some relaxing activities before going to bed, such as meditation or listening to soft music, to improve sleep quality."

[0205] The multimodal interaction engine sends the health feedback results back to the central control service, which processes the results and makes them suitable for display on Mr. Li's mobile app (which comes with the smart bracelet).

[0206] The central control service will add some visual elements, such as using charts to show the changing trends of sleep duration and deep sleep ratio, and using different colors to mark the gap with health goals, etc. Then, the final health feedback results will be sent to Mr. Li's mobile app for display, and Mr. Li can intuitively view his sleep health status and corresponding suggestions through his mobile phone.

[0207] In order to more clearly describe the solution provided by the embodiment of the present invention, a relatively complete explanation is provided below. Figure 2 , Figure 2A schematic diagram of a framework for touch screen and voice fusion interaction provided in an embodiment of the present invention.

[0208] This technical solution aims to realize a hybrid interaction solution of voice and touch screen. Elderly people can use the touch screen to complete basic life convenience services such as making appointments for dining, repairs, cleaning, physical therapy, and checking information. Elderly people collect audio streams through the microphone of the client and give them to the central control service. The central control service calls the ASR (speech to text) service. After obtaining the text information, it calls the multi-mode human-computer interaction engine (NLP natural language processing platform). The multi-mode human-computer interaction engine performs natural language recognition to identify the intent and calls the interface to determine the operation instructions supported by the current page, and then sends a text response to the central control service. The central control service then calls the TTS (text to speech) service to obtain the voice, which is then transmitted to the client. The client can play the voice through the sound playback device.

[0209] Touch screen and voice parallel processing mechanism:

[0210] Design a parallel processing mechanism that allows the system to receive touch screen and voice input at the same time, and parse and respond through independent processing threads. This mechanism ensures that the two input methods do not interfere with each other, improving the system's response speed and concurrent processing capabilities.

[0211] Operation results are cached synchronously:

[0212] After the touch screen and voice operations are completed, the system will synchronize the respective operation results to the Redis database in real time. As a high-performance key-value database, Redis can support fast read and write operations and data persistence, ensuring the accuracy and reliability of the operation results.

[0213] Integrating advanced ASR technology:

[0214] ASR is the process of converting speech into text. The specific technical process is as follows:

[0215] 1) Voice signal acquisition: Use microphones and other devices to convert people's voices into electrical signals. These electrical signals contain the original information of the voice and are the basis for subsequent processing;

[0216] Preprocessing: Noise removal: remove background noise through filtering and other methods to make the speech signal clearer, standardization: standardize the amplitude, frequency and other characteristics of the speech signal to facilitate subsequent unified processing, endpoint detection: determine the starting and ending points of the speech signal, remove the silent part, and reduce the amount of calculation;

[0217] Feature extraction: convert the preprocessed speech signal into a form that can be processed by the computer, and extract the feature vector that reflects the speech characteristics;

[0218] Acoustic modeling: Use the acoustic modeling methods commonly used in traditional ASR systems or deep neural networks for modeling;

[0219] Language modeling: Methods based on statistical language models evaluate the language rationality of text by calculating the probability of adjacent words appearing in the text and recurrent neural networks (RNN) and their variants capture long-range dependencies in the text;

[0220] Decoding and recognition: Combine the outputs of the acoustic model and the language model, and use the decoding algorithm to find the most likely text sequence as the recognition result;

[0221] Post-processing: Grammar correction: check for grammatical errors in the text and correct them, Punctuation processing: automatically add or correct punctuation based on the context, Text regularization: replace non-standard words in the recognition results with standard words.

[0222] Integrate advanced TTS technology:

[0223] TTS is the process of converting text to speech. The specific technical process is as follows:

[0224] 1) Text preprocessing

[0225] Text input: First, the user or system inputs the text information that needs to be converted.

[0226] Text normalization: Normalize the input text, including text formatting, punctuation processing, number and date conversion, etc., to meet the requirements of speech synthesis. For example, expand abbreviations to full forms, convert "100" to "onehundred", etc.

[0227] Word segmentation and part-of-speech tagging: Segment the text into words, phrases or sentences and tag them with parts of speech for subsequent language analysis.

[0228] 2) Language Analysis

[0229] Syntax and semantic analysis: Perform in-depth grammatical and semantic analysis on the preprocessed text to capture the deep meaning and tone characteristics of the text. This includes syntactic analysis, semantic role labeling, etc., which helps determine the correct tone, rhythm, and emphasis.

[0230] Phoneme conversion: Breaking down words into phonemes, the smallest units of sound in a language. This step usually involves looking up a phoneme dictionary, breaking down the word into phonemes, and determining the rules for their pronunciation.

[0231] Prosodic marking: Determine the prosodic features of the text, including syllable stress, rhythm, intonation changes (such as rising and falling tones), etc. This information will affect the naturalness and comprehensibility of the synthesized speech.

[0232] 3) Acoustic modeling and waveform synthesis

[0233] Acoustic modeling: Based on the phoneme sequence, the acoustic model is used to generate the corresponding acoustic feature parameters. These parameters describe the spectral and temporal characteristics of speech and are the key to synthesizing speech.

[0234] Waveform synthesis: Generate speech waveforms through waveform synthesis technology based on acoustic feature parameters. There are two main methods of waveform synthesis technology:

[0235] Concatenative synthesis: Use pre-recorded speech fragments (such as words, syllables, or phonemes) to concatenate into complete speech output. This method can generate very natural-sounding speech, but requires a lot of recording data and storage space.

[0236] Parametric synthesis: Use mathematical models to simulate the human vocal system and generate speech waveforms based on acoustic feature parameters. This method requires less database, but the quality of the generated speech may be rough. In recent years, end-to-end speech synthesis technology based on deep learning has gradually emerged. It can directly convert text features into speech waveforms through neural network models, further improving the quality and naturalness of synthesized speech.

[0237] 4) Voice post-processing

[0238] Sound enhancement: Perform volume adjustment, noise suppression, echo cancellation and other processing on the synthesized speech to improve the sound quality and listening experience.

[0239] Smoothing: For a TTS system based on unit splicing, the pre-recorded speech units need to be seamlessly spliced ​​together and smoothed to eliminate the unnatural feeling at the splicing points.

[0240] 5) Output and evaluation

[0241] Speech output: Output the synthesized speech to the user or system.

[0242] Effect evaluation: Evaluate the synthesized speech, including subjective and objective evaluation. Subjective evaluation is usually conducted through manual listening test to evaluate sound quality, naturalness, usability and other aspects; objective evaluation is tested through instruments or software, such as using PESQ (Perceptual Evaluation of Speech Quality) method to evaluate speech quality.

[0243] WebSocket Audio Streaming:

[0244] The WebSocket protocol is used to achieve real-time transmission of audio data, ensuring low latency and high real-time performance during touch screen and voice interaction. The two-way communication capability of WebSocket enables seamless connection and synchronous processing of audio data.

[0245] Database and large model support:

[0246] The system backend uses a relational database or a non-relational database (such as Redis) to store operation results and user data. At the same time, large models (such as deep learning models) are used for data analysis, prediction and optimization to improve the system's intelligence level and decision-making ability.

[0247] Innovation:

[0248] Parallel and synchronization of touch screen and voice: For the first time, parallel processing and real-time synchronous caching of touch screen and voice input are realized, breaking the limitations of the traditional single interaction mode.

[0249] Efficient caching of Redis database: Using Redis database to quickly cache and query operation results improves the system's response speed and data processing capabilities.

[0250] Intelligent support of large models: By introducing large models for data analysis and prediction, the system is provided with stronger intelligent support, which helps to optimize user experience and improve system performance.

[0251] Cross-platform compatibility and scalability: The system supports multiple operating systems and device platforms, and has good scalability, and can easily adapt to future technological developments and changes in market demand.

[0252] Beneficial effects:

[0253] Improve user experience: Parallel operation of touch screen and voice and real-time synchronous caching enable users to interact with the device in a more natural and flexible way, improving the user experience.

[0254] Improve work efficiency: The system's quick response and efficient data processing capabilities help users quickly complete operational tasks and improve work efficiency.

[0255] Enhanced system intelligence: The introduction of large models provides the system with intelligent data analysis and prediction capabilities, helping to optimize system performance and user experience.

[0256] Adaptability and scalability: The system supports cross-platform compatibility and scalability design, and can adapt to changes in different scenarios and needs.

[0257] The specific implementation is as follows:

[0258] 1. System initialization

[0259] Start the system and load modules such as ASR, TTS, and natural speech recognition, and connect to the Redis database and WebSocket server.

[0260] 2. Client touch screen or voice input

[0261] The client will display card information such as sleep records, medication plans, dining appointments, maintenance appointments, physical therapy appointments, blood pressure checks, blood sugar checks, etc.

[0262] 1) Sleep records are recorded by sleep monitoring devices sending data to the IoT platform. The data on the IoT platform is transmitted to the Dark Matter Intelligent Big Data Platform. The Big Data Platform analyzes the elderly’s recent sleep conditions, gives scores and sleep analysis reports, and then pushes them to the client to display the elderly’s sleep conditions.

[0263] 2) The medication plan is set by the elder through voice or operation management. The back-end service will push the elder's medication plan for today. The back-end scheduling platform will schedule at the specified time and send a notification to the client that it is now time to take the medicine.

[0264] 3) To make reservations for dining, repairs, cleaning, physical therapy, etc., you can enter the reservation list page by touching the card or using voice, and select the service you need. The entire process of touching or using voice is done simultaneously.

[0265] 3. Central control service receives data and processes it

[0266] The central control service will receive upstream and downstream requests and perform partial processing or forward them to other services for processing, as follows:

[0267] 1) Synchronize the results of touch screen and voice operations to the Redis database in real time, and set appropriate expiration time and index strategy.

[0268] 2) If the audio stream received is transmitted from the client, the central control service will call the ASR (speech to text) service to convert the audio stream into text, and then give the text to the multi-modal human-computer interaction engine for processing.

[0269] 3) If the text received is transmitted by the multi-modal human-computer interaction engine, the central control service will call the TTS (text-to-speech) service to convert the text into an audio stream and transmit it to the client.

[0270] 4. Multi-mode human-computer interaction engine for human-computer dialogue

[0271] The multimodal human-computer interaction engine (NLP natural language processing platform) enables computers to understand and process human language, thereby achieving natural interaction with humans.

[0272] Its main technologies are as follows:

[0273] Document processing technology:

[0274] Information extraction: Automatically extract key information from documents, such as entities, relationships, events, etc., to provide basic data for subsequent dialogue understanding and generation.

[0275] Semantic understanding: Use natural language processing technology to deeply understand the content of the document and identify the document's theme, sentiment, intent, etc.

[0276] Knowledge graph construction: The structured information in the document is constructed into a knowledge graph to facilitate fast query and reasoning.

[0277] Model training technology:

[0278] Machine learning: Use various machine learning algorithms to learn from training data, optimize the parameters and structure of the model, and improve the accuracy and generalization ability of the model.

[0279] Deep learning: In particular, neural network models such as LSTM and Transformer have performed well in text processing, speech recognition, image recognition and other fields, and are the core technology for model training.

[0280] Multimodal fusion: Fuse data from different modalities (such as text, speech, and images) to train a model that can handle multimodal input.

[0281] Multi-modal central control input technology:

[0282] The multimodal central control can receive audio and video streams and text information streams, and transfer them to each modal perception service for parallel perception. The perception results are then transferred to the interactive central control to obtain a reply, and the reply information is distributed to each modal presentation service for processing and presentation.

[0283] Multimodal and sensory control: Receive audio, video and text information streams, distribute them to the sensory processing services of each modality, and transfer the sensory results (such as ASR results, user recognition results, etc.) to the interactive central control.

[0284] Multimodal response distribution: distribute or control the responses returned by the interactive central control to various modal presentation services (such as controlling the TTS service for streaming voice broadcasting and controlling the virtual person TTSA for presentation).

[0285] In addition to the most basic text interaction, the parallel perception and performance control of voice modalities (ASR, TTS) have been standardized. If there are needs for visual perception, virtual human performance, etc., the relevant modal services can be connected to the multimodal central control in a plug-in manner to quickly expand the central control processing capabilities.

[0286] Multimodal central control needs to be used in combination with semantic understanding capabilities and dialogue capabilities.

[0287] Semantic understanding ability:

[0288] The NLU service can perform semantic understanding of the user's original text and identify intent and word slot (entity) information.

[0289] The ability to recognize intents and entities in specific scenarios can be achieved by customizing intents, word slots, and training NLU models.

[0290] The engine has preset some common entity recognition capabilities, which can be used quickly without training models. If necessary, it also supports custom expansion or exclusion based on the preset recognition capabilities.

[0291] Common entities: such as date, time, city, address, mobile phone number, name, etc.

[0292] The semantic understanding capability can be used as an intermediate processing link in human-computer interaction, or it can be directly called through the interactive interface in pure NLU mode (input: voice or text; output: intent, word slot recognition results).

[0293] Conversational skills:

[0294] Conversational capabilities refer to the robot's ability to respond based on the variable information and semantic understanding results of each request, and the context of the conversation. Conversational capabilities can be divided into the following types according to scenarios and implementation mechanisms, including FAQ question and answer, task-based conversation, common greetings and cover-ups, knowledge graph question and answer, open domain chat, cognitive conversation, and general conversation skills. In specific business scenarios, a combination of one or more of these conversational capabilities may be used based on actual conditions and needs.

[0295] Natural Language Processing Technology:

[0296] Semantic understanding: The system needs to be able to accurately understand the semantic information in the text or voice input by the user, including intent, context, and entities. This usually involves semantic analysis and intent recognition in natural language processing. Dialogue management: The dialogue management module is responsible for controlling the flow of the dialogue, including understanding the user's request, generating appropriate responses, and guiding the in-depth dialogue. This requires the system to have flexible dialogue strategies and strong context management capabilities.

[0297] Generative responses: Based on natural language generation technology, the system can generate natural, fluent and contextual responses to meet user needs.

[0298] Data fusion and decision-making technology:

[0299] Multimodal data fusion: The system needs to fuse data from different modalities to form a comprehensive understanding of user intentions and behaviors. This usually involves technologies such as data preprocessing, feature extraction, and fusion algorithms.

[0300] Decision making: Based on the fused multimodal data, the system needs to make appropriate decisions to generate the best response or perform corresponding operations. This requires the system to have strong decision-making capabilities and real-time response capabilities.

[0301] Real-time interaction and feedback technology:

[0302] Real-time processing technology: Ensure that the system can process user input and generate responses in real time, reducing delays and waiting time.

[0303] User feedback mechanism: Collect user feedback on system responses to evaluate system performance and improve models.

[0304] Personalization and Adaptive Technology:

[0305] User portrait: Build user portraits based on their historical behaviors and preferences to provide a basis for personalized conversations.

[0306] Adaptive learning: The system can continuously learn and optimize its own models and behaviors based on user feedback and interaction data to improve user experience.

[0307] 5. Digital Human Management Platform for Configuration Management

[0308] The digital human management platform manages all configurations of digital humans, as well as stores and reads intermediate process data during the conversation, thus ensuring that basic data required for use in other links has a unified place to centrally store and read.

[0309] Please refer to Figure 3 , Figure 3 A touch screen and voice fusion interaction device 110 provided in an embodiment of the present invention includes:

[0310] An acquisition module 1101 is used to acquire a target demand instruction input by a target user, where the target demand instruction is a touch screen instruction or a voice instruction;

[0311] The interaction module 1102 is used to send the target demand instruction to the central control service for processing to obtain a processed target demand instruction; send the processed target demand instruction to the multimodal interaction engine for processing to obtain a user demand feedback result; process the user demand feedback result through the central control service to obtain a final demand feedback result and display it.

[0312] It should be noted that the implementation principle of the aforementioned touch screen and voice fusion interaction device 110 can refer to the implementation principle of the aforementioned touch screen and voice fusion interaction method, which will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the touch screen and voice fusion interaction device 110 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above touch screen and voice fusion interaction device 110. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in software form.

[0313] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0314] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned touch screen and voice fusion interaction device 110. Figure 4 As shown, Figure 4The computer device 100 is a block diagram of a structure of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a touch screen and voice fusion interaction device 110, a memory 111, a processor 112 and a communication unit 113.

[0315] In order to realize data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be realized through one or more communication buses or signal lines. The touch screen and voice fusion interaction device 110 includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the touch screen and voice fusion interaction device 110 stored in the memory 111, such as the software function modules and computer programs included in the touch screen and voice fusion interaction device 110.

[0316] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, it controls the computer device where the readable storage medium is located to execute the aforementioned touch screen and voice fusion interaction device 110.

[0317] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A touch screen and voice fusion interaction method, characterized in that: include: Obtaining a target demand instruction input by a target user, wherein the target demand instruction is a touch screen instruction or a voice instruction; Sending the target demand instruction to the central control service for processing to obtain the processed target demand instruction; Sending the processed target demand instruction to the multimodal interaction engine for processing to obtain a user demand feedback result; The user demand feedback result is processed by the central control service to obtain the final demand feedback result and display it.

2. The method according to claim 1, characterized in that In the case where the target demand instruction is the voice instruction, sending the target demand instruction to the central control service for processing to obtain the processed target demand instruction includes: The central control service calls the ASR service to convert the voice command to obtain a text command corresponding to the voice command.

3. The method according to claim 2, characterized in that The processing of the target demand instruction is sent to the multimodal interaction engine for processing to obtain the user demand feedback result, including: Extracting information and understanding semantics of the text instruction through the multimodal interaction engine to obtain key information of the text instruction; The multimodal interaction engine calls a pre-trained dialogue model to process the key information of the text instruction to obtain a text feedback result corresponding to the text instruction.

4. The method according to claim 3, characterized in that The processing of the user demand feedback result by the central control service to obtain the final demand feedback result and display it includes: The central control service calls the TTS service and the TTSA service to convert the text feedback result to obtain the corresponding voice feedback result and the virtual human feedback result; Perform streaming voice broadcast based on the voice feedback result, and control the virtual human to execute the virtual human feedback result.

5. The method according to claim 1, characterized in that The method further comprises: The target demand instruction is synchronized to the Redis database in real time for caching, and an expiration time and index strategy for the target demand instruction are configured.

6. The method according to claim 3, characterized in that: The method further comprises: Obtaining the historical behavior and user profile of the target user; The historical behavior and the user portrait are used as optimization data for the conversation model.

7. The method according to claim 1, characterized in that The target demand instruction includes a health monitoring instruction, and the final demand feedback result includes a health feedback result.

8. A touch screen and voice fusion interaction device, characterized in that: include: An acquisition module is used to acquire a target demand instruction input by a target user, wherein the target demand instruction is a touch screen instruction or a voice instruction; The interactive module is used to send the target demand instruction to the central control service for processing to obtain the processed target demand instruction; send the processed target demand instruction to the multimodal interactive engine for processing to obtain the user demand feedback result; The user demand feedback result is processed by the central control service to obtain the final demand feedback result and display it.

9. A computer device, characterized in that: The computer device comprises a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images