Dialogue methods, devices, electronic devices, storage media, and program products

Abstract

This application discloses a dialogue method, apparatus, electronic device, storage medium, and program product, relating to the field of electronic device technology. The method, applied to an electronic device, includes: receiving dialogue content input by a user; determining target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device; determining response content based on the target memory data and the dialogue content; and replying to the user with the response content and the target memory data as response information. This application integrates the user's multimodal memory data in the electronic device, incorporating it as part of the response during interaction. This move goes beyond simple text responses, making interactions more diverse and personalized, and enhancing the user's interactive experience.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, and more specifically, to a dialogue method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Artificial Intelligence (AI) is an emerging scientific and technological field. A primary research goal of AI is to enable machines to perform complex tasks that typically require human intelligence. For example, current intelligent dialogue systems (such as chatbots) are a practical application of AI technology. However, current intelligent dialogue systems are limited to simple text replies, lacking diversity and personalization, resulting in a poor user experience. Summary of the Invention

[0003] In view of the above problems, this application proposes a dialogue method, apparatus, electronic device, storage medium, and program product to solve the above problems.

[0004] In a first aspect, embodiments of this application provide a dialogue method applied to an electronic device. The method includes: receiving dialogue content input by a user; determining target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device; determining response content based on the target memory data and the dialogue content; and replying to the user with the response content and the target memory data as response information.

[0005] Secondly, embodiments of this application provide a dialogue device applied to an electronic device. The device includes: receiving dialogue content input by a user; determining target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device; determining reply content based on the target memory data and the dialogue content; and replying to the user with the reply content and the target memory data as reply information.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor performs the above-described method.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the above-described method.

[0008] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method.

[0009] The dialogue method, apparatus, electronic device, storage medium, and program product provided in this application receive dialogue content input by a user, determine target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device, determine reply content based on the target memory data and the dialogue content, and reply to the user as reply information using the reply content and the target memory data. By integrating the user's multimodal memory data in the electronic device and incorporating it as part of the reply during interaction, it is no longer limited to a single text reply, making the interaction more diverse and personalized, and improving the user's interactive experience. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a dialogue method provided in an embodiment of this application is shown;

[0012] Figure 2 This paper illustrates a first interface diagram of an electronic device provided in an embodiment of this application;

[0013] Figure 3 A schematic diagram of a second interface of an electronic device provided in an embodiment of this application is shown;

[0014] Figure 4 A schematic diagram of a third interface of an electronic device provided in an embodiment of this application is shown;

[0015] Figure 5 A schematic diagram of a fourth interface of the electronic device provided in an embodiment of this application is shown;

[0016] Figure 6 A flowchart illustrating a dialogue method provided in an embodiment of this application is shown;

[0017] Figure 7 A flowchart illustrating a dialogue method provided in an embodiment of this application is shown;

[0018] Figure 8 A flowchart illustrating a dialogue method provided in an embodiment of this application is shown;

[0019] Figure 9A flowchart illustrating a dialogue method provided in an embodiment of this application is shown;

[0020] Figure 10 A flowchart illustrating a dialogue method provided in an embodiment of this application is shown;

[0021] Figure 11 A flowchart illustrating a dialogue method provided in an embodiment of this application is shown;

[0022] Figure 12 A block diagram of a dialogue device provided in one embodiment of this application is shown;

[0023] Figure 13 A block diagram of an electronic device for performing a dialogue method according to an embodiment of this application is shown;

[0024] Figure 14 An embodiment of this application shows a storage unit for storing or carrying program code that implements the dialogue method according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0026] Currently, electronic devices are increasingly equipped with companion-type AI emotional chat assistants and intelligent agents to achieve intelligent dialogue with users. Specifically, intelligent dialogue systems provide an emotional AI chat experience based on historical chat records and support for Large Language Models (LLM). However, current intelligent chat systems can only provide text feedback based on the user's historical data, resulting in a single interaction modality, low relevance to the user, poor empathy, and a weak emotional experience, making it difficult for users to establish an emotional connection with the chat assistant.

[0027] To address the aforementioned problems, the inventors, through extensive research, have discovered and proposed a dialogue method, apparatus, electronic device, storage medium, and program product according to an embodiment of this application. By integrating the user's multimodal memory data in the electronic device and incorporating it as part of the response during interaction, this approach moves beyond simple text replies, making interactions more diverse and personalized, and enhancing the user's interactive experience. The specific dialogue method will be described in detail in subsequent embodiments.

[0028] Please see Figure 1 , Figure 1A flowchart illustrating a dialogue method provided in an embodiment of this application is shown. This dialogue method integrates multimodal memory data of the user in an electronic device, incorporating it as part of the response during interaction. This move beyond simple text replies, making interactions more diverse and personalized, and enhancing the user's interactive experience. In a specific embodiment, this dialogue method is applied to, for example... Figure 12 The dialogue device 200 and the electronic device 100 equipped with the dialogue device 200 are shown. Figure 13 The following will use an electronic device as an example to illustrate the specific process of this embodiment. Of course, it is understood that the electronic device used in this embodiment may include smartphones, tablets, wearable electronic devices, smart TVs, smart screens, etc., and is not limited thereto. The following will focus on... Figure 1 The process shown will be described in detail. The dialogue method may specifically include the following steps:

[0029] Step S110: Receive the dialogue content input by the user.

[0030] In this embodiment, the electronic device can receive dialogue content input by the user.

[0031] In some implementations, the electronic device may have a built-in intelligent dialogue system (e.g., a chatbot). Accordingly, when a user wishes to establish a chat with the intelligent dialogue system, they can input dialogue content into the intelligent dialogue system through the electronic device, and the electronic device can receive the dialogue content input by the user.

[0032] Optionally, the dialogue content entered by the user may include: text dialogue content, voice dialogue content, image dialogue content, video dialogue content, etc., without limitation.

[0033] As one feasible approach, when the dialogue content is voice dialogue content, the dialogue content input by the user via voice can be received; when the dialogue content is text dialogue content, image dialogue content, or video dialogue content, the dialogue content input by the user via input box can be received, etc., without any limitation.

[0034] Step S120: Determine the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device.

[0035] In some implementations, the electronic device may pre-record and store the user's multimodal memory data. It is understood that this multimodal memory data is all data related to the user personally, reflecting the user's personal preferences, status, and other information. This multimodal memory data may include text data, image data, video data, audio recording data, health and exercise data, etc., and is not limited thereto.

[0036] Optionally, the multimodal memory data of a user on an electronic device may include: image and video data in the electronic device's gallery; image and video data recorded using the electronic device's camera; Uniform Resource Locator (URL) data saved to the system; text data recorded in notes and memos; audio recording data; trip data in the calendar; health data (such as step count) identified by the electronic device's sensors; location-based service data (including but not limited to location data obtained when taking pictures and videos with the camera); data automatically recorded when using the electronic device (such as time, weather, and music playback data); and health data detected and generated when using a reference device (such as a wearable device) (including but not limited to sleep data, stress data, and exercise data).

[0037] In this embodiment, upon receiving dialogue content input by the user, the memory data corresponding to the dialogue content can be determined as the target memory data from the user's multimodal memory data in the electronic device.

[0038] In some implementations, upon receiving user-input dialogue content, Natural Language Processing (NLP) techniques can be used to parse the dialogue content, extracting key information such as topic, entities (names, locations, times, etc.), and intent. Keywords or phrases potentially related to multimodal memory data can be identified from the dialogue content. These extracted keywords or phrases are then matched against the content in the multimodal memory data to find records corresponding to these keywords or phrases. Simultaneously, contextual information such as timestamps and locations can be considered to further filter multimodal memory data more relevant to the dialogue content. Based on the matching results and contextual information, the relevance of each memory record to the dialogue content is evaluated. The filtered memory data is then ranked according to factors such as relevance and timestamps (e.g., prioritizing the newest or most relevant data), ultimately determining the target memory data corresponding to the dialogue content.

[0039] Optionally, the number of target memory data can be one or more, and there is no limitation here.

[0040] Step S130: Determine the response content based on the target memory data and the dialogue content.

[0041] In this embodiment, once the target memory data corresponding to the dialogue content is determined, the response content can be determined based on the target memory data and the dialogue content. It is understood that the response content should be associated with both the target memory data and the dialogue content.

[0042] In some implementations, when target memory data corresponding to the dialogue content is determined, the target memory data can be interpreted to determine the information, events, and / or emotions contained in the target memory data, analyze the keywords, phrases, and contextual content in the dialogue content, determine the specific content that the user wants to know or discuss, and determine the response content based on the interpretation of the target memory data and the analysis of the dialogue content.

[0043] Step S140: Reply to the user with the reply content and the target memory data as reply information.

[0044] In this embodiment, if the reply content and the target memory data are obtained, the reply content and the target memory data can be used together as reply information to the user.

[0045] In some implementations, when both the response content and the target memory data are obtained, the target memory data and the response content can be displayed simultaneously on the interface, so that the target memory data and the response content are used together as the response information to reply to the user through the interface display.

[0046] As an feasible approach, if the target memory data and the response content are displayed together as the reply information through an interface, then the target memory data and the response content can be displayed simultaneously on the interface according to preset display parameters. Optionally, the preset display parameters may include: the target memory data is displayed above the response content, the target memory data is displayed below the response content, or the response content can be displayed around the target memory data, etc., which are not limited here.

[0047] In some implementations, when the reply content and the target memory data are obtained, the target memory data and the reply content can be read aloud sequentially by voice, so that the target memory data and the reply content are used together as reply information to the user by voice reading.

[0048] As an example, please refer to Figure 2 , Figure 2 A schematic diagram of a first interface of an electronic device provided in an embodiment of this application is shown. For example... Figure 2 As shown, when the user enters the dialogue content "I feel very lonely lately. It seems that my friends don't have time to play with me anymore after they get married," the determined target memory data can be "the user's dog video," and the reply content can be "Don't worry, my dog ​​and I are always here." The target memory data "the user's dog video" and the reply content "Don't worry, my dog ​​and I are always here" will be displayed together on the interface as a reply to the user.

[0049] As yet another example, please refer to Figure 3 , Figure 3A schematic diagram of a second interface of an electronic device provided in an embodiment of this application is shown. For example... Figure 3 As shown, when the user enters the dialogue "It's been raining every day this month, so annoying," the target memory data can be "a cherry blossom photo taken by the user last April," and the reply can be "But this month also has its own unique beauty, look at the cherry blossoms from last April." The target memory data "a cherry blossom photo taken by the user last April" and the reply "But this month also has its own unique beauty, look at the cherry blossoms from last April" are displayed together on the interface as a reply to the user.

[0050] As another example, please refer to Figure 4 , Figure 4 A schematic diagram of a third interface of an electronic device provided in an embodiment of this application is shown. For example... Figure 4 As shown, when the user enters the dialogue content "A friend gave me a collection of poems by XXX, which is wonderful", the determined target memory data can be "a line of poetry by this poet that the user copied last year", and the reply content can be "You have a special connection with this poet, as you copied her short poems last year". The target memory data "a line of poetry by this poet that the user copied last year" and the reply content "You have a special connection with this poet, as you copied her short poems last year" are displayed together on the interface as a reply to the user.

[0051] As another example, please refer to Figure 5 , Figure 5 A schematic diagram of a fourth interface of the electronic device provided in an embodiment of this application is shown. For example... Figure 5 As shown, when the user enters the dialogue content "I feel inexplicably bad", the determined target memory data can be "the user's health data", and the reply content can be "feeling down may just be because you didn't sleep well". The target memory data "the user's health data" and the reply content "feeling down may just be because you didn't sleep well" are displayed together on the interface as reply information to the user.

[0052] It is understandable that the above dialogue method has at least the following effects:

[0053] First, by integrating multimodal memory data, chats with intelligent assistants are no longer limited to simple text replies. They can proactively access data such as images and videos based on context, making interactions more diverse and providing users with a completely new communication experience, greatly enriching the forms of interaction with AI.

[0054] Secondly, by leveraging users' multimodal memory data, intelligent assistants can provide a unique interactive experience for each user. Based on the user's personal preferences and experiences, relevant memory data can be retrieved and used to respond in a customized manner, making the interaction more personalized and meeting the user's need for a unique experience.

[0055] Third, when a smart assistant retrieves a user's memory data during a conversation, it can help the user recall past experiences and emotions. For example, seeing old photos or hearing familiar recordings can trigger strong emotional responses, thereby enhancing the emotional resonance between the user and the smart assistant and making the communication deeper and more meaningful.

[0056] Fourth, by recalling memories as the content of responses, emotional communication and connection are established. This deep emotional connection makes users feel understood and cared for, thereby enhancing their trust in the intelligent assistant.

[0057] One embodiment of this application provides a dialogue method that receives dialogue content input by a user, determines target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device, determines reply content based on the target memory data and the dialogue content, and replies to the user with the reply content and the target memory data as reply information. By integrating the user's multimodal memory data in the electronic device and incorporating it as part of the reply during the interaction, it is no longer limited to a single text reply, making the interaction more diverse and personalized, and improving the user's interactive experience.

[0058] Please see Figure 6 , Figure 6 A schematic flowchart of a dialogue method provided in an embodiment of this application is shown. In this embodiment, the method is applied to an electronic device, and will be discussed below. Figure 6 The process shown will be described in detail. The dialogue method may specifically include the following steps:

[0059] Step S210: Receive the dialogue content input by the user.

[0060] For a detailed description of step S210, please refer to step S110, which will not be repeated here.

[0061] Step S220: Determine the contextual content that is related to the dialogue content.

[0062] In this embodiment, upon receiving user-input dialogue content, contextual content that is related to the dialogue content can be determined. Optionally, the contextual content that is related to the dialogue content may include previous dialogue records, related documents, emails, social media posts, etc.

[0063] In some implementations, upon receiving user-input dialogue content, keywords or phrases can be extracted from the dialogue content. Based on the context of the dialogue (such as the scene and time of the dialogue), the scope of the search context content can be determined. Then, using the keywords extracted from the dialogue, a search is performed within the determined search scope to find the context content corresponding to these keywords. The searched context content is evaluated to determine its relevance to the dialogue content. Based on the relevance of the context content to the dialogue content, the searched content is prioritized, and context content that is content-related to the dialogue content is determined based on the priority ranking.

[0064] Step S230: Based on the context content, determine the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device.

[0065] In this embodiment, when it is determined that there is contextual content related to the dialogue content, the target memory data corresponding to the dialogue content can be determined from the user's multimodal memory data in the electronic device based on the contextual content.

[0066] In some implementations, when contextual content related to the dialogue is identified, the search scope can be narrowed based on key elements extracted from the contextual content to focus on multimodal memory data that may contain these key elements. As one approach, for text-type memory data, keyword search or natural language processing techniques can be used for matching to find records similar to the contextual content as target memory data; for image and video-type memory data, image recognition or video analysis techniques can be used to identify images or video clips that match the scenes, objects, or people described in the contextual content as target memory data, etc., without limitation.

[0067] Step S240: Determine the response content based on the target memory data and the dialogue content.

[0068] Step S250: Reply to the user with the reply content and the target memory data as reply information.

[0069] For a detailed description of steps S240-S250, please refer to steps S130-S140, which will not be repeated here.

[0070] One embodiment of this application provides a dialogue method that receives dialogue content input by a user, determines contextual content related to the dialogue content, determines target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device based on the contextual content, determines response content based on the target memory data and the dialogue content, and replies to the user with the response content and the target memory data as response information. Compared to Figure 1 The dialogue method shown in this embodiment also determines the contextual content that is related to the dialogue content, and determines the target memory data corresponding to the dialogue content based on the context, thereby improving the accuracy of the determined target memory data and enhancing the interactive experience.

[0071] Please see Figure 7 , Figure 7 A schematic flowchart of a dialogue method provided in an embodiment of this application is shown. In this embodiment, the method is applied to an electronic device, and will be discussed below. Figure 7 The process shown will be described in detail. The dialogue method may specifically include the following steps:

[0072] Step S310: Receive the dialogue content input by the user.

[0073] For a detailed description of step S310, please refer to step S110, which will not be repeated here.

[0074] Step S320: Determine the multimodal memory data within the authorization range set by the user from the multimodal memory data of the electronic device.

[0075] Specifically, this embodiment supports users setting authorization ranges for multimodal memory data stored on electronic devices. It can be understood that by setting authorization ranges, the user's multimodal memory data on electronic devices can be divided into two parts: one part is the multimodal memory data within the authorized range, which can be accessed without authentication; the other part is the multimodal memory data outside the authorized range, which requires authentication for access. This protects user privacy and enhances information security.

[0076] In this embodiment, upon receiving dialogue content input by the user, the multimodal memory data within the user's authorized range can be determined from the user's multimodal memory data in the electronic device. For example, if the multimodal memory data within the user's authorized range includes image data, text data, and video data, and the multimodal memory data outside the authorized range includes audio recording data and health / exercise data, then it can be determined that the multimodal memory data within the user's authorized range includes image data, text data, and video data, but excludes audio recording data and health / exercise data.

[0077] Optionally, this embodiment also supports users withdrawing authorization and modifying the scope of authorization, etc., which are not limited here.

[0078] Step S330: Determine the target memory data corresponding to the dialogue content from the multimodal memory data within the authorized scope.

[0079] In this embodiment, when the authorized scope of multimodal memory data is determined, the target memory data corresponding to the dialogue content can be identified from the multimodal memory data within the authorized scope. It is understood that this embodiment searches for multimodal memory data within the user's authorized scope, thus avoiding the leakage of user privacy data and protecting user privacy, thereby enhancing information security.

[0080] Specifically, this embodiment supports users setting both the authorization scope and authorization time for multimodal memory data on electronic devices. Understandably, by setting the authorization scope and authorization time, the user's multimodal memory data on the electronic device can be divided into four parts. One part is multimodal memory data within the authorized scope and time, which can be accessed without authentication. Another part is multimodal memory data outside the authorized scope and time, which cannot be accessed. The third part is multimodal memory data outside the authorized scope, which requires authentication for access. This protects user privacy, improves information security, increases time control over multimodal memory data, and enhances the diversity and personalization of multimodal memory data usage.

[0081] In some implementations, when multimodal memory data within an authorized range is determined, multimodal memory data within an authorized time period set by the user can be determined from the multimodal memory data within the authorized range, and target memory data corresponding to the dialogue content can be determined from the multimodal memory data within the authorized time period.

[0082] For example, if a user sets the authorized multimodal memory data to include image data, text data, and video data, and sets the authorized time for image data to 8:00-12:00, the authorized time for text data to be unlimited, and the authorized time for video data to 15:00-18:00, then if the current time is 9:00, it can be determined that the multimodal memory data within the user-set authorized time includes image data and text data, but excludes video data. In this case, the target memory data corresponding to the dialogue content can be determined from the text data and image data.

[0083] Step S340: Determine the response content based on the target memory data and the dialogue content.

[0084] Step S350: Reply to the user with the reply content and the target memory data as reply information.

[0085] For a detailed description of steps S340-S350, please refer to steps S130-S140, which will not be repeated here.

[0086] An embodiment of this application provides a dialogue method that receives dialogue content input by a user, determines multimodal memory data within the user's authorized range from the user's multimodal memory data in the electronic device, determines target memory data corresponding to the dialogue content from the multimodal memory data within the authorized range, determines reply content based on the target memory data and the dialogue content, and replies to the user with the reply content and the target memory data as reply information. Compared to Figure 1 The dialogue method shown in this embodiment also supports users in setting the authorization scope of multimodal memory data and selecting memory data as part of the response within the authorization scope, which can protect user privacy, improve information security and user experience.

[0087] Please see Figure 8 , Figure 8 A schematic flowchart of a dialogue method provided in an embodiment of this application is shown. In this embodiment, the method is applied to an electronic device, and will be discussed below. Figure 8 The process shown will be described in detail. The dialogue method may specifically include the following steps:

[0088] Step S410: Determine the user's recorded data on the electronic device and / or a reference device communicating with the electronic device.

[0089] Optionally, the electronic device may be communicatively connected to a reference device. The electronic device and the reference device may belong to the same user and be used to record the same user's memory data. For example, the electronic device and the reference device may be devices logged into the same account. Furthermore, the electronic device and the reference device may be of the same type; for example, both may be smartphones or wearable devices. Alternatively, the electronic device and the reference device may be of different types; for example, the electronic device may be a smartphone and the reference device may be a wearable device, or vice versa. No limitation is imposed here.

[0090] In this embodiment, the electronic device can be a smartphone, and the reference device can be a wearable device, such as a smartwatch.

[0091] In this embodiment, the user's recorded data on the electronic device and / or the reference electronic device can be determined. That is, only the user's recorded data on the electronic device can be obtained, only the user's recorded data on the reference device can be obtained, or the user's recorded data on both the electronic device and the reference device can be obtained simultaneously; no limitation is made here.

[0092] The data recorded by the user on the electronic device and / or reference device can be understood as data actively recorded and saved by the user on the electronic device and / or reference device, including but not limited to: image and video data in the gallery of the electronic device; image and video data recorded using the camera of the electronic device; Uniform Resource Locator (URL) data saved to the system; text data recorded in notes and memos; audio recording data; and itinerary data in the calendar.

[0093] In some implementations, it is possible to determine the data recorded by the user at all times of the electronic device and / or reference device; it is possible to determine the data recorded by the electronic device and / or reference device within a specified time period; it is possible to determine the data recorded by the electronic device and / or reference device within a recent period, etc., without limitation.

[0094] In some implementations, the recorded data of the user at all locations of the electronic device and / or reference device can be determined; the recorded data of the electronic device and / or reference device at a specified location can be determined, etc., without limitation.

[0095] Step S420: Determine the user's usage data on the electronic device and / or the reference device.

[0096] In this embodiment, user usage data on the electronic device and / or the reference device can be determined. That is, user usage data on the electronic device can be obtained only, user usage data on the reference device can be obtained only, or user usage data on both the electronic device and the reference device can be obtained simultaneously. No limitation is made here.

[0097] Among them, user usage data on electronic devices and / or reference devices can be understood as data passively generated by users on electronic devices and / or reference devices, including but not limited to health data identified by the sensors of electronic devices (such as step count); data based on location services (including but not limited to location data obtained when taking pictures and videos with a camera); data automatically recorded when using electronic devices (such as time, weather, music playback data); and health data detected and generated when using reference devices (such as wearable devices) (including but not limited to sleep data, stress data, exercise data, etc.).

[0098] In some implementations, user usage data at all times of the electronic device and / or reference device can be determined; usage data of the electronic device and / or reference device within a specified time period can be determined; usage data of the electronic device and / or reference device within a recent period can be determined, etc., without limitation.

[0099] In some implementations, user usage data at all locations of the electronic device and / or reference device can be determined; user usage data at a specific location of the electronic device and / or reference device can be determined, etc., without limitation.

[0100] Step S430: Based on the recorded data and the usage data, obtain the user's multimodal memory data in the electronic device.

[0101] In this embodiment, given the memory data and usage data, multimodal memory data of the user in the electronic device can be obtained based on the recorded data and usage data.

[0102] In some implementations, when memory data and usage data are obtained, the memory data and usage data can be combined to obtain multimodal memory data of the user in the electronic device.

[0103] In some implementations, when both memory data and usage data are obtained, both memory data and usage data can be added to a memory dataset to obtain multimodal memory data of the user on the electronic device.

[0104] Step S440: Receive the dialogue content input by the user.

[0105] Step S450: Determine the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device.

[0106] Step S460: Determine the response content based on the target memory data and the dialogue content.

[0107] Step S470: Reply to the user with the reply content and the target memory data as reply information.

[0108] For a detailed description of steps S440-S470, please refer to steps S110-S140, which will not be repeated here.

[0109] An embodiment of this application provides a dialogue method that determines user-recorded data on an electronic device and / or a reference device communicating with the electronic device, determines user-usage data on the electronic device and / or the reference device, obtains multimodal memory data of the user on the electronic device based on the recorded data and usage data, receives dialogue content input by the user, determines target memory data corresponding to the dialogue content from the user's multimodal memory data on the electronic device, determines response content based on the target memory data and the dialogue content, and replies to the user with the response content and the target memory data as response information. Compared to Figure 1 The dialogue method shown in this embodiment also integrates the user's active recording data and passive usage data on electronic devices and / or reference devices as multimodal memory data, thereby improving the diversity and coverage of memory data and enhancing the interactive experience.

[0110] Please see Figure 9 , Figure 9 A schematic flowchart of a dialogue method provided in an embodiment of this application is shown. In this embodiment, the method is applied to an electronic device, and will be discussed below. Figure 9 The process shown will be described in detail. The dialogue method may specifically include the following steps:

[0111] Step S510: Receive dialogue content input by the user, wherein the dialogue content is input via voice.

[0112] For a detailed description of step S510, please refer to step S110, which will not be repeated here.

[0113] Step S520: Determine the tone information when the user inputs the dialogue content.

[0114] Optionally, users can interact with the intelligent dialogue system via voice. Accordingly, users can input dialogue content via voice, and electronic devices can receive the dialogue content input by the user via voice.

[0115] In this embodiment, the tone of voice when the user inputs dialogue content can be determined.

[0116] As a feasible approach, tone information of the user's input dialogue can be determined through speech recognition and text conversion. For example, speech recognition technology can be used to convert the user's voice input into text content. Once the text content is obtained, natural language processing technology can be used to analyze the text content and extract keywords, phrases, etc., in order to determine the tone information of the user's input dialogue content.

[0117] As another feasible approach, tone information when a user inputs dialogue content can be determined through speech feature extraction. For example, the user's speech signal can be preprocessed, such as by denoising and filtering, to extract features that reflect tone information, such as pitch, volume, speech rate, and rhythm. These extracted features can then be used to determine the tone information when the user inputs dialogue content.

[0118] As another feasible approach, affective computing and tone analysis can be used to determine the tone information of the user's input dialogue content. For example, machine learning or deep learning techniques can be used to train an affective recognition model, which can identify the user's emotional state, such as happiness, sadness, and anger, based on extracted speech features. On the basis of affective recognition, the user's tone information can be further classified, such as friendly, neutral, hostile, and polite, etc., without limitation.

[0119] As another feasible approach, the tone of voice when a user enters dialogue content can be determined by combining contextual information. For example, the user's previous dialogue history can be analyzed to understand the user's background, interests, and habits, and the tone of voice when the user enters dialogue content can be determined based on the dialogue history and dialogue content.

[0120] Step S530: Determine the target memory data corresponding to the tone information and the dialogue content from the user's multimodal memory data in the electronic device.

[0121] In this embodiment, given the dialogue content input by the user and the tone information of the user when inputting the dialogue content, the memory data corresponding to the tone information and dialogue content can be determined from the user's multimodal memory data in the electronic device as the target memory data.

[0122] In some implementations, given the user-input dialogue content and tone information, natural language processing techniques can be used to parse the dialogue content and extract key information such as keywords, phrases, themes, and logical relationships between sentences. The tone and speech rate can be analyzed to infer the user's emotional state, attitude, and intention. Subsequently, target memory data can be identified from the user's multimodal memory data on the electronic device that corresponds to the user's emotional state, attitude, and intention as represented by the key information and tone of the dialogue content.

[0123] Step S540: Determine the response content based on the target memory data and the dialogue content.

[0124] Step S550: Reply to the user with the reply content and the target memory data as reply information.

[0125] For a detailed description of steps S540-S550, please refer to steps S130-S140, which will not be repeated here.

[0126] One embodiment of this application provides a dialogue method that receives dialogue content input by a user via voice, determines the tone information of the user's input, determines target memory data corresponding to the tone information and dialogue content from the user's multimodal memory data in the electronic device, determines response content based on the target memory data and dialogue content, and replies to the user with the response content and target memory data as response information. Compared to Figure 1 The dialogue method shown in this embodiment also determines the user's tone information when the user inputs dialogue content by voice, and determines the memory data by referring to the tone information, thereby improving the accuracy of the determined memory data.

[0127] Please see Figure 10 , Figure 10 A schematic flowchart of a dialogue method provided in an embodiment of this application is shown. In this embodiment, the method is applied to an electronic device, and will be discussed below. Figure 10 The process shown will be described in detail. The dialogue method may specifically include the following steps:

[0128] Step S610: Receive the dialogue content input by the user.

[0129] For a detailed description of step S610, please refer to step S110, which will not be repeated here.

[0130] Step S620: Determine the facial expression information when the user inputs the dialogue content.

[0131] In this embodiment, facial expressions can be determined when the user inputs dialogue content.

[0132] Optionally, the electronic device may include a camera, which may include a front-facing camera, a rear-facing camera, a retractable camera, a rotating camera, etc., and is not limited thereto. Based on this, during the process of the user inputting dialogue content, the camera of the electronic device can capture the user's facial image while the user is inputting dialogue content, and the captured facial image can be analyzed to determine the user's facial expression information while inputting dialogue content.

[0133] In some implementations, during user input of dialogue content, an image of the user's input is captured by the electronic device's camera. A pre-trained face detection model (such as Haar features, HOG+SVM, or a deep learning model) can detect facial regions in the captured image. If a face is detected, it is extracted from the captured image and scaled to a size suitable for input into the facial expression recognition model. To reduce computational complexity and noise, the image can be converted to grayscale and normalized, which will not be elaborated further here. Once a face is extracted, features can be extracted (e.g., manual feature extraction or deep learning feature extraction). These extracted features are then input into the facial expression recognition model to obtain the facial expression information output by the model when the user inputs the dialogue content.

[0134] In some implementations, upon receiving user-input dialogue content, the electronic device's camera can capture an image and detect whether the user's facial information has been captured. Optionally, if the user's facial information is not captured, it can be assumed that the user may not be facing the electronic device's camera, and the camera can be rotated to capture the user's facial information. Optionally, if the user's facial information is not captured, it can be assumed that the user may not be facing the electronic device's camera, and a prompt message can be output, which may prompt the user to face the electronic device's camera.

[0135] Step S630: Determine the target memory data corresponding to the facial expression information and the dialogue content from the user's multimodal memory data in the electronic device.

[0136] In this embodiment, given the dialogue content input by the user and the facial expressions used when inputting the dialogue content, the memory data corresponding to the facial expressions and dialogue content can be determined from the user's multimodal memory data in the electronic device as the target memory data.

[0137] In some implementations, given the user-input dialogue content and tone of voice, natural language processing techniques can be used to parse the dialogue content and extract key information such as keywords, phrases, themes, and logical relationships between sentences. Facial expressions can be analyzed to infer the user's emotional state, attitude, and intention. Subsequently, memory data corresponding to the key information of the dialogue content and the user's emotional state, attitude, and intention as represented by facial expressions can be identified from the user's multimodal memory data on the electronic device and used as target memory data.

[0138] Step S640: Determine the response content based on the target memory data and the dialogue content.

[0139] Step S650: Reply to the user with the reply content and the target memory data as reply information.

[0140] For a detailed description of steps S640-S650, please refer to steps S130-S140, which will not be repeated here.

[0141] One embodiment of this application provides a dialogue method that receives dialogue content input by a user, determines facial expression information when the user inputs the dialogue content, determines target memory data corresponding to the facial expression information and dialogue content from the user's multimodal memory data in the electronic device, determines reply content based on the target memory data and dialogue content, and replies to the user with the reply content and target memory data as reply information. Compared to... Figure 1 In the dialogue method shown in this embodiment, the facial expression information of the user when inputting dialogue content is also determined, and the memory data is determined by referring to the facial expression information, thereby improving the accuracy of the determined memory data.

[0142] Please see Figure 11 , Figure 11 A schematic flowchart of a dialogue method provided in an embodiment of this application is shown. In this embodiment, the method is applied to an electronic device, and will be discussed below. Figure 11 The process shown will be described in detail. The dialogue method may specifically include the following steps:

[0143] Step S710: Receive the dialogue content input by the user.

[0144] Step S720: Determine the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device.

[0145] Step S730: Determine the response content based on the target memory data and the dialogue content.

[0146] For a detailed description of steps S710-S730, please refer to steps S110-S130, which will not be repeated here.

[0147] Step S740: If the target memory data includes the user's photo, then collect the user's facial information.

[0148] In this embodiment, when target memory data corresponding to the dialogue content is determined, the target memory data can be analyzed to determine whether it includes a user's photo. If it is determined that the target memory data includes a user's photo, the user's facial information can be collected; if it is determined that the target memory data does not include a user's photo, the user's facial information can be left uncollected, and the reply content and the target memory data can be used together as the reply information to the user.

[0149] In some implementations, when target memory data corresponding to the dialogue content is determined, the data type corresponding to the target memory data can be detected. If the data type of the target memory data is detected to meet a specified data type, the target memory data can be analyzed to determine whether it includes the user's photo. Optionally, the specified data type may include photo data types and video data types; if the data type of the target memory data is detected to not meet the specified data type, it is not necessary to determine whether the target memory data includes the user's photo, and the target memory data and the reply content can be directly used together as the reply information to the user.

[0150] In some implementations, the electronic device may pre-set and store the user's facial features. When target memory data corresponding to the dialogue content is determined, image recognition technology (such as facial recognition algorithms) can be used to identify whether the target memory data contains facial features. If facial features are detected in the target memory data, these features can be compared with the user's facial features to determine if they match. If the facial features in the target memory data match the user's facial features, it can be determined that the target memory data includes the user's photograph; if the facial features do not match, it can be determined that the target memory data does not include the user's photograph.

[0151] Optionally, the electronic device may include a camera, which may include a front-facing camera, a rear-facing camera, a retractable camera, a rotating camera, etc., and is not limited thereto. Based on this, if it is determined that the target memory data includes the user's photo, the user's facial information can be collected through the camera of the electronic device.

[0152] In some implementations, if it is determined that the target memory data includes a user's photo, an image can be captured using the camera of the electronic device, and it can be detected whether the user's facial information has been captured. Optionally, if the user's facial information is not captured, it can be assumed that the user may not be facing the camera of the electronic device, and the camera of the electronic device can be rotated to capture the user's facial information. Optionally, if the user's facial information is not captured, it can be assumed that the user may not be facing the camera of the electronic device, and a prompt message can be output, wherein the prompt message can prompt the user to face the camera of the electronic device.

[0153] Step S750: Reply to the user with the reply content, the target memory data, and the facial information as the reply information.

[0154] In this embodiment, upon obtaining the reply content, target memory data, and facial information, the reply content, target memory data, and facial information can be used together as the reply information to the user. It is understood that the target memory data includes the user's photo. Comparing and displaying the user's facial information with their photo allows the user to intuitively perceive changes in their appearance during the chat, enhancing the user's interactive experience.

[0155] One embodiment of this application provides a dialogue method that receives dialogue content input by a user, determines target memory data corresponding to the dialogue content from the user's multimodal memory data in an electronic device, determines reply content based on the target memory data and the dialogue content, and if the target memory data includes a photo of the user, collects the user's facial information, and replies to the user with the reply content, target memory data, and facial information. Compared to Figure 1 The dialogue method shown in this embodiment, when the target memory data includes the user's photo, also collects the user's face as reply information to enhance the comparison of the user's image and improve the user's intuitive feeling and interactive experience.

[0156] Please see Figure 12 , Figure 12 A block diagram of a dialogue device according to an embodiment of this application is shown. This dialogue device 200 is applied to the aforementioned electronic device, and will be discussed below. Figure 12 The diagram illustrates that the dialogue device 200 includes: a dialogue content receiving module 210, a target memory data determining module 220, a response content determining module 230, and a response information determining module 240, wherein:

[0157] The dialogue content receiving module 210 is used to receive dialogue content input by the user.

[0158] The target memory data determination module 220 is used to determine the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device.

[0159] Furthermore, the target memory data determination module 220 includes: a context content determination submodule and a first target memory data determination submodule, wherein:

[0160] The context content determination submodule is used to determine the context content that is related to the dialogue content.

[0161] The first target memory data determination submodule is used to determine the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device based on the context content.

[0162] Furthermore, the target memory data determination module 220 includes: a multimodal memory data determination submodule and a second target memory data determination submodule, wherein:

[0163] The multimodal memory data determination submodule is used to determine, from the multimodal memory data of the user in the electronic device, the multimodal memory data within the authorization range set by the user.

[0164] The second target memory data determination submodule is used to determine the target memory data corresponding to the dialogue content from the multimodal memory data within the authorized range.

[0165] Furthermore, the second target memory data determination submodule includes: a multimodal memory data determination unit and a target memory data determination unit, wherein:

[0166] A multimodal memory data determination unit is used to determine multimodal memory data within the authorized time period set by the user from the multimodal memory data within the authorized range.

[0167] The target memory data determination unit is used to determine the target memory data corresponding to the dialogue content from the multimodal memory data within the authorized time period.

[0168] Furthermore, the dialogue content is input via voice, and the target memory data determination module 220 includes: a tone information determination submodule and a third target memory data determination submodule, wherein:

[0169] The tone information determination submodule is used to determine the tone information when the user inputs the dialogue content.

[0170] The third target memory data determination submodule is used to determine the target memory data corresponding to the tone information and the dialogue content from the user's multimodal memory data in the electronic device.

[0171] Furthermore, the target memory data determination module 220 includes: an expression information determination submodule and a fourth target data determination submodule, wherein:

[0172] The facial expression information determination submodule is used to determine the facial expression information when the user inputs the dialogue content.

[0173] The fourth target memory data determination submodule is used to determine the target memory data corresponding to the facial expression information and the dialogue content from the user's multimodal memory data in the electronic device.

[0174] The response content determination module 230 is used to determine the response content based on the target memory data and the dialogue content.

[0175] The reply information determination module 240 is used to reply to the user with the reply content and the target memory data as reply information.

[0176] Furthermore, the response information determination module 240 includes: a face information acquisition submodule and a response information determination submodule, wherein:

[0177] The facial information collection submodule is used to collect the user's facial information if the target memory data includes the user's photo.

[0178] The response information determination submodule is used to reply to the user with the response content, the target memory data, and the facial information.

[0179] Furthermore, the dialogue device 200 also includes: a recording data determination module, a usage data determination module, and a multimodal memory data determination module, wherein:

[0180] A data recording determination module is used to determine the data recorded by the user on the electronic device and / or a reference device communicating with the electronic device.

[0181] A data determination module is used to determine the user's usage data in the electronic device and / or the reference device.

[0182] A multimodal memory data determination module is used to obtain the user's multimodal memory data in the electronic device based on the recorded data and the usage data.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0184] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0185] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0186] Please see Figure 13 This document illustrates a structural block diagram of an electronic device 100 provided in an embodiment of this application. The electronic device 100 can be a smartphone, tablet computer, e-reader, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0187] The processor 110 may include one or more processing cores. The processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and by calling data stored in the memory 120. Optionally, the processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 110 and may be implemented separately using a communication chip.

[0188] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0189] In some embodiments, the electronic device 100 may further include a touch screen, which is used to display information input by the user, information provided to the user, and various graphical user interfaces of the electronic device 100. These graphical user interfaces may be composed of graphics, text, icons, numbers, video, and any combination thereof. In one example, the touch screen may be a liquid crystal display (LCD) or an organic light-emitting diode (OLED), without limitation.

[0190] In some embodiments, the electronic device 100 may further include a camera for collecting user motion data. Optionally, the camera may include a front-facing camera, a rear-facing camera, a telescopic camera, a rotating camera, etc., and is not limited thereto.

[0191] In some embodiments, the electronic device 100 may further include sensors, including a light sensor that can be used to turn off the touchscreen display when an object approaches the touchscreen, such as when the body of the electronic device is moved to the ear. The sensor may also include a pressure sensor that can detect pressure generated by pressing on the electronic device; that is, the pressure sensor can detect pressure generated by contact or pressing between the user and the electronic device, such as pressure generated by contact or pressing between the user's ear or finger and the electronic device. The sensor may also include a gravity sensor that can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. This can be used for applications that identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Additionally, the electronic device may also be equipped with other sensors such as gyroscopes, barometers, and hygrometers, which are not limited here.

[0192] In some embodiments, the electronic device 100 may also include an artificial intelligence module, which may be integrated into the processor 110 of the electronic device 100 to improve the intelligence level and performance of the electronic device.

[0193] Of course, electronic devices may include many other components, which will not be elaborated here.

[0194] Please see Figure 14 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 300 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0195] The computer-readable storage medium 300 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 300 has storage space for program code 310 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 310 may be compressed, for example, in a suitable form.

[0196] In some embodiments, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described dialogue method.

[0197] In summary, the dialogue method, apparatus, electronic device, storage medium, and program product provided in this application receive dialogue content input by a user, determine target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device, determine reply content based on the target memory data and the dialogue content, and reply to the user as reply information using the reply content and the target memory data. By integrating the user's multimodal memory data in the electronic device and incorporating it as part of the reply during interaction, it is no longer limited to a single text reply, making the interaction more diverse and personalized, and improving the user's interactive experience.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A dialogue method, characterized in that, Applied to electronic devices, the method includes: Receive user-input dialogue content; From the user's multimodal memory data on the electronic device, determine the target memory data corresponding to the dialogue content; Based on the target memory data and the dialogue content, determine the response content; The reply content and the target memory data are used as reply information to reply to the user.

2. The method according to claim 1, characterized in that, The step of determining the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device includes: Determine the contextual content that is related to the dialogue content; Based on the context content, the target memory data corresponding to the dialogue content is determined from the user's multimodal memory data in the electronic device.

3. The method according to claim 1, characterized in that, The step of determining the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device includes: From the user's multimodal memory data in the electronic device, determine the multimodal memory data within the authorization range set by the user; From the multimodal memory data within the authorized scope, determine the target memory data corresponding to the dialogue content.

4. The method according to claim 3, characterized in that, Determining the target memory data corresponding to the dialogue content from the multimodal memory data within the authorized scope includes: From the multimodal memory data within the authorized range, determine the multimodal memory data within the authorized time period set by the user; From the multimodal memory data within the authorized time period, determine the target memory data corresponding to the dialogue content.

5. The method according to any one of claims 1-4, characterized in that, Before receiving the user-input dialogue content, the following is also included: Determine the user's recorded data on the electronic device and / or a reference device communicating with the electronic device; Determine the user's usage data on the electronic device and / or the reference device; Based on the recorded data and the usage data, the user's multimodal memory data in the electronic device is obtained.

6. The method according to any one of claims 1-4, characterized in that, The corresponding content is input via voice. Determining the target memory data corresponding to the dialogue content from the user's multimodal memory data on the electronic device includes: Determine the tone of voice of the user when they input the dialogue content; The target memory data corresponding to the tone information and the dialogue content is determined from the user's multimodal memory data in the electronic device.

7. The method according to any one of claims 1-4, characterized in that, The step of determining the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device includes: Determine the facial expressions of the user when they input the dialogue content; The target memory data corresponding to the facial expression information and the dialogue content is determined from the user's multimodal memory data on the electronic device.

8. The method according to any one of claims 1-4, characterized in that, The step of replying to the user with the reply content and the target memory data as reply information includes: If the target memory data includes a photo of the user, then the user's facial information is collected; The reply content, the target memory data, and the facial information are used as the reply information to reply to the user.

9. A dialogue device, characterized in that, Applied to electronic devices, the device includes: The dialogue content receiving module is used to receive dialogue content input by the user. The target memory data determination module is used to determine the target memory data corresponding to the dialogue content from the user's multimodal memory data in the electronic device; The response content determination module is used to determine the response content based on the target memory data and the dialogue content; The reply information determination module is used to reply to the user with the reply content and the target memory data as reply information.

10. An electronic device, characterized in that, The method includes a memory and a processor, the memory being coupled to the processor, the memory storing instructions that, when executed by the processor, the processor performs the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images