Interaction method and device, electronic equipment, medium and product

Abstract

The invention discloses an interaction method, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: receiving interaction information of a user in a real-time interaction process between the user and intelligent equipment; acquiring long-term memory information of the user from a long-term memory database of the user based on the interaction information; generating reply information of the interaction information based on the long-term memory information and a user portrait of the user; and sending the reply information to the intelligent device. Through the method and the device, the reply information can be associated with the long-term memory data of the user and the portrait of the user, and the memory-based recall rate between the user and the intelligent equipment is improved, so that the user intention is accurately identified to realize personalized response.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to an interaction method, device, electronic device, medium and product. Background Art

[0002] In related technologies, many chat systems of smart devices respond to users' questions rather than their preferences. The popularity of large models allows smart devices to generate reply information based on memories or chat templates generated from chat text, but reply information is still limited to text memory, and the accuracy of memory recall is low. Summary of the Invention

[0003] The embodiments of the present application provide an interaction method, apparatus, electronic device, medium, and product that can associate reply information with the user's long-term memory data and user portrait, thereby improving the memory-based recall rate between the user and the smart device, thereby accurately identifying the user's intention and realizing personalized response.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides an interaction method, the method comprising:

[0006] Receiving interaction information of the user during real-time interaction between the user and the smart device;

[0007] Acquiring the long-term memory information of the user from the long-term memory database of the user based on the interaction information;

[0008] generating reply information for the interaction information based on the long-term memory information and the user profile of the user;

[0009] Send the reply information to the smart device.

[0010] An embodiment of the present application provides an interactive device, the device comprising:

[0011] A receiving module, configured to receive interaction information of a user during real-time interaction between the user and the smart device;

[0012] an acquisition module, configured to acquire the long-term memory information of the user from the long-term memory database of the user based on the interaction information;

[0013] a processing module, configured to generate reply information for the interaction information based on the long-term memory information and the user profile of the user;

[0014] A sending module is used to send the reply information to the smart device.

[0015] In the above solution, the acquisition module is further used to acquire the first identity information and the second identity information of the user; the first identity information and the second identity information are identity information of different dimensions of the user;

[0016] The processing module is further configured to store the user identity of the user based on the first identity information and the second identity information;

[0017] A long-term memory database corresponding to the user identity is generated.

[0018] In the above solution, the processing module is further used to obtain historical interaction information between the user and the smart device during the historical interaction process; the historical interaction information includes historical text information and historical image information;

[0019] Analyze the historical image information using a large model to obtain description information of the user;

[0020] extracting a long-term memory vector based on the historical text information and the description information;

[0021] A long-term memory database corresponding to the user identity is generated based on the long-term memory vector.

[0022] In the above solution, the processing module is further configured to obtain the user picture information of the user at intervals of a preset period;

[0023] Extracting a long-term memory vector based on the user image information;

[0024] A long-term memory database corresponding to the user identity is generated based on the long-term memory vector.

[0025] In the above scheme, the processing module is also used to delete some long-term memory vectors in the long-term memory database based on the set interval time and / or the upper limit threshold of the long-term memory vector number, to obtain an updated long-term memory database; the some long-term memory vectors do not meet the index requirements, and the long-term memory vectors contained in the updated long-term memory database meet the index requirements.

[0026] In the above solution, the processing module is further used to perform cluster analysis on the long-term memory database through a network model to obtain the user label of the user;

[0027] Determining multiple portrait dimensions of the user based on the user tag;

[0028] A user portrait corresponding to the user identity is constructed based on the multiple portrait dimensions.

[0029] In the above solution, the processing module is further used to determine the inversion frequency of each of the multiple image dimensions;

[0030] The user portrait is updated based on the inversion frequency to obtain an updated user portrait.

[0031] In the above solution, the interaction information includes real-time text information and real-time image information; the processing module is further used to extract the first memory element of the user based on the real-time text information;

[0032] extracting a second memory element of the user based on the real-time image information;

[0033] determining a memory vector of the user based on the first memory element and the second memory element;

[0034] determining a similarity between the memory vector and long-term memory vectors in the long-term memory database;

[0035] The long-term memory vectors whose similarity is greater than a similarity threshold are used as the long-term memory information.

[0036] In the above solution, the processing module is further configured to receive feedback information from the user regarding the reply information;

[0037] The similarity threshold is adjusted based on the feedback information.

[0038] An embodiment of the present application provides an electronic device, including:

[0039] a memory for storing computer-executable instructions or computer programs;

[0040] The processor is used to implement an interaction method provided in an embodiment of the present application when executing computer-executable instructions or computer programs stored in the memory.

[0041] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which is used to implement an interaction method provided in an embodiment of the present application when executed by a processor.

[0042] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, an interaction method provided in an embodiment of the present application is implemented.

[0043] The embodiments of the present application have the following beneficial effects:

[0044] The system receives interaction information from a user during real-time interaction with a smart device; obtains the user's long-term memory information from the user's long-term memory database based on the interaction information; generates a reply to the interaction information based on the long-term memory information and the user's user profile; and sends the reply to the smart device. This system can associate the reply with the user's long-term memory data and user profile, improving the memory-based recall rate between the user and the smart device, thereby accurately identifying user intent and achieving personalized responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of the architecture of the interactive system 100 provided in an embodiment of the present application;

[0046] Figure 2 6 is a schematic structural diagram of an electronic device 600 provided in an embodiment of the present application;

[0047] Figure 3 This is a flow chart of an interactive method provided in an embodiment of the present application;

[0048] Figure 4 This is a flow chart of interactive information processing provided by an embodiment of the present application;

[0049] Figure 5 This is a flowchart of a user identity determination process provided by an embodiment of the present application;

[0050] Figure 6 This is a schematic diagram of a process for storing short-term memory provided by an embodiment of the present application;

[0051] Figure 7 This is a schematic diagram of a process for storing long-term memory provided by an embodiment of the present application;

[0052] Figure 8 This is another flowchart of storing long-term memory provided by an embodiment of the present application;

[0053] Figure 9 This is a schematic diagram of a long-term memory update process provided by an embodiment of the present application;

[0054] Figure 10 This is a flowchart of a user profile update process provided by an embodiment of the present application;

[0055] Figure 11 This is a flowchart of a memory recall process provided by an embodiment of the present application;

[0056] Figure 12 Schematic diagram of the process of updating the similarity threshold provided in the embodiment of the present application;

[0057] Figure 13This is a flowchart of another interactive method provided in an embodiment of the present application;

[0058] Figure 14 This is a flow chart of the third interaction method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0060] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0061] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0062] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0064] In related technologies, memory generation systems are generated based on text input by users or data from user voice chats. This type of memory generation system basically defines the identity of the subject being remembered through the user's logged-in account. This type of memory generation system is difficult to apply to robots and devices that need to interact with multiple people.

[0065] The memory system mainly uses templates for memory recall. The recall accuracy is low for different topics and scenarios. In addition, the update mechanism of memory and the remembered subject is simply based on the time dimension, which cannot improve the memory recall rate. For smart devices that need to accompany users for several years or even more than 10 years, long-term memory access consumes a lot of storage space and reduces the recall speed, which cannot meet users' real-time interaction needs.

[0066] In view of this, the embodiments of the present application provide an interactive method, device, electronic device, medium and product, which can associate reply information with the user's long-term memory data and user portrait, improve the memory-based recall rate between the user and the smart device, and thus accurately identify the user's intention to achieve personalized response.

[0067] For example, see Figure 1 , Figure 1 is a schematic diagram of the architecture of the interactive platform 100 provided in an embodiment of the present application, such as Figure 1 As shown, the interactive platform 100 includes: a server 200, a network 300, a robot end 400 (such as various types of robots), and a smart device 500. The robot end 400 and the smart device 500 are respectively connected to the server 200 through the network 300, wherein the network 300 can be a local area network or a wide area network, or a combination of the two; the smart device 500 is a device that can interact with the user, such as a smart watch, a tablet computer, a mobile phone, etc.

[0068] In some embodiments, the smart device 500 sends the interaction information of the user during the real-time interaction between the user and the smart device to the server through the network 300. Then, after receiving the interaction information of the user during the real-time interaction between the user and the smart device sent by the smart device 500, the server 200 obtains the user's long-term memory information from the user's long-term memory database based on the interaction information; generates reply information of the interaction information based on the long-term memory information and the user's user portrait; the server 200 transmits the reply information to the smart device 500 through the network 300; thereafter, the smart device 500 interacts with the user based on the reply information.

[0069] It should be noted that the technical solution provided in this application can be applied to various scenarios, such as robot voice interaction and other robot video real-time interaction scenarios.

[0070] In other embodiments, the embodiments of the present application can also be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize data calculation, storage, processing, and sharing.

[0071] Cloud technology is a general term for network, information, integration, management platform, and application technologies used in the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a key support. The backend services of technical network systems require a large amount of computing and storage resources.

[0072] For example, Figure 1 The server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The robot side 400 can be various types of robots (such as remote control robots, security detection robots, etc.). The robot side 400, the smart device 500 and the server 200 can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present application.

[0073] The following continues to describe the structure of the electronic device provided in the embodiment of the present application. Take the electronic device as an example, see Figure 2 , Figure 2 is a structural diagram of an electronic device 600 provided in an embodiment of the present application, Figure 2 The electronic device 600 shown includes: at least one processor 610, a memory 640, and at least one network interface 620. The various components in the electronic device 600 are coupled together via a bus system 630. It is understood that the bus system 630 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 630 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 630 is not described in detail. Figure 2 Various buses are labeled as bus system 630 .

[0074] The processor 610 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0075] The memory 640 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 640 may optionally include one or more storage devices physically located away from the processor 610.

[0076] The memory 640 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 640 described in the embodiments of the present application is intended to include any suitable type of memory.

[0077] In some embodiments, the memory 640 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0078] The operating system 641 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks.

[0079] The network communication module 642 is used to reach other computing devices via one or more (wired or wireless) network interfaces 620. Exemplary network interfaces 620 include Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB).

[0080] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The interactive device 643 stored in the memory 640 is shown, which can be software in the form of a program or plug-in, including the following software modules: a receiving module 6431, an acquisition module 6432, a processing module 6433, and a sending module 6434. These modules are logical and can be arbitrarily combined or further divided according to the functions implemented. It should be noted that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the interactive device 643 excludes the implementation of only including the receiving module 6431, the acquisition module 6432, the processing module 6433 and the sending module 6434. The functions of each module will be explained below.

[0081] The interactive method provided in the embodiment of the present application will be described in detail below in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0082] See also Figure 3 , Figure 3 This is a flow chart of the interactive method provided in the embodiment of the present application, combined with Figure 3 The steps shown are explained.

[0083] Step S101: receiving interaction information of a user during real-time interaction between the user and the smart device;

[0084] It should be noted that this application uses a robot as an example interactive device. When a user issues a voice command to the robot, the robot can collect voice data through its built-in microphone. For example, when a user sends a voice query to the robot, the robot's built-in microphone detects the voice input, collects the voice, and obtains voice data. After collecting the voice data, the robot can convert the voice data into real-time text information. Users can also input text data into the robot through its built-in screen, and the robot converts the text data into real-time text information. Real-time image information can be collected by the robot's built-in camera when receiving a user's voice command, or by an external camera device, without specific limitation. Real-time image information can also be automatically captured by the robot at preset intervals. For example, the robot can set an automatic photo timer, and after recognizing a face in the timed photo, it saves the image. The interactive information includes real-time text information and real-time image information. The robot sends the real-time text information and real-time image information to a server.

[0085] In some embodiments, when a user interacts with a robot for the first time, for example, by sending a voice command to the robot, the robot can collect voice data and activate its built-in camera to take a photo, thereby obtaining image data. The robot can convert the image data into real-time image information and parse the image data to obtain the user's first identity information, which may include the user's facial information, and use the facial information as the user's first identity identifier (ID1). The robot can also convert the voice data into real-time text information and parse the voice data to obtain the user's second identity information, which may include the user's voiceprint information, and use the voiceprint information as the user's second identity identifier (ID2). For the same user, ID1 and ID2 are the same.

[0086] Exemplary, reference Figure 4 As shown, the user sends a voice command to the robot, which activates its built-in camera to take a photo. Before sending the real-time text and image information to the server, the robot can parse the real-time image information to obtain the user's ID1. The robot can also convert the voice data into real-time text and parse the voice data to obtain the user's ID2. The robot then sends ID1, ID2, the real-time text information, and the real-time image information to the server.

[0087] In some embodiments, reference Figure 5As shown, after receiving all the data, the server can confirm whether ID1 and ID2 are the same; if so, ID1 is returned to the robot as the user's user identity; if not, the large language model can be used to extract the user's catchphrases, idioms, preferences and other multi-dimensional content from the real-time text information, and the multi-dimensional content can be matched with the user portrait in the server data to obtain the user's matching identity, which is recorded as ID3. If ID3 is the same as ID1, ID1 is returned to the robot as the user's user identity; if ID3 is the same as ID2, ID2 is returned to the robot as the user's user identity; if ID3 is different from both ID1 and ID2, the server returns the user's stranger identity to the robot.

[0088] Step S102: Acquire the user's long-term memory information from the user's long-term memory database based on the interaction information.

[0089] It should be noted that during historical interactions between the robot and the user, the robot sends historical text and image information to the server. After the server confirms the user's identity, it can store the historical text and image information based on the user's identity. Furthermore, the server can analyze the historical image information to obtain a description of the user. This description includes the user's profile information, information about objects around the user, behavior information, facial expressions, clothing information, and scene information, which are not specifically limited here.

[0090] In some embodiments, reference Figure 6 As shown in the figure, during historical interactions between a robot and a user, when the robot receives a voice query from the user, it collects voice data and uses its built-in camera to take a photo, generating image data. The robot then obtains the user's ID1 from the image data and ID2 from the voice data, and sends these IDs, along with real-time text and image information, to the server. After confirming the user's identity, the server stores the text and image information from this interaction as the user's short-term memory data and constructs a long-term memory database based on this short-term memory data.

[0091] In some embodiments, reference Figure 7As shown, after storing short-term memory data, the server can also analyze historical image information to obtain user description information. The description information includes the user's personal information, information about objects around the user, behavior information, expression information, clothing information, and scene information. The server summarizes the description information and text information to obtain a comprehensive description. The server performs vector calculation on the comprehensive description and uses the calculation result as the key. The comprehensive description and the image Uniform Resource Locator (URL) are stored as values ​​in the vector database. The scalar access is the current time to obtain the long-term memory vector to construct the long-term memory database.

[0092] In some embodiments, reference Figure 8 As shown, the server confirms the user's identity and returns it to the robot. The robot can collect user image data at preset intervals, convert the collected image data into user image information, and send it to the server. The server analyzes the user image to obtain a user description. The description information can include the user's personal information, information about objects around the user, behavior information, facial expressions, clothing information, and scene information. The server performs vector calculations on the description information and uses the calculation result as the key. The user's description information and image URL are stored as values ​​in the vector database. Scalar access is set to the current time to obtain a long-term memory vector to construct a long-term memory database.

[0093] In some embodiments, reference Figure 9 As shown, when updating the long-term memory database, the server deletes some long-term memory vectors from the database based on a set interval and / or an upper limit on the number of long-term memory vectors, thereby obtaining an updated long-term memory database. If some long-term memory vectors do not meet the index requirements, the long-term memory vectors included in the updated long-term memory database meet the index requirements. Index requirements include, but are not limited to, association density and timeliness coefficient.

[0094] Step S103: Generate reply information for the interaction information based on the long-term memory information and the user's user profile.

[0095] In some embodiments, the server may perform cluster analysis on the long-term memory database through a network model to obtain user tags of the users; determine multiple portrait dimensions of the users based on the user tags; and construct a user portrait corresponding to the user identity based on the multiple portrait dimensions.

[0096] For example, the server can extract long-term memory vectors from the long-term memory vector database and perform cluster analysis on the long-term memory vectors. When the number of unextracted long-term memory vectors reaches an upper limit, cluster analysis is performed on the long-term memory vectors using a network model to obtain user tags. Based on the user tags, multiple user profile dimensions are determined. The profile dimensions may include name, age, gender, preferred sports, disliked sports, music, books, and personality, without specific limitations here.

[0097] In some embodiments, the server may determine the inversion frequency of each portrait dimension among multiple portrait dimensions; update the user portrait based on the inversion frequency to obtain an updated user portrait.

[0098] Exemplary, reference Figure 10 As shown, determine whether the reversal frequency of content preferences and dislikes of the user portrait dimension is higher than a certain frequency; if not, update the user portrait; if so, for dimensions with high reversal frequency, guide the user to clarify through interaction, and update the user portrait based on the results of the user's clarification. If the user fails to clarify in time, only update the dimensions with lower reversal frequency and no reversal.

[0099] In some embodiments, reference Figure 11 As shown, the server can extract the user's first memory element based on the received real-time text information, extract the user's second memory element based on the real-time image information, combine the first memory element and the second memory element to calculate the memory vector associated with the user, determine the topic classification based on the memory vector, search the long-term memory database, and determine the similarity between the memory vector and the long-term memory vector in the long-term memory database; use the long-term memory vector with a similarity greater than the similarity threshold as the long-term memory information, merge the long-term memory record with the user portrait, and generate reply information based on the merged result.

[0100] Step S104: Send a reply message to the smart device.

[0101] It should be noted that the robot interacts with the user based on the reply information. The user can send feedback information, and the robot sends the feedback information to the server. The server adjusts the similarity threshold based on the feedback information.

[0102] In some embodiments, reference Figure 12 As shown, the server adjusts the similarity threshold according to the user's feedback information.

[0103] As can be seen from the above, the embodiments of the present application receive user interaction information during real-time interaction between a user and a smart device; obtain the user's long-term memory information from the user's long-term memory database based on the interaction information; generate a reply to the interaction information based on the long-term memory information and the user's user profile; and send the reply to the smart device. This can associate the reply information with the user's long-term memory data and user profile, improving the memory-based recall rate between the user and the smart device, thereby accurately identifying the user's intent and achieving personalized responses.

[0104] In some embodiments, before step S102, the following processing may be performed:

[0105] Obtaining the user's first identity information and second identity information; the first identity information and the second identity information are identity information of the user in different dimensions;

[0106] storing a user identity of the user based on the first identity information and the second identity information;

[0107] Generate a long-term memory database corresponding to user identities.

[0108] Exemplary, reference Figure 4 As shown, the user sends a voice command to the robot, which activates its built-in camera to take a photo. Before sending the real-time text and image information to the server, the robot can parse the real-time image information to obtain the user's ID1. The robot can also convert the voice data into real-time text and parse the voice data to obtain the user's ID2. The robot then sends ID1, ID2, the real-time text information, and the real-time image information to the server.

[0109] refer to Figure 5 As shown, after receiving all the data, the server can confirm whether ID1 and ID2 are the same; if so, ID1 is returned to the robot as the user's user identity; if not, the large language model can be used to extract the user's catchphrases, idioms, preferences and other multi-dimensional content from the real-time text information, and the multi-dimensional content can be matched with the user portrait in the server data to obtain the user's matching identity, which is recorded as ID3. If ID3 is the same as ID1, ID1 is returned to the robot as the user's user identity; if ID3 is the same as ID2, ID2 is returned to the robot as the user's user identity; if ID3 is different from both ID1 and ID2, the server returns the user's stranger identity to the robot.

[0110] In some embodiments, generating a long-term memory database corresponding to a user identity may be performed in the following manner:

[0111] Obtain historical interaction information between the user and the smart device during the historical interaction process; historical interaction information includes historical text information and historical image information;

[0112] Through the large model, historical image information is analyzed to obtain user description information;

[0113] Extract long-term memory vectors based on historical text information and description information;

[0114] Generate a long-term memory database corresponding to the user identity based on the long-term memory vector.

[0115] It should be noted that during historical interactions between the robot and the user, the robot sends historical text and image information to the server. After the server confirms the user's identity, it can store the historical text and image information based on the user's identity. Furthermore, the server can analyze the historical image information to obtain a description of the user. This description includes the user's profile information, information about objects around the user, behavior information, facial expressions, clothing information, and scene information, which are not specifically limited here.

[0116] refer to Figure 6 As shown in the figure, during historical interactions between a robot and a user, when the robot receives a voice query from the user, it collects voice data and uses its built-in camera to take a photo, generating image data. The robot then obtains the user's ID1 from the image data and ID2 from the voice data, and sends these IDs, along with real-time text and image information, to the server. After confirming the user's identity, the server stores the text and image information from this interaction as the user's short-term memory data and constructs a long-term memory database based on this short-term memory data.

[0117] refer to Figure 7 As shown, after storing short-term memory data, the server can also analyze historical image information to obtain user description information. The description information includes the user's personal information, information about objects around the user, behavior information, expression information, clothing information, and scene information. The server summarizes the description information and text information to obtain a comprehensive description. The server performs vector calculation on the comprehensive description and uses the calculation result as the key. The comprehensive description and the image Uniform Resource Locator (URL) are stored as values ​​in the vector database. The scalar access is the current time to obtain the long-term memory vector to construct the long-term memory database.

[0118] In some embodiments, generating a long-term memory database corresponding to a user identity may also be processed in the following manner:

[0119] Obtain user picture information of the user at preset intervals;

[0120] Extract long-term memory vectors based on user image information;

[0121] Generate a long-term memory database corresponding to the user identity based on the long-term memory vector.

[0122] Exemplary, reference Figure 8 As shown, the server confirms the user's identity and returns it to the robot. The robot can collect user image data at preset intervals, convert the collected image data into user image information, and send it to the server. The server analyzes the user image to obtain a user description. The description information can include the user's personal information, information about objects around the user, behavior information, facial expressions, clothing information, and scene information. The server performs vector calculations on the description information and uses the calculation result as the key. The user's description information and image URL are stored as values ​​in the vector database. Scalar access is set to the current time to obtain a long-term memory vector to construct a long-term memory database.

[0123] In some embodiments, the method further comprises:

[0124] Based on the set interval time and / or the upper limit threshold of the long-term memory vector number, some long-term memory vectors in the long-term memory database are deleted to obtain an updated long-term memory database; some long-term memory vectors do not meet the indicator requirements, and the long-term memory vectors included in the updated long-term memory database meet the indicator requirements.

[0125] Exemplary, reference Figure 9 As shown, when updating the long-term memory database, the server deletes some long-term memory vectors from the database based on a set interval and / or an upper limit on the number of long-term memory vectors, thereby obtaining an updated long-term memory database. If some long-term memory vectors do not meet the index requirements, the long-term memory vectors included in the updated long-term memory database meet the index requirements. Index requirements include, but are not limited to, association density and timeliness coefficient.

[0126] In some embodiments, after generating the long-term memory database corresponding to the user identity, it can also be processed in the following ways:

[0127] Perform cluster analysis on the long-term memory database through the network model to obtain the user's user label;

[0128] Determine multiple user portrait dimensions based on user tags;

[0129] Build a user portrait corresponding to the user identity based on multiple portrait dimensions.

[0130] For example, the server can extract long-term memory vectors from the long-term memory vector database and perform cluster analysis on the long-term memory vectors. When the number of unextracted long-term memory vectors reaches an upper limit, cluster analysis is performed on the long-term memory vectors using a network model to obtain user tags. Based on the user tags, multiple user profile dimensions are determined. The profile dimensions may include name, age, gender, preferred sports, disliked sports, music, books, and personality, without specific limitations here.

[0131] In some embodiments, the method further comprises:

[0132] determining a reversal frequency for each of the plurality of image dimensions;

[0133] The user profile is updated based on the reversal frequency to obtain an updated user profile.

[0134] Exemplary, reference Figure 10 As shown, determine whether the reversal frequency of content preferences and dislikes of the user portrait dimension is higher than a certain frequency; if not, update the user portrait; if so, for dimensions with high reversal frequency, guide the user to clarify through interaction, and update the user portrait based on the results of the user's clarification. If the user fails to clarify in time, only update the dimensions with lower reversal frequency and no reversal.

[0135] In some embodiments, the interactive information includes real-time text information and real-time image information; step S102 may be processed in the following manner:

[0136] Extracting the user's first memory element based on real-time text information;

[0137] Extracting the user's second memory element based on real-time image information;

[0138] determining a memory vector for the user based on the first memory element and the second memory element;

[0139] determining a similarity between the memory vector and a long-term memory vector in a long-term memory database;

[0140] The long-term memory vectors whose similarity is greater than the similarity threshold are used as long-term memory information.

[0141] Exemplary, reference Figure 11As shown, the server can extract the user's first memory element based on the received real-time text information, extract the user's second memory element based on the real-time image information, combine the first memory element and the second memory element to calculate the memory vector associated with the user, determine the topic classification based on the memory vector, search the long-term memory database, and determine the similarity between the memory vector and the long-term memory vector in the long-term memory database; use the long-term memory vector with a similarity greater than the similarity threshold as the long-term memory information, merge the long-term memory record with the user portrait, and generate reply information based on the merged result.

[0142] In some embodiments, the method further comprises:

[0143] Receive user feedback on the reply information;

[0144] Adjust the similarity threshold based on the feedback information.

[0145] It should be noted that the robot interacts with the user based on the reply information. The user can send feedback information, and the robot sends the feedback information to the server. The server adjusts the similarity threshold based on the feedback information.

[0146] refer to Figure 12 As shown, the server adjusts the similarity threshold based on the user's feedback information. If the user's feedback classification on the accuracy of the topic is incorrect, the similarity threshold can be increased.

[0147] It should be noted that the interaction between the robot and the user can be initiated by the user or by the robot.

[0148] In some embodiments, reference Figure 13 As shown, the user sends a voice command to the robot, which activates its built-in camera to take a photo. Before sending the real-time text and image information to the server, the robot can parse the real-time image information to obtain the user's ID1. The robot can also convert the voice data into real-time text and parse the voice data to obtain the user's ID2. The robot then sends ID1, ID2, the real-time text information, and the real-time image information to the server.

[0149] The server can extract the user's first memory element based on the received text information, extract the user's second memory element based on the image information, calculate the memory vector associated with the user based on the first memory element and the second memory element, determine the topic classification based on the memory vector, search the long-term memory database, and obtain the user's first long-term memory record.

[0150] The server generates a short-term memory corresponding to the user identity based on the text and image information. Furthermore, the service can perform a simplified recall of the long-term memory database corresponding to the user identity based on the text and image information to obtain a second long-term memory record. The server recalls the user's user profile based on the user identity.

[0151] The server generates a reply message by combining the first long-term memory record, the second long-term memory record, the short-term memory, the user portrait and the current time, and sends the reply message to the robot.

[0152] In some embodiments, reference Figure 14 As shown, when the robot's timer expires, it turns on its built-in camera to take a photo, or when its passive infrared sensor (PIR) is triggered, it turns on its built-in camera to take a photo. The robot recognizes a face in the image, identifies the user's ID1 based on the image, and sends the facial image and ID1 to the server. The server searches the long-term memory database based on the facial image to obtain a long-term memory record. The server then simplifies the search based on the facial image to obtain a simplified memory record. The server then combines the long-term memory record, the simplified memory record, the current time, and the user profile matching ID1 to generate a reply message and sends it to the robot.

[0153] The following continues to describe the exemplary structure of the interactive device 643 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the interaction processing 643 of the memory 640 may include: a receiving module 6431, an acquisition module 6432, a processing module 6433, and a sending module 6434. The receiving module 6431 is used to receive interaction information of a user during real-time interaction between a user and a smart device; the acquisition module 6432 is used to obtain the user's long-term memory information from the user's long-term memory database based on the interaction information; the processing module 6433 is used to generate reply information to the interaction information based on the long-term memory information and the user's user profile; and the sending module 6434 is used to send the reply information to the smart device.

[0154] In some embodiments, the acquisition module 6432 is further configured to acquire the user's first identity information and second identity information; the first identity information and the second identity information are identity information of different dimensions of the user;

[0155] The processing module 6433 is further configured to store the user identity of the user based on the first identity information and the second identity information;

[0156] Generate a long-term memory database corresponding to user identities.

[0157] In some embodiments, the processing module 6433 is further configured to obtain historical interaction information between the user and the smart device during the historical interaction process; the historical interaction information includes historical text information and historical image information;

[0158] Through the large model, historical image information is analyzed to obtain user description information;

[0159] Extract long-term memory vectors based on historical text information and description information;

[0160] Generate a long-term memory database corresponding to the user identity based on the long-term memory vector.

[0161] In some embodiments, the processing module 6433 is further configured to obtain user picture information of the user at predetermined intervals;

[0162] Extract long-term memory vectors based on user image information;

[0163] Generate a long-term memory database corresponding to the user identity based on the long-term memory vector.

[0164] In some embodiments, the processing module 6433 is also used to delete some long-term memory vectors in the long-term memory database based on the set interval time and / or the upper limit threshold of the long-term memory vector quantity, to obtain an updated long-term memory database; some long-term memory vectors do not meet the index requirements, and the long-term memory vectors included in the updated long-term memory database meet the index requirements.

[0165] In some embodiments, the processing module 6433 is further configured to perform cluster analysis on the long-term memory database using a network model to obtain user tags of the users;

[0166] Determine multiple user portrait dimensions based on user tags;

[0167] Build a user portrait corresponding to the user identity based on multiple portrait dimensions.

[0168] In some embodiments, the processing module 6433 is further configured to determine an inversion frequency of each of the plurality of image dimensions;

[0169] The user profile is updated based on the reversal frequency to obtain an updated user profile.

[0170] In some embodiments, the interaction information includes real-time text information and real-time image information; the processing module 6433 is further configured to extract the user's first memory element based on the real-time text information;

[0171] Extracting the user's second memory element based on real-time image information;

[0172] determining a memory vector for the user based on the first memory element and the second memory element;

[0173] determining a similarity between the memory vector and a long-term memory vector in a long-term memory database;

[0174] The long-term memory vectors whose similarity is greater than the similarity threshold are used as long-term memory information.

[0175] In some embodiments, the processing module 6433 is further configured to receive user feedback on the reply information;

[0176] Adjust the similarity threshold based on the feedback information.

[0177] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated here. Figure 3 explanation and understand.

[0178] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the interaction method described above in the present invention.

[0179] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the interactive method provided in the embodiment of the present application, for example, Figure 3 The interaction method is shown.

[0180] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); or various devices including one or any combination of the above memories.

[0181] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0182] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0183] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0184] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An interactive method, characterized in that: The method comprises: Receiving interaction information of the user during real-time interaction between the user and the smart device; Acquiring the long-term memory information of the user from the long-term memory database of the user based on the interaction information; generating reply information for the interaction information based on the long-term memory information and the user profile of the user; Send the reply information to the smart device.

2. The method according to claim 1, characterized in that Before acquiring the long-term memory information of the user from the long-term memory database of the user based on the interaction information, the method includes: Acquire first identity information and second identity information of the user; the first identity information and the second identity information are identity information of the user in different dimensions; storing the user identity of the user based on the first identity information and the second identity information; A long-term memory database corresponding to the user identity is generated.

3. The method according to claim 2, characterized in that Generating a long-term memory database corresponding to the user identity includes: Acquire historical interaction information between the user and the smart device during historical interactions; the historical interaction information includes historical text information and historical image information; Analyze the historical image information using a large model to obtain description information of the user; extracting a long-term memory vector based on the historical text information and the description information; A long-term memory database corresponding to the user identity is generated based on the long-term memory vector.

4. The method according to claim 2, characterized in that Generating a long-term memory database corresponding to the user identity includes: At preset intervals, obtaining user picture information of the user; Extracting a long-term memory vector based on the user image information; A long-term memory database corresponding to the user identity is generated based on the long-term memory vector.

5. The method according to claim 3 or 4, characterized in that The method further comprises: Based on a set interval duration and / or an upper limit threshold of the number of long-term memory vectors, some long-term memory vectors in the long-term memory database are deleted to obtain an updated long-term memory database; the some long-term memory vectors do not meet the indicator requirements, and the long-term memory vectors included in the updated long-term memory database meet the indicator requirements.

6. The method according to claim 3, characterized in that After generating the long-term memory database corresponding to the user identity, the method includes: Performing cluster analysis on the long-term memory database through a network model to obtain a user tag of the user; Determining multiple portrait dimensions of the user based on the user tag; A user portrait corresponding to the user identity is constructed based on the multiple portrait dimensions.

7. The method according to claim 6, characterized in that The method further comprises: determining a reversal frequency for each of the plurality of image dimensions; The user portrait is updated based on the inversion frequency to obtain an updated user portrait.

8. The method according to claim 3, characterized in that The interactive information includes real-time text information and real-time image information; and obtaining the user's long-term memory information from the user's long-term memory database based on the interactive information includes: extracting a first memory element of the user based on the real-time text information; extracting a second memory element of the user based on the real-time image information; determining a memory vector of the user based on the first memory element and the second memory element; determining a similarity between the memory vector and long-term memory vectors in the long-term memory database; The long-term memory vectors whose similarity is greater than a similarity threshold are used as the long-term memory information.

9. The method according to claim 8, characterized in that The method further comprises: receiving feedback information from the user regarding the reply information; The similarity threshold is adjusted based on the feedback information.

10. A multi-interaction device, characterized in that: The device comprises: A receiving module, configured to receive interaction information of a user during real-time interaction between the user and the smart device; an acquisition module, configured to acquire the long-term memory information of the user from the long-term memory database of the user based on the interaction information; a processing module, configured to generate reply information for the interaction information based on the long-term memory information and the user profile of the user; A sending module is used to send the reply information to the smart device.

11. An electronic device, characterized in that: include: a memory for storing computer-executable instructions or computer programs; The processor is configured to implement the interaction method according to any one of claims 1 to 9 when executing the computer-executable instructions or computer programs stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the interactive method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the interactive method according to any one of claims 1 to 9 is implemented.

Images