Object interaction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an object interaction method and device, electronic equipment, a storage medium and a program product, which can be applied to various interaction scenes, such as vehicle-mounted scenes. The method comprises the following steps: receiving interaction demand data from a client; determining background data matched with the interaction demand data; performing semantic fusion processing on the background data and the interaction demand data to generate interaction process data; the interaction process data is adjusted by using the historical data to obtain the target interaction process data, so that the target interaction process data can better fit the actual interaction condition, all historical data can be prevented from flowing into the subsequent processing link, the data processing amount of the subsequent link is reduced, and then the target interaction process data is sent to the client. The client only needs to generate interaction data and object interaction according to the target interaction process data, the required computing power is small, and low-delay object interaction can be achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an object interaction method, device, electronic device, storage medium and program product. Background Art

[0002] In recent years, with the rapid development of artificial intelligence, language models have developed rapidly. After being trained with a large amount of data, language models have the ability to understand and generate language. Therefore, they can imitate the interaction between humans and interact with objects. For example, they can answer questions from objects, or output content according to the requirements given by objects.

[0003] However, this type of language model has a large demand for computing resources and the model itself has a large number of parameters. During interaction, a large amount of data calculation is involved, making it difficult to achieve low-latency object interaction. Summary of the invention

[0004] The embodiments of the present application provide an object interaction method, device, electronic device, storage medium and program product, which can reduce the amount of data calculation during object interaction and achieve low-latency object interaction.

[0005] The present application provides an object interaction method, which is applied to a server, and the method includes:

[0006] receiving interaction demand data from the client;

[0007] Determine background data matching the interaction demand data;

[0008] Performing semantic fusion processing on the background data and the interaction demand data to generate interaction process data;

[0009] Adjusting the interactive process data using historical data to obtain target interactive process data, wherein the historical data is data generated by the server and the client before the current moment;

[0010] The target interaction process data is sent to the client so that the client generates interaction data using the target interaction process data to interact with the object.

[0011] The present application also provides an object interaction method, which is applied to a client, and the method includes:

[0012] Receive target interaction process data from the server;

[0013] Based on the target interaction process data, generating interaction demand data;

[0014] Performing semantic prediction on the interaction demand data to obtain interaction data;

[0015] In the process of predicting the interaction data, detecting demand change data, the demand change data being data obtained after the interaction demand data is updated;

[0016] Based on the detection result of the demand change data, the interaction data is output to interact with the object.

[0017] The present application also provides an object interaction device, which is applied to a server, and includes:

[0018] A demand receiving unit, used for receiving interactive demand data from a client;

[0019] A matching unit, used to determine background data that matches the interaction demand data;

[0020] A semantic fusion unit, used for performing semantic fusion processing on the background data and the interaction demand data to generate interaction process data;

[0021] An adjustment unit, configured to adjust the interactive process data using historical data to obtain target interactive process data, wherein the historical data is data generated by the server and the client before a current moment;

[0022] The sending unit is used to send the target interaction process data to the client, so that the client generates interaction data using the target interaction process data and interacts with the object.

[0023] The present application also provides an object interaction device, which is applied to a client, and includes:

[0024] A process data receiving unit, used for receiving target interactive process data from a server;

[0025] A demand generating unit, configured to generate interaction demand data based on the target interaction process data;

[0026] A prediction unit, used for performing semantic prediction on the interaction demand data to obtain interaction data;

[0027] A detection unit, configured to detect demand change data during the process of predicting the interaction data, wherein the demand change data is data obtained after the interaction demand data is updated;

[0028] The interactive unit is used to output the interactive data and interact with the object based on the detection result of the demand change data.

[0029] An embodiment of the present application also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps of any object interaction method provided in the embodiment of the present application.

[0030] An embodiment of the present application also provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps in any one of the object interaction methods provided in the embodiment of the present application.

[0031] The embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any object interaction method provided in the embodiment of the present application.

[0032] The embodiment of the present application can receive interaction demand data from the client, and determine the background data that matches the interaction demand data, and can perform semantic fusion processing on the background data and the interaction demand data to generate interaction process data, and adjust the interaction process data based on historical data to obtain target interaction process data, so that the target interaction process data is more in line with the actual interaction situation, and can avoid all historical data from flowing into the subsequent processing links, reducing the amount of data processing in the subsequent links. Then the target interaction process data is sent to the client, and the client only needs to generate interaction data and object interaction based on the target interaction process data, which requires less computing power and can achieve low-latency object interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1a It is a schematic diagram of an application scenario of the object interaction method provided in an embodiment of the present application;

[0035] Figure 1b is a flow chart of the object interaction method provided in the embodiment of the present application;

[0036] Figure 2a is a flowchart of an object interaction method provided by another embodiment of the present application;

[0037] Figure 2b is a schematic diagram of a training interaction model provided in an embodiment of the present application;

[0038] Figure 3a is a flowchart of an object interaction method provided in yet another embodiment of the present application;

[0039] Figure 3b It is a schematic diagram of the overall framework of the object interaction method provided in the embodiment of the present application;

[0040] Figure 4a is a schematic diagram of the structure of an object interaction device provided in an embodiment of the present application;

[0041] Figure 4b is a structural schematic diagram of an object interaction device provided by another embodiment of the present application;

[0042] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0044] Embodiments of the present application provide an object interaction method, device, electronic device, storage medium, and program product.

[0045] The object interaction device can be integrated into an electronic device, which can be a terminal, a server, etc. The terminal can include but is not limited to a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle terminal, an aircraft, etc. The server can be a single server or a server cluster composed of multiple servers.

[0046] In some embodiments, the object interaction device may also be integrated into multiple electronic devices. For example, the object interaction device may be integrated into multiple servers, and the object interaction method of the present application may be implemented by multiple servers.

[0047] In some embodiments, the server may also be implemented in the form of a terminal.

[0048] The embodiment of the present application can integrate multiple autoregressive language models, and interact with objects based on multiple autoregressive language models. Among them, the autoregressive language model is a probability-based generative model that can be used to predict the next word or character of a text sequence. The basic idea is to calculate the probability distribution of the next word or character by conditional probability given the previous word or character sequence, and select a word or character with the highest probability as the prediction result. The specific structure of this type of model mainly includes a recurrent neural network represented by RNN, or an encoder-decoder neural network represented by transformer. After training, this type of autoregressive language model can simulate the way humans talk to interact, for example, answer questions or give corresponding content as required, etc., but at present, this type of autoregressive language model has a large demand for computing power, and can usually only be deployed on the server side, coupled with the influence of the network, resulting in delays and freezes during interaction. In the embodiment of the present application, the autoregressive language model with large computing power consumption can be deployed on the server side, and the autoregressive language model with low computing power demand can be deployed on the client side to achieve low-latency interaction.

[0049] For example, refer to Figure 1a , shows a schematic diagram of an application scenario of the object interaction method. The application scenario may include a server 101 and a client 102, wherein the server 101 may refer to a device in an environment with sufficient computing power and a good network, such as a server. The client 102 may refer to a device with limited computing power and in an environment with an unstable network, such as a mobile device, a vehicle-mounted device, etc.

[0050] The server 101 and the client 102 can exchange data through the network to realize object interaction. For example, the server may receive interaction demand data from the client; determine the background data that matches the interaction demand data; perform semantic fusion processing on the background data and the interaction demand data to generate interaction process data; adjust the interaction process data using historical data to obtain target interaction process data, wherein the historical data is data generated by the server and the client before the current moment; and send the target interaction process data to the client so that the client can generate interaction data using the target interaction process data to interact with the object.

[0051] The client can obtain target interaction process data from the server; generate interaction demand data based on the target interaction process data; perform semantic prediction on the interaction demand data to obtain interaction data; in the process of predicting the interaction data, detect demand change data, which is the data obtained after the interaction demand data is updated; based on the detection result of the demand change data, output the interaction data and interact with the object.

[0052] It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0053] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0054] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0055] Computer Vision (CV) is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further processing graphics to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as swin-transformer, ViT, V-MOE, and MAE can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0056] The key technologies of speech technology include automatic speech recognition technology (ASR), text-to-speech technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech becomes one of the most promising human-computer interaction methods in the future. Large model technology has brought changes to the development of speech technology. Pre-trained models such as WavLM and UniSpeech that use the transformer architecture have strong generalization and versatility, and can excellently complete speech processing tasks in various directions.

[0057] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between people and computers using natural language. Natural language processing involves natural language, which is the language people use in daily life and is closely related to linguistic research; it also involves important technologies for model training in the field of artificial intelligence in computer science and mathematics. The pre-trained model is developed from the large language model in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0058] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0059] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless cars, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0060] The object interaction method provided in the embodiment of the present application can be applied to a variety of different scenarios, such as in-vehicle scenarios, intelligent question-and-answer scenarios, etc. The object interaction method can be used alone as a virtual assistant, or in combination with other applications.

[0061] For example, the virtual assistant can simulate a car radio, and the content and style of the car radio can be customized. For another example, the virtual assistant can simulate multiple objects talking to each other, and the specific content of the chat and the object's conversation style can also be customized. The content of the conversation can be the development history of a certain brand, news in the entertainment industry, etc. For another example, in a car scene, the virtual assistant can be used as a playmate for children to comfort them.

[0062] For another example, the virtual assistant can be combined with a news application program and can summarize and narrate the news to the subject according to the subject's needs. The virtual assistant can narrate the news as a virtual character or as multiple virtual characters imitating the roles of hosts and commentators to discuss with each other, and the subject can participate at any time.

[0063] For another example, the virtual assistant can be combined with a map application to proactively broadcast the current route to the subject in a broadcasting manner. In other words, it can customize traffic broadcasts based on the subject's journey from the starting point to the destination, and some news, jokes, local information, etc. can be interspersed in the broadcast.

[0064] For another example, the virtual assistant can be combined with a reading application. The virtual assistant can simulate multiple virtual roles, assuming there are two here, one is a reading role that can read the content of the book, and the other is a commentary role that can comment on or annotate the content of the reading, so as to simulate the form of a book club. If the subject encounters content in the book that he does not understand, he can interact with the virtual assistant so that the commentary role can explain it.

[0065] For another example, the virtual assistant can be combined with a social application to help the subject summarize the content of communication in a conversation group, or summarize the content updated by a media platform that the subject is interested in.

[0066] It can be understood that in the specific implementation of the present application, data related to the object is involved, such as the application used to access the object, the content in the object's conversation group, the collected audio and audio data of the object, etc. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0067] In this embodiment, an object interaction method involving artificial intelligence is provided, such as Figure 1bAs shown, the object interaction method can be applied to the server, and the specific process of the method can be as follows:

[0068] 110. Receive interaction demand data from the client.

[0069] Interaction demand data refers to data related to the interaction expectations of the object, and the interaction demand data may be data that limits the interaction content, interaction style, etc., so that the final interaction can meet the interaction expectations of the object. Interaction demand data may include at least one of interaction configuration data and real-time interaction data, wherein real-time interaction data refers to the latest interaction data when interacting with the object, and interaction configuration data refers to pre-configured data that limits at least one of the interaction topic, interaction style, and the number of virtual objects participating in the interaction.

[0070] The interactive theme refers to the theme of the interaction, such as sports, finance, entertainment, transportation, etc. The interactive style may refer to the style of the object interaction, such as humorous style, friendly style, polite style, etc. The number of virtual objects participating in the interaction refers to the number of virtual objects that can appear during the interaction. For example, the interaction may be in the form of two virtual objects, and the two virtual objects may be expressed in different styles. Real-time interactive data refers to interactive data detected in real time, such as the voice input by the user, or the data of the current interaction of the virtual object, etc.

[0071] The interaction demand data may be sent by the client to the server. In the aforementioned content, the interaction demand data may include interaction configuration data and real-time interaction data. Before the interaction begins, the client may receive the interaction configuration data, and the client may send the interaction configuration data to the server so that the server can process it and realize the interaction. During the interaction, real-time interaction data may be generated, and then the client may synchronize it to the server at certain intervals so that the server can continue to process it, making the entire interaction more reasonable.

[0072] 120. Determine background data that matches the interaction demand data.

[0073] Background data refers to the domain knowledge required for object interaction. For example, if the interactive theme is sports, the background data can include professional knowledge related to sports and the latest sports news. For another example, if the interactive theme is a TV series, the background data can include the actor information, plot, and creation basis of the TV series.

[0074] Background data can come from a variety of sources, for example, it can be obtained from a designated database, from a knowledge graph, or from the public Internet. In order to obtain rich background data, when determining the background data that matches the interactive demand data, the interactive demand data can be used to search in the first database to obtain the first sub-background data; the interactive demand data can be used to search in the second database to obtain the second sub-background data; the first sub-background data and the second sub-background data can be merged to obtain the background data.

[0075] The first database refers to a specially designed database with a dedicated API, which may include data related to multiple fields, such as traffic, music, live sports, weather, books, etc. The data that can be retrieved in the first database are all specialized for the function of object interaction, for example, all data have a fixed format, such as text format, and a system, program or component inside the first database has complete functions and resources and can run independently without relying on the external environment or other components.

[0076] Specifically, during retrieval, the interactive demand data may be used as a search term to obtain multiple data in the first database that match the interactive demand data. The multiple data may be directly used as the first sub-background data, or the first n data that are ranked first may be selected as the first sub-background data. The specific setting may be made according to actual needs.

[0077] The second database refers to the public Internet, which mainly includes search engines and a collection of classified websites that may be organized in a manner similar to the Yellow Pages. Different websites may be given different tags, such as type, feasibility, etc. In order to achieve fast search, the search behavior of each website may also be defined and standardized. For example, the search formulas corresponding to different websites are somewhat different, and the format requirements of the search formulas may be annotated. When using the corresponding website to retrieve the second sub-background data, the interactive demand data may be packaged into the corresponding format and then input into the corresponding website to improve the feasibility and effectiveness of obtaining the second sub-background data. Similarly, using the interactive demand data as the search term, multiple data matching the interactive demand data may be obtained from the second database. The multiple data may be directly used as the second sub-background data, or the top n data ranked at the top may be selected as the second sub-background data. The specific setting may be made according to actual needs.

[0078] In order to improve the processing efficiency when the background data is used later, the first sub-background data and the second sub-background data can be merged to obtain the background data. For example, the semantically incomplete data in the first sub-background data and the second sub-background data are deleted, and the same data are merged. For example, the first sub-background data and the second sub-background data both have the same data about the creator of A Dream of Red Mansions, and only one copy of the corresponding data is required. If the data about the creator of A Dream of Red Mansions in the first sub-background data and the second background data are different, the data in the first sub-background data can be used as the basis, or the data on the authoritative website can be used as the basis, and the other copy of the data can be deleted.

[0079] 130. Perform semantic fusion processing on the background data and the interaction demand data to generate interaction process data.

[0080] The interactive process data is data describing the entire process of object interaction, which may include data that needs to be output during interaction, and is used to guide the rhythm and process of the entire interaction. For example, when the object interaction is carried out in the form of a car radio, the interactive process data may refer to the radio script. For another example, when the object interaction is carried out in the form of a virtual assistant, the interactive process may refer to the outline of the interaction. That is, the overall direction of the object interaction can be guided by the interactive process data.

[0081] In order to make the object interaction process meet the object's expectations and be reasonable, the semantics of the background data and the interaction demand data can be integrated to generate the interaction process data. For example, before generating the interaction process data, the large language model can be pre-trained so that the large language model can simulate the knowledge and reasoning process of human experts to have the ability to understand semantics and automatically generate interaction process data. The trained model here can be recorded as an expert model. Then, the background data and interaction demand data can be input into the expert model to quickly obtain the interaction process data.

[0082] In some embodiments, the background data and the interaction demand data may be concatenated to obtain first data to be processed; the first data to be processed is encoded to obtain a first vector, wherein the first vector includes a background vector corresponding to the background data and an interaction demand vector corresponding to the interaction demand data; based on a first transformation matrix, the background vector and the interaction demand vector are semantically fused to obtain a fused semantic vector; the fused semantic vector is used to generate probability distributions corresponding to multiple first prediction data; and interaction process data is generated according to the probability distribution corresponding to the first prediction data.

[0083] Among them, the large language model can usually include an input layer, an attention layer and a feedforward layer. The input layer is used to encode the data of the input model and pass it to the attention layer for attention calculation, and finally the probability distribution is calculated through the feedforward layer. Among them, the attention layer can be the decoder of the transformer model, and the expert model also has a similar structure.

[0084] In order to input the background data and the interactive demand data into the expert model, the background data and the interactive demand data may be directly concatenated together to obtain the first data to be processed. Since the background data and the interactive demand data may be text, the first data to be processed is also a text sequence.

[0085] Then, the first data to be processed can be encoded through the input layer to obtain the first vector corresponding to the first data to be processed. Since the first data to be processed can be a text sequence, each word in the text sequence has its corresponding position. Since the positions of words in the sentence are different, the semantics of the sentence can change. In order to capture the semantics more accurately, the position information of each word in the first data to be processed can also be integrated during encoding.

[0086] As an implementation method, the first data to be processed may be segmented to obtain multiple words; the words are converted into corresponding text codes; based on the position information of the words in the sentence, the position code of the words is obtained; the text code and the position code corresponding to the words are merged to obtain the first sub-vector corresponding to the words; the first sub-vectors corresponding to all words constitute the first vector.

[0087] For example, during word segmentation, the first data to be processed can be divided into discrete tags, and then the tags can be mapped to low-dimensional vector representations, that is, text encodings of words. The position information of words in a sentence can also be encoded as a vector, that is, a position encoding, which can usually be generated using sine and cosine functions, and the dimension of the position encoding can be consistent with the dimension of the text encoding. Then, the position encoding of the word can be combined with the text encoding through an addition operation to obtain a final encoding representation containing the position information, that is, the first sub-vector, and the first sub-vector of each word in the first data to be processed is determined as the first vector.

[0088] It is understandable that the first vector may include a background vector corresponding to the background data and an interaction requirement vector corresponding to the interaction requirement data. The background vector may be composed of the first sub-vector corresponding to each word in the background data, and the interaction requirement vector may be composed of the first sub-vector corresponding to each word in the interaction requirement data.

[0089] The first conversion matrix is ​​a set of parameters learned by the expert model during the training process. Based on the first conversion matrix, the background vector and the interactive demand data can be semantically fused to obtain a fused semantic vector. In some embodiments, the first conversion matrix can include a first vector parameter matrix W Q1 , the first key vector parameter matrix W K1 and the first value vector parameter matrix W V1 Based on the first transformation matrix, the first vector can be linearly transformed to obtain the transformation vector corresponding to the first vector, that is, the first request vector Q1, the first key vector K1 and the first value vector V1.

[0090] Among them, Q1 includes the first sub-vector corresponding to each word and W Q1 The first sub-request vector obtained by multiplication, K1 includes the first sub-vector corresponding to each word and W K1 The first sub-key vector obtained by multiplication, V1 includes the first sub-vector corresponding to each word and W V1 The first sub-value vector obtained by multiplication.

[0091] Based on the first request vector, the first key vector and the first value vector corresponding to the first vector, a fused semantic vector can be calculated. The specific calculation method can refer to the following formula:

[0092]

[0093] in, represents the dimension of the first key vector K1; represents the transpose of the first key vector K1; Z1 is the fused semantic vector. If it is multi-head attention, there are multiple groups of Q1, K1 and V1, and multiple Z1 can be calculated and concatenated.

[0094] Then, the fused semantic vector can be used to generate probability distributions corresponding to multiple first prediction data. For example, the fused semantic vector can be subjected to nonlinear changes and feature extraction through a feedforward layer, and then the output of the feedforward layer can be subjected to a softmax operation to obtain the probability distribution of the first prediction data. The first prediction data can refer to the data learned by the expert model during the training phase, and the first prediction data with the highest probability is used as the output. Usually, the first prediction data is only a certain word. In order to obtain complete interactive process data, the current output and the previous input can be concatenated as new input, and the above process is repeated until the first prediction data is a terminator. All its outputs in the whole process are interactive process data.

[0095] In some embodiments, there may be multiple expert models. For example, an expert model may be trained for each field, such as an expert model in the financial field and an expert model in the sports field. Training a corresponding expert model for each field can reduce the difficulty of model training and provide more professional output. In a scenario where each field corresponds to an expert model, when generating interactive process data, the field to which the interaction belongs may be determined based on the interactive demand data. For example, in the financial field, an expert model in the financial field may be selected to generate interactive process data.

[0096] 140. Use historical data to adjust the interactive process data to obtain target interactive process data.

[0097] Historical data refers to data generated by the client and the server before the current moment, for example, it may include interaction demand data received before the current moment, determined background data, generated interaction process data, etc.

[0098] In some embodiments, the historical data may refer to data related to the current interaction. For example, one interaction may include multiple rounds of dialogue, and the historical data may be data related to the current object interaction generated by the server and the client before the current moment. In other embodiments, the historical data may refer to data generated by the server and the client within a period of time before the current moment. The period of time may be set according to actual needs, such as one day before the current moment, one week before the current moment, etc.

[0099] That is, after the server receives the interaction demand data for the first time, it can trigger this interaction, and all the data generated in this interaction can be stored and become historical data. It should be noted that as the interaction proceeds, the amount of historical data will also increase.

[0100] In order to make the interactive process data more in line with the actual interactive situation, the interactive process data can be adjusted using historical data to obtain target interactive process data. The adjustment may include deleting some data from the interactive process data and adding some historical data to the interactive process data.

[0101] The interactive process data can be used to guide the entire interactive process, but as the interaction progresses, some processes have already been executed. Therefore, this part of the data is no longer needed for the current interactive situation and can be deleted from the interactive process data. For example, the opening remarks in the interactive process data are no longer needed after the opening. Therefore, the unnecessary data in the interactive process data can be deleted, which can save the subsequent data processing volume and improve the processing efficiency, and also make the interactive process data more suitable for the current interactive situation.

[0102] For another example, the interactive process data contains some time-sensitive data, such as sports live broadcasts, news, road conditions, etc., and some expired data can also be deleted based on the current moment. For example, in a car-mounted scenario, traffic conditions change in real time. If a certain road section is congested at a certain moment, and the road section is now open to traffic, the congested data of the road section needs to be deleted.

[0103] Usually, during the interaction, people may bring up the content that they talked about before. Similarly, in order to make the interaction more reasonable, some data can be added from the historical data to the interaction process data. For example, in the first round of conversation, it includes "The weather is very good today, so Star A's concert can go on smoothly", and the subsequent conversation is about the plot of Star A's TV series B. At this time, the content related to Star A's concert may be mentioned again, but this part of the content has been deleted from the interaction process data. Therefore, the data related to Star A's concert in the previous conversation can be added to the interaction process data from the historical data, so that the topic can be turned to Star A's concert in the future, making the interaction more reasonable. For another example, the user asked a question about a previous news. At this time, the background data about this news that was previously provided can be added to the interaction process data.

[0104] It should be noted that adding some of the historical data to the interactive process data here does not change the historical data, that is, when adding data to the interactive process data, it does not delete the data from the historical data. Adjusting the interactive process data based on historical data makes the target interactive process data more relevant to the actual interactive situation. Due to the huge amount of historical data, it can also prevent all historical data from flowing into the subsequent processing link, which can reduce the data processing volume of the subsequent processing link.

[0105] In some implementations, the adjustment of the interactive process data to generate the target interactive process data may also be performed by a pre-trained large language model, which is subsequently referred to as a cleaning model for ease of description. That is, the cleaning model may be pre-trained to enable it to have language understanding capabilities, and then the historical data and interactive process data may be input into the cleaning model to quickly obtain the target interactive process data.

[0106] For example, the historical data and the interactive process data may be concatenated to obtain second data to be processed; the second data to be processed may be converted into a second vector; based on a second transformation matrix, attention calculation may be performed on the second vector to obtain an attention vector; the attention vector may be used to generate probability distributions corresponding to multiple second prediction data; and according to the probability distribution corresponding to the second prediction data, the interactive process data may be updated to obtain the target interactive process data.

[0107] The structure of the cleaning model is similar to that of the aforementioned large language model, and may also include an input layer, an attention layer, and a feedforward layer. In order to input the interactive process data and the historical data into the cleaning model, the historical data and the interactive process data may be concatenated together to obtain the second data to be processed. Since the historical data and the interactive process data may be text, the obtained second data to be processed is also a text sequence.

[0108] Then, the second data to be processed can be converted into a second vector through the input layer. Similar to the aforementioned encoding method of the first data to be processed, each word and the position of the word in the second data to be processed can be encoded at the same time to obtain a second sub-vector corresponding to each word. The second sub-vector of each word in the second data to be processed can constitute the second vector. The specific method of obtaining the second vector is similar to the aforementioned method of obtaining the first vector, which will not be repeated here.

[0109] The second conversion matrix is ​​a set of parameters learned by the cleaning model during the training process. Based on the second conversion matrix, the attention calculation can be performed on the second vector to integrate the semantics of the historical data and the interactive process data to obtain the second attention vector. In some embodiments, the second conversion matrix may include the second request vector parameter matrix W Q2 , the second key vector parameter matrix W K2 and the second value vector parameter matrix W V2 Based on the second transformation matrix, the second vector can be linearly transformed to obtain the transformation vector corresponding to the second vector, that is, the second request vector Q2, the second key vector K2 and the second value vector V2.

[0110] Among them, Q2 includes the second sub-vector corresponding to each word and W Q2 The second sub-request vector K2 obtained by multiplication includes the second sub-vector corresponding to each word and W K2 The second sub-key vector obtained by multiplication, V2 includes the second sub-vector corresponding to each word and W V2 The second sub-value vector obtained by multiplication.

[0111] Based on the second request vector, the second key vector and the second value vector corresponding to the second vector, an attention vector can be calculated.

[0112] The specific calculation method can refer to the following formula:

[0113]

[0114] in, represents the dimension of the second key vector K2; represents the transpose of the second key vector K2; Z2 is the attention vector. If it is multi-head attention, there are multiple groups of Q2, K2 and V2, and multiple Z2 can be calculated and concatenated.

[0115] Then, the attention vector can be used to generate probability distributions corresponding to multiple second prediction data. For example, the attention vector can be subjected to nonlinear changes and feature extraction through a feedforward layer, and then the output of the feedforward layer can be subjected to a softmax operation to obtain the probability distribution of the second prediction data. The second prediction data can refer to the data learned by the cleaning model during the training phase, and the second prediction data with the highest probability is used as the output. Usually, the second prediction data is only a certain word. In order to obtain complete interactive process data, the current output and the previous input can be concatenated as new input, and the above process is repeated until the second prediction data is a terminator. All its outputs in the whole process are the target interactive process data.

[0116] In some embodiments, the aforementioned expert model and cleansing model may be integrated into one model, and the target interactive process data may be directly obtained.

[0117] In other embodiments, the expert model and the cleaning model are two models. Since the amount of historical data can grow rapidly as the interaction progresses, the server needs to execute the above process multiple times during the interaction. The input of the cleaning model is historical data and interaction process data. It is the model with the largest input data volume in the entire interaction process, and the more computing power is required. However, the task of the cleaning model is simple and single. Therefore, separating it into a cleaning model can reduce the complexity of training, reduce the amount of calculation of the model, and save computing power.

[0118] 150. Send the target interaction process data to the client, so that the client generates interaction data using the target interaction process data to interact with the object.

[0119] The server can send the target interaction process data to the client so that the client can use the target interaction process data to generate interaction data and interact with objects. The objects may include users, AI assistants and other virtual objects.

[0120] Since the target interactive process data is compared with the interactive process data, deleting unnecessary data can reduce the amount of calculation for the subsequent client to generate interactive data using the target interactive process data, and introducing the required data can also ensure a good interactive experience. Among them, the relevant processing details of the client will be described in the subsequent embodiments.

[0121] In some implementations, in order to reduce the amount of data transmission and quickly transmit the target interactive process data to the client, the target interactive process data can be simplified by numbering. For example, the target interactive process data can be divided into a plurality of sub-interactive process data according to a preset unit; the data number corresponding to each of the sub-interactive process data is obtained; the target interactive process data is updated with the data number corresponding to the sub-interactive process data to obtain the interactive process data to be sent; and the interactive process data to be sent is sent to the client.

[0122] The target interactive process data may be a text sequence, and the preset unit may be a sentence or a paragraph, which may be set according to actual needs. Based on the preset unit, the target interactive process data may be divided into a plurality of sub-interactive process data, that is, each sub-interactive process data is a preset unit.

[0123] Then, the data number corresponding to each sub-interaction process data can be obtained, wherein the data number can be used to uniquely refer to a sub-interaction process data. In some embodiments, the sub-interaction process data can be numbered according to the generation time of the sub-interaction process data. The numbering can be performed when the sub-interaction process data is generated. After numbering, the data number and the corresponding sub-interaction process data can be used as historical data.

[0124] The target interactive process data can be updated by the data number corresponding to the sub-interactive process data to obtain the interactive process data to be sent. For example, the data to be replaced can be determined from the target interactive process data, and the data number corresponding to the data to be replaced can be used to replace the data to be replaced, and the remaining sub-interactive process data and the corresponding data number can be used as the interactive process data to be sent.

[0125] The data to be replaced refers to new data that the client does not have. In some implementations, since the server can generate the target interactive process data when needed, the server can obtain the interval between the last time the target interactive process data was generated and the current time. Then, the sub-interactive process data with a timestamp within this interval in the target interactive process data is obtained as the data to be replaced. For example, the server last generated the target interactive process data at 12 o'clock, and the current time is 12:05. When the sub-interactive process data in the target interactive process data is numbered, its generation time will be carried. Therefore, based on the data number, the sub-interactive process data with a generation time between 12 o'clock and 12:05 can be determined as the data to be replaced.

[0126] Then, the target interactive process data may include the data to be replaced and other sub-interactive process data. The data number of the data to be replaced can be directly used to replace the data to be replaced, and then the data numbers corresponding to other sub-interactive process data are added to obtain the interactive process data to be sent. For example, the target interactive process data is "The weather is so nice today. The blue sky and white clouds are like a pleasant picture. The warm sunshine shines on the earth, giving people a sense of tranquility and comfort." Among them, the data number of "The weather is so nice today" is 1, the data number of "The blue sky and white clouds are like a pleasant picture" is 2, and the data number of "The warm sunshine shines on the earth, giving people a sense of tranquility and comfort" is 3. If "The weather is so nice today. The blue sky and white clouds are like a pleasant picture." is the data to be replaced, the interactive process data to be sent can be "12The warm sunshine shines on the earth, giving people a sense of tranquility and comfort-3".

[0127] It should be noted that the data stored on the client and server and the corresponding data numbers are consistent. The data generated by the server is further numbered by the server, and the data on the client is numbered by the client. In order to ensure the uniqueness of the data number, both the server and the client can generate data numbers based on the data generation time, but the specific rules used are different, or the format of the data number is different. For example, the data number contains a timestamp and a number. The format of the data number on the server is timestamp-number, and the format of the data number on the client is number-timestamp.

[0128] The client can then generate interaction data and object interactions based on the received target interaction process data. It should be noted that the expert model used by the server and the training of the cleaning model can use human feedback reinforcement learning, that is, the reward model can be trained by human-generated data, and the output given by the reward model can be used to optimize the target model through the reinforcement learning algorithm. The target model is the expert model and the cleaning model, and the reward model can be set according to the actual task.

[0129] By adopting the object interaction solution provided in the embodiment of the present application, the server can receive interaction demand data from the client; determine the background data that matches the interaction demand data so as to have sufficient domain knowledge when subsequently generating interaction process data; semantically fuse the background data and the interaction demand data to generate interaction process data; and use historical data to adjust the interaction process data to obtain target interaction process data that is more in line with the interaction situation, which can avoid unnecessary data flowing to the next link, thereby reducing the amount of data processing during subsequent interactions and improving interaction efficiency.

[0130] Since the target interaction process data can be used to guide the entire interaction process, the server can send the target interaction process data to the client so that the client can independently generate interaction data and interact with the object based on the target interaction process data without relying on a question-and-answer model. In object interaction, placing complex data processing processes on the server, and the server providing the client with the necessary data to support its interaction with the object can reduce the computing power requirements of the client, making the latency of object interaction lower, thereby achieving efficient and reasonable object interaction.

[0131] In this embodiment, an object interaction method involving artificial intelligence is provided, such as Figure 2a As shown, the object interaction method can be applied to the client, and the specific process of the method can be as follows:

[0132] 210. Receive target interaction process data from the server.

[0133] The target interactive process data is generated by the server, and thus the client can receive the target interactive process data from the server.

[0134] In some implementations, if the server sends the target interactive process data to the client in the form of interactive process data to be sent, the client may restore the interactive process data to be sent to the target interactive process data based on the data number.

[0135] The client also has complete data numbers and historical data. Therefore, the client can query the data to be replaced corresponding to the data number based on the data number. As for other sub-interactive process data, the client can store its corresponding data number and sub-interactive process data as historical data to ensure that the historical data and data number can be synchronized with the server. Therefore, through the data number, the client can restore the interactive process data to be sent to the target interactive process data for subsequent processing.

[0136] 220. Generate interaction demand data based on the target interaction process data.

[0137] As described in the above embodiments, the interaction demand data may generally include interaction configuration data and real-time interaction data. After the client receives the target interaction process data, the target interaction process data may be used to update the interaction demand data. That is, the interaction demand data is continuously iterated during the object interaction process.

[0138] For example, before the interaction begins, the client can obtain the interaction configuration data, that is, the data set by the object and related to the interaction. At this time, the client can upload the interaction configuration data to the server as the interaction requirement data, so that the server can generate the corresponding target interaction process data and send it to the client.

[0139] After the client receives the target interaction process data, the target interaction process data is also used to guide the entire interaction process. Therefore, the target interaction process data can be used as new interaction demand data. In addition, after receiving the target interaction process, the input of the object can be detected. For example, when the user speaks, the corresponding voice can be detected. At this time, the input of the object can also be used as part of the interaction demand data. That is, the interaction demand data essentially describes some requirements for interaction so that interaction can be performed according to the interaction demand data.

[0140] Thus, when generating interactive demand data, environmental data can be detected at a first interval; if the environmental data is detected, the environmental data and the target interactive process data are integrated to obtain interactive demand data, and the interactive demand data is sent to the server; if the environmental data is not detected, the target interactive process data is used as the interactive demand data, and the interactive demand data is sent to the server at a second interval.

[0141] Among them, environmental data refers to audio data, video data, image data, etc. in the environment where the client is located. Among them, when the client detects environmental data, it needs to obtain corresponding permissions, such as audio collection permissions, video collection permissions, etc. The corresponding data can only be collected when the corresponding permissions are obtained, and the use of the collected audio and video must comply with laws and regulations. As an implementation method, before the interaction begins, prompt information for permission acquisition can be output on the client, such as "whether to enable voice collection permissions", "whether to enable video collection permissions", etc. The prompt information can be voice or directly displayed on the client. The object can submit corresponding feedback so that the client can determine whether to enable the corresponding permissions.

[0142] When the client has the corresponding data collection authority, it can collect environmental data according to the first interval. If environmental data is detected, it indicates that the current object may be interacting. Therefore, the environmental data and the target interaction process data can be integrated to obtain the interaction demand data, and the interaction demand data can be synchronized to the server in a timely manner.

[0143] The environmental data may include at least one of audio data and video data. When fusing the environmental data and the target interactive process data, the environmental data needs to be converted accordingly so that the expression of the environmental data is consistent with the target interactive process data for better fusion.

[0144] For example, when the environmental data includes audio data, the audio data can be identified based on voice identification technology to extract the identity information of the sender of the audio data, such as the location information and timbre information of the sender of the audio data. For example, in a car scene, the positions in the vehicle can be divided into the main driver, the co-driver and the back seat. Based on voice identification technology, it can be identified whether the sender of the audio data is in the main driver, the co-driver or the back seat. The audio data is converted into text data for better integration.

[0145] For another example, when the environmental data includes video data or image data, each frame in the video data can be identified based on image recognition technology to identify the body movements, expressions, etc. of each object in the video data or image data. If the video data carries audio data, it can be processed in the above manner to convert the audio data into text data, and combined with the expression information recognized by the image, etc., it can be added to the text data.

[0146] If the video data does not carry audio data, or there is a lot of noise in the audio data and voice identification cannot be performed, it can be converted into corresponding text data based on the recognized body movements and expressions for better integration. For example, when the body movement is sign language, the sign language can be converted into text data. For another example, the text data can be inferred based on the lip shape and expression.

[0147] Then, the text data converted from the environmental data and the target interactive process data are spliced ​​together to obtain the current interactive demand data. After the environmental data is detected, the interactive demand data can be uploaded to the server in a timely manner so that the server can answer relevant questions in a timely manner to ensure the normal interaction.

[0148] If no environmental data is detected, it indicates that the object has not interacted. Therefore, the target interaction process data can be directly used as the interaction demand data, and the interaction demand data can be synchronized to the server according to the second interval. Since no environmental data is detected, it indicates that the interaction can be carried out according to the target interaction process data, and the interaction demand data can be synchronized according to the second interval.

[0149] 230. Perform semantic prediction on the interaction demand data to obtain interaction data.

[0150] After obtaining the latest interaction demand data, the client can perform semantic prediction on the interaction demand data to obtain interaction data. Semantic prediction can be performed by a pre-trained large language model, which is referred to as an interaction model. That is, the interaction demand data is input into the interaction model, which can output the predicted interaction data.

[0151] In some embodiments, the interaction demand data may be encoded to obtain a demand vector; based on a preset transformation matrix and the demand vector, the interaction demand data may be semantically extracted to obtain a demand semantic vector; using the demand semantic vector, a probability distribution corresponding to candidate interaction data may be generated; and based on the probability distribution corresponding to the candidate interaction data, interaction data may be generated.

[0152] The interactive model also has an input layer, an attention layer, and a feedforward layer. The input layer can encode the interactive demand data to obtain a demand vector. Similarly, the interactive demand data can be a text sequence. Therefore, when encoding, each word in the interactive demand data can be text encoded, and the position information of each word in the sentence can be encoded to obtain a position code. Then each word can be represented as the final code obtained by summing its corresponding text code and position code. The final code of each word in the interactive demand data constitutes a demand vector.

[0153] The preset conversion matrix is ​​a set of parameters learned by the interactive model during the training process. Based on the preset conversion matrix and the demand vector, the interactive demand data can be semantically extracted to obtain the demand semantic vector. In some embodiments, the preset conversion matrix may include a preset request vector parameter matrix W Q , preset key vector parameter matrix W K And the preset value vector parameter matrix W V Based on the preset transformation matrix, the demand vector can be linearly transformed to obtain the transformation vector corresponding to the demand vector, namely the preset request vector Q, the preset key vector K and the preset value vector V.

[0154] Among them, Q includes the final encoding corresponding to each word and W Q The preset sub-request vector obtained by multiplication, K includes the final encoding corresponding to each word and W K The preset sub-key vector obtained by multiplication, V includes the final encoding corresponding to each word and W V The vector of preset subvalues ​​obtained by multiplication.

[0155] Based on the preset request vector, preset key vector and preset value vector corresponding to the demand vector, the demand semantic vector can be calculated. The specific calculation method can refer to the following formula:

[0156]

[0157] Among them, d K Represents the dimension of the preset key vector K; K T represents the transpose of the preset key vector K; Z is the required semantic vector. If it is multi-head attention, there are multiple groups of Q, K and V, and multiple Zs can be calculated and concatenated.

[0158] Then, the demand semantic vector can be used to generate the probability distribution corresponding to the candidate interaction data. For example, the demand semantic vector can be subjected to nonlinear changes and feature extraction through the feedforward layer, and then the output of the feedforward layer can be subjected to a softmax operation to obtain the probability distribution of the candidate interaction data. The candidate interaction data can refer to the data learned by the interaction model during the training phase, and the candidate interaction data with the highest probability is used as the output. Usually, the candidate interaction data is only a certain word. In order to obtain the complete interaction data, the current output and the previous input can be concatenated as the new input, and the above process is repeated until the candidate interaction data is a terminator. All its outputs in the whole process are interaction data.

[0159] 240. In the process of predicting the interaction data, detect demand change data, where the demand change data is data obtained after the interaction demand data is updated.

[0160] In the above process of predicting interaction data, the probability distribution of candidate interaction data is first obtained, and then the candidate interaction data with the highest probability distribution is selected for output, and then the predicted candidate interaction data is added to the input to continue prediction. For example, if the interaction demand data is "today", and the probability distribution of the candidate interaction data "weather" is the highest, then "weather" is added to the interaction demand data to change it to "today's weather" and continue prediction.

[0161] Therefore, in the process of predicting interaction data, the interaction model will continuously iterate the interaction demand data in order to obtain the interaction data. If the interaction demand data changes in the process of predicting interaction data, it may affect the final interaction data.

[0162] Among them, demand change data refers to the data obtained after the interactive demand data is updated, and the object that updates the interactive demand data may not be the interactive model that is predicting the interactive data. For example, when the interactive model is predicting the interactive data, the user suddenly speaks. At this time, the content of the user's speech can be added to the interactive demand data to obtain demand change data. For another example, there are two interactive models, denoted as interactive model 1 and interactive model 2. Assume that both interactive models use the interactive demand data as input and start to predict their respective interactive data, but interactive model 2 outputs first, then the candidate interactive data predicted by it is added to the interactive demand data. At this time, for interactive model 1, it detects demand change data.

[0163] In some embodiments, when detecting demand change data, it can be when updating interactive demand data, detecting whether the interactive demand data that needs to be updated is consistent with the input of the interactive model; if consistent, it is determined that no demand change data is detected; if inconsistent, it is determined that demand change data is detected.

[0164] As in the above example, if the input interactive demand data is "today's", the interactive model predicts "weather". When "weather" is added to the input, the interactive demand data "today's" can be re-acquired, which is consistent with its input, and no demand change data is detected. If the re-acquired interactive demand data is "today's vehicle", it is inconsistent with its input "today", and it can be considered that demand change data is detected.

[0165] 250. Based on the detection result of the demand change data, output the interaction data and the object for interaction.

[0166] The detection result may include detection of demand change data or non-detection of demand change data. Different detection results may affect the final interaction. That is, the interactive data may be output and the object may be interacted with according to the detection result. When the interactive data is output, it may be output in the form of text, audio, or images, etc., which may be set according to actual needs.

[0167] In some embodiments, if the demand change data is not detected during the process of generating the interactive data, the interactive data is output to interact with the object; if the demand change data is detected during the process of generating the interactive data, the interactive data is output to interact with the object based on the demand change data.

[0168] If no demand change data is detected, the interactive data can be directly output for interaction. If demand change data is detected, the interactive data can be output for interaction based on the actual demand change data. It should be noted that if the scenario of demand change data is detected, it is similar to the situation where a conversation between humans is suddenly interrupted. At this time, the interrupted party can choose to continue the conversation or choose not to talk anymore.

[0169] Similarly, if demand change data is detected, semantic information of the demand change data can be detected; if the semantic information meets preset conditions, the interactive data is output to interact with the object; if the semantic information does not meet the preset conditions, the interactive data is iterated using the demand change data.

[0170] Semantic information refers to the semantics contained in the demand change data, and the interactive model can capture its semantics. The preset conditions can be determined by the rules learned by the interactive model during training. When training the interactive model, the interactive model can be manually marked whether it is output, which can be based on whether the previous sentence is semantically complete, whether the current output is semantically coherent, etc. Therefore, after training, the interactive model can find the rules of whether to output, and determine whether to output in combination with the semantics of the demand change data.

[0171] That is, it can be to determine whether the demand change data is semantically complete. If it is, it can be determined that the semantic information meets the preset conditions. It can also be determined whether the demand change data and the currently predicted candidate interaction data are semantically coherent. If they are, it can be determined that the semantic information meets the preset conditions. Of course, it is also possible to first determine whether the demand change data is semantically complete; if it is semantically complete, directly determine that the semantic information meets the preset conditions; if the semantics of the demand change data are incomplete, determine whether the demand change data and the currently predicted candidate interaction data are semantically coherent; if they are semantically coherent, determine that the semantic information meets the preset conditions; if the semantics are incoherent, determine that the semantic information does not meet the preset conditions.

[0172] If the semantic information does not meet the preset conditions, the demand change data can be directly used as input to predict the corresponding interaction data, and whether to output the interaction data can be determined in a similar manner.

[0173] Through the above method, it is possible to simulate human conversations between interactive models, and it is also possible to simulate human conversations between interactive models and objects such as users, with good interactive effects. The style and number of interactive models can be set by the user. For example, two interactive models, one with a humorous style and the other with a formal and serious style, can simulate the radio host to output interactive data, and objects can also join in the interaction at any time.

[0174] Among them, the interactive model is obtained by training the large language model, and one style can be trained for one interactive model. The interactive model is a large language model. The goal of this type of model is to generate the next word that should appear. Assume that the words in the language used by humans to communicate are actually some abstractions of the real world, and the connection between words reflects the essential relationship between these objects in the real world. For example, a cat is an animal and a cat is a pet. Then, we can establish connections that animals may be pets and pets may be animals. For a large language model, if it is large enough and has enough parameters, these words can be abstracted into a state vector in the neural network that includes all the concepts involved in this word. For example, the vector corresponding to the word dog contains the connection between it and pets and animals that the model decoder can interpret. And the same model, if it is large enough and has enough parameters, can predict this autoregressive method with the maximum probability and output the combination of words to the object in a form that humans can understand, that is, the answer or continuation of the question in the context, to achieve object interaction.

[0175] When training the interaction model, a reinforcement learning mechanism based on human feedback can be used. For example, the interaction sample data can be occluded by the initial interaction model, and the prediction probability corresponding to the occluded data can be calculated; the model parameters corresponding to the initial interaction model can be updated using the prediction probability to obtain an intermediate interaction model; based on the intermediate interaction model, multiple predicted interaction data corresponding to the interaction sample data can be generated; each predicted interaction data can be mapped to an interaction score using a reward model, and the interaction score is related to whether the predicted sample data is output; based on the interaction score, the model parameters of the intermediate interaction model are updated to obtain an interaction model, and the interaction model is used to perform semantic prediction on the interaction demand data to obtain the step of interaction data, and then output the step of interaction between the interaction data and the object based on the detection result of the demand change data.

[0176] The above process can also be found in Figure 2b , shows a schematic diagram of training the interactive model. Human feedback reinforcement learning is a way to incorporate human subjective feedback as a reward mechanism into the reinforcement learning algorithm to optimize the model training effect. It can generally be decomposed into two steps: training the reward model through human-generated data, and optimizing the target model through the reinforcement learning algorithm using the output given by the reward model.

[0177] The initial interaction model can be a language model, and the interaction sample data refers to some data related to the interaction with the object, which can be used as the training data of the initial interaction model. The initial interaction model also includes an input layer, an attention layer, and a feedforward layer. When the initial interaction model is trained using the interaction sample data, the attention layer can be a decoder in the transformer model, which can perform occlusion processing on the interaction sample data, and then use the feedforward layer to calculate the prediction probability corresponding to the occluded data, that is, occlude part of the data and use the occluded data as a supervisory signal, so that the initial interaction model has the ability to predict the interaction data and can interact with the object to answer questions.

[0178] This part of the training is unsupervised pre-training. The interactive sample data is unlabeled text data, which can be denoted as U, where U = {u1, u2, ..., u n}, when training the initial interaction model, its parameters can be estimated by maximum likelihood:

[0179]

[0180] Where k is the size of the context window, P is the conditional probability, θ is the parameter of the conditional probability, and the parameter update adopts stochastic gradient descent. Given u1,u2,…,u k-1And the initial interaction model, try to predict the probability of the kth word. Repeat this process and add the logarithms of the obtained probabilities (after taking the logarithm, the probability itself is multiplied, that is, the joint probability), and you can get the first objective function.

[0181] Then, in order to make it output interactive data of a fixed style, it can be fine-tuned. Assume that the input X of the interactive sample data C carrying the label is a text sequence x 1 , x 2 , …, x m And label y. Among them, the text sequence x 1 , x 2 , …, x m Input the transformer decoder to get the output of the top layer at the last moment Pass it through the softmax layer (with parameter W y ) for classification, and finally use cross entropy to calculate the loss, so as to adjust the transformer decoder parameters and softmax parameters according to the labeled interactive sample data. This is equivalent to maximum likelihood estimation:

[0182]

[0183] W y Represents the parameters when predicting the output, and fine-tuning requires maximizing the following function:

[0184]

[0185] Normally, we should adjust the parameters to maximize L2(C), but in order to improve the training speed and generalization ability of the model, the pre-trained loss function is also considered during fine-tuning, and it maximizes the likelihood L1(C) and L2(C), that is, the final loss function is:

[0186] L3(C)=L2(C)+λ×L1(C);

[0187] Among them, λ is the weight hyperparameter. The L1(C) used here is still the loss of the previous language model, but the data used is not the previous unsupervised data U, but the sample data with labels, and only the specific data is used without labels.

[0188] That is, the conditional probability, i.e. the aforementioned predicted probability, is used to update the model parameters corresponding to the initial interaction model until the final loss function is maximized to obtain an intermediate interaction model. This intermediate interaction model can generate multiple stylized predicted interaction data corresponding to the interaction sample data. This part corresponds to Figure 2bIn step 1 of the figure, the unsupervised training part is omitted, that is, if the initial interaction model is a model that already has language generation, it only needs to be fine-tuned using supervised training on this basis.

[0189] Then, a reward model can be trained using the manually labeled data. The manually labeled data can be selected as an interaction sample data and input into the aforementioned intermediate interaction model to generate multiple predicted interaction data, which are then manually scored and sorted before being input into the reward model.

[0190] This reward model can map each of the predicted interaction data into an interaction score to sort the predicted interaction data. Since there are multiple interaction models, they can be preemptively output in the order of output time, and there will be a situation where one model has input from another model or human character during output, which is similar to the interruption in a human conversation. In order to adapt to this situation, the interaction score can be related to whether the predicted sample data is output, that is, when training the reward model, the loss function of the reward model is related to whether to continue output, and it uses the label representation of the sample interaction data to indicate non-output.

[0191] Finally, based on the interaction score of the reward model, the model parameters of the intermediate interaction model can be updated to obtain an interaction model, wherein the higher the interaction score, the more the corresponding output meets expectations. Specifically, the output of the intermediate interaction model can be sorted based on the interaction score, and the intermediate interaction model can be optimized by combining the sorting result and the reinforcement learning algorithm. At the same time, after the intermediate interaction model is updated, the reward model needs to be updated accordingly to better urge the intermediate interaction model to optimize its performance until the intermediate interaction model meets expectations.

[0192] Among them, the reinforcement learning algorithm can refer to the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm can initialize a strategy so that the intermediate interaction model can output predicted interaction data according to the strategy, and then the reward model scores it to obtain an interaction score. The PPO algorithm can adjust the strategy of generating output of the intermediate interaction model based on the interaction score of the reward model until it meets expectations and obtains an interaction model.

[0193] The object interaction solution provided in the embodiment of the present application can be applied in the aforementioned various scenarios. For example, the object interaction solution can be presented as an intelligent assistant, and can be used in combination with other various applications or used alone. For details, please refer to the description of the aforementioned corresponding part.

[0194] Through the method provided in the embodiment of the present application, it is possible to receive target interaction process data from the server, and generate interaction demand data using the target interaction process data; then, semantic prediction can be performed on the interaction demand data to obtain interaction data. Since the target interaction process data can be used to guide the entire interaction process, the client can generate interaction data autonomously without relying on a question-and-answer model. In the process of predicting interaction data, the demand change data can be detected, that is, whether it is interrupted. Based on the detection result of the demand change data, that is, according to whether it is interrupted, it can be determined how to output the interaction data and interact with the object, making the object interaction more reasonable. In addition, the client only needs to rely on the target interaction process data to maintain interaction with the object, which consumes less computing power and does not involve complex data processing, so that the latency of object interaction is lower.

[0195] The method described in the above embodiment will be further described in detail below.

[0196] In this embodiment, the method of the embodiment of the present application will be described in detail by taking the customization of a radio station in a vehicle-mounted scenario as an example.

[0197] like Figure 3a As shown, the specific process of an object interaction method is as follows:

[0198] 310. The server receives interaction demand data from the client.

[0199] 320. The server determines background data that matches the interaction demand data.

[0200] 330. The server inputs the background data and the interaction demand data into the expert model to generate interaction process data.

[0201] 340. The server inputs the historical data and the interactive process data into the cleansing model to obtain the target interactive process data.

[0202] 350. The server sends the target interaction process data to the client, and the client generates interaction demand data based on the target interaction process data.

[0203] 360. The client inputs the interaction demand data into the interaction model to generate interaction data.

[0204] 370. In the process of generating interactive data, the client detects demand change data, and based on the detection result of the demand change data, outputs the interactive data and interacts with the object.

[0205] For the above 310 to 370, reference may be made to the description of the corresponding part of the above embodiment. For a more clear description of the specific process of the object interaction method, reference may be made to Figure 3b , showing the overall framework of the object interaction method. Figure 3bProvide detailed explanation.

[0206] Among them, when the car radio is started, the user can customize the radio, that is, the user can set the radio's theme, style, number of virtual objects participating in the radio, etc. This part of data can be sent from the client to the server as the initial interaction demand data.

[0207] For example, a vehicle may generally include a display, and a radio setting interface may be provided based on the display. In order to achieve more personalized radio customization, multiple interactive models of different styles may be pre-trained and displayed in the setting interface for user selection. The user may set the number of virtual objects as needed, which is the number of interactive models, and select a corresponding style for each interactive model.

[0208] As an implementation method, in order to provide interactive models with richer styles, a model store can be provided, in which any interactive model trained by any user can be sold, and users can purchase it as needed and add it to the radio service for use.

[0209] After receiving the interactive demand data, the server can query the background data matching the interactive demand data from the database and the public Internet, that is, some domain knowledge related to the content that the radio station needs to play. Then, the interactive demand data and the background data can be input into the pre-trained expert model to obtain the interactive process data. The interactive process data is similar to the radio script, which can contain a text sequence of three parts: object configuration text, domain knowledge, and real-time interactive data. The interactive process data is then passed to the subsequent cleaning model.

[0210] The cleansing model on the server side takes interactive process data and historical data as input. It can add some required historical data to the interactive process data and delete unnecessary data in the interactive process data to obtain the target interactive data, and then send the target interactive process data to the client.

[0211] Among them, historical data is the data generated by the client and the server before the current moment in the entire interaction process. In order to achieve faster data transmission, the server can number the data generated by it according to sentences or paragraphs, etc. When transmitting data to the client, the data already in the client can be replaced with numbers to reduce the amount of data transmission and avoid transmitting large amounts of text data in an unstable network environment. Among them, the data generated by the client is numbered by the client, and the numbers are unique, and the client and the server have the same data and corresponding data numbers. After receiving the target interaction process data, the client can capture the environmental data in the vehicle. If the environmental data is detected, it can be merged with the target interaction process data as new interaction demand data, and then the interaction demand data is input into the trained interaction model to output the interaction data of the corresponding style.

[0212] It should be noted here that if there is only one interaction model, during the process of the interaction model predicting interaction data, if the user is detected speaking, the interaction model can choose to continue outputting or not outputting. If there are two interaction models, and the styles of the two interaction models are different, the interaction demand data can be input into the two interaction models at the same time, and the two interaction models can interrupt each other.

[0213] For example, two interaction models are recorded as interaction model a and interaction model b, and the interaction demand data of the input model is input. At this time, input can input a and b at the same time. At this time, the output is preemptive. For example, a outputs a1 first, then input will become input+a1.

[0214] When it becomes input+a1, b can choose to continue to output its predicted b1, or choose not to output its predicted b1. For example, when b chooses not to output, after a outputs, b can use input+a1 as input and continue to predict, and then continue to determine whether to output its predicted interaction data. This is similar to b interrupting what a is saying.

[0215] It can be understood that when interacting with an object in the form of a radio, the output interactive data can be voice, and the timbre and pitch of the voice can be determined according to the style learned by the interactive model during training, so as to achieve more vivid and reasonable interaction.

[0216] In the process of interacting with the object, if the user raises some new questions, these questions can be detected as environmental data, and together with the target interaction process data as interaction demand data, they are sent back to the server so that the server can answer them and finally output by the client.

[0217] From this, it can be seen that the expert model involves searching for domain knowledge of a certain topic given by the user and generating interactive process data. It involves more professional knowledge, and its computing power requirements and data processing complexity are both high. The input data volume of the cleaning model is huge, and the required computing power is also large, so it can be deployed on the server. The interactive model can directly use the target interactive process data to generate interactive data and object interaction, that is, the interactive model does not need to interpret the program or understand professional knowledge, but only needs to maintain the interaction. Its computing power requirements are small, it can run in real time, and it can be deployed on the client. And because the user asks a question and the interactive model cannot answer it, this part of the content can be sent to the server as new interactive demand data, and then sent to the client after the server answers it. This deployment method places the network request involving professional knowledge on the server, and does not require each interactive model to request the network, which can save network resources. It can maintain basic interaction on the vehicle side, and can also answer some questions through communication with the server side, which can achieve low-latency interaction.

[0218] In addition, the server will only execute subsequent steps when it receives interaction demand data, that is, the execution frequency of the server is inconsistent with that of the client. The client needs to run the interaction model all the time to maintain the interaction, while the expert model and cleanup model on the server are only triggered to run when they receive interaction demand data. The model in the server runs more frequently, which further saves computing resources while maintaining object interaction.

[0219] The output of the above expert model can be used as the input of the cleaning model, and the output of the cleaning model can be used as the input of the interactive model. That is, model integration is a stacking method, which can obtain more accurate prediction results. When dealing with complex tasks and data sets, the advantages of different models can be used to make up for the shortcomings of a single model, thereby improving the performance and effect of the entire interaction.

[0220] As can be seen from the above, the object interaction method provided by the embodiment of the present application divides the data processing of the entire object interaction into multiple models, and deploys models with high computing power requirements and high network request frequencies, such as expert models and cleaning models, on the server side, and deploys interactive models with low computing power requirements and low network request frequencies on the client side, so as to maintain real-time object interaction. Object interaction can be customized, and interactive data can be generated autonomously and continuously without relying on a question-and-answer model. Local low-latency object interaction can be achieved when computing power is controllable.

[0221] In order to better implement the above method, the embodiment of the present application also provides an object interaction device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0222] For example, in this embodiment, the method of the embodiment of the present application will be described in detail by taking the object interaction device being specifically integrated in the server as an example.

[0223] For example, Figure 4a As shown, the object interaction device may include a demand receiving unit 410, a matching unit 420, a semantic fusion unit 430, an adjustment unit 440 and a sending unit 450, as follows:

[0224] (I) Request receiving unit 410

[0225] Used to receive interaction demand data from the client.

[0226] (ii) Matching Unit 420

[0227] The background data used to determine the matching of the interaction demand data.

[0228] (III) Semantic Fusion Unit 430

[0229] It is used to perform semantic fusion processing on the background data and the interaction demand data to generate interaction process data.

[0230] In some implementations, the semantic fusion unit 430 may be specifically used for the following steps:

[0231] splicing the background data and the interactive demand data to obtain first data to be processed;

[0232] Performing encoding processing on the first data to be processed to obtain a first vector, wherein the first vector includes a background vector corresponding to the background data and an interaction demand vector corresponding to the interaction demand data;

[0233] Based on the first conversion matrix, semantically fuse the background vector and the interaction demand vector to obtain a fused semantic vector;

[0234] Using the fused semantic vector, generating probability distributions corresponding to a plurality of first prediction data;

[0235] Generate interactive process data according to the probability distribution corresponding to the first prediction data.

[0236] (IV) Adjustment unit 440

[0237] It is used to adjust the interactive process data by using historical data to obtain target interactive process data, wherein the historical data is data generated by the server and the client before the current moment.

[0238] In some implementations, the adjustment unit 440 may be specifically configured to perform the following steps:

[0239] splicing the historical data and the interactive process data to obtain second data to be processed;

[0240] Converting the second to-be-processed data into a second vector;

[0241] Based on the second transformation matrix, performing attention calculation on the second vector to obtain an attention vector;

[0242] Using the attention vector, generating probability distributions corresponding to a plurality of second prediction data;

[0243] According to the probability distribution corresponding to the second prediction data, the interactive process data is updated to obtain the target interactive process data.

[0244] (V) Transmitting Unit 450

[0245] It is used to send the target interaction process data to the client, so that the client generates interaction data by using the target interaction process data to interact with the object.

[0246] In some implementations, the sending unit 450 may be specifically configured to perform the following steps:

[0247] According to the preset unit, the target interactive process data is divided into a plurality of sub-interactive process data;

[0248] Performing numbering processing on each of the sub-interaction process data to obtain a data number corresponding to each of the sub-interaction process data;

[0249] The target interactive process data is updated with the data number corresponding to the sub-interactive process data to obtain the interactive process data to be sent;

[0250] The to-be-sent interactive process data is sent to the client.

[0251] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0252] From the above, it can be seen that the object interaction device deployed on the server side in this embodiment can semantically fuse background data and interaction demand data to generate interaction process data; and use historical data to adjust the interaction process data to obtain target interaction process data that is more in line with the interaction situation, thereby avoiding unnecessary data from flowing to the next link, thereby reducing the amount of data processing during subsequent interactions and improving interaction efficiency.

[0253] In order to better implement the above method, the embodiment of the present application also provides an object interaction device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0254] For example, in this embodiment, the method of the embodiment of the present application will be described in detail by taking the object interaction device being specifically integrated in the terminal as an example.

[0255] For example, Figure 4b As shown, the object interaction device may include a process data receiving unit 510, a demand generating unit 520, a prediction unit 530, a detection unit 540 and an interaction unit 550, as follows:

[0256] (I) Process data receiving unit 510

[0257] Used to receive target interaction process data from the server.

[0258] (II) Demand generation unit 520

[0259] Used to generate interaction demand data based on the target interaction process data.

[0260] In some implementations, the demand generation unit 520 may be specifically used for the following steps:

[0261] detecting environmental data at a first interval;

[0262] If the environment data is detected, the environment data and the target interaction process data are integrated to obtain interaction demand data, and the interaction demand data is sent to the server;

[0263] If the environment data is not detected, the target interaction process data is used as the interaction requirement data, and the interaction requirement data is sent to the server at a second interval.

[0264] (III) Prediction Unit 530

[0265] Used to perform semantic prediction on the interaction demand data to obtain interaction data.

[0266] In some implementations, the prediction unit 530 may be specifically used for the following steps:

[0267] Encoding the interaction demand data to obtain a demand vector;

[0268] Based on a preset conversion matrix and the demand vector, semantic extraction is performed on the interactive demand data to obtain a demand semantic vector;

[0269] Using the demand semantic vector, generating a probability distribution corresponding to the candidate interaction data;

[0270] Generate interaction data according to the probability distribution corresponding to the candidate interaction data.

[0271] (IV) Detection unit 540

[0272] Used to detect demand change data during the process of predicting the interaction data, where the demand change data is data obtained after the interaction demand data is updated.

[0273] (V) Interactive Unit 550

[0274] Used to output the interaction data and interact with the object based on the detection result of the demand change data.

[0275] In some implementations, the interaction unit 550 may be specifically used for the following steps:

[0276] If the demand change data is not detected during the process of predicting the interaction data, outputting the interaction data and interacting with the object;

[0277] If the demand change data is detected during the process of predicting the interaction data, the interaction data is outputted to interact with the object according to the demand change data.

[0278] In some implementations, the step of outputting the interaction data and interacting with the object according to the demand change data may include the following steps:

[0279] Detecting semantic information of the demand change data;

[0280] If the semantic information meets the preset conditions, output the interactive data and interact with the object;

[0281] If the semantic information does not satisfy the preset condition, the interaction data is iterated using the demand change data.

[0282] In some embodiments, the object interaction device may further include a training unit for:

[0283] The interactive sample data is occluded through the initial interaction model, and the prediction probability corresponding to the occluded data is calculated;

[0284] Using the predicted probability, updating the model parameters corresponding to the initial interaction model to obtain an intermediate interaction model;

[0285] Based on the intermediate interaction model, generate a plurality of predicted interaction data corresponding to the interaction sample data;

[0286] Using a reward model, mapping each of the predicted interaction data into an interaction score, wherein the interaction score is related to whether the predicted sample data is output;

[0287] Based on the interaction score, the model parameters of the intermediate interaction model are updated to obtain a target interaction model, and the target interaction model is used to perform semantic prediction on the interaction demand data to obtain the interaction data, and then output the interaction data based on the detection result of the demand change data to interact with the object.

[0288] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0289] As can be seen from the above, the object interaction device deployed on the client in the embodiment of the present application can autonomously generate interaction data based on the target interaction process data, without relying on a question-and-answer model. In the process of predicting interaction data, it can detect demand change data, that is, detect whether it is interrupted. Based on the detection result of demand change data, that is, according to whether it is interrupted, it can be determined how to output interaction data and interact with objects, making object interaction more reasonable. In addition, the client only needs to rely on the target interaction process data to maintain and interact with objects, which consumes less computing power and does not involve complex data processing, so that the latency of object interaction is low.

[0290] The embodiment of the present application also provides an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc. The server can be a single server or a server cluster composed of multiple servers, etc.

[0291] In some embodiments, the object interaction device may also be integrated into multiple electronic devices. For example, the object interaction device may be integrated into multiple servers, and the object interaction method of the present application may be implemented by multiple servers.

[0292] like Figure 5 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0293] The electronic device may include components such as a processor 610 with one or more processing cores, a memory 620 with one or more computer-readable storage media, a power supply 630, an input module 640, and a communication module 650. Those skilled in the art will appreciate that Figure 5The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0294] The processor 610 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 620, and calling data stored in the memory 620, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. In some embodiments, the processor 610 may include one or more processing cores; in some embodiments, the processor 610 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 610.

[0295] The memory 620 can be used to store software programs and modules. The processor 610 executes various functional applications and data processing by running the software programs and modules stored in the memory 620. The memory 620 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 620 may also include a memory controller to provide the processor 610 with access to the memory 620.

[0296] The electronic device also includes a power supply 630 for supplying power to various components. In some embodiments, the power supply 630 can be logically connected to the processor 610 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 630 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0297] The electronic device may further include an input module 640, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0298] The electronic device may further include a communication module 650. In some embodiments, the communication module 650 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 650, thereby providing the user with wireless broadband Internet access. For example, the communication module 650 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0299] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 610 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 620 according to the following instructions, and the processor 610 will run the application programs stored in the memory 620, thereby realizing various functions, as follows:

[0300] receiving interaction demand data from the client;

[0301] Determine background data matching the interaction demand data;

[0302] Performing semantic fusion processing on the background data and the interaction demand data to generate interaction process data;

[0303] Adjusting the interactive process data using historical data to obtain target interactive process data, wherein the historical data is data generated by the server and the client before the current moment;

[0304] The target interaction process data is sent to the client so that the client generates interaction data using the target interaction process data to interact with the object.

[0305] Or, implement the following functionality:

[0306] Receive target interaction process data from the server;

[0307] Based on the target interaction process data, generating interaction demand data;

[0308] Performing semantic prediction on the interaction demand data to obtain interaction data;

[0309] In the process of predicting the interaction data, detecting demand change data, the demand change data being data obtained after the interaction demand data is updated;

[0310] Based on the detection result of the demand change data, the interaction data is output to interact with the object.

[0311] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0312] As can be seen from the above, the embodiment of the present application can divide the data processing of the entire object interaction into multiple models, and deploy processing flows with high computing power requirements and high network request frequencies on the server side, and deploy processing flows with low computing power requirements and low network request frequencies on the client side, so as to maintain real-time object interaction. Object interaction can be customized, and interactive data can be generated autonomously and continuously without relying on a question-and-answer model. Local low-latency object interaction can be achieved when computing power is controllable.

[0313] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0314] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any object interaction method provided in the embodiment of the present application. For example, the instructions can execute the following steps:

[0315] receiving interaction demand data from the client;

[0316] Determine background data matching the interaction demand data;

[0317] Performing semantic fusion processing on the background data and the interaction demand data to generate interaction process data;

[0318] Adjusting the interactive process data using historical data to obtain target interactive process data, wherein the historical data is data generated by the server and the client before the current moment;

[0319] The target interaction process data is sent to the client so that the client generates interaction data using the target interaction process data to interact with the object.

[0320] Alternatively, perform the following steps:

[0321] Receive target interaction process data from the server;

[0322] Based on the target interaction process data, generating interaction demand data;

[0323] Performing semantic prediction on the interaction demand data to obtain interaction data;

[0324] In the process of predicting the interaction data, detecting demand change data, the demand change data being data obtained after the interaction demand data is updated;

[0325] Based on the detection result of the demand change data, the interaction data is output to interact with the object.

[0326] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0327] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the methods provided in various optional implementations of the object interaction aspect or model training aspect provided in the above embodiments.

[0328] Since the instructions stored in the storage medium can execute the steps in any object interaction method provided in the embodiments of the present application, the beneficial effects that can be achieved by any object interaction method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0329] The above is a detailed introduction to an object interaction method, device, electronic device, storage medium and program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An object interaction method, characterized in that: include: Determining a plurality of interaction models, each of the interaction models having a corresponding interaction style; Acquire interactive data of the dialog interactions between the plurality of interactive models; When outputting the interactive data of the dialog interaction between the multiple interactive models, if the user's voice is detected, new interactive data is output based on the user's voice to achieve the dialog interaction between the user and the multiple interactive models.

2. The method according to claim 1, characterized in that The determining of a plurality of interaction models, each interaction model having a corresponding interaction style, includes: Displaying a setting interface, wherein the setting interface displays a plurality of candidate interactive models of different styles; Determine the interaction model selected by the user in the setting interface, each of the interaction models having a corresponding interaction style.

3. The method according to claim 1, characterized in that When outputting the interactive data of the dialog interaction between the plurality of interactive models, if a user's voice is detected, outputting new interactive data based on the user's voice includes: When outputting the interactive data of the dialog interaction between the plurality of interactive models, if a user voice of a user is detected, the user voice is added to the interactive demand data to obtain demand change data; Outputting new interaction data based on the demand change data; The new interaction data is output.

4. The method according to claim 1, characterized in that: Before determining the interaction model selected by the user in the setting interface, the method further includes: A model store is displayed, in which trained interaction models are sold. The trained interaction models are added to the dialogue interaction after being purchased.

5. The method according to claim 1, characterized in that The object interaction method is applied to simulate a vehicle radio, the interaction content of the vehicle radio and the interaction style of each virtual object in the vehicle radio are customized by the user, and each interaction model is used to simulate a virtual object.

6. The method according to claim 1, characterized in that The object interaction method is applied to news broadcasts, in which the virtual objects in the news broadcasts are used to summarize and narrate the news. The interactive content of the news broadcasts and the interactive style of the virtual objects in the news broadcasts are customized by the users. Each of the interactive models is used to simulate a virtual object, and the virtual objects include a host and a commentator.

7. The method according to claim 1, characterized in that The object interaction method is applied to route broadcasting, and the virtual objects in the route broadcasting are used to actively broadcast the current route to the user in a broadcasting manner. The interactive content of the route broadcasting and the interactive style of the virtual objects in the route broadcasting are customized by the user, and each of the interactive models is used to simulate a virtual object.

8. The method according to claim 1, characterized in that: The object interaction method is applied to a reading application, in which a plurality of virtual objects are simulated. Each of the interaction models is used to simulate a virtual object. The virtual objects include a reading role and a commenting role. The reading role is used to read the content of a book, and the commenting role is used to comment on or annotate the content of the read book.

9. The method according to claim 1, characterized in that: The object interaction method is applied to a social application, which is used to help users summarize the communication content in a conversation group, or summarize the updated content of a media platform that is of interest to the user.

10. An object interaction device, characterized in that: include: Displaying a setting interface, wherein the setting interface displays a plurality of candidate interactive models of different styles; Determine the interaction model selected by the user in the setting interface, each of the interaction models having a corresponding interaction style; Acquire interactive data of the dialog interactions between the plurality of interactive models; When outputting the interactive data of the dialog interaction between the multiple interactive models, if the user's voice is detected, new interactive data is output based on the user's voice to achieve the dialog interaction between the user and the multiple interactive models.

11. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the object interaction method as claimed in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the object interaction method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the steps in the object interaction method according to any one of claims 1 to 9 are implemented.

Images