Human-computer interaction method, device, electronic device and storage medium based on digital human

Abstract

The present disclosure provides a human-computer interaction method, device, electronic device and storage medium based on digital human, which relates to the field of artificial intelligence. The specific implementation scheme is: in response to the user's question information, determine the response text information corresponding to the question information; wherein the question information represents the question raised by the user to the target object, and the response text information represents the reply answer to the question represented by the question information; perform text analysis and processing on the response text information to obtain the digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information and expression information; based on the preset digital human image, broadcast the digital human broadcast information. It is realized that through the digital human, the reply answer to the question is provided to the user flexibly, conveniently and intuitively, thereby improving the user's human-computer interaction experience.

Description

Technical Field

[0001] The present disclosure relates to the field of human-computer interaction in the field of artificial intelligence, and in particular to a human-computer interaction method, device, electronic device and storage medium based on digital human. Background Art

[0002] With the development of terminal devices, people often need to obtain information through terminal devices. Terminal devices can provide users with answers to questions raised by users.

[0003] Furthermore, there is an urgent need for a method that can flexibly, conveniently and intuitively provide users with reply answers to improve the user experience. Summary of the invention

[0004] The present disclosure provides a human-computer interaction method, device, electronic device and storage medium based on digital human.

[0005] According to a first aspect of the present disclosure, a human-computer interaction method based on a digital human is provided, comprising:

[0006] In response to a user's question information, determining response text information corresponding to the question information; wherein the question information represents a question raised by the user to a target object, and the response text information represents a reply answer to the question represented by the question information;

[0007] Performing text analysis on the response text information to obtain the digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information, and expression information;

[0008] Based on the preset digital human image, the digital human broadcast information is broadcasted.

[0009] According to a second aspect of the present disclosure, a human-computer interaction device based on a digital human is provided, comprising:

[0010] A determination module, configured to determine, in response to a user's question information, a response text information corresponding to the question information; wherein the question information represents a question raised by the user to a target object, and the response text information represents a reply answer to the question represented by the question information;

[0011] An analysis module, used for performing text analysis on the response text information to obtain the digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information and expression information;

[0012] The broadcasting module is used to broadcast the digital human broadcasting information based on the preset digital human image.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] According to the technology disclosed in the present invention, a method is provided that can flexibly, conveniently and intuitively provide reply answers to users, thereby improving the user's human-computer interaction experience.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flowchart of a human-computer interaction method based on a digital human provided by an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of a human-computer interaction method based on a digital human provided by an embodiment of the present disclosure;

[0024] Figure 3 is a flowchart of a human-computer interaction method based on a digital human provided by an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of a visualization interface provided by an embodiment of the present disclosure;

[0026] Figure 5 is a flowchart of a human-computer interaction method based on a digital human provided by an embodiment of the present disclosure;

[0027] Figure 6 is a structural block diagram of a human-computer interaction device based on a digital human provided by an embodiment of the present disclosure;

[0028] Figure 7 is a structural block diagram of a human-computer interaction device based on a digital human provided by an embodiment of the present disclosure;

[0029] Figure 8 is a block diagram of an electronic device for implementing the human-computer interaction method based on digital human according to an embodiment of the present disclosure;

[0030] Fig. 9 It is a block diagram of an electronic device used to implement the human-computer interaction method based on digital human in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] When using a terminal device, a user often needs to obtain relevant information of a target object through the terminal device. For example, the user can obtain instructions of target objects such as various medicines, household appliances, and daily necessities, so as to facilitate the use of the target object.

[0033] Taking obtaining drug instructions as an example, in the current related technologies, when users obtain drug instructions, the terminal device reads and displays the text of the drug instructions. This method can solve the problem that users with reading disabilities cannot read the medication instructions, but it only reads out the text on the instructions. If users have questions about the explanation of professional medical terms, difficult-to-understand words, or medication rules, they cannot be answered. There is a lack of question-and-answer sessions, and users can only solve them by seeking professionals or registering at hospitals. In other words, it is impossible to provide users with answers to their questions flexibly, conveniently, and intuitively, and the user experience is poor.

[0034] The present invention discloses a human-computer interaction method, device, electronic device and storage medium based on a digital human, which are applied to the human-computer interaction field in the field of artificial intelligence to provide users with responses in a flexible, convenient and intuitive manner, thereby improving the user experience.

[0035] It should be noted that the model in this embodiment is not targeted at a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0036] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0037] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 1-Figure 9 The embodiment is further refined.

[0038] Figure 1 FIG. 1 is a flow chart of a human-computer interaction method based on a digital human provided according to an embodiment of the present disclosure. The method can be executed by a human-computer interaction device based on a digital human. Figure 1 As shown, the method comprises the following steps:

[0039] S101. In response to a user's question information, determine response text information corresponding to the question information; wherein the question information represents a question raised by the user to a target object, and the response text information represents a reply answer to the question represented by the question information.

[0040] Exemplarily, the user sends a question message through a terminal device, for example, the user can input the question message in the form of voice or text. The question message is a question raised by the user for the target object, and the target object is an object with an instruction manual, for example, the target object can be a medicine, a household appliance, a cosmetic, etc. Taking the target object as medicine A, the question raised by the user can be what are the adverse effects of taking medicine A, what are the ingredients of medicine A, or what is the applicable population of medicine A, etc.

[0041] Different question information may correspond to different response text information, and the response text information refers to the reply answer to the question represented by the question information. That is, the user asks a question by sending a question information, and the response text information is the answer to the question raised by the user. The response text information is the content in the specification of the target object, that is, the corresponding response text information can be found from the specification of the target object.

[0042] In response to the user's question information, the response text information corresponding to the question information is determined. For example, the association relationship between different question information and response text information can be preset, and the response text information corresponding to the question information is searched according to the preset association relationship.

[0043] The question information may correspond to one question or multiple questions. That is, when the user inputs the question information to the terminal device, the user may input the question information of one question at a time, or may input the question information of multiple questions at a time. When the user inputs the question information of multiple questions, the questions may be separated by preset symbols, for example, commas, question marks, semicolons, etc. may be used for separation. In response to the user's question information, each question is obtained according to the preset spacing symbols. For each question, the corresponding response text information may be determined. For example, if the user's question information is "I want to know how to take medicine A? Will there be any adverse reactions after taking it?", the response text information of "I want to know how to take medicine A?" and the response text information of "Will there be any adverse reactions after taking it?" can be determined.

[0044] S102, performing text analysis on the response text information to obtain digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information, and expression information.

[0045] For example, the response text information is the text content in the instruction manual of the target object, which may contain professional medical terms and other sentences that are difficult for users to understand. Therefore, after obtaining the response text information, the response text information can be subjected to text analysis and processing to obtain the digital human broadcast information that is easier for users to understand. The digital human broadcast information may include at least one of the digital human's voice information, action information, and expression information. Voice information refers to the voice played by the digital human, action information refers to the digital human's gestures, and expression information refers to the digital human's facial expressions.

[0046] For different response text information, the digital human can use different tones, actions or expressions to play the corresponding response text information. For example, if the response text information is "People with allergies are prohibited from taking this medicine", the digital human's expression information and tone information can be a serious expression and tone, and the action information can be a waving action; if the response text information is "This medicine should be taken after meals", the digital human's expression information and tone information can be a calm expression and tone.

[0047] In this embodiment, a preset neural network model may be used to perform text analysis on the response text information. For example, an NLP (Natural Language Processing) model may be used to perform text analysis. This embodiment does not specifically limit the model structure of the preset neural network model.

[0048] S103: Based on the preset digital human image, broadcast the digital human broadcast information.

[0049] Exemplarily, the image of the digital person is pre-set, for example, the gender, hairstyle, clothing and facial features of the digital person can be set, and the voice of the digital person can also be pre-set. After obtaining the broadcast information of the digital person, the broadcast information of the digital person can be broadcast through the image of the digital person. For example, the voice information corresponding to the response text information can be read by the preset voice of the digital person, the gesture of the digital person can be changed through the action information, and the facial expression of the digital person can be changed through the expression information. For example, the action information is the action of waving hands, and the motion track of the hand corresponding to the waving hand action is pre-stored, and the hand motion change of the digital person is realized according to the preset hand motion track.

[0050] In this embodiment, based on the preset digital human image, broadcasting the digital human broadcasting information includes: determining the digital human image selected by the user according to the user's selection operation on the digital human image; displaying the action corresponding to the action information and the expression corresponding to the expression information on the visual interface through the digital human image selected by the user, and playing the voice information.

[0051] Specifically, multiple digital human images can be pre-set, and users can select digital human images when performing human-computer interaction. For example, users can see various digital human images through the visual interface of the terminal device, select one of the digital human images, and use the digital human image selected by the user to broadcast the digital human broadcast information.

[0052] The user can make a selection operation on a certain digital human image. For example, the user can click on a digital human image to indicate that the digital human image is selected. According to the user's selection operation on the digital human image, the digital human image selected by the user is determined. The digital human image selected by the user is displayed on the visual interface, and the expression and action of the digital human image are changed according to the expression information and action information corresponding to the digital human broadcast information. The preset voice of the digital human image is used to play the voice information of the digital human broadcast information.

[0053] The beneficial effect of this setting is that users can choose different digital human images, express actions and expressions through digital humans, achieve flexible and intuitive display of response text information, and enhance the user's human-computer interaction experience.

[0054] In this embodiment, the method further includes: scrolling and playing the response text information in a preset area of ​​the visual interface according to a preset text scrolling speed.

[0055] Specifically, the response text information can be played in the form of voice or displayed in the form of text on the visual interface of the terminal device. The response text information can be played in a preset area of ​​the visual interface. For example, a conversation box area can be preset, and the conversation box area is used to display the question information and the response text information.

[0056] Due to the limited size of the preset area, the response text information may not be fully displayed in the preset area. The text scrolling speed can be preset, and the text scrolling speed refers to the scrolling speed of the text of the response text information in the preset area. The response text information can be displayed as multiple lines of text, and the text scrolling speed can be the scrolling speed of the line, so that the response text information can appear in the preset area in sequence starting from the first line. For example, the text scrolling speed is 2 seconds / line, that is, one line of text is moved every two seconds. The text scrolling speed can be preset and can be related to the playback speed of the voice, so that the text displayed in the preset area is the text corresponding to the voice currently played by the digital human.

[0057] The beneficial effect of this setting is that the response text information is broadcast in a scrolling manner, so that the user can see the entire response text information as the voice is played, avoiding the problem of a small preset area affecting the display of the response text information, helping users to obtain responses to questions more conveniently and quickly, and improving user experience.

[0058] The disclosed embodiment can realize the answering of questions between the user and the terminal device by responding to the user's question information and determining the response text information corresponding to the question information. The user can ask questions according to the actual situation and obtain the reply answer to the question, that is, the response text information. By performing text analysis and processing on the response text information, the digital human broadcast information is obtained. The digital human broadcast information may include various forms of information such as voice information, action information and expression information, and the information is broadcast through the digital human image. The problem that the manual of an item is small and difficult to understand, and the problem that people with reading disabilities cannot read the manual are solved. When the user has questions, valuable answers are obtained through human-computer interaction, which helps the user to obtain answers more conveniently and quickly, and realizes a flexible, convenient and intuitive way to provide users with reply answers, thereby improving the user experience.

[0059] Figure 2 A flowchart of a human-computer interaction method based on a digital human is provided for an embodiment of the present disclosure. This embodiment is an optional embodiment based on the above embodiment.

[0060] In this embodiment, in response to the user's question information, the response text information corresponding to the question information is determined, which can be further refined as: in response to the user's question information, based on the preset structured data matched by the target object, the response text information corresponding to the question information is determined; wherein the preset structured data represents structured data corresponding to the explanatory text information of the target object, the format of the preset structured data is a key-value pair format, the preset structured data represents the relationship between the question information and the response text information, and the explanatory text information represents the text of the explanation of the target object.

[0061] like Figure 2 As shown, the method comprises the following steps:

[0062] S201. In response to a user's question information, based on preset structured data matched by a target object, determine response text information corresponding to the question information; wherein the preset structured data represents structured data corresponding to the explanatory text information of the target object, the format of the preset structured data is a key-value pair format, the preset structured data represents the relationship between the question information and the response text information, and the explanatory text information represents a text that explains the target object.

[0063] Exemplarily, a user inputs question information in the form of voice or text through a terminal device to ask questions about the usage or precautions of the target object. In response to the user's question information, the question raised by the user is determined. The preset structured data matching the target object is obtained, and the response text information corresponding to the question information is determined according to the preset structured data. The format of the preset structured data can be a key-value pair format, that is, a KV (Key-Value) structure. The preset structured data is structured data corresponding to the explanatory text information of the target object, which characterizes the relationship between the question information and the response text information. The explanatory text information is a text that explains the target object, that is, the text of the instruction manual of the target object.

[0064] Convert the target object's explanatory text information into structured data of KV structure. The key in the KV structure is the question information that the user may ask about the explanatory text information, and the value is the reply answer corresponding to the question information, that is, the response text information. Extract the possible questions that the user may ask from the explanatory text information, and determine the answer to the question from the explanatory text information, and associate the question and answer as structured data for storage. When responding to the user's question information, search for the response text information corresponding to the user's question information from the structured data.

[0065] Users can ask questions to a target object according to actual needs. Based on the user's question information, the response text information corresponding to the question information can be determined instead of showing the entire manual to the user. Through the preset structured data, the response text information can be quickly found, which solves the problem of only being able to broadcast the entire manual, and answers the user's questions, making it easier for the user to understand the manual and improving the user experience.

[0066] In this embodiment, the method also includes: in response to a consultation request initiated by a user, determining identification information of a target object indicated by the consultation request; wherein the consultation request represents that the user is consulting the target object; and obtaining preset structured data matching the target object corresponding to the identification information.

[0067] Specifically, when a user issues a question, the user may consult a target object. A user may initiate a consultation request through a terminal device, and a consultation request refers to the user consulting the instructions of a target object. The consultation request may include or exclude the question information. For example, a user may first initiate a consultation request, indicating that the user wants to consult the target object, and then enter the question information to ask a question.

[0068] The consultation request may indicate to consult a certain target object, that is, the target object that the user wants to consult may be determined according to the consultation request. The target object may be represented by identification information, and each target object has unique identification information. For example, the identification information may be the name, code, etc. of the target object. After responding to the consultation request, the identification information of the target object indicated by the consultation request may be determined. For example, if the consultation request contains identification information, the consultation request may be directly parsed to obtain the identification information.

[0069] Different target objects correspond to different preset structured data. After the identification information is determined, the preset structured data corresponding to the identification information can be obtained, that is, the preset structured data matching the target object corresponding to the identification information is obtained. For example, the preset structured data of each object is determined in advance, and each identification information is associated with the corresponding structured data and stored. After the identification information of the consultation request is determined, the matching structured data is found according to the preset association relationship.

[0070] The beneficial effect of this arrangement is that users can consult different target objects according to actual needs. By determining the identification information, they can accurately find the corresponding structured data, and thus determine the response text information based on the structured data, thereby improving the pertinence and accuracy of the answers to questions.

[0071] In this embodiment, in response to a consultation request initiated by a user, identification information of a target object indicated by the consultation request is determined, including: in response to a consultation request issued by a user, an image of the target object is obtained from the consultation request; wherein the consultation request carries an image of the target object; identification information of the target object is identified from the image of the target object; or, in response to a consultation request issued by a user, identification information of the target object is obtained from the consultation request; wherein the consultation request is a text message or a voice message, and the consultation request carries the identification information of the target object.

[0072] Specifically, when initiating a consultation request, the user can upload an image of the target object. For example, the user can obtain a cover image by photographing the cover of a medicine box. The consultation request may include an image of the target object. After responding to the consultation request issued by the user, the image of the target object can be obtained from the consultation request. According to a preset image recognition algorithm, identification information of the target object is identified from the image of the target object. For example, a QR code or an item name in the image can be identified as identification information. In this embodiment, the preset image recognition algorithm is not specifically limited.

[0073] Alternatively, when initiating a consultation request, the user directly sends the identification information of the target object to the terminal device, that is, the consultation request may include the identification information of the target object. After responding to the consultation request issued by the user, the identification information of the target object may be obtained from the consultation request. The consultation request may be a text message or a voice message. For example, the user may input the identification information such as the name or code of the target object through a keyboard, or may input the identification information of the target object through voice.

[0074] The beneficial effect of such a setting is that after the user initiates a consultation request, the identification information can be determined according to the consultation request, so as to provide a reply to the user according to the target object. The user can initiate a consultation request in a variety of ways, which improves the flexibility of human-computer interaction and enhances the user's experience.

[0075] In this embodiment, the preset structured data matching the target object corresponding to the identification information is obtained, including: determining the explanatory text information corresponding to the identification information of the target object indicated by the consultation request as the target explanatory text based on the association relationship between the preset identification information and the explanatory text information; and performing data structure conversion processing on the target explanatory text to obtain the preset structured data matching the target object corresponding to the identification information.

[0076] Specifically, the instructions for each object are stored in advance, that is, the explanatory text information of each object is stored. One identification information can correspond to its own explanatory text information, and the association relationship between the identification information and the explanatory text information is preset. After the identification information of the target object is determined, the explanatory text information corresponding to the identification information can be determined according to the preset association relationship. The determined explanatory text information is determined as the target explanatory text.

[0077] The target interpretation text represents the entire text content of the instruction manual, and the target interpretation text needs to be converted into structured data of a preset data structure. The target interpretation text can be converted into a data structure through a preset large language model to obtain structured data of a KV structure. One target interpretation text can be converted into multiple KV structures, and all KV structures corresponding to the target interpretation text are determined as structured data corresponding to the identification information. The large language model can also convert the professional target interpretation text into easy-to-understand language to make the Value in the structured data easier to understand. In this embodiment, the model structure of the large language model is not specifically limited.

[0078] The beneficial effect of such a setting is that the corresponding target explanation text is determined and converted into structured data, which facilitates the subsequent answering of questions for users based on the structured data. Users can understand the instructions more easily, and responses can be provided to users more flexibly, conveniently and intuitively.

[0079] In this embodiment, the target interpreted text is converted into a data structure to obtain preset structured data that matches the target object corresponding to the identification information, including: splitting the target interpreted text to obtain at least one text block; wherein the text block is used to represent the local text content in the target interpreted text; determining the text content corresponding to the text block as the reply answer to the question, and performing semantic analysis on the text content corresponding to the text block to obtain the question corresponding to the text block; determining the key-value pair structure corresponding to the text block based on the text content corresponding to the text block and the question corresponding to the text block; wherein the keyword in the key-value pair structure is the question corresponding to the text block, and the key value in the key-value pair structure is the text content corresponding to the text block; determining the key-value pair structure corresponding to all text blocks of the target interpreted text as the preset structured data that matches the target object corresponding to the identification information.

[0080] Specifically, after obtaining the target interpretation text, structured data corresponding to the target interpretation text can be generated based on a preset large language model. The large language model can perform semantic analysis and other processing on the target interpretation text. For example, the target interpretation text can be split into multiple parts, each part being a text block, and at least one text block can be obtained. The text block can be used to represent the local text content in the target interpretation text. For example, a paragraph in the target interpretation text is a text block.

[0081] The text content corresponding to the text block is determined as the reply answer to the question, and the corresponding question is derived based on the reply answer to the question. That is, the text content corresponding to the text block can be semantically analyzed and processed through a preset large language model to obtain the question corresponding to the text block. For example, if the text content of the text block is the precautions for taking medicine, then it can be determined that the question corresponding to the text block is "What are the precautions when taking medicine?"

[0082] According to the text content corresponding to the text block and the question corresponding to the text block, determine the key-value pair structure corresponding to the text block. The Key in the key-value pair structure is the question corresponding to the text block, and the Value in the key-value pair structure is the text content corresponding to the text block. A target interpretation text can be split into multiple text blocks, that is, multiple KV structures can be obtained. The KV structure corresponding to all text blocks of the target interpretation text is determined as the preset structured data that matches the target object corresponding to the identification information.

[0083] The corresponding structured data may be generated after responding to the user's consultation request, or the structured data of each object may be generated in advance, and the structured data corresponding to the identification information may be searched after responding to the consultation request. If the structured data corresponding to the target object is generated after responding to the user's consultation request, the generated structured data may be stored, and if the target object is consulted again later, the generated structured data may be directly obtained, thereby improving the efficiency of human-computer interaction.

[0084] The beneficial effect of this setting is that the target explanation text is disassembled, and each part is the answer to a question. The answer is analyzed to get the corresponding question, and the question is used as the key and the answer as the value, which facilitates the subsequent targeted answers to different questions raised by users, improves the efficiency of human-computer interaction, and enhances the user's human-computer interaction experience.

[0085] In this embodiment, based on the preset structured data matched by the target object, the response text information corresponding to the question information is determined, including: the question information is determined as a keyword, and if there is a key value corresponding to the question information in the preset structured data, the key value corresponding to the question information is determined as the response text information corresponding to the question information.

[0086] Specifically, after determining the preset structured data, the user's question information can be obtained from the consultation request, or the user's new question information can be received. After receiving the user's question information, the response text information corresponding to the question information is searched from the preset structured data.

[0087] The question information can be used as a key to search the preset structured data for a value corresponding to the question information. If so, the value corresponding to the question information is determined as the response text information corresponding to the question information. If not, it is determined that a reply answer to the question information does not exist in the preset structured data, and a prompt message can be sent to the user on the visual interface of the terminal device, prompting the user that there is no answer, or prompting the user to change the question.

[0088] When searching for whether there is a Value corresponding to the question information in the preset structured data, the question information can be compared with each Key in the preset structured data to determine whether there is a Key in the preset structured data that is consistent with the question information. If so, the Value corresponding to the Key is determined to be the Value corresponding to the question information. The similarity between the question information and each Key in the preset structured data can also be calculated. If the similarity exceeds a preset similarity threshold, the question information is considered to be consistent with the Key in the preset structured data, or the Key corresponding to the highest similarity is determined to be the Key consistent with the question information. In this embodiment, a preset NLP algorithm can be used to analyze the question information to determine the similarity between the question information and each Key in the preset structured data.

[0089] The beneficial effect of this setting is that by searching through structured data, it is possible to answer different questions raised by users, thereby improving the flexibility of human-computer interaction and the accuracy of response answers.

[0090] In this embodiment, the method further includes: if the key value corresponding to the question information does not exist in the preset structured data, performing an online query on the question information to obtain response text information corresponding to the question information.

[0091] Specifically, if the value corresponding to the question information does not exist in the preset structured data, it is determined that the response text information cannot be provided to the user according to the preset structured data. A prompt message can be sent to the user to inform the user that the answer cannot be obtained, or an online query can be performed for the user's question information, and the result of the online query can be determined as the response text information. For example, the question can be answered based on the support of big data such as medical classics. Information matching the question information can be searched from a preset database as the response text information.

[0092] The beneficial effect of this setting is that if there is no corresponding Value, an online query will be performed to ensure that answers can be provided to users, helping users to obtain relevant information such as medication instructions more conveniently and quickly, thereby improving the user experience.

[0093] S202, performing text analysis on the response text information to obtain digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information and expression information.

[0094] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0095] S203: Based on the preset digital human image, broadcast the digital human broadcast information.

[0096] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0097] The disclosed embodiment can realize the answering of questions between the user and the terminal device by responding to the user's question information and determining the response text information corresponding to the question information. The user can ask questions according to the actual situation and get the reply answers to the questions. By performing text analysis and processing on the response text information, the digital human broadcast information is obtained. The digital human broadcast information may include various forms of information such as voice information, action information and expression information, and the information is broadcast through the digital human image. The problem that the words in the manual of the item are small and difficult to understand, and the problem that people with reading disabilities cannot read the manual, is solved. When the user has questions, valuable answers are obtained through human-computer interaction, which helps the user to get answers more conveniently and quickly, and realizes a flexible, convenient and intuitive way to provide reply answers to users, thereby improving the user experience.

[0098] Figure 3 A flowchart of a human-computer interaction method based on a digital human is provided for an embodiment of the present disclosure. This embodiment is an optional embodiment based on the above embodiment.

[0099] In this embodiment, the method may additionally include: determining the language unit corresponding to the voice information currently being played; and marking the language unit corresponding to the voice information currently being played with a preset reading mark in a preset area of ​​the visual interface.

[0100] like Figure 3 As shown, the method comprises the following steps:

[0101] S301. In response to a user's question information, determine response text information corresponding to the question information; wherein the question information represents a question raised by the user to a target object, and the response text information represents a reply answer to the question represented by the question information.

[0102] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0103] S302, performing text analysis on the response text information to obtain digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information and expression information.

[0104] Exemplarily, the digital human broadcast information may include voice information, action information and expression information. The response text information is information in text form, that is, the information in text form can be converted into voice information, action information and expression information. The response text information is subjected to text analysis processing, which may be based on a preset NLP model to perform semantic analysis on the response text information to obtain action information and expression information, and to perform TTS (Text To Speech) processing on the response text information to obtain voice information.

[0105] In this embodiment, the digital human broadcast information is voice information; performing text analysis processing on the response text information to obtain the digital human broadcast information includes: performing word segmentation processing on the response text information to divide the response text information into at least one language unit; wherein the language unit represents a local text in the response text information; performing text-to-speech conversion processing on the language unit to obtain voice information corresponding to the response text information.

[0106] Specifically, when determining the voice information corresponding to the response text information, the response text information can be segmented, that is, the response text information can be cut. The response text information is segmented into at least one language unit, and the language unit represents the local text in the response text information, which is a part of the response text information after cutting. For example, according to a preset NLP model, a semantic analysis can be performed on the response text information to determine the words in the response text information, and the response text information can be cut into individual words, each segmented word is a language unit, and the response text information can be cut into multiple language units.

[0107] After the language unit is obtained, each language unit is converted into a TTS, that is, the text of the language unit is converted into audio data. The audio data of each language unit are connected in sequence to obtain voice information corresponding to the response text information.

[0108] The beneficial effect of such a setting is that the text content is divided into TTS language units, which makes it easier to determine the voice of each language unit, thereby obtaining a complete voice response to the text information and improving the efficiency and accuracy of determining the voice information.

[0109] In this embodiment, the digital human broadcast information is action information; text analysis processing is performed on the response text information to obtain the digital human broadcast information, including: semantic analysis of the language unit to determine the emotional information of the language unit; wherein the emotional information represents the emotion expressed by the language unit; according to the preset association relationship between the emotional information and the action information, the action information corresponding to the emotional information of the language unit is determined as the action information corresponding to the response text information.

[0110] Specifically, the digital human can make different movements for different response text messages. Different movements can be used to express different emotions. By showing movements, the digital human can help users understand the meaning of the response text message.

[0111] When determining the action information corresponding to the response text information, the language unit may be subjected to semantic analysis, for example, a preset NLP may be used for semantic analysis. The semantic analysis may include sentiment analysis, that is, the sentiment information of the language unit may be determined. The sentiment information may represent the emotion expressed by the language unit, for example, the sentiment information corresponding to the language unit may be a more serious emotion, or a more relaxed emotion. The sentiment information may be an emotion identifier representing the emotion, for example, the sentiment information is 01, indicating a serious emotion; the sentiment information is 02, indicating a relaxed emotion.

[0112] The association relationship between the emotion information and the action information is preset. After the emotion information corresponding to the language unit is determined, the action information corresponding to the emotion information of the language unit can be determined according to the preset association relationship. The action information of all language units is determined as the action information corresponding to the response text information. For example, a language unit in the response text information is "no drinking", and the emotion expressed by the language unit is more serious, and the corresponding action information can be a waving action.

[0113] The beneficial effect of such a setting is that the content of the response text information is analyzed, the sentence sentiment is analyzed based on the content, and the corresponding action information is obtained, so that the user can understand the response text information more intuitively and improve the user experience.

[0114] In this embodiment, the digital human broadcast information is expression information; the response text information is subjected to text analysis processing to obtain the digital human broadcast information, including: according to the association relationship between the preset emotion information and the expression information, the expression information corresponding to the emotion information of the language unit is determined as the expression information corresponding to the response text information.

[0115] Specifically, expression information is used to represent the digital human's facial expression. The digital human can make different expressions for different response text information. Different expressions can be used to express different emotions. By showing expressions, the digital human can help users understand the meaning of the response text information.

[0116] When determining the expression information corresponding to the response text information, the language unit may be subjected to semantic analysis, for example, a preset NLP may be used for semantic analysis. The semantic analysis may include sentiment analysis, that is, the sentiment information of the language unit may be determined. The sentiment information may represent the emotion expressed by the language unit, for example, the sentiment information corresponding to the language unit may be a more serious emotion, or a more relaxed emotion. The sentiment information may be an emotion identifier representing the emotion, for example, the sentiment information is 01, which represents a serious emotion; and the sentiment information is 02, which represents a relaxed emotion.

[0117] The association relationship between the emotion information and the expression information is preset. After the emotion information corresponding to the language unit is determined, the expression information corresponding to the emotion information of the language unit can be determined according to the preset association relationship. The action information of all language units is determined as the expression information corresponding to the response text information. For example, a language unit in the response text information is "no drinking", and the emotion expressed by the language unit is more serious, and the corresponding expression information can be a frowning expression.

[0118] The beneficial effect of such a setting is that the content of the response text information is analyzed, the sentence emotion is analyzed according to the content, and the corresponding expression information is obtained, so that the user can understand the response text information more intuitively and improve the user experience.

[0119] S303: Based on the preset digital human image, broadcast the digital human broadcast information.

[0120] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0121] The preset digital human voice is used to broadcast voice information, and the preset digital human appearance is used to broadcast the digital human's limb movement information and facial expression information. The playback time of voice information, movement information, and expression information corresponds to each other. When a certain language unit is played, the digital human makes corresponding movements and expressions to facilitate users to better understand.

[0122] S304, determining the language unit corresponding to the currently played voice information; marking the language unit corresponding to the currently played voice information with a preset reading mark in a preset area of ​​the visual interface.

[0123] For example, when the digital human broadcast information is broadcasted through the digital human image, the language unit corresponding to the currently played voice information can be determined in real time, that is, it is determined which language unit the currently played voice belongs to.

[0124] A preset area is designed on the visual interface to display response text information. The reading mark is preset, and the position of the reading mark in the preset area is changed in real time so that the reading mark always follows the currently played language unit. For example, the reading mark can be a rectangular frame or a highlight mark. Taking the highlight mark as an example, the currently played language unit can be highlighted in the preset area, so that the user can intuitively see the current playback progress. The response text information can be scrolled in the preset area, and the currently broadcast text content is always highlighted on the screen, helping users to obtain the reply answer to the question information more conveniently and quickly, and enhancing the user's experience of human-computer interaction.

[0125] Figure 4 A schematic diagram of a visualization interface provided in this embodiment. Figure 4 In the example, the user asked the question "What are the precautions for taking medicine A?", and the response text message was "You need to take the medicine on time and in the dosage, and you cannot stop taking the medicine or change the dosage at will. In terms of diet, you should avoid drinking alcohol, as it may cause symptoms of nausea and weakness, affect the efficacy of the medicine, and even cause the risk of lactic acid poisoning." Among them, "You need to take the medicine on time and in the dosage, and you cannot stop taking the medicine or change the dosage at will" is displayed in bold and underlined on the interface. Bold and underlined are preset reading marks, indicating that the currently played voice is "You need to take the medicine on time and in the dosage, and you cannot stop taking the medicine or change the dosage at will."

[0126] The disclosed embodiment can realize the answering of questions between the user and the terminal device by responding to the user's question information and determining the response text information corresponding to the question information. The user can ask questions according to the actual situation and get the reply answers to the questions. By performing text analysis and processing on the response text information, the digital human broadcast information is obtained. The digital human broadcast information may include various forms of information such as voice information, action information and expression information, and the information is broadcast through the digital human image. The problem that the words in the manual of the item are small and difficult to understand, and the problem that people with reading disabilities cannot read the manual, is solved. When the user has questions, valuable answers are obtained through human-computer interaction, which helps the user to get answers more conveniently and quickly, and realizes a flexible, convenient and intuitive way to provide reply answers to users, thereby improving the user experience.

[0127] Figure 5 A flowchart of a human-computer interaction method based on a digital human is provided for an embodiment of the present disclosure. This embodiment is an optional embodiment based on the above embodiment.

[0128] In this embodiment, the target object is a drug, and the explanatory text information is a drug instruction sheet; in response to the user's question information, based on the preset structured data matched by the target object, the response text information corresponding to the question information is determined, which can be refined as follows: in response to the user's question information regarding the drug, the drug identification of the drug is determined; wherein the drug identification is used to characterize the drug; the drug instruction sheet corresponding to the drug identification is obtained; based on the drug instruction sheet corresponding to the drug identification, the preset structured data matched with the drug is determined; based on the preset structured data matched with the drug, the response text information corresponding to the question information is determined.

[0129] like Figure 5 As shown, the method comprises the following steps:

[0130] S501. In response to a question from a user regarding a drug, determine a drug identifier of the drug; wherein the drug identifier is used to characterize the drug.

[0131] For example, a user can consult a drug. For example, if the user is not clear about the instructions for use of a drug, the user can ask questions about the drug. The question information may include the drug identification of the drug. Each drug may have a unique drug identification, for example, the drug identification may be the name of the drug.

[0132] In response to the question information sent by the user, the drug identification of the drug is obtained from the question information. For example, if the question information sent by the user is "What is the method of taking drug A", the drug identification can be obtained as "drug A".

[0133] In this embodiment, determining the drug identification of the drug includes: obtaining a medicine box cover image of the drug uploaded by a user; identifying the drug name from the medicine box cover image according to a preset image recognition algorithm, and determining the drug name as the drug identification of the drug.

[0134] Specifically, when the user sends a question, the user can use the camera on the terminal device to take a picture of the medicine box cover to obtain the medicine box cover image, and upload the medicine box cover image when sending the question.

[0135] After responding to the user's question information about the drug, the medicine box cover image uploaded by the user is obtained. The medicine box cover image contains the drug identification of the drug, for example, the drug name can be printed on the medicine box cover. Based on the preset image recognition algorithm, the drug identification of the medicine box cover image is identified, for example, the drug name can be identified. The drug name is identified from the medicine box cover image, and the identified drug name is determined as the drug identification of the drug. In this embodiment, the preset image recognition algorithm is not specifically limited.

[0136] The beneficial effect of this setting is that the user can automatically identify the drug identification by taking a photo and uploading the cover image of the medicine box, without the need for the user to manually enter the drug identification, thus reducing user operations. This effectively avoids users from entering incorrect drug identifications, improves the accuracy and efficiency of drug identification, and further improves the accuracy and efficiency of responding to text information, thus improving the user's human-computer interaction experience.

[0137] S502: Obtain the drug instructions corresponding to the drug identification.

[0138] For example, each drug has its own drug instructions, which are text information, including information such as an introduction to the drug and instructions for use of the drug. Drug instructions for different drugs are stored in a preset database in advance. When the drug that the user wants to consult is determined, the corresponding drug instructions can be obtained from the preset database.

[0139] In this embodiment, obtaining the drug instructions corresponding to the drug identification includes: determining the drug instructions associated with the drug name of the drug according to a preset association relationship between the drug name and the drug instructions.

[0140] Specifically, the association relationship between the drug identification and the drug instructions is preset, for example, the association relationship between the drug name and the drug instructions can be set. After the drug name is determined, the drug instructions corresponding to the drug name can be found according to the preset association relationship.

[0141] The beneficial effect of such a setting is that, according to the preset association relationship, the required drug instructions can be quickly found, which makes it easier to answer users through the drug instructions and improve the efficiency and accuracy of human-computer interaction.

[0142] S503: Determine preset structured data matching the drug according to the drug instructions corresponding to the drug identification.

[0143] Exemplarily, the preset structured data is a key-value pair structure. After obtaining the drug instructions, the drug instructions are converted into data of the key-value pair structure as the preset structured data matching the drug consulted by the user. A drug instruction may correspond to multiple key-value pair structures, each of which represents a question that the user may ask and the answer to the question. The answer to the question is the information extracted from the drug instructions, and the question that the user may ask is a question inferred based on the answer to the question.

[0144] In this embodiment, according to the drug instructions corresponding to the drug identification, the preset structured data matching the drug is determined, including: according to the preset large language model, the drug instructions are split and processed to obtain at least one text block; wherein the text block is used to represent the local text content in the drug instructions; the medical terms in the text block are obtained, and the medical terms are converted into a preset common language; wherein the medical terms have the same semantics as the corresponding common language; the text content corresponding to the converted text block is determined as the reply answer to the question, and the text content corresponding to the text block is semantically analyzed to obtain the question corresponding to the text block; according to the text content corresponding to the converted text block and the question corresponding to the text block, a key-value pair structure corresponding to the text block is determined; wherein the keyword in the key-value pair structure is the question corresponding to the text block, and the key value in the key-value pair structure is the text content corresponding to the converted text block; the key-value pair structure corresponding to all text blocks of the drug instructions is determined as the preset structured data matching the drug.

[0145] Specifically, a large language model is pre-trained and constructed, and the large language model can be used to analyze and process text data. According to the preset large language model, the drug instructions are split and processed to obtain at least one text block. The content in each text block is a part of the drug instructions, that is, the text block can be used to represent the local text content in the drug instructions.

[0146] There may be professional medical terms in the drug instructions, and users may have difficulty understanding the medical terms. In order to help users better understand the drug instructions, the medical terms in each text block can be obtained. For example, a medical terminology database can be preset, and a variety of medical terms are stored in the medical terminology database. The content in the text block is matched with the medical terms in the medical terminology database to obtain the medical terms in the text block. According to the preset large language model, the medical terms are converted into a preset common language, that is, the medical terms are converted into more easy-to-understand terms. The semantics of medical terms are the same as those of the corresponding common language, that is, when the medical terms are converted, the meaning of the medical terms is not changed. The association relationship between different medical terms and common languages ​​can also be preset, and the common language corresponding to the medical terms in the text block is searched according to the association relationship.

[0147] The text content corresponding to the converted text block is determined as the reply answer to the question, so that the reply answer to the question is easier for the user to understand. According to the preset large language model, the text content corresponding to the text block is semantically analyzed to obtain the question corresponding to the text block. It can be a semantic analysis of the content of the converted text block, or it can be a semantic analysis of the content of the text block before conversion. According to the content of the text block, determine the question corresponding to the content, that is, determine what kind of question corresponds to the reply answer of the content of the text block.

[0148] According to the text content corresponding to the converted text block and the question corresponding to the text block, the key-value pair structure corresponding to the text block is determined. The keywords in the key-value pair structure are the questions corresponding to the text block, and the key values ​​in the key-value pair structure are the text content corresponding to the converted text block. All key-value pair structures of the drug instructions are determined as preset structured data matching the drug.

[0149] The beneficial effect of this setting is that the drug instructions are converted into a key-value pair structure, allowing users to ask questions about different parts of the drug instructions, providing users with targeted answers, and by converting medical terms, the replies are easier to understand, thereby improving the user's human-computer interaction experience.

[0150] S504: Determine response text information corresponding to the question information based on the preset structured data matched with the drug.

[0151] Exemplarily, the response text information corresponding to the question information is searched from the preset structured data of the drug targeted by the user, and the found response text information may be a part of the content in the drug instruction manual.

[0152] In this embodiment, the question information represents the questions raised by the user regarding the drug instructions; based on the preset structured data matching the drug, the response text information corresponding to the question information is determined, including: determining the questions raised by the user regarding the drug instructions as keywords, and if there is a key value corresponding to the question in the preset structured data, then determining the key value corresponding to the question as the response text information corresponding to the question.

[0153] Specifically, the user asks a question about a drug, and the question information is about the drug instructions. After determining the question raised by the user, you can search the preset structured data to see if there is a reply answer corresponding to the question. In the preset structured data, the keywords are questions that the user may ask, and the key-value pairs are the replies to the questions. The question in the question information can be compared with the keywords in the preset structured data to find out whether there are keywords similar or consistent with the question in the question information. If so, the key value of the keyword is determined as the response text information corresponding to the question information; if not, it is determined that the drug instructions do not have the answer the user wants.

[0154] The beneficial effect of this setting is that it can provide targeted responses to different questions raised by users, and improve the efficiency and accuracy of responses by presetting structured data. It also does not need to display the complete drug instructions to users, which improves the flexibility and convenience of human-computer interaction.

[0155] S505: Perform text analysis on the response text information to obtain digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information, and expression information.

[0156] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0157] S506: Based on the preset digital human image, broadcast the digital human broadcast information.

[0158] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0159] Figure 6 The structure block diagram of a human-computer interaction device based on a digital human provided by the embodiment of the present disclosure. For the convenience of explanation, only the part related to the embodiment of the present disclosure is shown. Figure 6 The human-computer interaction device 600 based on digital human includes: a determination module 601, an analysis module 602 and a broadcasting module 603.

[0160] The determination module 601 is used to determine, in response to the user's question information, the response text information corresponding to the question information; wherein the question information represents the question raised by the user to the target object, and the response text information represents the reply answer to the question represented by the question information;

[0161] The analysis module 602 is used to perform text analysis on the response text information to obtain the digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information and expression information;

[0162] The broadcast module 603 is used to broadcast the digital human broadcast information based on the preset digital human image.

[0163] Figure 7 A structural block diagram of a human-computer interaction device based on a digital human provided in an embodiment of the present disclosure, such as Figure 7 As shown, the human-computer interaction device 700 based on digital human includes a determination module 701 , an analysis module 702 and a reporting module 703 , wherein the determination module 701 includes an information determination unit 7011 .

[0164] The information determination unit 7011 is used to respond to the user's question information and determine the response text information corresponding to the question information based on the preset structured data matched by the target object; wherein the preset structured data represents the structured data corresponding to the explanatory text information of the target object, the format of the preset structured data is a key-value pair format, the preset structured data represents the relationship between the question information and the response text information, and the explanatory text information represents the text of the explanation of the target object.

[0165] In one example, the device further includes:

[0166] A consultation module, configured to determine, in response to a consultation request initiated by a user, identification information of a target object indicated by the consultation request; wherein the consultation request indicates that the user is consulting the target object;

[0167] The data acquisition module is used to acquire preset structured data matching the target object corresponding to the identification information.

[0168] In one example, a consulting module includes:

[0169] an identification recognition unit, configured to, in response to a consultation request issued by a user, obtain an image of a target object from the consultation request; wherein the consultation request carries an image of the target object; and recognize identification information of the target object from the image of the target object;

[0170] Alternatively, the identification determination unit is used to obtain the identification information of the target object from the consultation request in response to the consultation request issued by the user; wherein the consultation request is a text message or a voice message, and the consultation request carries the identification information of the target object.

[0171] In one example, the data acquisition module includes:

[0172] A text determination unit, configured to determine, according to a preset association relationship between the identification information and the interpretation text information, the interpretation text information corresponding to the identification information of the target object indicated by the consultation request as the target interpretation text;

[0173] The structure conversion unit is used to perform data structure conversion processing on the target interpretation text to obtain preset structured data matching the target object corresponding to the identification information.

[0174] In one example, the structure conversion unit includes:

[0175] A splitting subunit, used for splitting the target interpretation text to obtain at least one text block; wherein the text block is used to represent a local text content in the target interpretation text;

[0176] A semantic analysis subunit is used to determine the text content corresponding to the text block as a reply answer to the question, and perform semantic analysis on the text content corresponding to the text block to obtain the question corresponding to the text block;

[0177] A key-value pair determination subunit is used to determine a key-value pair structure corresponding to the text block according to the text content corresponding to the text block and the question corresponding to the text block; wherein the keyword in the key-value pair structure is the question corresponding to the text block, and the key value in the key-value pair structure is the text content corresponding to the text block;

[0178] The data determination subunit is used to determine the key-value pair structure corresponding to all text blocks of the target interpreted text as preset structured data matching the target object corresponding to the identification information.

[0179] In one example, the information determination unit 7011 is specifically configured to:

[0180] The question information is determined as a keyword, and if a key value corresponding to the question information exists in the preset structured data, the key value corresponding to the question information is determined as response text information corresponding to the question information.

[0181] In one example, the device further includes:

[0182] The networking module is used to perform a network query on the question information to obtain response text information corresponding to the question information if the key value corresponding to the question information does not exist in the preset structured data.

[0183] In one example, the information broadcast by the digital human is voice information; the analysis module 702 includes:

[0184] A word segmentation unit, used to perform word segmentation processing on the response text information, and divide the response text information into at least one language unit; wherein the language unit represents a local text in the response text information;

[0185] The first obtaining unit is used to perform text-to-speech conversion processing on the language unit to obtain voice information corresponding to the response text information.

[0186] In one example, the digital human broadcast information is action information; the analysis module 702 includes:

[0187] An emotion determination unit, configured to perform semantic analysis on the language unit to determine the emotion information of the language unit; wherein the emotion information represents the emotion expressed by the language unit;

[0188] The second obtaining unit is used to determine the action information corresponding to the emotional information of the language unit according to the preset association relationship between the emotional information and the action information, which is the action information corresponding to the response text information.

[0189] In one example, the information broadcast by the digital human is expression information; the analysis module 702 includes:

[0190] The third obtaining unit is used to determine the expression information corresponding to the emotion information of the language unit according to the preset association relationship between the emotion information and the expression information, which is the expression information corresponding to the response text information.

[0191] In one example, the device further includes:

[0192] A unit determination module, used to determine the language unit corresponding to the currently played voice information;

[0193] The marking module is used to mark the language unit corresponding to the currently played voice information with a preset reading mark in a preset area of ​​the visual interface.

[0194] In one example, the broadcast module 703 includes:

[0195] An image selection unit, used to determine the digital human image selected by the user according to the user's selection operation on the digital human image;

[0196] The display unit is used to display the action corresponding to the action information and the expression corresponding to the expression information on a visual interface through the digital human image selected by the user, and play the voice information.

[0197] In one example, the device further includes:

[0198] The text scrolling module is used to play the response text information in a scrolling manner within a preset area of ​​the visual interface according to a preset text scrolling speed.

[0199] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0200] In one example, the target object is a drug, and the explanatory text information is a drug instruction sheet; the information determination unit includes:

[0201] The identification determination subunit is used to determine the drug identification of the drug in response to the question information issued by the user regarding the drug; wherein the drug identification is used to characterize the drug;

[0202] The instruction obtaining subunit is used to obtain the drug instruction corresponding to the drug identification;

[0203] The instruction sheet conversion subunit is used to determine the preset structured data matching the drug according to the drug instruction sheet corresponding to the drug identifier;

[0204] The information search subunit is used to determine the response text information corresponding to the question information based on the preset structured data matching the drug.

[0205] In one example, the identification subunit is specifically used for:

[0206] Obtaining the medicine box cover image of the medicine uploaded by the user;

[0207] According to a preset image recognition algorithm, the name of the medicine is recognized from the medicine box cover image, and the name of the medicine is determined as the medicine identification of the medicine.

[0208] In one example, the instruction manual acquisition subunit is specifically used for:

[0209] According to the preset association relationship between the drug name and the drug instructions, the drug instructions associated with the drug name of the drug are determined.

[0210] In one example, the instruction manual conversion subunit is specifically used for:

[0211] According to a preset large language model, the drug instruction manual is split to obtain at least one text block; wherein the text block is used to represent a local text content in the drug instruction manual;

[0212] Acquire the medical terms in the text block, and convert the medical terms into a preset common language; wherein the medical terms have the same semantics as the corresponding common language;

[0213] Determine the text content corresponding to the converted text block as the reply answer to the question, perform semantic analysis on the text content corresponding to the text block, and obtain the question corresponding to the text block;

[0214] Determine a key-value pair structure corresponding to the text block according to the text content corresponding to the converted text block and the question corresponding to the text block; wherein the keyword in the key-value pair structure is the question corresponding to the text block, and the key value in the key-value pair structure is the text content corresponding to the converted text block;

[0215] The key-value pair structures corresponding to all text blocks of the drug instructions are determined as preset structured data matching the drug.

[0216] In one example, the question information represents the question raised by the user regarding the drug instructions; the information search subunit is specifically used to:

[0217] Questions raised by users regarding drug instructions are determined as keywords, and if a key value corresponding to the question exists in the preset structured data, the key value corresponding to the question is determined as response text information corresponding to the question.

[0218] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present disclosure. The electronic device may be a microcontroller, such as Figure 8 As shown, the electronic device 800 includes: at least one processor 802; and a memory 801 communicatively connected to the at least one processor 802; wherein the memory stores instructions executable by the at least one processor 802, and the instructions are executed by the at least one processor 802 so that the at least one processor 802 can execute the human-computer interaction method based on digital human disclosed in the present invention.

[0219] The electronic device 800 further includes a receiver 803 and a transmitter 804. The receiver 803 is used to receive instructions and data sent by other devices, and the transmitter 804 is used to send instructions and data to external devices.

[0220] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0221] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0222] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0223] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0224] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0225] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a human-computer interaction method based on a digital person. For example, in some embodiments, a human-computer interaction method based on a digital person may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the human-computer interaction method based on a digital person described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the human-computer interaction method based on a digital human in any other appropriate manner (for example, by means of firmware).

[0226] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0227] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0228] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0229] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0230] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0231] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0232] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0233] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A human-computer interaction method based on digital human, comprising: In response to a user's question information, determining response text information corresponding to the question information; wherein the question information represents a question raised by the user to a target object, and the response text information represents a reply answer to the question represented by the question information; Performing text analysis on the response text information to obtain the digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information, and expression information; Based on the preset digital human image, broadcast the digital human broadcast information; The step of determining, in response to the user's question information, response text information corresponding to the question information includes: In response to the user's question information, based on the preset structured data matched by the target object, the response text information corresponding to the question information is determined; wherein the preset structured data represents the structured data corresponding to the explanatory text information of the target object, the format of the preset structured data is a key-value pair format, the preset structured data represents the relationship between the question information and the response text information, and the explanatory text information represents the text of the explanation of the target object; The method further comprises: In response to a consultation request initiated by a user, determining identification information of a target object indicated by the consultation request; wherein the consultation request represents that the user is consulting the target object; According to the association relationship between the preset identification information and the interpretation text information, the interpretation text information corresponding to the identification information of the target object indicated by the consultation request is determined as the target interpretation text; The target interpretation text is subjected to data structure conversion processing to obtain preset structured data matching the target object corresponding to the identification information.

2. The method according to claim 1, wherein: The step of determining, in response to a consultation request initiated by a user, identification information of a target object indicated by the consultation request includes: In response to a consultation request issued by a user, an image of a target object is obtained from the consultation request; wherein the consultation request carries the image of the target object; and identification information of the target object is identified from the image of the target object; Alternatively, in response to a consultation request issued by a user, the identification information of the target object is obtained from the consultation request; wherein the consultation request is a text message or a voice message, and the consultation request carries the identification information of the target object.

3. The method according to claim 2, wherein: The step of converting the target interpretation text into a data structure to obtain preset structured data that matches the target object corresponding to the identification information includes: Splitting the target interpretation text to obtain at least one text block; wherein the text block is used to represent a local text content in the target interpretation text; Determine the text content corresponding to the text block as the reply answer to the question, perform semantic analysis on the text content corresponding to the text block, and obtain the question corresponding to the text block; Determine a key-value pair structure corresponding to the text block according to the text content corresponding to the text block and the question corresponding to the text block; wherein the keyword in the key-value pair structure is the question corresponding to the text block, and the key value in the key-value pair structure is the text content corresponding to the text block; The key-value pair structures corresponding to all text blocks of the target interpreted text are determined as preset structured data matching the target object corresponding to the identification information.

4. The method according to claim 3, wherein: The step of determining the response text information corresponding to the question information based on the preset structured data matched by the target object includes: The question information is determined as a keyword, and if a key value corresponding to the question information exists in the preset structured data, the key value corresponding to the question information is determined as response text information corresponding to the question information.

5. The method according to claim 4, further comprising: If the key value corresponding to the question information does not exist in the preset structured data, an online query is performed on the question information to obtain response text information corresponding to the question information.

6. The method according to any one of claims 1 to 5, wherein: The digital human broadcast information is voice information; the text analysis and processing of the response text information to obtain the digital human broadcast information includes: Performing word segmentation processing on the response text information to divide the response text information into at least one language unit; wherein the language unit represents a local text in the response text information; The language unit is subjected to text-to-speech conversion processing to obtain voice information corresponding to the response text information.

7. The method according to claim 6, wherein: The digital human broadcast information is action information; the text analysis and processing of the response text information to obtain the digital human broadcast information includes: Performing semantic analysis on the language unit to determine the emotional information of the language unit; wherein the emotional information represents the emotion expressed by the language unit; According to the preset association relationship between the emotion information and the action information, the action information corresponding to the emotion information of the language unit is determined as the action information corresponding to the response text information.

8. The method according to claim 7, wherein: The digital human broadcast information is expression information; the text analysis and processing of the response text information to obtain the digital human broadcast information includes: According to the preset association relationship between the emotion information and the expression information, the expression information corresponding to the emotion information of the language unit is determined as the expression information corresponding to the response text information.

9. The method according to any one of claims 7-8, further comprising: Determine the language unit corresponding to the currently played voice information; In a preset area of ​​the visual interface, a preset reading mark is used to mark the language unit corresponding to the currently played voice information.

10. The method according to any one of claims 1-5, 7-8, wherein: The broadcasting of the digital human broadcasting information based on the preset digital human image includes: Determining the digital human image selected by the user according to the user's selection operation on the digital human image; The action corresponding to the action information and the expression corresponding to the expression information are displayed on a visual interface through the digital human image selected by the user, and the voice information is played.

11. The method according to any one of claims 1-5, 7-8, further comprising: According to the preset text scrolling speed, the response text information is played in a scrolling manner within a preset area of ​​the visual interface.

12. The method according to claim 1, wherein: The target object is a drug, and the explanatory text information is a drug instruction manual; the response to the user's question information, based on the preset structured data matched by the target object, determines the response text information corresponding to the question information, including: In response to a question from a user regarding a drug, determining a drug identifier of the drug; wherein the drug identifier is used to characterize the drug; Obtaining a drug instruction sheet corresponding to the drug identification; Determining preset structured data matching the drug according to the drug instructions corresponding to the drug identifier; According to the preset structured data matched with the drug, the response text information corresponding to the question information is determined.

13. The method according to claim 12, wherein: The step of determining the drug identification of the drug includes: Obtaining the medicine box cover image of the medicine uploaded by the user; According to a preset image recognition algorithm, the name of the medicine is recognized from the medicine box cover image, and the name of the medicine is determined as the medicine identification of the medicine.

14. The method according to claim 12 or 13, wherein: The obtaining of the drug instructions corresponding to the drug identification includes: According to the preset association relationship between the drug name and the drug instructions, the drug instructions associated with the drug name of the drug are determined.

15. The method according to any one of claims 12 to 13, wherein: The step of determining preset structured data matching the drug according to the drug instructions corresponding to the drug identifier includes: According to a preset large language model, the drug instruction manual is split to obtain at least one text block; wherein the text block is used to represent a local text content in the drug instruction manual; Acquire the medical terms in the text block, and convert the medical terms into a preset common language; wherein the medical terms have the same semantics as the corresponding common language; Determine the text content corresponding to the converted text block as the reply answer to the question, perform semantic analysis on the text content corresponding to the text block, and obtain the question corresponding to the text block; Determine a key-value pair structure corresponding to the text block according to the text content corresponding to the converted text block and the question corresponding to the text block; wherein the keyword in the key-value pair structure is the question corresponding to the text block, and the key value in the key-value pair structure is the text content corresponding to the converted text block; The key-value pair structures corresponding to all text blocks of the drug instructions are determined as preset structured data matching the drug.

16. The method according to any one of claims 12 to 13, wherein: The question information represents the question raised by the user regarding the drug instructions; the response text information corresponding to the question information is determined according to the preset structured data matched with the drug, including: Questions raised by users regarding drug instructions are determined as keywords, and if a key value corresponding to the question exists in the preset structured data, the key value corresponding to the question is determined as response text information corresponding to the question.

17. A human-computer interaction device based on digital human, comprising: A determination module, configured to determine, in response to a user's question information, a response text information corresponding to the question information; wherein the question information represents a question raised by the user to a target object, and the response text information represents a reply answer to the question represented by the question information; An analysis module, used for performing text analysis on the response text information to obtain the digital human broadcast information; wherein the digital human broadcast information includes at least one of the digital human's voice information, action information and expression information; A broadcasting module, used for broadcasting the digital human broadcasting information based on a preset digital human image; The determining module comprises: An information determination unit is used to determine, in response to a user's question information, response text information corresponding to the question information based on preset structured data matched by the target object; wherein the preset structured data represents structured data corresponding to the explanatory text information of the target object, the format of the preset structured data is a key-value pair format, the preset structured data represents the relationship between the question information and the response text information, and the explanatory text information represents text of an explanation of the target object; A consultation module, configured to determine, in response to a consultation request initiated by a user, identification information of a target object indicated by the consultation request; wherein the consultation request indicates that the user is consulting the target object; Data acquisition module, including: A text determination unit, configured to determine, according to a preset association relationship between the identification information and the interpretation text information, the interpretation text information corresponding to the identification information of the target object indicated by the consultation request as the target interpretation text; The structure conversion unit is used to perform data structure conversion processing on the target interpretation text to obtain preset structured data matching the target object corresponding to the identification information.

18. The device according to claim 17, wherein: The consulting module comprises: an identification recognition unit, configured to, in response to a consultation request issued by a user, obtain an image of a target object from the consultation request; wherein the consultation request carries an image of the target object; and recognize identification information of the target object from the image of the target object; Alternatively, the identification determination unit is used to obtain the identification information of the target object from the consultation request in response to the consultation request issued by the user; wherein the consultation request is a text message or a voice message, and the consultation request carries the identification information of the target object.

19. The device according to claim 18, wherein: The structure conversion unit comprises: A splitting subunit, used for splitting the target interpretation text to obtain at least one text block; wherein the text block is used to represent a local text content in the target interpretation text; A semantic analysis subunit is used to determine the text content corresponding to the text block as a reply answer to the question, and perform semantic analysis on the text content corresponding to the text block to obtain the question corresponding to the text block; A key-value pair determination subunit is used to determine a key-value pair structure corresponding to the text block according to the text content corresponding to the text block and the question corresponding to the text block; wherein the keyword in the key-value pair structure is the question corresponding to the text block, and the key value in the key-value pair structure is the text content corresponding to the text block; The data determination subunit is used to determine the key-value pair structure corresponding to all text blocks of the target interpreted text as preset structured data matching the target object corresponding to the identification information.

20. The device according to claim 19, wherein The information determination unit is specifically used to: The question information is determined as a keyword, and if a key value corresponding to the question information exists in the preset structured data, the key value corresponding to the question information is determined as response text information corresponding to the question information.

21. The apparatus according to claim 20, further comprising: The networking module is used to perform a network query on the question information to obtain response text information corresponding to the question information if the key value corresponding to the question information does not exist in the preset structured data.

22. The device according to any one of claims 17 to 21, wherein: The digital human broadcast information is voice information; The analysis module comprises: A word segmentation unit, used to perform word segmentation processing on the response text information, and divide the response text information into at least one language unit; wherein the language unit represents a local text in the response text information; The first obtaining unit is used to perform text-to-speech conversion processing on the language unit to obtain voice information corresponding to the response text information.

23. The device according to claim 22, wherein: The digital human broadcast information is action information; The analysis module comprises: An emotion determination unit, configured to perform semantic analysis on the language unit to determine the emotion information of the language unit; wherein the emotion information represents the emotion expressed by the language unit; The second obtaining unit is used to determine the action information corresponding to the emotional information of the language unit according to the preset association relationship between the emotional information and the action information, which is the action information corresponding to the response text information.

24. The device according to claim 23, wherein: The digital human broadcast information is expression information; The analysis module comprises: The third obtaining unit is used to determine the expression information corresponding to the emotion information of the language unit according to the preset association relationship between the emotion information and the expression information, which is the expression information corresponding to the response text information.

25. The apparatus according to any one of claims 23-24, further comprising: A unit determination module, used to determine the language unit corresponding to the currently played voice information; The marking module is used to mark the language unit corresponding to the currently played voice information with a preset reading mark in a preset area of ​​the visual interface.

26. The device according to any one of claims 23-24, wherein: The broadcast module comprises: An image selection unit, used to determine the digital human image selected by the user according to the user's selection operation on the digital human image; The display unit is used to display the action corresponding to the action information and the expression corresponding to the expression information on a visual interface through the digital human image selected by the user, and play the voice information.

27. The device according to any one of claims 17-21, 23-24, further comprising: The text scrolling module is used to play the response text information in a scrolling manner within a preset area of ​​the visual interface according to a preset text scrolling speed.

28. The device according to claim 17, wherein: The target object is a drug, and the explanatory text information is a drug instruction sheet; The information determining unit comprises: The identification determination subunit is used to determine the drug identification of the drug in response to the question information issued by the user regarding the drug; wherein the drug identification is used to characterize the drug; The instruction obtaining subunit is used to obtain the drug instruction corresponding to the drug identification; The instruction sheet conversion subunit is used to determine the preset structured data matching the drug according to the drug instruction sheet corresponding to the drug identifier; The information search subunit is used to determine the response text information corresponding to the question information based on the preset structured data matching the drug.

29. The device according to claim 28, wherein The identification determination subunit is specifically used for: Obtaining the medicine box cover image of the medicine uploaded by the user; According to a preset image recognition algorithm, the name of the medicine is recognized from the medicine box cover image, and the name of the medicine is determined as the medicine identification of the medicine.

30. The device according to claim 28 or 29, wherein: The instruction manual acquisition subunit is specifically used for: According to the preset association relationship between the drug name and the drug instructions, the drug instructions associated with the drug name of the drug are determined.

31. The device according to any one of claims 28-29, wherein: The instruction manual conversion subunit is specifically used for: According to a preset large language model, the drug instruction manual is split to obtain at least one text block; wherein the text block is used to represent a local text content in the drug instruction manual; Acquire the medical terms in the text block, and convert the medical terms into a preset common language; wherein the medical terms have the same semantics as the corresponding common language; Determine the text content corresponding to the converted text block as the reply answer to the question, perform semantic analysis on the text content corresponding to the text block, and obtain the question corresponding to the text block; Determine a key-value pair structure corresponding to the text block according to the text content corresponding to the converted text block and the question corresponding to the text block; wherein the keyword in the key-value pair structure is the question corresponding to the text block, and the key value in the key-value pair structure is the text content corresponding to the converted text block; The key-value pair structures corresponding to all text blocks of the drug instructions are determined as preset structured data matching the drug.

32. The device according to any one of claims 28-29, wherein: The question information represents the question raised by the user regarding the drug instructions; the information search subunit is specifically used for: Questions raised by users regarding drug instructions are determined as keywords, and if a key value corresponding to the question exists in the preset structured data, the key value corresponding to the question is determined as response text information corresponding to the question.

33. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-16.

35. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 16.

Images