Information processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an information processing method and device, equipment and a storage medium. The method comprises the following steps: presenting an interactive interface with a virtual object, wherein the interactive interface comprises an image input control; receiving a first input image via the image input control; in response to the fact that the first input image is determined to be matched with the first interaction scene, first response content for the first input image is provided in the interaction interface, and the first response content is generated by a first processing module corresponding to the first interaction scene. Therefore, according to the embodiment of the invention, the interaction scene matched with the input image can be automatically identified, the adaptability between the generated response content and the input image and the interaction scene of the input image is ensured, and the user experience is improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to information processing methods, apparatuses, devices, and computer-readable storage media. Background Art

[0002] With the development of computer technology, the Internet has become an important platform for people to exchange information, and various forms of electronic devices can greatly enrich people's daily lives. For example, some electronic devices can provide users with virtual scenes, such as users interacting with virtual characters.

[0003] In such a virtual scene, it is difficult to automatically and accurately determine what kind of scene the user wants to interact with based on the information input by the user, which affects the user's interactive experience. Summary of the invention

[0004] In a first aspect of the present disclosure, an information processing method is provided. The method includes: presenting an interactive interface with a virtual object, the interactive interface including an image input control; receiving a first input image via the image input control; and in response to the first input image being determined to match a first interactive scene, providing first response content for the first input image in the interactive interface, the first response content being generated by a first processing module corresponding to the first interactive scene.

[0005] In a second aspect of the present disclosure, an information processing device is provided. The device includes: an interface presentation module, an image receiving module, and a first content providing module, wherein the presentation module is configured to present an interactive interface with a virtual object, the interactive interface including an image input control; the receiving module is configured to receive a first input image via the image input control; the first content providing module is configured to provide a first response content for the first input image in the interactive interface in response to the first input image being determined to match a first interactive scene, the first response content being generated by a first processing module corresponding to the first interactive scene.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer executable instructions, which, when executed by a device, cause the device to perform the method provided according to the first aspect.

[0009] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram showing an example environment in which embodiments according to the present disclosure may be implemented;

[0012] FIG. 2A to FIG. 2C shows an example interface according to some embodiments of the present disclosure;

[0013] FIG. 3A to FIG. 3C An example scenario according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A flowchart illustrating an example process of information processing according to some embodiments of the present disclosure;

[0015] Figure 5 A schematic structural block diagram showing an example apparatus for information processing according to some embodiments of the present disclosure; and

[0016] Figure 6 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0020] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0021] In this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or it is necessary to perform a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0022] Information processing is an important part of the online information interaction process. With the development of artificial intelligence technology, the interaction scenarios between people and virtual objects are also expanding. For example, in the field of artificial intelligence, people have more and more demands for pan-medical interactions, such as medical consultation, health encyclopedia, report interpretation, symptom identification, medication consultation and other scenarios.

[0023] In the related technology, the generative model supports general cameras uploading image data, but it cannot automatically identify the user's interaction scene requirements based on the received images, that is, it cannot automatically and accurately identify and call the matching scene sub-model to adapt to user needs; and the interaction scenarios it can adapt to are relatively single, which makes it difficult to meet the diverse needs of users, and the user experience is poor.

[0024] The embodiment of the present disclosure proposes an information processing scheme. According to the scheme, an interactive interface with a virtual object can be presented, the interactive interface including an image input control; a first input image is received via the image input control; and in response to the first input image being determined to match a first interactive scene, a first response content for the first input image is provided in the interactive interface, the first response content being generated by a first processing module corresponding to the first interactive scene.

[0025] According to the solution provided in the embodiments of the present disclosure, it is possible to automatically determine the interactive scene that matches the image input by the user, thereby mobilizing the corresponding processing module to process the input image to accurately generate and provide corresponding response content, effectively ensuring the accuracy of the response content provided and improving the user experience.

[0026] In some embodiments of the present disclosure, a group of interactive entrances corresponding to a group of preset interactive scenes can also be provided in the interactive interface of the scheme. The group of preset interactive scenes includes a first interactive scene. The diversity of interactive scenes is improved through a group of preset interactive scenes to meet the diversity of user interaction needs, and the scalability of the interactive scenes is improved to improve the user experience.

[0027] Various example implementations of the solution are described in detail below in conjunction with the accompanying drawings.

[0028] Example Environment

[0029] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0030] In this example environment 100, the electronic device 110 may run an application 120 that supports interface interaction. The application 120 may be any appropriate type of application for interface interaction, examples of which may include, but are not limited to, chat applications, virtual interaction applications, or other appropriate applications. The user 140 may interact with the application 120 via the electronic device 110 and / or its attached devices.

[0031] exist Figure 1In the environment 100, if the application 120 is active, the electronic device 110 can present an interface 150 for supporting interaction with the virtual object through the application 120.

[0032] In some embodiments, the electronic device 110 communicates with the server 130 to provide services for the application 120. The electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a handheld computer, a portable game terminal, a VR / AR device, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio receiver, an e-book device, a game device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.).

[0033] The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 130 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, etc. The server 130 may provide background services for the application 120 in the electronic device 110 that supports interaction with virtual objects.

[0034] A communication connection may be established between the server 130 and the electronic device 110. The communication connection may be established in a wired manner or a wireless manner. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In the embodiments of the present disclosure, the server 130 and the electronic device 110 may implement signaling interaction through the communication connection between the two.

[0035] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0036] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0037] Example Interaction

[0038] FIG. 2A to FIG. 2C 200A to 200C according to some embodiments of the present disclosure. The interfaces 200A to 200C may be composed of, for example, Figure 1 The electronic device 110 is provided as shown.

[0039] In some embodiments, Figure 2A As shown, the electronic device 110 can present an interactive interface 200A with a virtual object. The virtual object can refer to Figure 2A The electronic device 110 may provide an image input control 211 in the interactive interface 200A. The user may input an image through the image input control 211 to interact with the virtual object.

[0040] Exemplarily, the electronic device 110 may provide an information input component 210 in the interactive interface 200A, and the information input component 210 includes an image input control 211 .

[0041] The user can trigger the image input control 211 in the interactive interface 200A to input an image and implement information interaction with the virtual object.

[0042] For example, the image input by the user can be a picture, such as a photo of any content. Taking the health consultation scenario as an example, the image input by the user can be a symptom picture of abnormal body surface, a text picture of an examination report, or an image of the appearance of a drug or drug packaging, etc., without limitation.

[0043] In some implementations, the image input by the user may also be an audio or video image, such as an audio or video clip with voice description taken for symptoms, or an audio or video clip with voice information taken for medicines, etc., which is not limited here.

[0044] According to an embodiment of the present disclosure, a user can input an image stored locally in the electronic device 110 via the image input control 211, or can take a new image via the image input control 211, or can call an image generation application (such as a camera) in the electronic device 110 via the image input control 211 to generate a new image, or can call an installed storage application (such as a cloud disk) in the electronic device 110 via the image input control 211 to search for and obtain an image, without limitation herein.

[0045] Reference Figure 2AAs shown, in the embodiment of the present disclosure, the information input component 210 may also include at least one of a voice input control 212, a text input control 213, and other input controls. Among them, the other input controls may be Figure 2A An input control triggered by the "+" as shown in the figure, for example, an emoticon input control.

[0046] In some embodiments, the user can input voice information in the interactive interface 200A through the voice input control 212 in the information input component 210. The voice information input by the user can be independent interactive information such as questions and replies, or can be associated description information of the input image input before or after the input image, such as question content about any information shown in the input image.

[0047] In some embodiments, the user can input text information in the interactive interface 200A through the text input control 213 in the information input component 210. The text information input by the user can be independent interactive information such as questions and replies, or can be associated description information of the input image input after or before the input of the image, such as the question content about any information shown in the input image.

[0048] In some embodiments, the user can also input other information by clicking "+" to trigger other input controls. For example, by clicking "+" to trigger the expression input control, and inputting the corresponding emoticon or expression image as a reply or response to the information provided by the virtual object.

[0049] Reference Figure 2B As shown, the electronic device 110 can present an interactive interface 200B with a virtual object. The interactive interface 200B includes an image input control, and may also include a set of interactive entrances 230 corresponding to a set of preset interactive scenes. Exemplarily, the preset interactive scenes may include health encyclopedias, medical consultations, symptom identification, report interpretation, medication consultation and other health consultation scenes, and may also include AI raw images, document generation and other interactive scenes, which are not limited here.

[0050] In some embodiments, a user can input an image 201 through an image input control in the interactive interface 200B. After receiving the image 201 input by the user, the electronic device 110 can identify the content of the image 201, and then determine a target interactive scene matching the image based on the recognition result, and call a processing module corresponding to the target interactive scene to process the image 201 input by the user, thereby obtaining response content 220 for the image and displaying it in the interactive interface 200B.

[0051] Exemplarily, after receiving the image 201 input by the user, the electronic device 110 provides the image 201 to the visual language model to determine the target interaction scene from multiple preset interaction scenes, thereby determining the target processing module to process the image 201 to ensure the adaptability of the response content 220 to the image 201 and its target interaction scene.

[0052] In some embodiments, the user can also select a target interactive entrance from a group of interactive entrances 230 and trigger it, for example, select the interactive entrance corresponding to "Report Interpretation" and click it, and then input an interactive image or other interactive information through the target interactive entrance. After the electronic device 110 receives the interactive image or other interactive information, it presents the corresponding response content on the interactive interface 200B.

[0053] Exemplarily, the user inputs image 201 by triggering the target interactive entrance, and the electronic device 110 obtains the image 201 in response to the target interactive entrance being triggered, and then, based on the target interactive scene corresponding to the target interactive entrance, mobilizes the processing module corresponding to the target interactive scene to process the image 201, generates response content 220 for the image 201, and displays it in the interactive interface 200B.

[0054] In some embodiments, the process of generating response content 220 includes: after the electronic device 110 receives the image 201, it determines a target response mode for the image 201 based on context information associated with the interactive interface, and the target response mode indicates whether to provide follow-up content; and obtains the response content 220 generated based on the target response mode.

[0055] Exemplarily, after determining the target response mode for image 201 , the electronic device calls a processing module corresponding to the target response mode to process image 201 , and then obtains response content 220 for image 201 from the processing module.

[0056] In some embodiments, electronic device 110 determines that image 201 corresponds to a first response mode in response to context information associated with the interactive interface indicating that image 201 corresponds to a first round of interaction with a virtual object, and the first response mode indicates that no follow-up content is provided.

[0057] In some embodiments, the electronic device 110 determines that the image 201 corresponds to the first response mode, that is, no follow-up content is provided, in response to the context information associated with the interactive interface indicating that the image 201 is independent of the health consultation process, for example, the content of the image 201 is completely different from the context information or belongs to different content. For example, the context is about the examination report on the knee joint, and the image 201 is a hand symptom map.

[0058] In some embodiments, the electronic device 110 determines that the image 201 corresponds to the first response mode, i.e., no follow-up content is provided, in response to context information associated with the interactive interface indicating that the correlation between the historical messages of the interactive interface 200B and the image 201 is lower than a threshold.

[0059] The relevance between the context information and the image 201 may be evaluated by a relevance level or a relevance score.

[0060] Exemplarily, the relevance level between the context information and the image 201 can be divided according to a preset level division standard, including the following relevance levels in sequence: completely different, slightly related, moderately related, closely related, and almost the same. Accordingly, the relevance threshold between the two items can be any one of moderately related, closely related, or almost the same in the above relevance levels.

[0061] Exemplarily, a deep learning model or a large language model can be used to identify and analyze the context information and image 201 associated with the interactive interface, and calculate the correlation score between the two. For example, any one of a ten-point system, a percentage system, or a percentage can be selected to determine the correlation score. Accordingly, the correlation threshold between the two is a specific score value, such as 7.5 points on a ten-point system, 80 points or 80% on a percentage system, etc.

[0062] In some embodiments, the electronic device 110 determines that the first input image corresponds to the second response mode in response to the context information associated with the interactive interface indicating that the image 201 is associated with the health consultation process, or the correlation between the historical message of the interactive interface 200B and the image 201 reaches a threshold, and the second response mode indicates providing follow-up content. The follow-up content may include follow-up content for the content shown in the image 201. For example, for the symptom image, a follow-up question such as "How long has this condition lasted?" may be asked.

[0063] Reference Figure 2C As shown, the electronic device 110 presents an interactive interface 200C with a virtual object. The interactive interface 200C includes an image input control, and may also include a set of interactive entrances 230 corresponding to a set of preset interactive scenes. Exemplarily, the preset interactive scenes may include health encyclopedias, medical consultations, symptom identification, report interpretation, medication consultation and other health consultation scenes, and may also include AI raw images, document generation and other interactive scenes, which are not limited here.

[0064] In some embodiments, the user can input the image 201 through the image input control in the interactive interface 200C, or input the image 201 by triggering the target interactive entry in a group of interactive entries 230. After receiving the image 201, the electronic device 110 responds to the context information associated with the interactive interface 200C indicating that the image 201 is associated with the health consultation process, or responds to the correlation between the historical message of the interactive interface 200C and the image 201 reaching a threshold, and determines that the image 201 corresponds to the second response mode, that is, providing the follow-up content. The electronic device 110 calls the corresponding follow-up module to generate the follow-up content 221 and presents it in the interactive interface 200C; then receives the reply content 202 given by the user to the follow-up content, and then calls the corresponding processing module based on the reply content 202, the image 201 and the historical message to generate the response content 220, and displays it in the interactive interface 200C.

[0065] In some embodiments, the electronic device 110 may generate the follow-up question content 221 only once, or may generate the follow-up question content again for the reply content 202 until a preset number of times or a preset follow-up question condition is met, then stop the follow-up question and generate the corresponding response content 220.

[0066] In this way, the embodiments of the present disclosure can accurately identify the target interaction scene desired by the user based on the user input image, and generate and provide response content accordingly, ensuring the adaptability of the response content to the user input image and its target interaction scene, thereby improving the user experience.

[0067] In addition, the embodiments of the present disclosure also provide a set of interactive entrances corresponding to a set of preset interactive scenes, so that users can manually select the target interactive entrance to trigger the corresponding interactive scene, thereby providing users with more interactive options and improving user experience; and in this mode, the interactive scenes and their corresponding interactive entrances can be expanded at any time, thereby improving the scalability of the interactive scenes.

[0068] Example scenario

[0069] FIG. 3A to FIG. 3C Example scenes 300A to 300C according to some embodiments of the present disclosure are shown. Scenes 300A to 300C are schematic diagrams of the process of the electronic device 110 generating and presenting response content according to the image input by the user in different scenes.

[0070] In some embodiments, Figure 3A As shown, the solution provided by the present disclosure can be applied to the symptom interpretation scenario 300A in the health consultation process.

[0071] Reference Figure 3AAs shown, after receiving the symptom image input by the user through the health interception process 310, the electronic device 110 determines that the symptom image matches the symptom interpretation scenario, and calls the symptom interpretation module 301 to process and respond to the symptom image.

[0072] Exemplarily, the symptom interpretation module 301 processes the symptom image as follows:

[0073] In block 321 , the symptom interpretation module 301 determines whether the symptom image corresponds to the virtual object for the first round of interaction, and if so, executes block 322 ;

[0074] In block 322, the symptom interpretation module 301 calls the visual language model symptom interpretation robot program (bot) A to process the symptom image and execute block 323; wherein the visual language model symptom interpretation botA only processes according to the image content and does not question the image content, for example, it recognizes the symptoms in the image and converts them into text descriptions, and can also analyze and suggest the recognized symptoms;

[0075] In block 323 , the symptom interpretation module 301 obtains response content for the symptom image according to the processing result of block 322 , including outputting at least one information such as symptom description, cause, and given suggestions, and then executes block 330 ;

[0076] In block 330 , the symptom interpretation module 301 streams the response content to the electronic device 110 ; finally, the electronic device 110 performs an output process 340 to present the response content to the interactive interface;

[0077] Returning to block 321 , if the symptom interpretation module 301 determines that the symptom image corresponds to a virtual object and is not the first round of interaction, block 324 is executed;

[0078] In block 324, the symptom interpretation module 301 determines whether the consultation process is currently in progress. If not, block 322 is executed; if yes, blocks 325 and 327 are executed. The method of determining whether the consultation process is currently in progress may include: determining whether the context information associated with the interactive interface is relevant to the symptom image. If so, the consultation process is in progress; if not, the consultation process is not in progress.

[0079] In block 325, the symptom interpretation module 301 calls the visual language model to determine the correlation score between the current graphic message and the historical message, and executes block 326 according to the correlation score; wherein the current graphic message includes the current symptom image and context information, and the historical message may be the historical message of the current interactive interface;

[0080] Exemplarily, determining the correlation score between the current graphic message and the historical message can be achieved through image recognition, text conversion, semantic analysis, similarity comparison and other processes, which can be referred to in the relevant public technology and will not be described in detail here;

[0081] In block 326, the symptom interpretation module 301 compares the correlation score determined in block 325 with a preset threshold to determine whether the correlation score is greater than the threshold; if not, return to block 322; if so, execute block 328;

[0082] In block 328, the symptom interpretation module 301 calls the visual language model symptom recognition botB to recognize the symptoms in the symptom image, obtain the symptom description, and generate corresponding follow-up questions, and then executes block 329; wherein the follow-up questions are the follow-up questions generated for the symptom image in combination with historical messages, context information, symptom description and other information;

[0083] In block 329 , the symptom interpretation module 301 organizes and outputs the symptom description and follow-up question content obtained in block 328 , obtains response content, and executes block 330 ;

[0084] Returning to box 324 , in response to determining that the consultation process is currently in progress, the symptom interpretation module 301 may also call the visual language model to rewrite the consultation prompt information of the symptom image in combination with the context information, and execute box 328 .

[0085] In some embodiments, Figure 3B As shown, the solution provided by the present disclosure can be applied to the report interpretation scenario 300B in the health consultation process.

[0086] Reference Figure 3B As shown, after receiving the report image input by the user through the health interception process 310, the electronic device 110 determines that the report image matches the report interpretation scenario, and calls the report interpretation module 302 to process and respond to the report image.

[0087] Exemplarily, the report interpretation module 302 processes the report image as follows:

[0088] In block 351 , the report interpretation module 302 first calls an OCR (Optical Character Recognition) model to extract text information from the report image, and then executes block 352 ;

[0089] At block 352 , the report interpretation module 302 determines whether the report image corresponds to the first round of interaction with the virtual object, if not, executing block 353 ; if yes, executing block 356 ;

[0090] In block 353 , the report interpretation module 302 calls the main bot to find context information associated with the interactive interface and executes block 354 ;

[0091] In block 354, the report interpretation module 302 determines whether the consultation process is currently in progress. If so, block 355 is executed; if not, block 356 is executed. The method of determining whether the consultation process is currently in progress may include: determining whether the context information associated with the interactive interface is relevant to the report image. If so, the consultation process is in progress; if not, the consultation process is not in progress.

[0092] In block 355 , the report interpretation module 302 calls the consulting bot to combine the text information and the context information, generate response content for the report image, and execute block 330 ;

[0093] In block 330 , the report interpretation module 302 streams the response content to the electronic device 110 ; finally, the electronic device 110 executes an output process 340 to present the response content to the interactive interface;

[0094] Returning to block 352 , if the report interpretation module 302 determines that the report image corresponds to the first round of interaction with the virtual object, block 356 is executed;

[0095] In box 356 , the report interpretation module 302 calls the main bot to process the text information, obtain the response content for the report image, and execute box 330 .

[0096] In some embodiments, Figure 3A As shown, the solution provided by the present disclosure can be applied to the drug identification scenario 300C in the health consultation process.

[0097] Reference Figure 3C As shown, after receiving the drug image input by the user through the health interception process 310, the electronic device 110 determines that the drug image matches the drug recognition scene, and calls the drug recognition module 303 to process and respond to the drug image.

[0098] Exemplarily, the process of drug recognition module 303 processing drug images is as follows:

[0099] In block 361 , the drug recognition module 303 determines that the drug image is suitable for the drug recognition module, and executes blocks 362 and 367 ;

[0100] In block 362, the OCR model is first called to identify the drug object in the drug image, and then block 363 is executed;

[0101] In block 363 , the drug identification module 303 retrieves the associated information of the drug object in the drug knowledge base according to the drug object identified in block 362 , and executes block 364 ;

[0102] In block 364, the drug identification module 303 outputs the drug instructions corresponding to the drug object and executes block 365 or block 371;

[0103] In block 365 , the drug identification module 303 calls a model (which may be a generative model, for example) to analyze and summarize the drug object and the drug instructions to obtain comprehensive information, and then executes block 366 ;

[0104] In block 366 , the drug identification module 303 generates a response content for the drug image based on the comprehensive information obtained in block 365 , and can give a correct reply to the user, and executes block 330 ;

[0105] In block 330 , the drug identification module 303 streams the response content to the electronic device 110 ; finally, the electronic device 110 performs an output process 340 to present the response content to the interactive interface;

[0106] In block 367 , the drug identification module 303 determines whether the drug image corresponds to the virtual object for the first round of interaction, and if so, executes block 365 ; if not, executes block 368 ;

[0107] In block 368, whether the drug identification module 303 is currently in the consultation process, if not, execute block 365; if yes, execute block 369; wherein, the manner of determining whether the drug identification module is currently in the consultation process may include: determining whether the context information associated with the interactive interface is relevant to the drug image, if relevant, then the drug identification module is in the consultation process, and if not relevant, then the drug identification module is not in the consultation process;

[0108] In block 369 , the drug recognition module 303 calls a model (such as a visual language model or other generative model) to determine a correlation score between the current drug information and the context information, and executes block 370 based on the correlation score;

[0109] Exemplarily, determining the correlation score between the current drug information and the context information can be achieved through image recognition, text conversion, semantic analysis, similarity comparison and other processes, which can be referred to in the relevant public technology and will not be described in detail here;

[0110] In block 370, the drug identification module 303 compares the correlation score determined in block 369 with a preset threshold to determine whether the correlation score is greater than the threshold; if not, return to block 365; if so, execute block 371;

[0111] In block 371 , the drug identification module 303 calls a model (e.g., a generative model) to process the drug object and the drug instructions in combination with the context information to obtain the query content, and then executes block 372 ;

[0112] In box 372 , the drug identification module 303 may give a reply and a follow-up question to the user based on the inquiry content obtained in box 371 , combined with information such as the drug object and its instructions, generate response content for the drug image, and execute box 330 .

[0113] Example Process

[0114] Figure 4 4 is a flowchart showing an example process 400 of information processing according to some embodiments of the present disclosure. The process 400 may be implemented at the electronic device 110. Figure 1 4. The process 400 is described below.

[0115] like Figure 4 As shown, in box 410, the electronic device 110 presents an interaction interface with the virtual object, and the interaction interface includes an image input control.

[0116] At block 420 , the electronic device 110 receives a first input image via the image input control.

[0117] In box 430, in response to the first input image being determined to match the first interactive scene, the electronic device 110 provides first response content for the first input image in the interactive interface, where the first response content is generated by a first processing module corresponding to the first interactive scene.

[0118] According to the information processing method of the embodiment of the present disclosure, the electronic device can automatically determine the first interactive scene that matches the first input image, and mobilize the corresponding first processing module to generate the first response content, and then provide the first response content in the interactive interface. By accurately matching the first interactive scene, the adaptation accuracy of the interactive scene is improved, and then the first processing module corresponding to the first interactive scene is mobilized to generate the first response content, the accuracy of information processing is improved, thereby ensuring the adaptability of the obtained first response content to the interactive scene of the first input image, and improving the user experience.

[0119] In some embodiments, the interactive interface presented by the electronic device 110 also includes a group of interactive entrances corresponding to a group of preset interactive scenes, and the group of preset interactive scenes includes the above-mentioned first interactive scene. Correspondingly, the information processing method disclosed in the present invention also includes: based on the triggering of the target interactive entrance in the group of interactive entrances, obtaining the second input image; and providing the second response content for the second input image in the interactive interface, the second response content is generated by the second processing module corresponding to the second interactive scene, and the second interactive scene corresponds to the target interactive entrance.

[0120] In some embodiments, the information processing method of the present disclosure further includes: providing a first input image to a visual language model to determine a first interaction scene that matches the first input image from a plurality of preset interaction scenes.

[0121] In some embodiments, the first response content is generated based on the following process: based on context information associated with the interactive interface, determining a target response mode for the first input image, the target response mode indicating whether to provide follow-up content; and obtaining the first response content generated based on the target response mode.

[0122] In some embodiments, determining a target response mode for a first input image based on context information associated with an interactive interface includes any of the following: in response to the context information indicating that the first input image corresponds to a first round of interaction with a virtual object, determining that the first input image corresponds to a first response mode, the first response mode indicating that no follow-up content is provided; in response to the context information indicating that the interactive interface is independent of the health consultation process, determining that the first input image corresponds to the first response mode; in response to the context information indicating that the correlation between historical messages of the interactive interface and the first input image is lower than a threshold, determining that the first input image corresponds to the first response mode.

[0123] In some embodiments, a target response mode for a first input image is determined based on contextual information associated with an interactive interface, including: in response to a context indication that the first input image is associated with a health consultation process or that the correlation between historical messages of the interactive interface and the first input image reaches a threshold, determining that the first input image corresponds to a second response mode, and the second response mode indicates providing follow-up content.

[0124] In some embodiments, different target response modes correspond to different processing entities for generating the first response content.

[0125] In some embodiments, the first interactive scene indicates identification of symptoms associated with the first input image, and the first response content includes at least one of the following: a symptom description associated with the symptoms in the first input image; cause information associated with the symptoms in the first input image; and suggestion information associated with the symptoms in the first input image.

[0126] In some embodiments, the first interaction scenario indicates obtaining an explanation corresponding to the report in the first input image, and the first response content includes an explanation text about the report in the first input image.

[0127] In some embodiments, the first interactive scene causes the identification of a drug object in the first input image, and the first response content includes a description text of the drug object.

[0128] In some embodiments, the first response content is determined based on a drug insert associated with the drug object.

[0129] Example devices and equipment

[0130] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 5 1 shows a schematic structural block diagram of an example apparatus 500 for xxx according to some embodiments of the present disclosure. The apparatus 500 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware or any combination thereof.

[0131] like Figure 5 As shown, the device 500 includes: an interface presentation module 510, an image receiving module 520 and a first content providing module 530. The presentation module 510 is configured to present an interactive interface with a virtual object, the interactive interface including an image input control; the image receiving module 520 is configured to receive a first input image via the image input control; and the first content providing module 530 is configured to provide a first response content for the first input image in the interactive interface in response to the first input image being determined to match the first interactive scene, the first response content being generated by a first processing module corresponding to the first interactive scene.

[0132] In some embodiments, the interactive interface also includes a group of interactive entrances corresponding to a group of preset interactive scenes, and the group of preset interactive scenes includes a first interactive scene; and the information processing device provided by the present disclosure may also include an image acquisition module and a second content providing module, wherein the image acquisition module is configured to acquire a second input image based on triggering a target interactive entrance in a group of interactive entrances; the second content providing module is configured to provide second response content for the second input image in the interactive interface, and the second response content is generated by a second processing module corresponding to the second interactive scene, and the second interactive scene corresponds to the target interactive entrance.

[0133] In some embodiments, the information processing device provided by the present disclosure also includes a scene determination module, which is configured to provide a first input image to the visual language model to determine a first interaction scene that matches the first input image from a plurality of preset interaction scenes.

[0134] In some embodiments, the first response content is generated based on the following process: based on context information associated with the interactive interface, determining a target response mode for the first input image, whether the target response mode knowledge provides follow-up content; and obtaining the first response content generated based on the target response mode.

[0135] In some embodiments, based on context information associated with the interaction interface, a target response mode for the first input image is determined, including any of the following: in response to the context information indicating that the first input image corresponds to a first round of interaction with a virtual object, determining that the first input image corresponds to a first response mode, the first response mode indicating that no follow-up content is provided; in response to the context information indicating that the interaction interface is independent of the health consultation process, determining that the first input image corresponds to the first response mode; in response to the context information indicating that the correlation between the historical messages of the interaction interface and the first input image is lower than a threshold, determining that the first input image corresponds to the first response mode.

[0136] In some embodiments, based on contextual information associated with the interactive interface, a target response mode for the first input image is determined, including: in response to the contextual information indicating that the first input image is associated with a health consultation process or the correlation of the historical message and the first input image reaches a threshold, determining that the first input image corresponds to a second response mode, and the second response mode indicates providing follow-up content.

[0137] In some embodiments, different target response modes correspond to different processing entities for generating the first response content.

[0138] In some embodiments, the first interactive scene indicates identification of a symptom associated with the first input image, and the first response content includes at least one of the following: a symptom description associated with the symptom in the first input image; cause information associated with the symptom; and suggestion information associated with the symptom.

[0139] In some embodiments, the first interaction scenario indicates obtaining an explanation corresponding to a report in the first input image, and the first response content includes an explanation text about the report.

[0140] In some embodiments, the first interactive scene indicates identifying a drug object in the first input image, and the first response content includes a description text of the drug object.

[0141] In some embodiments, the first response content is determined based on a drug instruction sheet associated with the drug object in the first input image.

[0142] Figure 6 6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The electronic device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to implement Figure 1 terminal device 110.

[0143] like Figure 6 As shown, the electronic device 600 is in the form of a general electronic device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0144] The electronic device 600 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 600.

[0145] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0146] The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0147] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0148] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0149] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0151] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0152] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0153] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. An information processing method, comprising: Presenting an interactive interface with the virtual object, the interactive interface including an image input control; Receiving a first input image via the image input control; as well as In response to the first input image being determined to match a first interactive scene, first response content for the first input image is provided in the interactive interface, where the first response content is generated by a first processing module corresponding to the first interactive scene.

2. The method according to claim 1, wherein: The interactive interface further includes a group of interactive entrances corresponding to a group of preset interactive scenes, the group of preset interactive scenes includes the first interactive scene, and the method further includes: Based on triggering a target interactive portal in the group of interactive portals, acquiring a second input image; and A second response content for the second input image is provided in the interactive interface, wherein the second response content is generated by a second processing module corresponding to a second interactive scene, and the second interactive scene corresponds to the target interactive entry.

3. The method according to claim 1, further comprising: The first input image is provided to a visual language model to determine the first interaction scene matching the first input image from a plurality of preset interaction scenes.

4. The method according to claim 1, wherein: The first response content is generated based on the following process: Determining a target response mode for the first input image based on context information associated with the interactive interface, the target response mode indicating whether to provide follow-up content; as well as The first response content generated based on the target response mode is obtained.

5. The method according to claim 4, wherein: The determining, based on the context information associated with the interactive interface, a target response mode for the first input image includes any one of the following: In response to the context information indicating that the first input image corresponds to a first round of interaction with the virtual object, determining that the first input image corresponds to a first response mode, the first response mode indicating that no follow-up content is provided; In response to the context information indicating that the interactive interface is independent of a health consultation process, determining that the first input image corresponds to the first response mode; In response to the context information indicating that the correlation between the historical messages of the interactive interface and the first input image is lower than a threshold, it is determined that the first input image corresponds to the first response mode.

6. The method according to claim 5, wherein: The determining, based on context information associated with the interactive interface, a target response mode for the first input image includes: In response to the context information indicating that the first input image is associated with the health consultation process or the correlation between the historical message and the first input image reaches the threshold, it is determined that the first input image corresponds to a second response mode, and the second response mode indicates providing follow-up content.

7. The method according to claim 4, wherein: Different target response modes correspond to different processing entities for generating the first response content.

8. The method according to claim 1, wherein: The first interaction scenario indicates identification of a symptom associated with the first input image, and the first response content includes at least one of the following: a symptom description associated with the symptom in the first input image; causal information associated with the described symptoms; Advisory information associated with the described symptom.

9. The method according to claim 1, wherein: The first interactive scenario indicates obtaining an explanation corresponding to the report in the first input image, and the first response content includes an explanation text about the report.

10. The method according to claim 1, wherein: The first interactive scenario indicates identifying a drug object in the first input image, and the first response content includes a description text of the drug object.

11. The method according to claim 10, wherein: The first response content is determined based on a drug instruction sheet associated with the drug object.

12. An information processing device, comprising: An interface presentation module, configured to present an interaction interface with a virtual object, wherein the interaction interface includes an image input control; An image receiving module, configured to receive a first input image via the image input control; as well as The first content providing module is configured to provide first response content for the first input image in the interaction interface in response to the first input image being determined to match the first interaction scene, wherein the first response content is generated by a first processing module corresponding to the first interaction scene.

13. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Images