Information processing method and device, electronic equipment and readable storage medium

Abstract

The invention relates to an information processing method and device, electronic equipment and a readable storage medium, and the method comprises the steps: responding to inquiry information input by a user, and transmitting the inquiry information to a server; receiving response information returned by the server for the inquiry information; wherein information elements contained in the response information are determined based on characters in the response information, and under the condition that the characters in the response information have corresponding entities, the response information contains picture elements and character elements; and outputting the response information corresponding to the inquiry information.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology, and more specifically, to an information processing method, device, electronic device and readable storage medium. Background Art

[0002] With the development of Internet technology, search technology based on large language models has become more and more mature. On the one hand, the search results of large language models are mostly text, and on the other hand, they cannot output search results in the corresponding form according to user needs. Summary of the invention

[0003] One object of the present disclosure is to provide a new technical solution for information processing.

[0004] According to a first aspect of the present disclosure, there is provided an information processing method, comprising: In response to a user inputting query information, sending the query information to a server; Receive response information returned by the server in response to the query information; wherein the information elements included in the response information are determined based on the text in the response information, and when there is a corresponding entity in the text in the response information, the response information includes a picture element and a text element; The response information corresponding to the inquiry information is output.

[0005] Optionally, when there is no corresponding entity for the text in the response information, the response information includes text elements but does not include image elements.

[0006] Optionally, the information elements included in the response information are determined based on text in the response information, including: The type of information element contained in the response information is determined by inputting the text in the response information into a set recognition model; wherein, when the set recognition model determines that there is a corresponding entity in the text in the response information, the response information includes image elements and text elements.

[0007] Optionally, when the set recognition model determines that there is no corresponding entity in the text in the response information, the response information contains text elements but does not contain image elements.

[0008] Optionally, the text in the response information is determined by inputting the search results obtained based on the query information into a set text processing model.

[0009] Optionally, the response information includes a picture element and a text element, and the picture in the response information is obtained from an information source corresponding to the text in the response information.

[0010] Optionally, when the matching degree between the picture in the information source and the text in the response information is greater than or equal to a set threshold, the picture in the response information is obtained from the information source; When the matching degree is less than the set threshold, the picture in the response information is searched based on the text in the response information.

[0011] Optionally, the response information includes a picture element and a text element, the information source corresponding to the text in the response information does not have a picture, and the picture in the response information is searched based on the text in the response information.

[0012] Optionally, the response information includes multiple information items, and the output order of the multiple information items is determined based on the mention rate of each information item, wherein the mention rate of the information item is determined based on the number of times the information item appears in the search results obtained by matching the query information.

[0013] Optionally, the outputting the response information corresponding to the inquiry information includes: Displaying a first set of response information and interactive controls corresponding to the inquiry information; In response to the interactive control being triggered, a second set of response information corresponding to the inquiry information is displayed successively after the first set of response information.

[0014] Optionally, the outputting the response information corresponding to the inquiry information includes: The content of the response information is output in time division until a termination identifier is detected, and it is determined that the output of the response information is completed.

[0015] According to a second aspect of the present disclosure, there is provided an information processing method, comprising: Receive the query information input by the user sent by the client; Performing information search according to the query information, and determining response information corresponding to the query information according to the search results; wherein the information elements included in the response information are determined based on the text in the response information, and when the text in the response information has a corresponding entity, the response information includes a picture element and a text element; The response information is returned to the client for display.

[0016] According to a third aspect of the present disclosure, there is provided an information processing device, including: A sending module, configured to send the query information to a server in response to the query information input by the user; A receiving module, configured to receive a response message returned by the server in response to the query message; wherein the information elements included in the response message are determined based on the text in the response message, and when the text in the response message has a corresponding entity, the response message includes a picture element and a text element; An output module is used to output the response information corresponding to the inquiry information.

[0017] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate to execute the method according to the first aspect or the second aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0019] According to a sixth aspect of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0020] The information processing method provided by the present invention determines the information elements contained in the response information based on the text in the response information. When there is a corresponding entity in the text in the response information, the response information includes image elements and text elements, and the response information including the image elements and text elements is output, thereby realizing diversified output of the response information and improving the user experience.

[0021] Features and advantages of the embodiments of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the specification and, together with the description, serve to explain the principles of the embodiments of the specification.

[0023] Figure 1 A schematic diagram showing a flow chart of an information processing method according to some embodiments of the present disclosure; Figure 2 A schematic diagram showing response information display according to some embodiments of the present disclosure is shown; Figure 3 A schematic diagram showing response information display according to some embodiments of the present disclosure is shown; Figure 4 A schematic diagram showing response information display according to some embodiments of the present disclosure is shown; Figure 5A schematic diagram showing response information display according to some embodiments of the present disclosure is shown; Figure 6 A schematic diagram showing response information display according to some embodiments of the present disclosure is shown; Figure 7 A schematic diagram showing response information display according to some embodiments of the present disclosure is shown; Figure 8 A schematic diagram showing response information display according to some embodiments of the present disclosure is shown; Fig. 9 A schematic diagram showing a flow chart of an information processing method according to some embodiments of the present disclosure; Fig.10 A structural block diagram of an information processing device according to some embodiments of the present disclosure is shown; Fig.11 A structural block diagram of an information processing device according to some embodiments of the present disclosure is shown; Fig.12 A structural block diagram of an electronic device according to some embodiments of the present disclosure is shown; Fig.13 A structural block diagram of an electronic device according to some other embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings.

[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments of the present specification and its application or uses.

[0026] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0027] To solve the above technical problems, an embodiment of the present disclosure provides an information processing method, which determines the information elements contained in the response information based on whether there is a corresponding entity for the text in the response information. When there is a corresponding entity for the text in the response information, the response information includes image elements and text elements, and the response information including the image elements and text elements is output, thereby realizing diversified output of the response information and improving the user experience.

[0028] Figure 1 The flowchart of the information processing method according to one embodiment of the present disclosure is shown. The method can be applied to an APP (application). The APP can be a client APP or a web APP. Figure 1 As shown, the method includes steps S110 to S130.

[0029] Step S110 , in response to the user inputting the query information, sending the query information to the server.

[0030] The inquiry information can be input by triggering the text input box of the APP interface, or by triggering the voice input control of the APP interface. When inputting through the voice input control, the obtained voice signal is converted into text as the inquiry information.

[0031] Step S120, receiving response information returned by the server in response to the query information; wherein the information elements included in the response information are determined based on the text in the response information, and when there is a corresponding entity for the text in the response information, the response information includes image elements and text elements.

[0032] In some embodiments, when there is no corresponding entity for the text in the response information, the response information includes text elements but does not include image elements.

[0033] In some embodiments, the information elements included in the response information are determined based on the text in the response information and specifically include: the type of information elements included in the response information is determined by inputting the text in the response information into a set recognition model; wherein, when the set recognition model determines that there is a corresponding entity in the text in the response information, the response information includes image elements and text elements.

[0034] The recognition model is a model trained based on a large number of training samples to identify whether the input text has a corresponding entity. The training samples are texts that include entity words and texts that do not include entity words. Entity words are words that have a physical counterpart. The physical object here is an object that objectively exists in the real world and has a material form.

[0035] Entity words can be set according to the specific classification of physical objects. For example, entity words are set according to the source and production method of physical objects. Entity words include natural physical objects and man-made physical objects. Natural physical objects include various animals, plants and minerals. Man-made physical objects include industrial products (for example, machines, vehicles, electronic equipment, etc.), handicrafts (for example, embroidery, weaving, carving, etc.), and works of art (for example, paintings, sculptures, photographs, etc.). For example, entity words are set according to the purpose of physical objects. Entity words include food physical objects, daily necessities physical objects, learning and work physical objects, and transportation physical objects. Food physical objects include grains, vegetables, fruits, and meat. Daily necessities physical objects include household items (for example, furniture, tableware, kitchen utensils, etc.), clothing and shoes (for example, clothes, shoes, backpacks, etc.). Learning and working physical objects include office equipment (for example, computers, printers, scanners, etc.), stationery (for example, pens, folders, etc.), books, and experimental equipment (for example, test tubes, microscopes, etc.). Transportation physical objects include cars, airplanes, and ships.

[0036] In some embodiments, the text input into the set recognition model may be all the text in the response information, or may be keywords extracted from the text in the response information.

[0037] For example, the query information is "How many kinds of special foods are there in Guangzhou?", and the text in the response information obtained based on the query information is "Boiled Chicken", "Rice Rolls", "Clay Pot Rice", "Cantonese Roast Goose", "Chaoshan Beef", "Barbecued Pork", "Shrimp Dumplings", "Wonton Noodles", "Sweet Soup", and "Beef Offal". The text in the above response information is input into the set recognition model to determine whether there is a corresponding entity for the text in the response information. Based on the text in the above response information, it can be determined that there is a corresponding entity. The response information contains image elements and text elements, that is, each specific food corresponds to a picture.

[0038] For example, the query information is "What are the landmark buildings in Beijing?", and the text in the response information obtained based on the query information is "Forbidden City", "Great Wall", "Temple of Heaven". The text in the above response information is input into the set recognition model to determine whether there is a corresponding entity for the text in the response information. Based on the text in the above response information, it is determined that there is a corresponding entity, and the response information contains image elements and text elements, that is, each landmark building corresponds to a picture.

[0039] For example, the inquiry information is "What is the process for reimbursement of medical insurance in other places?", and the text in the corresponding response information does not have a corresponding entity. The response information contains text elements but not image elements.

[0040] In some embodiments, the text in the response information is determined by inputting the search results obtained based on the query information into a set text processing model. The text processing model is used to understand the user's intention based on the query information and obtain the intention understanding result. The text processing model is also used to analyze and organize the text in the search results obtained based on the query information, and generate text content that matches the intention understanding result as the text in the response information.

[0041] The text processing model is a model trained with a large amount of text data. The text processing model is trained with a large amount of text data so that the text processing model learns the basic rules of the language, such as syntax, semantics, and contextual relationships, and enables the text processing model to analyze and organize text data based on user intent to obtain text content that meets user needs.

[0042] In some embodiments, after receiving the query information, the server performs information search according to the query information to obtain at least one search result, and inputs all search results into a set text processing model to determine the text in the response information corresponding to the query information.

[0043] In some embodiments, after receiving the query information, the server performs an information search according to the query information to obtain at least one search result. The server screens and obtains search results whose matching degree exceeds a preset threshold value according to the matching degree between the query information and each search result. The search results whose matching degree exceeds the preset threshold value are input into a set text processing model to determine the text in the response information corresponding to the query information.

[0044] In some embodiments, when the response information includes image elements, the images in the response information are obtained from the information source corresponding to the text in the response information. For example, when all search results are input into a set text processing model to determine the text in the response information corresponding to the query information, the information source corresponding to the text in the response information is the information source corresponding to all search results obtained by the server based on the query information search. For example, when search results with a matching degree exceeding a preset threshold are input into a set text processing model to determine the text in the response information corresponding to the query information, the information source corresponding to the text in the response information is the information source corresponding to the search results with a matching degree exceeding a preset threshold.

[0045] In some embodiments, the response information includes image elements and text elements, the information source corresponding to the text in the response information does not have an image, and the image in the response information is searched based on the text in the response information.

[0046] The pictures and texts in the response information are matched, that is, the pictures and texts in the response information are related. In the case where the pictures are obtained from the information source corresponding to the texts in the response information, it is determined whether the pictures in the information source match the texts in the response information to ensure the accuracy of the pictures in the response information, meet the needs of users, and improve user experience.

[0047] Specifically, first, determine the degree of match between the picture in the information source and the text in the response information. When the degree of match between the picture in the information source and the text in the response information is greater than or equal to a set threshold, the picture in the response information is obtained from the information source. When the degree of match is less than a set threshold, the picture in the response information is searched based on the text in the response information. The degree of match between the picture in the information source and the text in the response information can be determined in the following manner: extract keywords based on the text in the response information, determine corresponding text feature vectors based on the keywords, and determine corresponding picture feature vectors based on the pictures in the information source, and then calculate the similarity between the text feature vector and the picture feature vector corresponding to each picture as the degree of match between the picture in the information source and the text in the response information.

[0048] In some embodiments, when the image is obtained by the server based on the text search in the response information, the server extracts keywords based on the text in the response information, and obtains multiple images based on the keyword search. After the server obtains multiple images based on the keyword search, the corresponding text feature vector is determined according to the keyword, and the corresponding image feature vector is determined according to each image obtained by the search, and then the similarity between the text feature vector and the image feature vector corresponding to each image is calculated, and the image with the highest similarity value is screened according to the similarity to be used as the image matching the text in the response information.

[0049] In some embodiments, the feature vector of the text corresponding to the keyword is obtained by inputting the keyword into a set multimodal model, and the feature vector of the picture is obtained by inputting the corresponding picture into the multimodal model. The multimodal model can extract features of different modal information (e.g., pictures, texts), and has the ability to capture the relevance of different modal information.

[0050] For example, if the query information is "How many kinds of specialties are there in Guangzhou", the server sorts and analyzes the text in the search results obtained based on the query information, and obtains "Boiled Chicken" as the text in the response information. The server obtains pictures from the information sources corresponding to the text in the response information, and when it is determined that the matching degree between the pictures in the information source and "Boiled Chicken" is greater than or equal to the set threshold, the pictures in the information source are used as the pictures in the response information. When it is determined that the matching degree between the pictures in the information source and "Boiled Chicken" is less than the set threshold, the server searches for "Boiled Chicken" to obtain the corresponding pictures.

[0051] For example, the query information is "Who are the main actors in the movie ××?" The server organizes and analyzes the text in the search results obtained based on the query information, and obtains "actor Zhang ××," "actor Li ××," and "actor Wang ××" as the text in the response information. Pictures are obtained from the information sources corresponding to the text in the response information. The picture is a group photo of all the actors in the movie, not individual pictures of each actor. In this way, the matching degrees of the pictures in the information source and "actor Zhang ××," "actor Li ××," and "actor Wang ××" are all less than the set threshold. The pictures in the information source cannot be used as the pictures corresponding to each actor, but the corresponding pictures are obtained by searching for "actor Zhang ××," "actor Li ××," and "actor Wang ××" respectively.

[0052] Step S130: outputting response information corresponding to the inquiry information.

[0053] In some embodiments, the number of information items included in the response information may be one or more. When the number of information items included in the response information is more than one, and the response information includes image elements and text elements, each information item includes an image and text.

[0054] For example, if the query is "How many kinds of specialties are there in Guangzhou", see Figure 2 The output response information contains 10 information items, each of which includes a picture and text. Based on "Boiled Chicken", it corresponds to a picture and related text content, based on "Rice Roll", it corresponds to a picture and related text content, for "Claypot Rice", it corresponds to a picture and related text content, and for other information items, they all include a picture and related text content. For details, see Figure 2 , I will not go into details here.

[0055] For example, the inquiry information is "What is the process for medical insurance reimbursement in other places?" Figure 3 The output response information contains an information item, which only includes text content.

[0056] In some embodiments, step S130 specifically includes: outputting the content of the response information in time-sharing mode until a termination identifier is detected and the output of the response information is determined to be completed. Outputting the content of the response information in time-sharing mode allows the user to see the processing results and progress in real time, avoiding sudden feedback of the content of the response information after a long wait, making the user feel more smooth and comfortable.

[0057] Specifically, at the first time, the first response information corresponding to the inquiry information is output on the display interface, and at the second time, the second response information corresponding to the inquiry information is output on the display interface, and the output is performed in the above-mentioned output mode until the output of the response information corresponding to the inquiry information is completed. The first time and the second time are time points or time periods, and the second time is later than the first time. The first response information and the second response information are different component information of the response information corresponding to the inquiry information.

[0058] For example, based on the query "How many kinds of special foods are there in Guangzhou?" Figure 4 A schematic diagram showing the time-sharing output of response information. Figure 4 Four display interfaces corresponding to different time periods are shown. The response information displayed on the four display interfaces corresponding to different time periods increases in sequence, in a corresponding time-sharing output manner.

[0059] It should be noted that the response information can be output in a time-sharing manner or as a whole. The whole output is to output the response information as a whole so that the response information can be presented at one time.

[0060] In some embodiments, the response information includes multiple information items, and the output order of the multiple information items is determined based on the mention rate of each information item, wherein the mention rate of the information item is determined based on the number of times the information item appears in the search results obtained by matching the query information. The higher the mention rate of the information item, the higher the output order of the corresponding information item in the display interface. Giving priority to displaying information items with high mention rates can enable users to find the required information more quickly and improve user experience.

[0061] In some embodiments, the mention rate of an information item is determined based on the number of times the information item appears in all search results matched with the query information.

[0062] In some embodiments, the mention rate of the information item is determined based on the number of times the information item appears in some of the search results among all the search results matched with the query information. The search results among all the search results are the search results whose matching degree between the query information and the search results exceeds a preset threshold.

[0063] For example, based on the query information of "How many kinds of special foods are there in Guangzhou", the response information obtained includes 10 information items. Based on the fact that the higher the mention rate of an information item, the higher the output order of the corresponding information item, the output order of the 10 information items is "Boiled Chicken", "Rice Roll", "Clay Pot Rice", "Cantonese Roast Goose", "Chaoshan Beef", "Barbecued Pork", "Shrimp Dumplings", "Wonton Noodles", "Sweet Soup", and "Beef Offal".

[0064] In some embodiments, when the response information includes multiple information items, the mention rate of each information item is displayed on the display interface. By displaying the mention rate of each information item, the user can clearly understand each information item, which helps the user better understand the source of the information and improves the user experience.

[0065] See also Figure 5 Based on the query information of "How many kinds of special food are there in Guangzhou", the response information obtained includes 10 information items, and the output order of the 10 information items is "Boiled Chicken", "Rice Roll", "Claypot Rice", "Cantonese Roast Goose", "Chaoshan Beef", "Barbecued Pork", "Shrimp Dumpling", "Wonton Noodles", "Sweet Soup", and "Beef Offal". On the display interface, the mention rates of the 10 information items are displayed respectively.

[0066] In some embodiments, step S130 specifically includes: displaying a first set of response information and an interactive control corresponding to the inquiry information; in response to the interactive control being triggered, displaying a second set of response information corresponding to the inquiry information after the first set of response information. This method of displaying the response information step by step by triggering the interactive control enables the user to obtain information according to their own needs, thereby improving the user experience. In addition, when the user does not need the second set of response information, the system does not need to load and display this information, thereby saving bandwidth and computing resources.

[0067] The number of information items in the first group of response information can be set. The information items in the second group of response information are the remaining response information after removing the first group of response information from the response information corresponding to the inquiry information.

[0068] For example, based on the query information "How many kinds of special foods are there in Guangzhou", the response information obtained includes 10 information items. The first group of response information includes "Boiled Chicken", "Rice Roll", "Claypot Rice", "Cantonese Roast Goose", and "Chaoshan Beef". The second group of response information includes "Barbecued Pork", "Shrimp Dumplings", "Wonton Noodles", "Sweet Soup", and "Beef Offal". See Figure 6 , the first set of response information corresponding to the query information is displayed on the display interface, namely "Boiled Chicken", "Rice Noodle Roll", "Claypot Rice", "Guangzhou Roast Goose", and "Chaoshan Beef". At the same time, the display interface also displays an interactive control, namely the interactive control corresponding to "Expand More". When the user triggers the interactive control, see Figure 7 , after "Boiled Chicken", "Rice Roll", "Clay Pot Rice", "Guangzhou Roast Goose", and "Chaoshan Beef", "Barbecued Pork", "Shrimp Dumpling", "Wonton Noodles", "Sweet Soup", and "Beef Offal" are displayed successively, and the corresponding words of the interactive control are changed to "Collapse". When the user triggers the interactive control, "Barbecued Pork", "Shrimp Dumpling", "Wonton Noodles", "Sweet Soup", and "Beef Offal" are no longer displayed successively after "Boiled Chicken", "Rice Roll", "Clay Pot Rice", "Guangzhou Roast Goose", and "Chaoshan Beef", and the corresponding words of the interactive control are changed to "Expand More".

[0069] In some embodiments, the display interface also displays a selection control for the information element type of the response information. The selection control for the information element type includes a selection control corresponding to text and a selection control corresponding to an image. The method also includes: in response to the user triggering the selection of the information element type of the response information, displaying the response information corresponding to the selected information element type. In this way, different needs of users based on the information element type of the response information can be met, thereby improving the user experience.

[0070] according to Figure 8As shown, the display interface also displays a selection control for the information element type of the response information, namely, "text" and "picture". The user can click the selection button corresponding to "text" and the selection button corresponding to "picture" to trigger the information type of the response information.

[0071] A flowchart of an information processing method according to an embodiment of the present disclosure. The method can be applied to a server. Fig. 9 As shown, the method includes steps S910 to S930.

[0072] Step S910: receiving inquiry information input by a user sent by a client.

[0073] The inquiry information can be input by triggering the text input box of the APP interface, or by triggering the voice input control of the APP interface. When inputting through the voice input control, the obtained voice signal is converted into text as the inquiry information.

[0074] Step S920, perform information search according to the query information, and determine response information corresponding to the query information according to the search results; wherein, the information elements included in the response information are determined based on the text in the response information, and when there is a corresponding entity in the text in the response information, the response information includes image elements and text elements.

[0075] In some embodiments, when there is no corresponding entity for the text in the response information, the response information includes text elements but does not include image elements.

[0076] In some embodiments, the server inputs the text in the response information into a set recognition model to determine whether the text in the response information has a corresponding entity. The set recognition model is a model trained based on a large number of training samples to identify whether the input text has a corresponding entity. The training samples are texts that include entity words and texts that do not include entity words. Entity words are words that have a physical counterpart. The physical object here is an object that objectively exists in the real world and has a material form.

[0077] In some embodiments, the server inputs the search results obtained based on the query information into a set text processing model to determine the text in the response information.

[0078] The text processing model is used to understand the user's intention based on the query information and obtain the intention understanding result. The text processing model is also used to analyze and organize the text in the search results obtained based on the query information, and generate text content that matches the intention understanding result as the text in the response information.

[0079] The text processing model is a model trained with a large amount of text data. The text processing model is trained with a large amount of text data so that the text processing model learns the basic rules of the language, such as syntax, semantics, and contextual relationships, and enables the text processing model to analyze and organize text data based on user intent to obtain text content that meets user needs.

[0080] In some embodiments, after receiving the query information, the server performs information search according to the query information to obtain at least one search result, and inputs all search results into a set text processing model to determine the text in the response information corresponding to the query information.

[0081] In some embodiments, after receiving the query information, the server performs an information search according to the query information to obtain at least one search result. The server screens and obtains search results whose matching degree exceeds a preset threshold value according to the matching degree between the query information and each search result. The search results whose matching degree exceeds the preset threshold value are input into a set text processing model to determine the text in the response information corresponding to the query information.

[0082] In some embodiments, when the response information includes a picture, the server obtains the picture in the response information from an information source corresponding to the text in the response information.

[0083] The pictures and text in the response information are matched, that is, the pictures and text in the response information are related. In the case where the pictures are obtained from the information source corresponding to the text in the response information, the server determines whether the pictures in the information source match the text in the response information to ensure the accuracy of the pictures in the response information, meet the needs of users, and improve user experience.

[0084] Specifically, first, the server determines the degree of match between the picture in the information source and the text in the response information. The degree of match between the picture in the information source and the text in the response information can be determined in the following manner: extract keywords based on the text in the response information, and determine the corresponding text feature vector based on the keywords, and determine the corresponding picture feature vector based on the picture in the information source, and then calculate the similarity between the text feature vector and the picture feature vector corresponding to each picture as the degree of match between the picture in the information source and the text in the response information. When the degree of match between the picture in the information source and the text in the response information is greater than or equal to a set threshold, the server obtains the picture in the response information from the information source. When the degree of match is less than the set threshold, the server searches based on the text in the response information to obtain the picture in the response information.

[0085] In some embodiments, when the response information includes a picture, and the information source corresponding to the text in the response information does not have the picture, the server searches based on the text in the response information to obtain the picture in the response information.

[0086] In some embodiments, when the image is obtained by the server based on the text search in the response information, the server extracts keywords based on the text in the response information, and obtains multiple images based on the keyword search. After the server obtains multiple images based on the keyword search, the corresponding text feature vector is determined according to the keyword, and the corresponding image feature vector is determined according to each image obtained by the search, and then the similarity between the text feature vector and the image feature vector corresponding to each image is calculated, and the image with the highest similarity value is screened according to the similarity to be used as the image matching the text in the response information.

[0087] In some embodiments, the feature vector of the text corresponding to the keyword is obtained by inputting the keyword into a set multimodal model, and the feature vector of the picture is obtained by inputting the corresponding picture into the multimodal model. The multimodal model can extract features of different modal information (e.g., pictures, texts), and has the ability to capture the relevance of different modal information.

[0088] In some embodiments, when the response information includes multiple information items, the server determines a corresponding mention rate based on each information item. The mention rate of an information item is determined based on the number of times the information item appears in the search results obtained by matching the query information.

[0089] Step S930: Return the response information to the client for display.

[0090] The information processing method provided by the embodiment of the present invention realizes diversified output of response information and improves user experience.

[0091] The present disclosure also provides an information processing device. Fig.10 FIG. 4 shows a structural block diagram of an information processing device according to some embodiments. Fig.10 As shown, the information processing device 1000 may include a sending module 1010 , a receiving module 1020 , and an output module 1030 .

[0092] The sending module 1010 is used to send the query information to the server in response to the query information input by the user.

[0093] The receiving module 1020 is used to receive the response information returned by the server in response to the query information; wherein, the information elements contained in the response information are determined based on the text in the response information, and when there is a corresponding entity in the text in the response information, the response information includes image elements and text elements.

[0094] The output module 1030 is used to output response information corresponding to the inquiry information.

[0095] In some embodiments, when there is no corresponding entity for the text in the response information, the response information includes text elements but does not include image elements.

[0096] In some embodiments, the information elements included in the response information are determined based on the text in the response information, including: the type of information elements included in the response information is determined by inputting the text in the response information into a set recognition model; wherein, when the set recognition model determines that there is a corresponding entity in the text in the response information, the response information includes image elements and text elements.

[0097] In some embodiments, when the set recognition model determines that there is no corresponding entity in the text in the response information, the response information includes text elements and does not include image elements.

[0098] In some embodiments, the text in the response information is determined by inputting the search results obtained based on the query information into a set text processing model.

[0099] In some embodiments, the response information includes a picture element and a text element, and the picture in the response information is obtained from an information source corresponding to the text in the response information.

[0100] In some embodiments, when the degree of match between the image in the information source and the text in the response information is greater than or equal to a set threshold, the image in the response information is obtained from the information source; when the degree of match is less than the set threshold, the image in the response information is searched based on the text in the response information.

[0101] In some embodiments, the response information includes image elements and text elements, the information source corresponding to the text in the response information does not have an image, and the image in the response information is searched based on the text in the response information.

[0102] In some embodiments, the response information includes multiple information items, and the output order of the multiple information items is determined based on the mention rate of each information item, wherein the mention rate of the information item is determined based on the number of times the information item appears in the search results obtained by matching the query information.

[0103] In some embodiments, the output module 1030 is used to display a first set of response information and interactive controls corresponding to the inquiry information; in response to the interactive controls being triggered, a second set of response information corresponding to the inquiry information is displayed successively after the first set of response information.

[0104] In some embodiments, the output module 1030 is used to output the content of the response information in a time-sharing manner until a termination identifier is detected and it is determined that the output of the response information is completed.

[0105] The present disclosure also provides an information processing device. Fig.11 FIG. 4 shows a structural block diagram of an information processing device according to some embodiments. Fig.11 As shown, the information processing device 1100 may include a receiving module 1110 , a response information determining module 1120 , and a response information sending module 1130 .

[0106] The receiving module 1110 is used to receive query information input by a user and sent by a client.

[0107] The response information determination module 1120 is used to perform information search based on the query information, and determine the response information corresponding to the query information based on the search results; wherein, the information elements contained in the response information are determined based on the text in the response information, and when there is a corresponding entity in the text in the response information, the response information includes image elements and text elements.

[0108] The response information sending module 1130 is used to return the response information to the client for displaying the response information.

[0109] The present disclosure also provides an electronic device for implementing any of the above method embodiments. Fig.12 The structure block diagram of the electronic device 1200 according to some embodiments is shown. The electronic device 1200 may be a PC, a workstation, a notebook computer, a server, etc., which is not limited here.

[0110] like Fig.12 As shown, the electronic device 1200 includes a processor 1210 and a memory 1220 for storing instructions executable by the processor 1210. The processor 1210 is configured to implement the information processing method according to any embodiment of the present disclosure when executing the instructions stored in the memory 1220.

[0111] The processor 1210 is used to execute computer instructions, which can be written in an instruction set of an architecture such as x86, Arm, RISC, MIPS, SSE, etc. The memory 1220 includes, for example, ROM (read-only memory), RAM (random access memory), a non-volatile memory such as a hard disk, etc., which is not limited here.

[0112] Fig.13 FIG. 2 shows a structural block diagram of an electronic device according to some other embodiments. Fig.13As shown, the electronic device 1200 may include a display device 1230 , an interface device 1240 , a communication device 1250 , an input device 1260 , and the like in addition to a processor 1210 and a memory 1220 .

[0113] The interface device 1240 includes, for example, a USB interface, a bus interface, a network interface, etc. The communication device 1250 can, for example, perform wired or wireless communication. The communication device 1250 may include at least one short-range communication module, such as any module for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 1250 may also include a remote communication module, such as any module for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 1230 can display an application interface. The input device 1260 may include a touch screen, a keyboard, a mouse, a microphone, a camera, etc., which are not limited here.

[0114] The present disclosure also provides a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the information processing method provided in any of the above embodiments is implemented.

[0115] The present disclosure also provides a computer program product. When instructions in the computer program product are executed by a processor, the information processing method provided in any of the above embodiments is implemented.

[0116] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the device embodiment, its related parts can be referred to the partial description of the method embodiment.

[0117] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The embodiments of the present specification may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer instructions for causing a processor to implement various aspects of the embodiments of the present specification.

[0119] A computer-readable storage medium may be a tangible device that can hold and store computer instructions used by a computer instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which computer instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0120] The computer instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network layer, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network layer may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network layer adapter card or network layer interface in each computing / processing device receives the computer instructions from the network layer and forwards the computer instructions for storage in a computer-readable storage medium in each computing / processing device.

[0121] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the multiple embodiments of this specification. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a computer instruction, and a part of a module, a program segment or a computer instruction contains one or more executable computer instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0122] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An information processing method, characterized in that: include: In response to a user inputting query information, sending the query information to a server; Receive response information returned by the server in response to the query information; wherein the information elements contained in the response information are determined based on the text in the response information, and when the text in the response information has a corresponding entity, the response information includes a picture element and a text element; The response information corresponding to the inquiry information is output.

2. The method according to claim 1, characterized in that In the case that there is no corresponding entity for the text in the response information, the response information includes text elements but does not include image elements.

3. The method according to claim 1, characterized in that The information elements included in the response information are determined based on the text in the response information, including: The type of information element contained in the response information is determined by inputting the text in the response information into a set recognition model; wherein, when the set recognition model determines that there is a corresponding entity in the text in the response information, the response information includes image elements and text elements.

4. The method according to claim 3, characterized in that In the case where the set recognition model determines that there is no corresponding entity in the text in the response information, the response information includes text elements but does not include image elements.

5. The method according to claim 1, characterized in that The text in the response information is determined by inputting the search results obtained based on the query information into a set text processing model.

6. The method according to claim 1, characterized in that The response information includes a picture element and a text element, and the picture in the response information is obtained from an information source corresponding to the text in the response information.

7. The method according to claim 6, characterized in that When the matching degree between the picture in the information source and the text in the response information is greater than or equal to a set threshold, the picture in the response information is obtained from the information source; When the matching degree is less than the set threshold, the picture in the response information is searched based on the text in the response information.

8. The method according to claim 1, characterized in that The response information includes a picture element and a text element. The information source corresponding to the text in the response information does not have a picture. The picture in the response information is searched based on the text in the response information.

9. The method according to claim 1, characterized in that: The response information includes multiple information items, and the output order of the multiple information items is determined based on the mention rate of each information item, wherein the mention rate of the information item is determined based on the number of times the information item appears in the search results obtained by matching the query information.

10. The method according to claim 1, characterized in that The outputting the response information corresponding to the inquiry information includes: Displaying a first set of response information and interactive controls corresponding to the inquiry information; In response to the interactive control being triggered, a second set of response information corresponding to the inquiry information is displayed successively after the first set of response information.

11. The method according to any one of claims 1 to 10, characterized in that: The outputting the response information corresponding to the inquiry information includes: The content of the response information is output in time division until a termination identifier is detected, and it is determined that the output of the response information is completed.

12. An information processing method, characterized in that: include: Receive the query information input by the user sent by the client; Performing information search according to the query information, and determining response information corresponding to the query information according to the search results; wherein the information elements included in the response information are determined based on the text in the response information, and when the text in the response information has a corresponding entity, the response information includes a picture element and a text element; The response information is returned to the client for display.

13. An information processing device, characterized in that: include: A sending module, configured to send the query information to a server in response to the query information input by the user; A receiving module, configured to receive a response message returned by the server in response to the query message; wherein the information elements included in the response message are determined based on the text in the response message, and when the text in the response message has a corresponding entity, the response message includes a picture element and a text element; An output module is used to output the response information corresponding to the inquiry information.

14. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute the method according to any one of claims 1 to 12.

15. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images