Image-text question and answer method and device, storage medium and electronic equipment

Abstract

The invention discloses an image-text question and answer method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining image data, obtaining an inquiry statement corresponding to the image data, determining an inquiry type corresponding to the inquiry statement, determining a target image feature corresponding to the inquiry statement based on the inquiry type, and generating reply data corresponding to the inquiry statement based on the target image feature and the inquiry type, and outputting the reply data on a display interface of the terminal device. By adopting the embodiment of the invention, the convenience and the accuracy of the automatic reply capability of the terminal equipment are improved by simplifying the step of converting the inquiry content into the characters and inputting the characters into the terminal equipment by the user and the error caused by character expression in the conversion process.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a graphic question-and-answer method, device, storage medium, and electronic device. Background Art

[0002] Nowadays, with the continuous development of science and technology, the functions of wearable devices have been continuously enriched, and they have gradually become an indispensable technological tool in children's learning and daily life. Children can use wearable devices to assist their learning. Summary of the invention

[0003] The embodiments of this specification provide a method, device, storage medium and electronic device for image-text question and answer. By conducting question and answer based on image data, the inquiry method of the terminal device's automatic reply capability is enriched, and the convenience and accuracy of the terminal device's automatic reply capability are improved by simplifying the steps for users to convert inquiry content into text and input it into the terminal device, as well as the errors caused by text expression during the conversion process.

[0004] In a first aspect, an embodiment of this specification provides a graphic question-and-answer method, which is applied to a terminal device and includes:

[0005] Acquire image data, and acquire a query statement corresponding to the image data;

[0006] Determining a query type corresponding to the query statement, and determining a target image feature corresponding to the query statement based on the query type;

[0007] generating answer data corresponding to the inquiry statement based on the target image feature and the inquiry type;

[0008] The reply data is output on a display interface of the terminal device.

[0009] Through the above-mentioned technical scheme, it is possible to obtain image data and inquiry statements, determine the target image features of the image data according to the inquiry type of the inquiry statement, generate reply data for the inquiry statement based on the target image features and the inquiry type, and display the reply data, thereby realizing question and answer based on image data, enriching the inquiry method of the terminal device's automatic reply capability, and further improving the convenience and accuracy of the terminal device's automatic reply capability by simplifying the steps for users to convert inquiry content into text and input it into the terminal device, as well as the errors caused by text expression during the conversion process.

[0010] In combination with the first aspect and the above implementation manner, in some possible implementation manners, determining the target image feature corresponding to the query statement based on the query type includes:

[0011] Determining, based on the query type, a confidence level of interest corresponding to each image feature in the image data;

[0012] The image feature whose focus confidence is a target confidence is determined as a target image feature corresponding to the query statement, and the target confidence is greater than or equal to a preset confidence.

[0013] Through the above technical solution, the attention confidence corresponding to each image feature in the image data can be determined according to the inquiry type to determine the target image features in the image data related to the inquiry statement, thereby improving the accuracy of answering the inquiry statement based on the image data.

[0014] In combination with the first aspect and the above implementation manner, in some possible implementation manners, generating the response data corresponding to the inquiry statement based on the target image feature and the inquiry type includes:

[0015] generating answer data corresponding to the query statement based on the target image feature and the query type, wherein the answer data includes at least one answer result;

[0016] The answer data is screened based on the adoption rate of each of the answer results to determine the reply data corresponding to the query statement.

[0017] Through the above technical solution, it is possible to generate answer data corresponding to the inquiry type based on the target image features, including answer data indicating an answer result, and filter the answer data based on the adoption rate of the answer result to obtain reply data, thereby improving the accuracy of the reply data generated for replying the inquiry statement.

[0018] In combination with the first aspect and the above implementation manner, in some possible implementation manners, generating answer data corresponding to the query sentence based on the target image feature and the query type includes:

[0019] If the inquiry type is an identification type, obtaining at least one identification data corresponding to the target image feature, and determining the identification data as the answer data corresponding to the inquiry statement;

[0020] If the inquiry type is a creation type, generating at least one creation data corresponding to the target image feature, and determining the creation data as the answer data;

[0021] If the inquiry type is a translation type, a translation language corresponding to the inquiry sentence is obtained, at least one translation data corresponding to the target image feature is generated, and the translation data is determined as the answer data.

[0022] In combination with the first aspect and the above implementation manner, in some possible implementation manners, generating at least one creation data corresponding to the target image feature, and determining the creation data as the answer data, includes:

[0023] Determine a work template based on the query statement;

[0024] At least one creation data corresponding to the target image feature is generated based on the work template, and each creation data is determined as the answer data.

[0025] In combination with the first aspect and the above implementations, in some possible implementations, screening the answer data based on the adoption rate of each answer result to determine the reply data corresponding to the query statement includes:

[0026] Obtaining the adoption rate of each of the answer results in the answer data;

[0027] The target answer result with the highest adoption rate among the answer results is selected, and the target answer result is determined as the response data corresponding to the query statement.

[0028] Through the above technical solution, the answer results can be screened according to the adoption rate of the answer results, and the answer result with the highest adoption rate can be selected as the reply data, thereby improving the accuracy of the reply data.

[0029] In combination with the first aspect and the above implementation manner, in some possible implementation manners, after acquiring the image data, the method further includes:

[0030] identifying the image data set to obtain at least one image feature;

[0031] If the query statement is not obtained within the preset query time, generating brief description data corresponding to the image data based on each of the image features;

[0032] The brief description data and preset inquiry options are output on a display interface of the terminal device.

[0033] Through the above technical solution, the summary data of the image data and the inquiry options can be output when no inquiry statement is received, so that the user can input the inquiry statement, thereby improving the convenience of the user to use the terminal device to make inquiries.

[0034] In combination with the first aspect and the above implementation manner, in some possible implementation manners, outputting the reply data on a display interface of the terminal device includes:

[0035] If the reply data is translation data, outputting the translation data and a preset playback query sentence on a display interface of the terminal device;

[0036] In response to a play selection operation for the play query sentence, audio data corresponding to the play selection operation is played.

[0037] In a second aspect, an embodiment of the present specification provides a graphic question-and-answer device, the device comprising:

[0038] A data acquisition unit, used to acquire image data and obtain a query statement corresponding to the image data;

[0039] a feature determination unit, configured to determine a query type corresponding to the query statement, and determine a target image feature corresponding to the query statement based on the query type;

[0040] A response generating unit, configured to generate response data corresponding to the inquiry statement based on the target image feature and the inquiry type;

[0041] A data display unit is used to output the reply data on the display interface of the terminal device.

[0042] In a third aspect, an embodiment of the present specification provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0043] In a fourth aspect, an embodiment of the present specification provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of a scenario of a graphic question and answer provided in an embodiment of this specification;

[0046] Figure 2 A flowchart of a graphic question-and-answer method provided in an embodiment of this specification;

[0047] Figure 3 A flowchart of a graphic question-and-answer method provided in an embodiment of this specification;

[0048] Figure 4 This is an example schematic diagram of an output summary data conversion provided in an embodiment of this specification;

[0049] Figure 5 An example schematic diagram of image data provided in an embodiment of this specification;

[0050] Figure 6 An example schematic diagram of inputting a query sentence provided in an embodiment of this specification;

[0051] Figure 7 An example schematic diagram of outputting reply data provided in an embodiment of this specification;

[0052] Figure 8 A schematic diagram of the structure of a graphic question-and-answer device provided in an embodiment of this specification;

[0053] Fig. 9 A schematic diagram of the structure of a graphic question-and-answer device provided in an embodiment of this specification;

[0054] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0055] In order to make the features and advantages of this specification more obvious and easy to understand, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this specification.

[0056] See also Figure 1 , which is a schematic diagram of a scenario of a graphic question-and-answer method according to an embodiment of this specification. Figure 1 As shown, the text-and-picture question-and-answer method provided in the embodiment of this specification can be applied to the terminal device 10 to implement the text-and-picture question-and-answer process. The system architecture provided in the embodiment of this specification mainly includes the terminal device 10 and the user 20. Among them, the terminal device 10 can be a terminal device with a text-and-picture question-and-answer function, including but not limited to wearable devices such as watches, as well as conference tablets, mobile phones, personal computers, laptops, smart furniture, vehicle-mounted devices, wearable devices and other terminal devices.

[0057] In the related art, when learning is assisted by wearable devices, the method used needs to convert questions into text and then input them into the AI ​​(Artificial Intelligence) attached to the wearable device for inquiry. Therefore, there is an urgent need for a method to answer questions that are not easy to convert into text.

[0058] In an embodiment of the present specification, the terminal device 10 obtains image data input by the user 20, and a query statement corresponding to the image data input by the user 20, determines the query type corresponding to the query statement, determines the target image features corresponding to the query statement based on the query type, generates reply data corresponding to the query statement based on the target image features and the query type, and outputs the reply data on the display interface of the terminal device 10, thereby improving the convenience and accuracy of the terminal device's automatic reply capability by simplifying the steps for the user to convert the query content into text and input it into the terminal device, as well as the errors caused by text expression during the conversion process.

[0059] based on Figure 1 The system architecture shown below will be combined with Figure 2-Figure 6 , the graphic question and answer method provided in the embodiments of this specification is introduced in detail.

[0060] See also Figure 2 , is a flowchart of a graphic question-and-answer method provided in the embodiment of this specification. Figure 2 As shown, the method is applied to a terminal device and may include the following steps S101 to S104.

[0061] S101, obtaining image data and obtaining a query statement corresponding to the image data;

[0062] In one embodiment, image data input by a user is obtained. The image data input method may be image data captured by the user through a camera component of a terminal device, or may be image data selected from an image data set stored in the terminal device. An inquiry statement input by the user for the image data is obtained. The inquiry statement may be an instruction for instructing the terminal device to respond to the image data.

[0063] The inquiry statement may be text data, voice data, or a preset statement selected by the user on the display interface of the terminal device.

[0064] S102, determining the query type corresponding to the query statement, and determining the target image features corresponding to the query statement based on the query type;

[0065] In one embodiment, semantic recognition is performed on the input query statement to determine the query type corresponding to the query statement. The query type may be the type of response to the image data indicated by the query statement, such as a recognition type, a creation type, or a translation type. Image features of the image data are screened based on the query type to determine the target image features corresponding to the query statement. The image features may be features included in the image data, such as objects, organisms, etc. in the image data, or text data in the image data.

[0066] Exemplarily, if the query type is a translation type, the text data of the image data is used as the target image feature corresponding to the image data.

[0067] S103, generating answer data corresponding to the inquiry sentence based on the target image features and the inquiry type;

[0068] In one embodiment, after determining the target image features of the image data and the query type of the query statement, the server is queried for answer data related to the target image features. The server may be a device including a database and data processing functions. The answer data is filtered based on the query type to obtain corresponding reply data. The reply data may be data generated to reply to the query statement.

[0069] Specifically, the method for generating reply data may be: generating answer data including at least one answer result, the answer result may be a result for answering the inquiry statement corresponding to the target image feature generated for the inquiry type, and screening each answer result to determine the reply data in the answer data.

[0070] S104, outputting reply data on a display interface of the terminal device;

[0071] In one embodiment, the display format of the display interface of the terminal device is obtained, and the display parameters of the reply data are adjusted based on the display format. The display format may be the format of the reply data displayed on the display interface of the terminal device, such as the format of a dialog box or the font of text, etc. The display parameters may be parameters corresponding to the display format, specifically the length and width of the dialog box, as well as the pattern of the dialog box, etc. The reply data is output on the display interface of the terminal device using the display format.

[0072] In the embodiments of the present specification, by acquiring image data and an inquiry statement, determining the target image features of the image data according to the inquiry type of the inquiry statement, generating response data for the inquiry statement based on the target image features and the inquiry type, and displaying the response data, it is possible to conduct question and answer on the image data, enrich the inquiry method of the automatic reply capability of the terminal device, and further improve the convenience and accuracy of the automatic reply capability of the terminal device by simplifying the steps for the user to convert the inquiry content into text and input it into the terminal device, as well as the errors caused by the text expression during the conversion process.

[0073] See also Figure 3 , is a flowchart of a graphic question-and-answer method provided in the embodiment of this specification. Figure 3 As shown, the method is applied to a terminal device and may include the following steps S201 to S207.

[0074] S201, obtaining image data and obtaining a query statement corresponding to the image data;

[0075] In one embodiment, image data input by a user is obtained. The image data input method may be image data captured by the user through a camera component of a terminal device, or may be image data selected from an image data set stored in the terminal device. An inquiry statement input by the user for the image data is obtained. The inquiry statement may be an instruction for instructing the terminal device to respond to the image data.

[0076] The inquiry statement may be text data, voice data, or a preset statement selected by the user on the display interface of the terminal device.

[0077] Furthermore, in order to facilitate the user to enter a query statement after inputting the image data, a feasible method may be: identifying the image data group to obtain at least one image feature; if the query statement is not obtained within the preset query time, generating brief description data corresponding to the image data based on each image feature, and outputting the brief description data and preset query options on the display interface of the terminal device.

[0078] The image features may be features obtained after feature extraction of the image data, and specifically may be recognition of objects, persons or texts in the image data according to an image recognition algorithm, and taking each object, person and text as an image feature corresponding to the image data. It is understandable that when extracting texts from the image data as image features, in order to facilitate subsequent processing of the image features, one feasible method may be to take all the texts included in the image data as an image feature, and another feasible method may be to divide the texts included in the image data into different words or phrases according to semantics, and determine each word or phrase as an image feature.

[0079] The specific duration of the inquiry time can be set according to the actual situation, for example, it can be 3 seconds, etc. It can be understood that if it is detected that the user is performing text input or voice input, a feasible method can be: if the inquiry time is reached and the user is performing text input or voice input, then the brief description data and inquiry options are not output; if the inquiry time is reached and the user is not in the input state such as text input or voice input, and no inquiry statement is received, then the brief description data is generated, and the brief description data and inquiry options are output on the display interface of the terminal device.

[0080] Specifically, the summary data may be data generated based on the correlation between various image features and used to summarize the content of the image data.

[0081] For example, Figure 4 As shown, Figure 4If the image data in the image includes a duck and water, the image features included in the image data may be a duck and water, and the generated summary data may be: a duck swimming in the water.

[0082] The query options may be pre-set fixed options, such as "What is this", "Create a short article based on the image", etc.; they may also be query options generated based on image features, for example, if the image features of the image data include text data, the generated query options may be "Translate the text in the image", "Create an article with the text in the image as the title", etc. The specific method of generating the query options may be set according to actual conditions.

[0083] It should be noted that if the user selects the query option output on the display interface, the query option is used as the query sentence corresponding to the image data. The method for determining the user's selection of the query option may be to detect the user's touch operation on the display interface, or to perform voice recognition on the received user input voice data to determine the query option corresponding to the voice data, etc., which may be set according to actual conditions.

[0084] S202, determining the query type corresponding to the query statement;

[0085] In one embodiment, after obtaining the query statement input by the user, semantic recognition is performed on the query statement to determine the query type corresponding to the query statement. The query type may be the type corresponding to the response to the image data indicated by the query statement, such as a recognition type, a creation type, or a translation type.

[0086] Among them, the recognition type can be a type of identifying an object or person included in the image data and introducing the object or person; the creation type can be a type of creating according to the object, person or text included in the image data, such as creating an article or a poem; the translation type can be a type of translating the text in the image data. It can be understood that the specific type of the inquiry type can be set according to the actual situation, and can also be a navigation type, a number dialing type, etc.

[0087] S203, determining the attention confidence corresponding to each image feature in the image data based on the query type;

[0088] In one embodiment, a comparison is made between each image feature in the image data based on the query type, and an attention confidence corresponding to each image feature is set. The attention confidence may be used to characterize the degree of association between each image feature and the query type. The greater the attention confidence of an image feature, the greater the degree of association between the image feature and the query type.

[0089] For example, Figure 5 As shown, Figure 5 The image data in includes a duck, water and the text "Duck swimming in the water", and the query type is a translation type, then the attention confidence of the text is higher than the attention confidence of the duck and the water.

[0090] S204, determining the image feature whose focus confidence is the target confidence as the target image feature corresponding to the query sentence;

[0091] In one embodiment, the attention confidence of each image feature in the image data is compared with a preset confidence, and the image feature whose attention confidence is a target confidence is determined as the target image feature corresponding to the query statement. The target confidence may be an attention confidence that is greater than or equal to the preset confidence.

[0092] It should be noted that, since the required image features may be one or more when answering the query statement based on the image data, the selected target image features may be one or more. Therefore, when setting the preset confidence, the situation where multiple image features in the image data are selected can be considered to avoid the preset confidence being too high, which may result in the inability to obtain sufficient image features to answer the query statement.

[0093] S205, generating answer data corresponding to the query sentence based on the target image features and the query type;

[0094] In one embodiment, after determining the target image features corresponding to the target image data, answer data related to the target image features and belonging to the inquiry type is obtained in the server based on the inquiry type. The server may be a device including a database and data processing functions. The answer data includes at least one answer result, and the answer result may be a result for answering the inquiry statement corresponding to the target image features generated for the inquiry type.

[0095] Specifically, the method for generating answer data according to the inquiry type may be: if the inquiry type is an identification type, obtaining at least one identification data corresponding to the target image feature, and determining the identification data as the answer data corresponding to the inquiry statement.

[0096] The identification data may be data for introducing the target image features obtained after querying the server according to the target image features. The introduction scope included in the identification data may be determined according to the specific content in the query statement. For example, the identification data may only include the introduction of the target image features, or may include the target image features and data related to the target image features. The related data may be poems, locations, etc. of the target image features.

[0097] For example, Figure 6The image data contains a duck and water. If the query statement is "What is this? Are there any poems related to the image data?", the generated recognition data is "The image data shows a duck swimming in the water. A duck is a kind of bird with feathers all over its body, a large and round head, no crest, wattle or earlobes; its beak is long and flat, its neck is relatively long, its body is wide and long, its wings are small while the secondary flight feathers are relatively long, its legs are short, and there are webs between its toes, which is conducive to swimming and it is good at foraging and playing in the water. Ducks are docile in temperament and easily frightened." Poems related to the image data include "Three or two branches of peach blossoms outside the bamboo, the ducks know first that the spring river is warm." etc.

[0098] Furthermore, if the query type is the creation type, at least one creation data corresponding to the target image features is generated, and the creation data is determined as the answer data.

[0099] The creation data can be article data corresponding to the query statement generated based on the target image features, specifically, it can be a composition, a diary or a poem, etc.

[0100] Specifically, a feasible method for generating creation data can be: determining a work template based on the query statement, generating at least one creation data corresponding to the target image features based on the work template, and determining each creation data as the answer data. Among them, the work template can be a template corresponding to the work type indicated by the query statement, for example, it can be a prose, a diary or a seven-character regulated poem, etc.

[0101] Furthermore, if the query type is the translation type, the translation language corresponding to the query statement is obtained, at least one translation data corresponding to the target image features is generated, and the translation data is determined as the answer data.

[0102] The translation data can be data generated by translating the target image features according to the translation language indicated by the query statement. Exemplarily, as Figure 5 shown, the translation language indicated in the query statement is Chinese, Figure 5 and the target image feature in

[0103] is "Duck swimming in the water", then the target image feature is translated into the Chinese translation statement "The duck is swimming in the water".

[0104] In one embodiment, the adoption rate of each answer result in the answer data is obtained. The adoption rate can be a probability representing the adoption of the answer result. The higher the adoption rate, the higher the correlation between the answer result and the query statement and the target image features. The target answer result with the highest adoption rate among each answer result is selected, and the target answer result is determined as the reply data corresponding to the query statement. The reply data can be data generated for replying to the query statement.

[0105] It should be noted that the adoption rate can be obtained based on the query type. If the query type is an identification type, the adoption rate can be obtained based on the ratio of the number of times each answer result in the server is retrieved to the number of times it is adopted as the reply data. For example, "answer result 1" is retrieved 10 times and adopted as the reply data 5 times, and the adoption rate of "answer result 1" is 0.5; if the query type is a creation type or a translation type, the accuracy of the semantics and vocabulary in the generated answer result is evaluated based on a pre-trained evaluation model, and the evaluation result output by the evaluation model for the answer result is determined as the adoption rate of the answer result. The higher the adoption rate, the higher the accuracy of the semantics and vocabulary in the answer result. Among them, the evaluation model can be a pre-trained model that can recognize the semantics and vocabulary of a sentence and determine the coherence, readability and accuracy of the sentence.

[0106] For example, if the adoption rate of "answer result 1" is 0.5, the adoption rate of "answer result 2" is 0.6, and the adoption rate of "answer result 3" is 0.57, then since 0.6>0.57>0.5, "answer result 2" is determined as the target answer result, and the target answer result is determined as the reply data corresponding to the query statement.

[0107] S207, outputting the reply data on the display interface of the terminal device;

[0108] In one embodiment, the display format of the display interface of the terminal device is obtained, and the display parameters of the reply data are adjusted based on the display format. The display format may be the format of the reply data displayed on the display interface of the terminal device, such as the format of a dialog box or the font of text, etc. The display parameters may be parameters corresponding to the display format, specifically the length and width of the dialog box, as well as the pattern of the dialog box, etc. The reply data is output on the display interface of the terminal device using the display format.

[0109] For example, Figure 7 As shown, Figure 7 The image data in the image includes a duck and water, and also includes the English "Duckswimming in the water". The query sentence "Translate the English in the image" is input, and the response data "Duck swimming in the water" is displayed on the display interface of the terminal device.

[0110] Furthermore, if the reply data is translation data, the translation data and a preset play query sentence are output on the display interface of the terminal device, and in response to a play selection operation for the play query sentence, audio data corresponding to the play selection operation is played.

[0111] For example, Figure 7After outputting the reply data, the preset playback query sentences "play original text" and "play translated text" are also output, and after receiving the user's selection operation for the playback query sentence, the audio data corresponding to the selection operation is played.

[0112] In the embodiment of the present specification, by acquiring image data and an inquiry statement, determining the target image features of the image data according to the inquiry type of the inquiry statement, generating the answer data corresponding to the inquiry statement based on the target image features and the inquiry type, and displaying the answer data, it is possible to answer questions based on the image data, enrich the inquiry method of the automatic answering capability of the terminal device, and further improve the convenience and accuracy of the automatic answering capability of the terminal device by simplifying the steps of converting the inquiry content into text and inputting it into the terminal device, as well as the errors caused by the text expression during the conversion process. Further, the attention confidence corresponding to each image feature in the image data is determined according to the inquiry type to determine the target image features in the image data related to the inquiry statement, thereby improving the accuracy of answering the inquiry statement based on the image data; in addition, the answer results are screened by the adoption rate of the answer results, and the answer result with the highest adoption rate is selected as the answer data, thereby improving the accuracy of the answer data; further, by outputting the brief description data and inquiry options of the image data when the inquiry statement is not received, it is convenient for the user to input the inquiry statement, thereby improving the convenience of the user to use the terminal device to inquire.

[0113] based on Figure 1 The system architecture shown below will be combined with Figure 8 and Fig. 9 , the graphic question-and-answer device provided in the embodiment of this specification is introduced in detail. It should be noted that, Figure 8 and Fig. 9 The graphic question and answer device in the manual is used to execute the Figure 2-Figure 7 For the convenience of explanation, only the part related to the embodiment of this specification is shown. For the specific technical details not disclosed, please refer to this specification. Figure 2-Figure 7 The embodiment shown.

[0114] See also Figure 8 , is a schematic diagram of the structure of a graphic question-and-answer device provided in the embodiment of this specification. Figure 8 As shown, the graphic question-and-answer device 1 of the embodiment of this specification may include: a data acquisition unit 11 , a feature determination unit 12 , a reply generation unit 13 and a data display unit 14 .

[0115] A data acquisition unit 11 is used to acquire image data and obtain a query statement corresponding to the image data;

[0116] A feature determination unit 12, configured to determine a query type corresponding to the query statement, and determine a target image feature corresponding to the query statement based on the query type;

[0117] A reply generating unit 13, configured to generate reply data corresponding to the inquiry statement based on the target image feature and the inquiry type;

[0118] The data display unit 14 is used to output the reply data on the display interface of the terminal device.

[0119] Optionally, the feature determination unit 12 is further configured to:

[0120] Determining, based on the query type, a confidence level of interest corresponding to each image feature in the image data;

[0121] The image feature whose focus confidence is a target confidence is determined as a target image feature corresponding to the query statement, and the target confidence is greater than or equal to a preset confidence.

[0122] Optionally, the reply generating unit 13 is further configured to:

[0123] generating answer data corresponding to the query statement based on the target image feature and the query type, wherein the answer data includes at least one answer result;

[0124] The answer data is screened based on the adoption rate of each of the answer results to determine the reply data corresponding to the query statement.

[0125] Optionally, the reply generating unit 13 is further configured to:

[0126] If the inquiry type is an identification type, obtaining at least one identification data corresponding to the target image feature, and determining the identification data as the answer data corresponding to the inquiry statement;

[0127] If the inquiry type is a creation type, generating at least one creation data corresponding to the target image feature, and determining the creation data as the answer data;

[0128] If the inquiry type is a translation type, a translation language corresponding to the inquiry sentence is obtained, at least one translation data corresponding to the target image feature is generated, and the translation data is determined as the answer data.

[0129] Optionally, the reply generating unit 13 is further configured to:

[0130] Determine a work template based on the query statement;

[0131] At least one creation data corresponding to the target image feature is generated based on the work template, and each creation data is determined as the answer data.

[0132] Optionally, the reply generating unit 13 is further configured to:

[0133] Obtaining the adoption rate of each of the answer results in the answer data;

[0134] The target answer result with the highest adoption rate among the answer results is selected, and the target answer result is determined as the response data corresponding to the query statement.

[0135] Optional, such as Fig. 9 As shown, the graphic question-and-answer device 1 further includes a query output unit 15, which is used to:

[0136] identifying the image data set to obtain at least one image feature;

[0137] If the query statement is not obtained within the preset query time, generating brief description data corresponding to the image data based on each of the image features;

[0138] The brief description data and preset inquiry options are output on a display interface of the terminal device.

[0139] Optionally, the data display unit 14 is further used for:

[0140] If the reply data is translation data, outputting the translation data and a preset playback query sentence on a display interface of the terminal device;

[0141] In response to a play selection operation for the play query sentence, audio data corresponding to the play selection operation is played.

[0142] In the embodiment of the present specification, by acquiring image data and an inquiry statement, determining the target image features of the image data according to the inquiry type of the inquiry statement, generating the answer data corresponding to the inquiry statement based on the target image features and the inquiry type, and displaying the answer data, it is possible to answer questions based on the image data, enrich the inquiry method of the automatic answering capability of the terminal device, and further improve the convenience and accuracy of the automatic answering capability of the terminal device by simplifying the steps of converting the inquiry content into text and inputting it into the terminal device, as well as the errors caused by the text expression during the conversion process. Further, the attention confidence corresponding to each image feature in the image data is determined according to the inquiry type to determine the target image features in the image data related to the inquiry statement, thereby improving the accuracy of answering the inquiry statement based on the image data; in addition, the answer results are screened by the adoption rate of the answer results, and the answer result with the highest adoption rate is selected as the answer data, thereby improving the accuracy of the answer data; further, by outputting the brief description data and inquiry options of the image data when the inquiry statement is not received, it is convenient for the user to input the inquiry statement, thereby improving the convenience of the user to use the terminal device to inquire.

[0143] The present specification also provides a computer storage medium, which can store a plurality of program instructions, which are suitable for being loaded and executed by a processor as described above. Figure 2-Figure 7 The method steps of the embodiment shown in the figure can be found in the specific implementation process. Figure 2-Figure 7 The specific description of the illustrated embodiment will not be repeated here.

[0144] See also Fig.10 , is a schematic diagram of the structure of an electronic device provided in the embodiment of this specification. Fig.10 As shown, the electronic device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, an input / output interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As shown in FIG. Fig.10 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, an input and output interface module, and a graphic question and answer application program.

[0145] exist Fig.10In the electronic device 1000 shown, the input-output interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user.

[0146] In one embodiment, the processor 1001 may be used to call the graphic question and answer application stored in the memory 1005, and specifically perform the following operations:

[0147] Acquire image data, and acquire a query statement corresponding to the image data;

[0148] Determining a query type corresponding to the query statement, and determining a target image feature corresponding to the query statement based on the query type;

[0149] generating answer data corresponding to the inquiry statement based on the target image feature and the inquiry type;

[0150] The reply data is output on a display interface of the terminal device.

[0151] Optionally, when determining the target image feature corresponding to the query statement based on the query type, the processor 1001 specifically performs the following operations:

[0152] Determining, based on the query type, a confidence level of interest corresponding to each image feature in the image data;

[0153] The image feature whose focus confidence is a target confidence is determined as a target image feature corresponding to the query statement, and the target confidence is greater than or equal to a preset confidence.

[0154] Optionally, when the processor 1001 generates the reply data corresponding to the inquiry statement based on the target image feature and the inquiry type, the processor 1001 specifically performs the following operations:

[0155] generating answer data corresponding to the query statement based on the target image feature and the query type, wherein the answer data includes at least one answer result;

[0156] The answer data is screened based on the adoption rate of each of the answer results to determine the reply data corresponding to the query statement.

[0157] Optionally, when the processor 1001 generates answer data corresponding to the query statement based on the target image feature and the query type, the processor 1001 specifically performs the following operations:

[0158] If the inquiry type is an identification type, obtaining at least one identification data corresponding to the target image feature, and determining the identification data as the answer data corresponding to the inquiry statement;

[0159] If the inquiry type is a creation type, generating at least one creation data corresponding to the target image feature, and determining the creation data as the answer data;

[0160] If the inquiry type is a translation type, a translation language corresponding to the inquiry sentence is obtained, at least one translation data corresponding to the target image feature is generated, and the translation data is determined as the answer data.

[0161] Optionally, when the processor 1001 generates at least one creation data corresponding to the target image feature and determines the creation data as the answer data, the processor 1001 specifically performs the following operations:

[0162] Determine a work template based on the query statement;

[0163] At least one creation data corresponding to the target image feature is generated based on the work template, and each creation data is determined as the answer data.

[0164] Optionally, when the processor 1001 screens the answer data based on the adoption rate of each answer result and determines the reply data corresponding to the query statement, the processor 1001 specifically performs the following operations:

[0165] Obtaining the adoption rate of each of the answer results in the answer data;

[0166] The target answer result with the highest adoption rate among the answer results is selected, and the target answer result is determined as the response data corresponding to the query statement.

[0167] Optionally, after executing the image data acquisition, the processor 1001 further performs the following operations:

[0168] identifying the image data set to obtain at least one image feature;

[0169] If the query statement is not obtained within the preset query time, generating brief description data corresponding to the image data based on each of the image features;

[0170] The brief description data and preset inquiry options are output on a display interface of the terminal device.

[0171] Optionally, when outputting the reply data on the display interface of the terminal device, the processor 1001 specifically performs the following operations:

[0172] If the reply data is translation data, outputting the translation data and a preset playback query sentence on a display interface of the terminal device;

[0173] In response to a play selection operation for the play query sentence, audio data corresponding to the play selection operation is played.

[0174] In the embodiment of the present specification, by acquiring image data and an inquiry statement, determining the target image features of the image data according to the inquiry type of the inquiry statement, generating the answer data corresponding to the inquiry statement based on the target image features and the inquiry type, and displaying the answer data, it is possible to answer questions based on the image data, enrich the inquiry method of the automatic answering capability of the terminal device, and further improve the convenience and accuracy of the automatic answering capability of the terminal device by simplifying the steps of converting the inquiry content into text and inputting it into the terminal device, as well as the errors caused by the text expression during the conversion process. Further, the attention confidence corresponding to each image feature in the image data is determined according to the inquiry type to determine the target image features in the image data related to the inquiry statement, thereby improving the accuracy of answering the inquiry statement based on the image data; in addition, the answer results are screened by the adoption rate of the answer results, and the answer result with the highest adoption rate is selected as the answer data, thereby improving the accuracy of the answer data; further, by outputting the brief description data and inquiry options of the image data when the inquiry statement is not received, it is convenient for the user to input the inquiry statement, thereby improving the convenience of the user to use the terminal device to inquire.

[0175] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0176] The above disclosure is only the preferred embodiment of this specification, which certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.

Claims

1. A picture-text question-answering method, characterized in that: The method is applied to a terminal device, comprising: Acquire image data, and acquire a query statement corresponding to the image data; Determining a query type corresponding to the query statement, and determining a target image feature corresponding to the query statement based on the query type; generating answer data corresponding to the inquiry statement based on the target image feature and the inquiry type; The reply data is output on a display interface of the terminal device.

2. The method according to claim 1, characterized in that The determining, based on the query type, the target image feature corresponding to the query statement includes: Determining, based on the query type, a confidence level of interest corresponding to each image feature in the image data; The image feature whose focus confidence is a target confidence is determined as a target image feature corresponding to the query statement, and the target confidence is greater than or equal to a preset confidence.

3. The method according to claim 1, characterized in that The generating the answer data corresponding to the inquiry statement based on the target image feature and the inquiry type includes: generating answer data corresponding to the query statement based on the target image feature and the query type, wherein the answer data includes at least one answer result; The answer data is screened based on the adoption rate of each of the answer results to determine the reply data corresponding to the query statement.

4. The method according to claim 3, characterized in that The generating the answer data corresponding to the query statement based on the target image feature and the query type includes: If the inquiry type is an identification type, obtaining at least one identification data corresponding to the target image feature, and determining the identification data as the answer data corresponding to the inquiry statement; If the inquiry type is a creation type, generating at least one creation data corresponding to the target image feature, and determining the creation data as the answer data; If the inquiry type is a translation type, a translation language corresponding to the inquiry sentence is obtained, at least one translation data corresponding to the target image feature is generated, and the translation data is determined as the answer data.

5. The method according to claim 4, characterized in that The step of generating at least one creation data corresponding to the target image feature and determining the creation data as the answer data comprises: Determine a work template based on the query statement; At least one creation data corresponding to the target image feature is generated based on the work template, and each creation data is determined as the answer data.

6. The method according to claim 3, characterized in that: The step of screening the answer data based on the adoption rate of each answer result to determine the reply data corresponding to the query statement includes: Obtaining the adoption rate of each of the answer results in the answer data; The target answer result with the highest adoption rate among the answer results is selected, and the target answer result is determined as the response data corresponding to the query statement.

7. The method according to claim 1, characterized in that After acquiring the image data, the method further includes: identifying the image data set to obtain at least one image feature; If the query statement is not obtained within the preset query time, generating brief description data corresponding to the image data based on each of the image features; The brief description data and preset inquiry options are output on a display interface of the terminal device.

8. A graphic question-and-answer device, characterized in that: The device comprises: A data acquisition unit, used to acquire image data and obtain a query statement corresponding to the image data; a feature determination unit, configured to determine a query type corresponding to the query statement, and determine a target image feature corresponding to the query statement based on the query type; A response generating unit, configured to generate response data corresponding to the inquiry statement based on the target image feature and the inquiry type; A data display unit is used to output the reply data on the display interface of the terminal device.

9. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method according to any one of claims 1 to 7.

Images