Information provision, product inquiry, question and answer processing methods, equipment, storage media

Abstract

The embodiment of the present application discloses an information providing method. The method includes: when processing the matching between texts, in addition to using the picture text data in the picture, the original picture data itself is also combined, so as to avoid the problem of missing other information in the picture except the text, improve the accuracy of determining the position of the answer text data in the picture, and use the picture area related to the answer text data in the picture as a sub-picture for the question text data, compared with only providing the answer text data, other relevant information in the picture except the answer text data can also be provided, so as to achieve the effect of providing picture-type information for the question text data and improve the integrity of the information provided for the question text data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and specifically to an information providing method, a commodity query method, a question and answer processing method, a computer device, and a computer-readable storage medium. Background Art

[0002] With the development of information services, automatic question-and-answer systems have appeared in some application systems. These automatic question-and-answer systems can automatically match questions raised by users based on the preset question-and-answer database and give corresponding answers. This can improve the efficiency of information consultation to a certain extent and reduce the manpower costs of enterprises.

[0003] For example, the most common retrieval method in the field of automatic question answering is question-answer pair retrieval. However, on the one hand, question-answer pairs need to be manually refined and summarized, which is very cumbersome. It is necessary to list the questions in news, encyclopedias, and business documents one by one, and then manually write the answers. On the other hand, humans can often only list high-frequency questions and cannot cover long-tail question-answer pairs well.

[0004] The applicant has found through research that the existing automatic question-answering technologies, whether they are retrieval-based automatic question-answering or automatic question-answering based on machine reading comprehension, can only match text-based answers, which are relatively simple and may even have incomplete answers. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide an information providing method, a product query method, a question and answer processing method, a computer device, and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0006] According to one aspect of the present application, there is provided a method for providing information, comprising:

[0007] Get the user's query request;

[0008] Based on the query request, determining a corresponding query image;

[0009] Matching the identified text in the query image with the query request to determine a matching query image;

[0010] The matched query image is provided to the user.

[0011] Optionally, providing the matching query image to the user includes:

[0012] Extracting a picture area related to the query request in the matched query picture as a sub-picture for the query request;

[0013] The sub-picture is provided to the user.

[0014] According to another aspect of the present application, a commodity query method is provided, comprising:

[0015] Get the user's product query request;

[0016] Based on the product query request, determine a corresponding product image;

[0017] Matching the identified text in the product image with the product query request to determine a matching product image;

[0018] The matching product images are provided to the user.

[0019] According to another aspect of the present application, a question-answering processing method is provided, comprising:

[0020] Get the user's question and answer request;

[0021] From the image library, matching the text in the identified query image with the question-answering request to determine a matching query image;

[0022] The matched query image is provided to the user as a result corresponding to the question-answering request.

[0023] According to another aspect of the present application, there is provided a method for providing information, comprising:

[0024] Extracting information data of the picture, wherein the information data includes picture text data and picture original data;

[0025] Determine answer text data matching the question text data in the picture text data according to the information data;

[0026] A picture area in the picture related to the answer text data is extracted as a sub-picture for the question text data, and the sub-picture is provided.

[0027] Optionally, the information data of the extracted picture includes:

[0028] Identifying image text data in the image;

[0029] The image text data is preprocessed, and the preprocessing includes at least one of word segmentation and filtering.

[0030] Optionally, before extracting the information data of the picture, the method further includes:

[0031] Receiving the question text data;

[0032] The picture is obtained according to the data object targeted by the question text data.

[0033] Optionally, determining answer text data matching the question text data in the picture text data according to the information data comprises:

[0034] The answer text data is determined based on the original image data, the image text data, and the question text data using a text positioning data set.

[0035] Optionally, before determining the answer text data using a text positioning data set based on the original picture data, the picture text data, and the question text data, the method includes:

[0036] Based on the original image data samples, the image text data samples with answer text position labels, and the question text data samples, a text positioning data set for positioning the answer text data is created.

[0037] Optionally, the creating a text positioning data set for positioning the answer text data based on the original picture data sample, the picture text data sample with the answer text position label, and the question text data sample comprises:

[0038] Generate a picture vector of the picture original data sample, a picture text vector of the picture text data sample, and a question text vector of the question text data sample;

[0039] Combining the picture vector and the picture text vector to obtain a picture text vector;

[0040] The text positioning data set is created based on the image-text vector, question text vector and answer text position label.

[0041] Optionally, the determining the answer text data using a text positioning data set according to the original image data, the image text data, and the question text data comprises:

[0042] According to the original image data, the image text data, and the question text data, the text positioning data set is used to obtain at least one candidate position of the answer text data and a corresponding candidate confidence;

[0043] Comparing the question text data and the picture text data, obtaining a no-candidate confidence level that the answer text data cannot be located in the picture text data;

[0044] Determine answer candidate positions corresponding to candidate confidences greater than the no-candidate confidence, and filter out the answer text data from the answer candidate positions.

[0045] Optionally, extracting a picture area in the picture related to the answer text data as a sub-picture for the question text data comprises:

[0046] Detecting content-related areas in the picture, and dividing them to obtain at least one picture area;

[0047] Extracting the image region where the answer text data is located as the image region related to the answer text data;

[0048] According to the image area, the image is cropped to obtain a sub-image.

[0049] Optionally, detecting a content-related area in the picture and dividing it to obtain at least one picture area includes:

[0050] Detecting at least one object region and at least one text region in the image;

[0051] merging at least one related region among the at least one object region and the at least one text region;

[0052] The at least one picture region is obtained by dividing according to the at least one object region, the at least one text region and the at least one related region.

[0053] Optionally, detecting at least one object region in the object picture includes:

[0054] Performing object detection on the image to obtain an object target area;

[0055] Dividing the image into multiple super-pixel regions according to pixel features of the image;

[0056] The object target region and a superpixel region related to the object target region are merged to obtain at least one object region.

[0057] Optionally, detecting at least one text area in the object picture includes:

[0058] Performing text detection on the image to obtain a text target area;

[0059] The text target regions whose distance or similarity between regions meets a preset condition are merged to obtain the at least one text region.

[0060] Optionally, before detecting at least one object region and at least one text region in the image, the method includes:

[0061] Detecting edge information in the image;

[0062] Dividing the image into at least one sub-region according to the edge information;

[0063] The detecting at least one object region and at least one text region in the image comprises:

[0064] At least one object region and at least one text region in each sub-region are detected respectively.

[0065] Optionally, before providing the sub-picture, the method further includes:

[0066] Image processing is performed on the answer text data in the sub-image so that the answer text data has an obvious display effect.

[0067] According to another aspect of the present application, there is provided a method for providing information, comprising:

[0068] Get the original image data;

[0069] Determine the image target data matching the problem data in the image original data;

[0070] A picture area in the picture related to the picture target data is extracted as an answer picture for the question data, and the answer picture is provided.

[0071] According to another aspect of the present application, a question-answering processing method is provided, comprising:

[0072] Get the original data of the question and the image;

[0073] Determine the image target data in the image original data that matches the problem data;

[0074] Extracting a picture region in the picture related to the picture target data as an answer picture for the question data;

[0075] Provide a picture of the answer.

[0076] According to another aspect of the present application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements one or more of the above methods when executing the computer program.

[0077] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, one or more of the above methods are implemented.

[0078] According to the embodiment of the present application, when processing the match between texts, in addition to using the picture text data in the picture, the original picture data itself is also combined, avoiding the problem of missing information other than text in the picture, improving the accuracy of determining the position of the answer text data in the picture, and using the picture area related to the answer text data in the picture as a sub-picture for the question text data. Compared with only providing the answer text data, other relevant information in the picture besides the answer text data can also be provided, thereby achieving the effect of providing picture-type information for the question text data and improving the completeness of the information provided for the question text data.

[0079] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0081] Figure 1 A schematic diagram of a reading comprehension model is shown;

[0082] Figure 2 A schematic diagram showing the information provision process;

[0083] Figure 3 A flowchart of an information providing method embodiment according to Embodiment 1 of the present application is shown;

[0084] Figure 4 A flowchart of a commodity query method embodiment according to the second embodiment of the present application is shown;

[0085] Figure 5 A flowchart of a question-answering processing method according to Embodiment 3 of the present application is shown;

[0086] Figure 6 A flowchart of an information providing method embodiment according to Embodiment 4 of the present application is shown;

[0087] Figure 7 A flowchart of an information providing method embodiment according to Embodiment 5 of the present application is shown;

[0088] Figure 8 A flowchart of an information providing method embodiment according to Embodiment 6 of the present application is shown;

[0089] Fig. 9 A flowchart of an information providing method embodiment according to Embodiment 7 of the present application is shown;

[0090] Fig.10 A flowchart of an embodiment of a question-answering processing method according to Embodiment 8 of the present application is shown;

[0091] Fig.11 A schematic diagram of a picture cropping process is shown;

[0092] Fig.12 A structural block diagram of an information providing device according to Embodiment 9 of the present application is shown;

[0093] Fig.13 A structural block diagram of a commodity query device embodiment according to Embodiment 10 of the present application is shown;

[0094] Fig.14 A structural block diagram of an embodiment of a question-answer processing device according to the eleventh embodiment of the present application is shown;

[0095] Fig.15 A structural block diagram of an information providing device according to Embodiment 12 of the present application is shown;

[0096] Fig.16 A structural block diagram of an information providing device according to Embodiment 13 of the present application is shown;

[0097] Fig.17 A structural block diagram of an embodiment of a question-answer processing device according to Embodiment 14 of the present application is shown;

[0098] Fig.18 An exemplary system is shown that can be used to implement the various embodiments described in this disclosure. DETAILED DESCRIPTION

[0099] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0100] In order to enable those skilled in the art to better understand the present application, the concepts involved in the present application are explained below:

[0101] The information data of the picture includes picture text data, picture original data, etc., for example, the picture original data of the picture itself, the picture text data obtained by text recognition of the picture, or any other applicable information data, which is not limited in the embodiments of the present application.

[0102] Information data is used to provide matching data for question text data. Question text data includes but is not limited to questions submitted by users in automatic question-answering systems, and text entered by users in human-computer dialogue systems. For example, in the customer service system of an e-commerce platform, a buyer inquires about a certain product and asks a question, which is question text data. After receiving the question raised by the buyer, the specific product inquired by the buyer is located. According to the product ID, the product details page image can be found in the database, and the product details page image can be recognized by text to obtain the text therein. The details page image and the text therein are information data.

[0103] The text data in the image text data that matches the question text data is recorded as the answer text data. For example, the image text data is the introduction text in the image of the product details page, and the question text data is a question about the shape of the product. By comparing the image text data and the question text data, it can be found that one of the sentences in the image text data is about the shape of the product. This sentence matches the question text data and can be used as the answer text data.

[0104] In addition to the answer text data, the image may also include an image area related to the answer text data. For example, in the product details page image, the product is usually introduced in a combination of pictures and texts, and the sentences introducing the shape of the product are accompanied by relevant product photos or structural diagrams. A simple approach is to use the area in the image that is closer to the answer text data as the image area related to the answer text data. Determining the image area related to the answer text data based on the distance is only one implementation method, and any applicable method may also be included. The embodiments of the present application do not limit this.

[0105] The image area in the image is extracted as another image, recorded as a sub-image. Since the answer text data matches the question text data, and the image area is related to the answer text data, a sub-image can be provided for the question text data. For example, for a question about the shape of a product asked by a user, a sub-image can be provided to answer the question.

[0106] In an optional embodiment of the present application, the answer text data can be determined by using a text positioning data set. The text positioning data set can locate the answer text data that matches the question text data in the image text data. The text positioning data set allows the machine to read the text and automatically match it based on the understanding of the inherent meaning of the text, that is, the input is the original image data, the image text data, and the question text data, and the output is the answer text data.

[0107] For example, Figure 1 The schematic diagram of the reading comprehension model (i.e., text location data set) shown in the figure has an image input in the image modality in addition to the document Context (i.e., image text data) and question Query (i.e., question text data) inputs in the text modality. The image is encoded using a convolutional neural network (CNN) and the text is encoded using a recurrent neural network (RNN). The reading comprehension model mainly consists of a CNN Encoder layer, an Embedding layer, an RNN Encoder layer, an Attention layer, and the final prediction layer. The inputs are the product detail page image, the text identified in the detail page image, and the question about the product. The output is the start and end positions of the answer in the text and the confidence of the answer.

[0108] In an optional embodiment of the present application, in order to create a text location data set that can locate answer text data, a supervised learning method is required to train based on original image data samples, image text data samples with answer text location labels, and question text data samples to obtain a text location data set. Figure 1 The reading comprehension model shown can be trained using manually annotated data based on the back-propagation algorithm.

[0109] In an optional embodiment of the present application, in the process of creating a text positioning data set, in order to allow the original image data to affect the result, one implementation method is to combine the image vector of the original image data sample and the image text vector of the image text data sample, and the obtained image vector is affected by both the image text data sample and the original image data sample, and can represent the image text data sample and the original image data sample. Then, based on the image vector, the question text vector of the question text data sample and the answer text position label, a text positioning data set is created.

[0110] For example, Figure 1In the reading comprehension model shown in the figure, during the model training process, the original data sample of the image is encoded into a vector through a convolutional neural network, and the image text data sample is encoded into a vector through a recurrent neural network. The two vectors are added together to obtain a vector, which is recorded as the image-text vector. The reading comprehension model may include a multi-layer structure, and the vector of the image can be added to the vector of the text at any one or more layers. The specific settings can be made according to actual needs. Different settings have different effects on the positioning results of the answer text data, but they can all be combined with the original data of the image itself to avoid the problem of missing other information in the image except the text, and improve the accuracy of determining the location of the answer text data in the image.

[0111] In an optional embodiment of the present application, in the process of determining the answer text data, the text location data set can generate at least one candidate position of the answer text data in the image text data, and the corresponding candidate confidence, the candidate confidence is used to characterize the credibility of the text data at the corresponding candidate position as the answer text data, which can also be said to be a probability. In addition to the situation where the answer text data that matches the question text data exists in the image text data, there is also a situation where the answer text data cannot be located in the image text data. The confidence in this situation is recorded as no candidate confidence.

[0112] The no-candidate confidence can be obtained by comparing the question text data and the image text data. For example, the vector corresponding to the image text data and the vector corresponding to the question text data are converted into two vectors of the same length, and then the inner product of these two vectors of the same length is performed, and the obtained number is used as the no-candidate confidence, that is, the correlation between the question text data and the image text data is calculated.

[0113] In an optional embodiment of the present application, when dividing an image into regions, the image can be divided into multiple super-pixel regions according to the pixel features of the image. Pixel features include but are not limited to features such as texture, color, and brightness, and a super-pixel region refers to an irregular pixel block with certain visual significance composed of adjacent pixels with similar pixel features.

[0114] For example, SLIC (Simple Linear Iterative Clustering) segmentation converts color images into 5-dimensional feature vectors in CIELAB color space and XY coordinates, and then constructs a distance metric for the 5-dimensional feature vector to perform local clustering of image pixels. The SLIC algorithm can generate compact, approximately uniform superpixel regions, which helps to cut low-texture areas in product images.

[0115] In an optional embodiment of the present application, some images have some edge information, and the image information that can be used to divide the image into several areas is recorded as edge information. For example, some product detail page images are obtained by simple splicing, and some images can directly detect the segmentation line at the splicing point, that is, edge information, by edge extraction.

[0116] According to an embodiment of the present application, when providing information for question text data, only text-based answers can be matched, which are relatively simple and may even have incomplete answers. Figure 2 The schematic diagram of the information provision process shown in the figure, the present application provides an information provision mechanism, by extracting the information data of the picture, the information data includes the picture text data and the picture original data, according to the information data, determining the answer text data in the picture text data that matches the question text data, extracting the picture area related to the answer text data in the picture as a sub-picture for the question text data, and providing the sub-picture, so that when processing the matching between texts, in addition to using the picture text data in the picture, the picture original data itself is also combined, avoiding the problem of missing other information in the picture except the text, improving the accuracy of determining the position of the answer text data in the picture, and using the picture area related to the answer text data in the picture as a sub-picture for the question text data, compared with only providing the answer text data, other relevant information in the picture except the answer text data can also be provided, achieving the effect of providing picture-type information for the question text data, and improving the integrity of the information provided for the question text data. The present application is applicable to but not limited to the above application scenarios.

[0117] Reference Figure 3 , shows a flow chart of an information providing method embodiment according to Embodiment 1 of the present application, the method may specifically include the following steps:

[0118] Step 101: Obtain a user's query request.

[0119] In an embodiment of the present application, a user's query request includes a user's request to query certain information, for example, a question submitted by a user in a customer service system regarding a certain product, or any other applicable query request, which is not limited in this embodiment of the present application.

[0120] Step 102: Determine a corresponding query image based on the query request.

[0121] In an embodiment of the present application, based on a query request, a corresponding query image can be determined. There are many specific implementation methods. For example, before the query request, the object targeted by the user's query request is obtained first. Then, after obtaining the query request, one or more query images of the object can be determined, or the query request is parsed, and the query request includes the name, identifier, or link of an object, based on which one or more query images of the object can be determined, or any other applicable implementation method, which is not limited by the embodiment of the present application.

[0122] Step 103: Match the identified text in the query image with the query request to determine a matching query image.

[0123] In an embodiment of the present application, text recognition is performed on the query image, and the text in the query image can be identified. The identified text is matched with the query request to determine the matching query image. The specific implementation method of the matching may include multiple methods, for example, using a text positioning data set, taking the identified text and the question text data in the query request as input, and obtaining the starting and ending positions of the answer text data in the image text data. If the answer text data can be obtained, it is determined that the query image matches the query request; or calculating the similarity between the text in multiple query images and the text in the query request, and determining the query image corresponding to the text with the highest similarity as the matching query image. Specifically, any applicable implementation method may be included, and the embodiment of the present application does not limit this.

[0124] Step 104: providing the matched query image to the user.

[0125] In an embodiment of the present application, optionally, an implementation method of providing the matching query image to the user may include: extracting an image area related to the query request in the matching query image as a sub-image for the query request; and providing the sub-image to the user.

[0126] Some parts of the query image are not relevant to the query request, and providing the entire query image to the user is not accurate. To this end, the image area related to the query request in the matched query image is extracted as a sub-image for the query request, for example, the image area related to the answer text data in the query image is used as the sub-image for the query request.

[0127] According to an embodiment of the present application, a query request from a user is obtained; based on the query request, a corresponding query image is determined; the identified text in the query image is matched with the query request to determine a matching query image; and the matching query image is provided to the user, thereby overcoming the limitation of only being able to query text, achieving the effect of providing image-type information in response to the query request, and improving the completeness of the information provided in response to the query request.

[0128] Reference Figure 4 , shows a flow chart of a commodity query method embodiment according to the second embodiment of the present application, the method may specifically include the following steps:

[0129] Step 201: Obtain a product query request from a user.

[0130] In an embodiment of the present application, a user's product query request includes a user's request to query certain information about a product, for example, a question submitted by a user in a customer service system regarding a certain product, or any other applicable product query request, which is not limited in this embodiment of the present application.

[0131] Step 202: Determine corresponding product images based on the product query request.

[0132] In the embodiment of the present application, based on the product query request, the product targeted by the product query request can be determined, and one or more product images of the product can be used as the corresponding product images. For example, all images in the product details page can be determined as the corresponding product images.

[0133] Step 203: Match the identified text in the product image with the product query request to determine a matching product image.

[0134] In the embodiment of the present application, the specific implementation method of this step can be found in the description in the aforementioned embodiment and will not be further described here.

[0135] Step 204: providing the matched product images to the user.

[0136] In the embodiment of the present application, the specific implementation method of this step can be found in the description in the aforementioned embodiment and will not be further described here.

[0137] According to an embodiment of the present application, a product query request from a user is obtained; based on the product query request, a corresponding product image is determined; the text in the identified product image is matched with the product query request to determine a matching product image; and the matching product image is provided to the user, thereby overcoming the limitation of only being able to query text, achieving the effect of providing image information in response to the product query request, and improving the completeness of the information provided in response to the product query request.

[0138] Reference Figure 5 , shows a flowchart of an embodiment of a question-answering processing method according to Embodiment 3 of the present application, and the method may specifically include the following steps:

[0139] Step 301: Obtain the user's question and answer request.

[0140] In an embodiment of the present application, a user's question and answer request includes a user's request for information consultation regarding a certain product. For example, a user submits a question regarding a certain product in a customer service system, submits a question and answer request to the customer service system, or any other applicable question and answer request. This embodiment of the present application does not limit this.

[0141] Step 302: From the image library, match the text in the identified query image with the question-answer request to determine a matching query image.

[0142] In an embodiment of the present application, a plurality of query images are included in the image library. Text recognition is performed on the query images in the image library to identify the text in the query images. The identified text is matched with the question and answer request to determine the matching query image. The specific implementation method of the matching may include multiple methods, for example, using a text positioning data set, taking the identified text and the question text data in the question and answer request as input, and obtaining the starting and ending positions of the answer text data in the image text data. If the answer text data can be obtained, it is determined that the query image matches the question and answer request; or calculating the similarity between the text in multiple query images and the text in the question and answer request, and determining the query image corresponding to the text with the highest similarity as the matching query image. Specifically, any applicable implementation method may be included, and the embodiment of the present application does not limit this.

[0143] Step 303: Provide the matched query image to the user as a result corresponding to the question-answering request.

[0144] In the embodiment of the present application, since the text in the query image matches the question and answer request, the matched query image can be used as the result corresponding to the question and answer request and provided to the user.

[0145] According to an embodiment of the present application, a question and answer request of a user is obtained; the text in an identified query image is matched with the question and answer request from an image library to determine a matching query image; and the matching query image is provided to the user as a result corresponding to the question and answer request. This overcomes the limitation that only text questions can be answered, achieves the effect of providing image information in response to the question and answer request, and improves the completeness of the information provided in response to the question and answer request.

[0146] Reference Figure 6, shows a flow chart of an information providing method embodiment according to Embodiment 4 of the present application, the method may specifically include the following steps:

[0147] Step 401, extracting information data of a picture, wherein the information data includes picture text data and picture original data.

[0148] In the embodiment of the present application, the information data of the picture includes picture text data and picture original data. For the picture text data, the picture can be subjected to text recognition, and the text obtained can be extracted as the picture text data. The recognized text data can also be subjected to certain preprocessing, and the preprocessed text data can be used as the picture text data. The picture text data is not necessarily all the text in the picture, which depends on the text recognition and other processing of the picture. The present application does not limit the implementation method of extracting the picture text data of the picture.

[0149] As for the original data of the image, the image itself can be directly extracted as the original data of the image, or the image can be processed in a certain way first and then the processed original data of the image can be extracted. For example, the resolution, file size, and dimensions of the image can be processed so that the image meets the input requirements. The specific method of extracting the original data of the image can be determined according to actual needs, and the embodiments of the present application do not limit this.

[0150] Step 402: Determine answer text data matching the question text data in the picture text data based on the information data.

[0151] In the embodiment of the present application, since it is necessary to find answer text data that matches the question text data from the picture, in addition to the picture text data, other information on the picture also has reference significance. It is not possible to only match the picture text data with the question text data. It is necessary to determine the answer text data in the picture text data that matches the question text data based on the information data, that is, the picture text data and the original picture data.

[0152] For example, by using a text positioning data set, taking the image text data, the original image data and the question text data as input, the starting and ending positions of the answer text data in the image text data are obtained, that is, the answer text data is determined.

[0153] In an embodiment of the present application, in an implementation method for determining answer text data matching question text data in image text data based on information data, it is necessary to create a text positioning data set for positioning the answer text data based on the original image data samples, the image text data samples with answer text position labels, and the question text data samples in advance. When it is necessary to determine the matching answer text data for the question text data, the original image data, the image text data, and the question text data are input to perform calculations through the text positioning data set to obtain the answer text data. Specifically, any applicable implementation method may be included, and the embodiment of the present application does not limit this.

[0154] Step 403: extract the image area related to the answer text data in the image as a sub-image for the question text data, and provide the sub-image.

[0155] In the embodiment of the present application, compared to providing only text information for question text data, in order to provide more complete information, relevant image areas in the image may be provided.

[0156] In an embodiment of the present application, the implementation methods of extracting the image area related to the answer text data in the image as a sub-image for the question text data may include multiple methods, for example, detecting the content-related area in the image, dividing it to obtain at least one image area, extracting the image area where the answer text data is located as the image area related to the answer text data, and cropping the image to obtain a sub-image based on the image area, or detecting the image area related to the answer text data in the image, extracting the image area, and cutting it to obtain a sub-image, or any other applicable implementation method, which is not limited to the embodiments of the present application.

[0157] According to the embodiment of the present application, when processing the match between texts, in addition to using the picture text data in the picture, the original picture data itself is also combined, avoiding the problem of missing information other than text in the picture, improving the accuracy of determining the position of the answer text data in the picture, and using the picture area related to the answer text data in the picture as a sub-picture for the question text data. Compared with only providing the answer text data, other relevant information in the picture besides the answer text data can also be provided, thereby achieving the effect of providing picture-type information for the question text data and improving the completeness of the information provided for the question text data.

[0158] Reference Figure 7 , shows a flow chart of an information providing method embodiment according to Embodiment 5 of the present application, the method may specifically include the following steps:

[0159] Step 501, receiving the question text data.

[0160] In the embodiment of the present application, the question text data includes questions submitted by users in the automatic question-answering system, context input by users in the human-computer dialogue system, etc. For example, the question Query input by the user is received, that is, the question text data.

[0161] Step 502: Acquire the picture according to the data object targeted by the question text data.

[0162] In the embodiment of the present application, the question text data may be submitted for the data object, so the image corresponding to the data object may match useful information for the question text data. For example, the question Query entered by the user is about a certain product, and the product detail page image is found in the database according to the product ID.

[0163] Step 503, identifying the picture text data in the picture.

[0164] Step 504: preprocess the image text data, wherein the preprocessing includes at least one of word segmentation and filtering.

[0165] In an embodiment of the present application, the image text data identified in the image needs to be preprocessed before it can be used. The preprocessing includes at least one of word segmentation and filtering. The embodiment of the present application does not limit the processing method included in the preprocessing.

[0166] Among them, the text is segmented to obtain corresponding multiple words. English word segmentation is relatively simple, because there are spaces between English words, while Chinese word segmentation is more complicated. There are many specific word segmentation processing methods, such as a word segmentation method based on character matching, a word segmentation method based on word frequency statistics, or a rule-based word segmentation method. Any applicable word segmentation method is specifically adopted, and the embodiments of the present application are not limited to this. Filtering the text includes filtering out meaningless words in the text, for example, sentences with very few words, sentences with many typos, etc., or other any applicable filtering methods, which are not limited to the embodiments of the present application.

[0167] Step 505: Determine the answer text data using a text positioning data set based on the original image data, the image text data, and the question text data.

[0168] In an embodiment of the present application, when a text positioning data set is used to match answer text data, the input data is picture original data, picture text data, and question text data. The picture original data, picture text data, and question text data are first encoded to obtain corresponding vectors, the picture text data and the picture vector are combined, and then the combined vector and the vector corresponding to the question text data are interactively processed word by word, all combinations of start position and end position in the picture text data are matched with the question text data, and finally the start position and end position of the answer text data are determined.

[0169] In an embodiment of the present application, optionally, an implementation method of determining the answer text data using a text positioning data set according to the original image data, the image text data, and the question text data may include:

[0170] Sub-step S1, obtaining at least one candidate position of the answer text data and a corresponding candidate confidence using the text positioning data set according to the original image data, the image text data, and the question text data;

[0171] Sub-step S2, comparing the question text data and the picture text data, and obtaining a no-candidate confidence level that the answer text data cannot be located in the picture text data;

[0172] Sub-step S3, determining the answer candidate positions corresponding to the candidate confidences greater than the no-candidate confidences, and filtering out the answer text data from the answer candidate positions.

[0173] In the process of matching the answer text data with the text location data set, at least one candidate position of the answer text data and the corresponding candidate confidence are generated. The candidate confidence of at least one candidate position can usually be sorted from large to small, and the candidate position with a larger candidate confidence is more likely to be used as the answer text data matching the question text data.

[0174] To further ensure the accuracy of the answer text data, the question text data and the image text data may be compared to obtain the no-candidate confidence that the answer text data cannot be located in the image text data, that is, the degree to which the image text data is irrelevant to the image text data.

[0175] Then, the candidate confidence and the no-candidate confidence are compared. If the candidate confidence of at least one candidate position is less than the no-candidate confidence, it means that the answer text data that matches the question text data cannot be located from the image text data, and the process of determining the answer text data from the image text data ends. Other images and image text data can be used for determination. If there is a candidate confidence greater than the no-candidate confidence, the answer text data is screened out from the corresponding answer candidate position.

[0176] Step 506: Detect content-related areas in the picture and divide them into at least one picture area.

[0177] In an embodiment of the present application, relevant content in a picture is detected, and the relevant content is divided into an area to divide the picture into at least one picture area. For example, if a text in a picture is detected to describe the shape of a product, and there is a photo or a structural diagram of the product next to the text, then the text and the photo or structural diagram of the product next to it are related content and can be divided into one picture area. There may be multiple ways to detect a picture and divide it into at least one picture area, which is not limited by the embodiment of the present application.

[0178] In the embodiment of the present application, optionally, an implementation method of detecting a content-related area in the picture and dividing it to obtain at least one picture area may include:

[0179] Sub-step S4, detecting at least one object region and at least one text region in the image;

[0180] Sub-step S5, merging at least one relevant region of the at least one object region and the at least one text region;

[0181] Sub-step S6, dividing the at least one picture region according to the at least one object region, the at least one text region and the at least one related region.

[0182] The picture includes multiple parts such as objects, texts, and backgrounds. At least one object area and at least one text area are obtained through detection. Some object areas and some text areas are related contents, and the areas obtained by merging are recorded as related areas. For example, if the distance between a text area and an object area on the picture is less than a preset distance threshold, the text area and the object area can be merged, or the name of the object included in the object area can be identified. If the object name appears in the text in the text area, the text area and the object area are related contents, and can be merged to obtain a related area, or at least one object area and at least one text area can be regarded as multiple sub-graphs of the picture, and the connected sub-graphs therein are merged as related areas. Any applicable merging method can be adopted according to actual needs, and the embodiments of the present application do not limit this.

[0183] Then the image is divided into at least one unmerged object region, at least one unmerged text region, and at least one merged related region, thus completing the division of the image.

[0184] In an embodiment of the present application, optionally, an implementation manner of detecting at least one object region in the object picture may include:

[0185] Sub-step S7, performing object detection on the image to obtain an object target area;

[0186] Sub-step S8, dividing the image into a plurality of super-pixel regions according to the pixel features of the image;

[0187] Sub-step S9, merging the object target region and the superpixel region related to the object target region to obtain at least one object region.

[0188] By performing object detection on an image, we can segment the area where the object is located in the image, which is recorded as the object target area. However, in general, the edge processing of the object target area determined by object detection is poor, and it is difficult to segment the area where the object is located exactly. Based on the pixel characteristics of the image, the image is divided into multiple superpixel areas, which can better maintain the outline of the object. Therefore, the object target area and the superpixel area can be combined to obtain a better object area.

[0189] The object target region and the superpixel region related to the object target region are merged. For example, if the edge of the object target region passes through a superpixel region, the superpixel region can be merged with the object target region, and the combination of the two regions is taken as the object region. The embodiment of the present application does not limit the method of merging the object target region and the superpixel region.

[0190] In an embodiment of the present application, optionally, an implementation manner of detecting at least one text area in the object picture may include:

[0191] Sub-step S10, performing text detection on the image to obtain a text target area;

[0192] Sub-step S11, merging text target regions whose distance or similarity between regions meets a preset condition to obtain the at least one text region.

[0193] When performing text detection on a picture, multiple text areas in the picture can be detected and recorded as text target areas. In order to prevent the text information in the picture from being cut too scattered, the text target area can be hierarchically clustered, and the areas can be merged according to the distance or similarity between the areas. For example, the areas whose distance between the areas is less than a preset distance threshold are merged, or the texts in the areas are compared for similarity, and the areas corresponding to the texts whose similarity is higher than a preset similarity threshold are merged to obtain at least one text area. The preset conditions can be set according to actual needs, and the embodiments of the present application do not limit this.

[0194] In the embodiment of the present application, optionally, before detecting at least one object region and at least one text region in the picture, the following may also be included:

[0195] Sub-step S12, detecting edge information in the image;

[0196] Sub-step S13, dividing the image into at least one sub-region according to the edge information;

[0197] Sub-step S14, detecting at least one object region and at least one text region in the image includes:

[0198] Sub-step S15, detecting at least one object region and at least one text region in each sub-region respectively.

[0199] Some images have edge information and can be divided into multiple regions. Before detecting at least one object region and at least one text region in the image, it is necessary to first divide the image into at least one sub-region based on the edge information, and then detect at least one object region and at least one text region in each sub-region respectively, so as to avoid the problem of being unable to separate the regions divided by the image itself when detecting the object region or the text region.

[0200] Step 507: extract the image region where the answer text data is located as the image region related to the answer text data.

[0201] In the embodiment of the present application, after the image is divided into at least one image area, the corresponding image area is determined according to the answer text data. For example, the starting position of the answer text data can be obtained by using a text positioning data set. The starting position in the image text data can correspond to the coordinates in the image, that is, the image area where the answer text data is located can be determined. Since the content in the image area is related, the image area can be used as the image area related to the answer text data.

[0202] Step 508: crop the image to obtain sub-images according to the image area.

[0203] In an embodiment of the present application, a picture area in a picture is cropped to obtain a sub-picture.

[0204] Step 509: perform image processing on the answer text data in the sub-image so that the answer text data has an obvious display effect.

[0205] In an embodiment of the present application, in order to make the answer text data have an obvious display effect, the answer text data in the sub-picture can be processed by image processing, for example, highlighting, underlining, changing obvious colors, etc. The embodiment of the present application does not limit the method of image processing.

[0206] Step 510: providing the sub-picture.

[0207] According to an embodiment of the present application, by receiving question text data, according to the data object targeted by the question text data, the picture is obtained, the picture text data in the picture is identified, and the picture text data is preprocessed, the preprocessing includes at least one of word segmentation and filtering, according to the original picture data, the picture text data, and the question text data, the answer text data is determined using a text positioning data set, the content-related area in the picture is detected, at least one picture area is obtained by dividing, the picture area where the answer text data is located is extracted as the picture area related to the answer text data, according to the picture area, the picture is cropped to obtain a sub-picture, and the sub-picture is provided. When processing the matching between texts, in addition to using the picture text data in the picture, the original picture data itself is also combined to avoid the problem of missing other information in the picture except the text, and the accuracy of determining the position of the answer text data in the picture is improved, and the picture area related to the answer text data in the picture is used as the sub-picture for the question text data. Compared with only providing the answer text data, other relevant information in the picture except the answer text data can also be provided, so as to achieve the effect of providing picture-type information for the question text data and improve the completeness of the information provided for the question text data.

[0208] Furthermore, by performing image processing on the answer text data in the sub-picture, the answer text data has an obvious display effect.

[0209] Furthermore, at least one candidate position of the answer text data and a corresponding candidate confidence are obtained by using the text positioning data set according to the original image data, the image text data, and the question text data; the question text data and the image text data are compared to obtain a no-candidate confidence that the answer text data cannot be located in the image text data; the answer candidate position corresponding to the candidate confidence that is greater than the no-candidate confidence is determined, and the answer text data is filtered out from the answer candidate position, so that the answer text data finally determined is verified by the no-candidate confidence, thereby avoiding the problem of too low confidence of the answer text data and improving the accuracy of determining the answer text data from the image text data.

[0210] Reference Figure 8 , shows a flow chart of an information providing method embodiment according to Embodiment 6 of the present application, the method may specifically include the following steps:

[0211] Step 601: Generate a picture vector of the picture original data sample, a picture text vector of the picture text data sample, and a question text vector of the question text data sample.

[0212] In an embodiment of the present application, in order to create a text positioning data set, the original data sample of the image is first encoded to obtain a picture vector, the picture text data sample is encoded to obtain a picture text vector, and the question text data sample is encoded to obtain a question text vector.

[0213] Step 602: Combine the picture vector and the picture text vector to obtain a picture text vector.

[0214] In the embodiment of the present application, in order to allow the original image data to affect the result during the machine learning process, the image vector and the image text vector are combined to obtain the image text vector. For example, the image vector and the image text vector (i.e., the vector of each word of the image text data sample) are added to obtain the image text vector.

[0215] Step 603: Create the text positioning data set based on the image-text vector, question text vector and answer text position label.

[0216] In an embodiment of the present application, the implementation method of creating a text positioning data set based on image-text vectors, question text vectors and answer text position labels is the same as the implementation method of creating a text positioning data set based on picture text vectors, question text vectors and answer text position labels.

[0217] Step 604: extract information data of the picture, where the information data includes picture text data and picture original data.

[0218] In the embodiment of the present application, the specific implementation method of this step can be found in the description in the aforementioned embodiment and will not be further described here.

[0219] Step 605, based on the original image data, image text data, and question text data, the answer text data is determined using a text positioning data set.

[0220] In the embodiment of the present application, the specific implementation method of this step can be found in the description in the aforementioned embodiment and will not be further described here.

[0221] Step 606: extract the image area related to the answer text data in the image as a sub-image for the question text data, and provide the sub-image.

[0222] In the embodiment of the present application, the specific implementation method of this step can be found in the description in the aforementioned embodiment and will not be further described here.

[0223] According to an embodiment of the present application, a text positioning data set for locating the answer text data is created based on a picture original data sample, a picture text data sample with an answer text position label, and a question text data sample, and information data of the picture is extracted, the information data including picture text data and picture original data. According to the picture original data, picture text data, and question text data, the answer text data is determined using a text positioning data set, and a picture area in the picture related to the answer text data is extracted as a sub-picture for the question text data, and a sub-picture is provided, so that when processing the matching between texts, in addition to using the picture text data in the picture, the picture original data itself is also combined, avoiding the problem of missing information other than text in the picture, improving the accuracy of determining the position of the answer text data in the picture, and using the picture area in the picture related to the answer text data as a sub-picture for the question text data. Compared with only providing the answer text data, other relevant information in the picture except the answer text data can also be provided, thereby achieving the effect of providing picture-type information for the question text data and improving the completeness of the information provided for the question text data.

[0224] Reference Fig. 9 , shows a flow chart of an information providing method embodiment according to Embodiment 7 of the present application, the method may specifically include the following steps:

[0225] Step 701, obtaining original image data.

[0226] In an embodiment of the present application, in order to automatically answer questions using image-type data, the original image data may be obtained first. For example, if the question data is for a certain product, then one or more original image data related to the product may be obtained.

[0227] Step 702: determine the image target data in the original image data that matches the problem data.

[0228] In the embodiment of the present application, the question data may include multiple forms such as text, voice, video, or any other applicable question data, which is not limited in the embodiment of the present application. The original data of the picture is matched with the question data, and the matched data is recorded as the picture target data. The picture target data includes the answer text data in the picture, the object in the picture, the color block in the picture, etc., or any other applicable picture target data, which is not limited in the embodiment of the present application.

[0229] In an embodiment of the present application, there may be multiple ways to determine the image target data in the image original data that matches the problem data. For example, first, based on the image original data samples and the problem data samples with image target data labels, a positioning data set for positioning the image target data is created, and then the image target data is determined using the positioning data set based on the image original data and the problem data, or any other applicable implementation method, which is not limited in the embodiment of the present application.

[0230] Step 703: extract a picture area in the picture related to the picture target data as an answer picture for the question data, and provide the answer picture.

[0231] In an embodiment of the present application, in order to provide more complete information, a picture area in the picture related to the picture target data can be provided as an answer picture for the question data.

[0232] In an embodiment of the present application, there may be multiple ways to extract a picture area related to picture target data in a picture as an answer picture for question data, for example, detecting a content-related area in a picture, dividing it to obtain at least one picture area, extracting the picture area where the picture target data is located as the picture area related to the picture target data, and cropping the picture to obtain an answer picture based on the picture area, or detecting a picture area related to the picture target data in a picture, extracting the picture area, and cropping to obtain an answer picture, or any other applicable implementation method, which is not limited in the embodiment of the present application.

[0233] According to the embodiments of the present application, by obtaining the original data of the image, determining the image target data in the original data that matches the question data, extracting the image area in the image related to the image target data as the answer image for the question data, and providing the answer image, the problem of missing information in the image itself is avoided, and the image area in the image related to the image target data is used as the answer image for the question data, thereby achieving the effect of providing image-type answers to the question data and improving the completeness of the information provided for the question data.

[0234] Reference Fig.10 , shows a flowchart of an embodiment of a question-answering processing method according to Embodiment 8 of the present application, and the method may specifically include the following steps:

[0235] Step 801, obtaining question data and original image data.

[0236] In the embodiment of the present application, the question-answering processing method can be applied to the intelligent question-answering system, first obtaining the question data submitted by the user and the original image data. After obtaining the answer image for the question data according to the question-answering processing method, the answer image is provided to the user. For example, in the customer service system of an e-commerce platform, the user submits question data for a certain product, obtains the question data, and then obtains the original image data related to the product.

[0237] Step 802: Determine the image target data in the original image data that matches the problem data.

[0238] Step 803: extracting a picture area in the picture related to the picture target data as an answer picture for the question data.

[0239] Step 804, providing the answer picture.

[0240] According to an embodiment of the present application, by obtaining question data and original image data, determining image target data in the original image data that matches the question data, extracting an image area in the image related to the image target data as an answer image for the question data, and providing the answer image, the problem of missing information in the image itself is avoided, and using the image area in the image related to the image target data as an answer image for the question data, the effect of providing image-type answers to the question data is achieved, and the completeness of the information provided for the question data is improved.

[0241] In order to enable those skilled in the art to better understand the present application, a verification method of the present application is described below through specific examples.

[0242] Reference Fig.11 Schematic diagram of image cropping processing shown.

[0243] Step 1: Get the large image of the product details page.

[0244] Step 2: Perform OCR (Optical Character Recognition) on the image of the large picture of the details page to obtain at least one text target area in the image, perform hierarchical clustering on the text target areas and merge them into at least one text area; perform SLIC segmentation on the image of the large picture of the details page to obtain at least one superpixel area; perform object detection on the image of the large picture of the details page to obtain at least one object target area; perform edge extraction on the image of the large picture of the details page to obtain at least one sub-area of ​​the image.

[0245] Step 3, by means of voting, at least one text region, at least one superpixel region, at least one object target region, and at least one sub-region are integrated to obtain at least one image region.

[0246] Step 4: Highlight the answer text data.

[0247] Step 5, extract the image area where the answer text data is located, and crop the sub-image corresponding to the image area.

[0248] Step 6: The cropped sub-image is provided to the user as the hit sub-image of the user's question.

[0249] Reference Fig.12 , shows a structural block diagram of an information providing device embodiment according to Embodiment 9 of the present application, which may specifically include:

[0250] The request acquisition module 901 is used to acquire the query request of the user;

[0251] A picture determination module 902, configured to determine a corresponding query picture based on the query request;

[0252] A matching module 903, configured to match the identified text in the query image with the query request to determine a matching query image;

[0253] The providing module 904 is used to provide the matched query image to the user.

[0254] In the embodiment of the present application, optionally, the providing module includes:

[0255] An extraction submodule, configured to extract a picture region related to the query request in the matched query picture as a sub-picture for the query request;

[0256] A submodule is provided, for providing the sub-image to the user.

[0257] According to an embodiment of the present application, a query request from a user is obtained; based on the query request, a corresponding query image is determined; the identified text in the query image is matched with the query request to determine a matching query image; and the matching query image is provided to the user, thereby overcoming the limitation of only being able to query text, achieving the effect of providing image-type information in response to the query request, and improving the completeness of the information provided in response to the query request.

[0258] Reference Fig.13 , shows a structural block diagram of a commodity query device embodiment according to Embodiment 10 of the present application, which may specifically include:

[0259] The request acquisition module 1001 is used to acquire the product query request of the user;

[0260] The picture determination module 1002 is used to determine the corresponding product picture based on the product query request;

[0261] A matching module 1003 is used to match the identified text in the product image with the product query request to determine a matching product image;

[0262] The providing module 1004 is used to provide the matched product image to the user.

[0263] According to an embodiment of the present application, a product query request from a user is obtained; based on the product query request, a corresponding product image is determined; the text in the identified product image is matched with the product query request to determine a matching product image; and the matching product image is provided to the user, thereby overcoming the limitation of only being able to query text, achieving the effect of providing image information in response to the product query request, and improving the completeness of the information provided in response to the product query request.

[0264] Reference Fig.14 , shows a structural block diagram of an embodiment of a question-answer processing device according to Embodiment 11 of the present application, which may specifically include:

[0265] The request acquisition module 1101 is used to acquire the user's question and answer request;

[0266] A matching module 1102 is used to match the text in the identified query image with the question-answering request from the image library to determine a matching query image;

[0267] The providing module 1103 is used to provide the matched query image as a result corresponding to the question and answer request to the user.

[0268] According to an embodiment of the present application, a question and answer request of a user is obtained; the text in an identified query image is matched with the question and answer request from an image library to determine a matching query image; and the matching query image is provided to the user as a result corresponding to the question and answer request. This overcomes the limitation that only text questions can be answered, achieves the effect of providing image information in response to the question and answer request, and improves the completeness of the information provided in response to the question and answer request.

[0269] Reference Fig.15 , shows a structural block diagram of an information providing device embodiment according to Embodiment 12 of the present application, which may specifically include:

[0270] The data extraction module 1201 is used to extract information data of the picture, wherein the information data includes picture text data and picture original data;

[0271] A data determination module 1202, configured to determine answer text data matching the question text data in the picture text data according to the information data;

[0272] The sub-picture providing module 1203 is used to extract the picture area related to the answer text data in the picture as a sub-picture for the question text data, and provide the sub-picture.

[0273] In an embodiment of the present application, optionally, the data extraction module includes:

[0274] A text recognition submodule, used to recognize picture text data in the picture;

[0275] The preprocessing submodule is used to preprocess the image text data, and the preprocessing includes at least one of word segmentation and filtering.

[0276] In the embodiment of the present application, optionally, the device further includes:

[0277] A data receiving module, used for receiving the question text data before extracting the information data of the picture;

[0278] The picture acquisition module is used to acquire the picture according to the data object targeted by the question text data.

[0279] In the embodiment of the present application, optionally, the data determination module includes:

[0280] The data determination submodule is used to determine the answer text data using a text positioning data set based on the original image data, the image text data, and the question text data.

[0281] In an embodiment of the present application, optionally, the device includes:

[0282] A set creation module is used to create a text positioning data set for locating the answer text data based on the picture original data samples, the picture text data samples with answer text position labels and the question text data samples before determining the answer text data using the text positioning data set based on the picture original data, the picture text data and the question text data.

[0283] In an embodiment of the present application, optionally, the set creation module includes:

[0284] A vector generation submodule, used to generate a picture vector of the picture original data sample, a picture text vector of the picture text data sample, and a question text vector of the question text data sample;

[0285] A vector combining submodule, used for combining the picture vector and the picture text vector to obtain a picture text vector;

[0286] The set creation submodule is used to create the text positioning data set based on the image-text vector, question text vector and answer text position label.

[0287] In the embodiment of the present application, optionally, the data determination submodule includes:

[0288] A confidence obtaining unit, which obtains at least one candidate position of the answer text data and a corresponding candidate confidence according to the original image data, the image text data, and the question text data using the text positioning data set;

[0289] A data comparison unit, used to compare the question text data and the picture text data, and obtain a no-candidate confidence level that the answer text data cannot be located in the picture text data;

[0290] The screening unit is used to determine the answer candidate position corresponding to the candidate confidence greater than the no-candidate confidence, and screen out the answer text data from the answer candidate position.

[0291] In the embodiment of the present application, optionally, the sub-picture providing module includes:

[0292] A region division submodule, used to detect content-related regions in the image and divide them to obtain at least one image region;

[0293] A region extraction submodule, used for extracting the image region where the answer text data is located as the image region related to the answer text data;

[0294] The cropping submodule is used to crop the image to obtain a sub-image according to the image area.

[0295] In the embodiment of the present application, optionally, the area division submodule includes:

[0296] A region detection unit, configured to detect at least one object region and at least one text region in the image;

[0297] a region merging unit, configured to merge at least one relevant region of the at least one object region and the at least one text region;

[0298] The region division unit is used to divide the at least one image region according to the at least one object region, the at least one text region and the at least one related region.

[0299] In the embodiment of the present application, optionally, the area detection unit includes:

[0300] An object detection subunit, used to perform object detection on the image to obtain an object target area;

[0301] A superpixel division subunit, used to divide the image into a plurality of superpixel regions according to pixel features of the image;

[0302] The object region merging subunit is used to merge the object target region and a superpixel region related to the object target region to obtain at least one object region.

[0303] In the embodiment of the present application, optionally, the area detection unit includes:

[0304] A text detection subunit, used to perform text detection on the image to obtain a text target area;

[0305] The text region merging subunit is used to merge the text target regions whose distance or similarity between regions meets a preset condition to obtain the at least one text region.

[0306] In an embodiment of the present application, optionally, the device includes:

[0307] an edge detection module, configured to detect edge information in the image before detecting at least one object region and at least one text region in the image;

[0308] A sub-region division module, used for dividing the image into at least one sub-region according to the edge information;

[0309] The area detection unit comprises:

[0310] The region detection subunit is used to detect at least one object region and at least one text region in each subregion respectively.

[0311] In the embodiment of the present application, optionally, the device further includes:

[0312] The processing module is used to perform image processing on the answer text data in the sub-image before providing the sub-image, so that the answer text data has an obvious display effect.

[0313] According to the embodiment of the present application, when processing the match between texts, in addition to using the picture text data in the picture, the original picture data itself is also combined, avoiding the problem of missing information other than text in the picture, improving the accuracy of determining the position of the answer text data in the picture, and using the picture area related to the answer text data in the picture as a sub-picture for the question text data. Compared with only providing the answer text data, other relevant information in the picture besides the answer text data can also be provided, thereby achieving the effect of providing picture-type information for the question text data and improving the completeness of the information provided for the question text data.

[0314] Reference Fig.16 , shows a structural block diagram of an information providing device embodiment according to Embodiment 13 of the present application, which may specifically include:

[0315] The data acquisition module 1301 is used to acquire the original data of the image;

[0316] The target data determination module 1302 is used to determine the image target data matching the problem data in the image original data;

[0317] The picture providing module 1303 is used to extract a picture area related to the picture target data in the picture as an answer picture for the question data, and provide the answer picture.

[0318] According to the embodiments of the present application, by obtaining the original data of the image, determining the image target data in the original data that matches the question data, extracting the image area in the image related to the image target data as the answer image for the question data, and providing the answer image, the problem of missing information in the image itself is avoided, and the image area in the image related to the image target data is used as the answer image for the question data, thereby achieving the effect of providing image-type answers to the question data and improving the completeness of the information provided for the question data.

[0319] Reference Fig.17 , shows a structural block diagram of an embodiment of a question-answer processing device according to Embodiment 14 of the present application, which may specifically include:

[0320] Data acquisition module 1401, used to acquire question data and original image data;

[0321] The target data determination module 1402 is used to determine the image target data matching the problem data in the image original data;

[0322] The picture extraction module 1403 is used to extract the picture area related to the picture target data in the picture as the answer picture for the question data;

[0323] The picture providing module 1404 is used to provide the answer picture.

[0324] According to an embodiment of the present application, by obtaining question data and original image data, determining image target data in the original image data that matches the question data, extracting an image area in the image related to the image target data as an answer image for the question data, and providing the answer image, the problem of missing information in the image itself is avoided, and using the image area in the image related to the image target data as an answer image for the question data, the effect of providing image-type answers to the question data is achieved, and the completeness of the information provided for the question data is improved.

[0325] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0326] Embodiments of the present disclosure may be implemented as a system using any suitable hardware, firmware, software, or any combination thereof to perform the desired configuration. Fig.18 An exemplary system (or device) 1500 that can be used to implement various embodiments described in the present disclosure is schematically illustrated.

[0327] For one embodiment, Fig.18 An exemplary system 1500 is shown having one or more processors 1502, a system control module (chip set) 1504 coupled to at least one of the processor(s) 1502, a system memory 1506 coupled to the system control module 1504, a non-volatile memory (NVM) / storage device 1508 coupled to the system control module 1504, one or more input / output devices 1510 coupled to the system control module 1504, and a network interface 1512 coupled to the system control module 1506.

[0328] Processor 1502 may include one or more single-core or multi-core processors, and processor 1502 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.) In some embodiments, system 1500 can be used as a browser as described in the embodiments of the present application.

[0329] In some embodiments, system 1500 may include one or more computer-readable media (e.g., system memory 1506 or NVM / storage device 1508) having instructions and one or more processors 1502 that are combined with the one or more computer-readable media and configured to execute the instructions to implement the modules to perform the actions described in the present disclosure.

[0330] For one embodiment, the system control module 1504 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1502 and / or any suitable device or component in communication with the system control module 1504 .

[0331] The system control module 1504 may include a memory controller module to provide an interface to the system memory 1506. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0332] The system memory 1506 may be used, for example, to load and store data and / or instructions for the system 1500. For one embodiment, the system memory 1506 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the system memory 1506 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0333] For one embodiment, system control module 1504 may include one or more input / output controllers to provide interfaces to NVM / storage device 1508 and input / output device(s) 1510 .

[0334] For example, NVM / storage device 1508 may be used to store data and / or instructions. NVM / storage device 1508 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0335] NVM / storage device 1508 may include storage resources that are physically part of the device on which system 1500 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1508 may be accessed over a network via (one or more) input / output devices 1510.

[0336] (One or more) input / output devices 1510 may provide an interface for the system 1500 to communicate with any other appropriate device, and the input / output device 1510 may include a communication component, an audio component, a sensor component, etc. The network interface 1512 may provide an interface for the system 1500 to communicate through one or more networks, and the system 1500 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof for wireless communication.

[0337] For one embodiment, at least one of the processor(s) 1502 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the system control module 1504. For one embodiment, at least one of the processor(s) 1502 may be packaged together with the logic of one or more controllers of the system control module 1504 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1502 may be integrated on the same die with the logic of one or more controllers of the system control module 1504. For one embodiment, at least one of the processor(s) 1502 may be integrated on the same die with the logic of one or more controllers of the system control module 1504 to form a system on chip (SoC).

[0338] In various embodiments, system 1500 may be, but is not limited to: a browser, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, system 1500 may have more or fewer components and / or a different architecture. For example, in some embodiments, system 1500 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.

[0339] Wherein, if the display includes a touch panel, the display screen can be implemented as a touch screen display to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0340] An embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a terminal device, the terminal device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0341] In one example, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as in an embodiment of the present application when executing the computer program.

[0342] In one example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, one or more methods of the embodiments of the present application are implemented.

[0343] The embodiments of the present application disclose an information providing method and device. Example 1 includes an information providing method, including:

[0344] Get the user's query request;

[0345] Based on the query request, determining a corresponding query image;

[0346] Matching the identified text in the query image with the query request to determine a matching query image;

[0347] The matched query image is provided to the user.

[0348] Example 2 may include the method of Example 1, wherein providing the matching query image to the user comprises:

[0349] Extracting a picture area related to the query request in the matched query picture as a sub-picture for the query request;

[0350] The sub-picture is provided to the user.

[0351] Example 3 includes a product query method, including:

[0352] Get the user's product query request;

[0353] Based on the product query request, determine a corresponding product image;

[0354] Matching the identified text in the product image with the product query request to determine a matching product image;

[0355] The matching product images are provided to the user.

[0356] Example 4 includes a question-answering processing method, including:

[0357] Get the user's question and answer request;

[0358] From the image library, matching the text in the identified query image with the question-answering request to determine a matching query image;

[0359] The matched query image is provided to the user as a result corresponding to the question-answering request.

[0360] Example 5 includes an information providing method, comprising:

[0361] Extracting information data of the picture, wherein the information data includes picture text data and picture original data;

[0362] Determine answer text data matching the question text data in the picture text data according to the information data;

[0363] A picture area in the picture related to the answer text data is extracted as a sub-picture for the question text data, and the sub-picture is provided.

[0364] Example 6 may include the method of Example 5, wherein the information data of the extracted picture includes:

[0365] Identifying image text data in the image;

[0366] The image text data is preprocessed, and the preprocessing includes at least one of word segmentation and filtering.

[0367] Example 7 may include the method described in Example 5 and / or Example 6, wherein, before extracting the information data of the picture, the method further includes:

[0368] Receiving the question text data;

[0369] The picture is obtained according to the data object targeted by the question text data.

[0370] Example 8 may include the method described in one or more of Examples 5 to 7, wherein determining, based on the information data, answer text data matching the question text data in the picture text data comprises:

[0371] The answer text data is determined based on the original image data, the image text data, and the question text data using a text positioning data set.

[0372] Example 9 may include the method described in one or more of Examples 5 to 8, wherein, before determining the answer text data using a text location data set based on the original image data, the image text data, and the question text data, the method includes:

[0373] Based on the original image data samples, the image text data samples with answer text position labels, and the question text data samples, a text positioning data set for positioning the answer text data is created.

[0374] Example 10 may include the method described in one or more of Examples 5 to 9, wherein creating a text location data set for locating the answer text data based on the original image data sample, the image text data sample with the answer text location label, and the question text data sample comprises:

[0375] Generate a picture vector of the picture original data sample, a picture text vector of the picture text data sample, and a question text vector of the question text data sample;

[0376] Combining the picture vector and the picture text vector to obtain a picture text vector;

[0377] The text positioning data set is created based on the image-text vector, question text vector and answer text position label.

[0378] Example 11 may include the method described in one or more of Examples 5 to 10, wherein determining the answer text data using a text location data set based on the original image data, the image text data, and the question text data comprises:

[0379] According to the original image data, the image text data, and the question text data, the text positioning data set is used to obtain at least one candidate position of the answer text data and a corresponding candidate confidence;

[0380] Comparing the question text data and the picture text data, obtaining a no-candidate confidence level that the answer text data cannot be located in the picture text data;

[0381] Determine answer candidate positions corresponding to candidate confidences greater than the no-candidate confidence, and filter out the answer text data from the answer candidate positions.

[0382] Example 12 may include the method described in one or more of Examples 5 to 11, wherein extracting the image region related to the answer text data in the image as a sub-image for the question text data comprises:

[0383] Detecting content-related areas in the picture, and dividing them to obtain at least one picture area;

[0384] Extracting the image region where the answer text data is located as the image region related to the answer text data;

[0385] According to the image area, the image is cropped to obtain a sub-image.

[0386] Example 13 may include the method described in one or more of Examples 5 to 12, wherein detecting a content-related area in the picture and dividing the area to obtain at least one picture area comprises:

[0387] Detecting at least one object region and at least one text region in the image;

[0388] merging at least one related region among the at least one object region and the at least one text region;

[0389] The at least one picture region is obtained by dividing according to the at least one object region, the at least one text region and the at least one related region.

[0390] Example 14 may include the method of one or more of Examples 5 to 13, wherein detecting at least one object region in the object picture comprises:

[0391] Performing object detection on the image to obtain an object target area;

[0392] Dividing the image into multiple super-pixel regions according to pixel features of the image;

[0393] The object target region and a superpixel region related to the object target region are merged to obtain at least one object region.

[0394] Example 15 may include the method of one or more of Examples 5-14, wherein detecting at least one text region in the object picture comprises:

[0395] Performing text detection on the image to obtain a text target area;

[0396] The text target regions whose distance or similarity between regions meets a preset condition are merged to obtain the at least one text region.

[0397] Example 16 may include the method described in one or more of Examples 5 to 15, wherein, before detecting at least one object region and at least one text region in the picture, the method includes:

[0398] Detecting edge information in the image;

[0399] Dividing the image into at least one sub-region according to the edge information;

[0400] The detecting at least one object region and at least one text region in the image comprises:

[0401] At least one object region and at least one text region in each sub-region are detected respectively.

[0402] Example 17 may include the method described in one or more of Examples 5 to 16, wherein, before providing the sub-picture, the method further comprises:

[0403] Image processing is performed on the answer text data in the sub-image so that the answer text data has an obvious display effect.

[0404] Example 18 includes an information providing method, comprising:

[0405] Get the original image data;

[0406] Determine the image target data matching the problem data in the image original data;

[0407] A picture area in the picture related to the picture target data is extracted as an answer picture for the question data, and the answer picture is provided.

[0408] Example 19 includes a question-answering processing method, including:

[0409] Get the original data of the question and the image;

[0410] Determine the image target data matching the problem data in the original image data;

[0411] Extracting a picture region in the picture related to the picture target data as an answer picture for the question data;

[0412] Provide a picture of the answer.

[0413] Example 20 includes an information providing device, comprising:

[0414] A request acquisition module is used to obtain the user's query request;

[0415] A picture determination module, used to determine a corresponding query picture based on the query request;

[0416] A matching module, used to match the identified text in the query image with the query request to determine a matching query image;

[0417] A providing module is used to provide the matched query image to the user.

[0418] Example 21 may include the apparatus of Example 20, wherein

[0419] In the embodiment of the present application, optionally, the providing module includes:

[0420] An extraction submodule, configured to extract a picture region related to the query request in the matched query picture as a sub-picture for the query request;

[0421] A submodule is provided, for providing the sub-image to the user.

[0422] Example 22 includes a commodity query device, comprising:

[0423] The request acquisition module is used to obtain the user's product query request;

[0424] A picture determination module, used to determine a corresponding product picture based on the product query request;

[0425] A matching module, used to match the identified text in the product image with the product query request to determine a matching product image;

[0426] A providing module is used to provide the matching product image to the user.

[0427] Example 23 includes a question-answering processing apparatus, comprising:

[0428] Request acquisition module, used to obtain user's question and answer requests;

[0429] A matching module, used to match the text in the identified query image with the question-answering request from the image library to determine a matching query image;

[0430] A providing module is used to provide the matched query image as a result corresponding to the question and answer request to the user.

[0431] Example 24 includes an information providing device, comprising:

[0432] A data extraction module is used to extract information data of the picture, wherein the information data includes picture text data and picture original data;

[0433] A data determination module, used to determine answer text data matching the question text data in the picture text data according to the information data;

[0434] The sub-picture providing module is used to extract a picture area related to the answer text data in the picture as a sub-picture for the question text data, and provide the sub-picture.

[0435] Example 25 may include the apparatus of Example 24, wherein the data extraction module comprises:

[0436] A text recognition submodule, used to recognize picture text data in the picture;

[0437] The preprocessing submodule is used to preprocess the image text data, and the preprocessing includes at least one of word segmentation and filtering.

[0438] Example 26 may include the apparatus of Example 24 and / or Example 25, wherein the apparatus further comprises:

[0439] A data receiving module, used for receiving the question text data before extracting the information data of the picture;

[0440] The picture acquisition module is used to acquire the picture according to the data object targeted by the question text data.

[0441] Example 27 may include the apparatus of one or more of Examples 24-26, wherein the data determination module comprises:

[0442] The data determination submodule is used to determine the answer text data using a text positioning data set based on the original image data, the image text data, and the question text data.

[0443] Example 28 may include the apparatus of one or more of Examples 24-27, wherein the apparatus comprises:

[0444] A set creation module is used to create a text positioning data set for locating the answer text data based on the picture original data samples, the picture text data samples with answer text position labels and the question text data samples before determining the answer text data using the text positioning data set based on the picture original data, the picture text data and the question text data.

[0445] Example 29 may include the apparatus of one or more of Examples 24-28, wherein the set creation module comprises:

[0446] A vector generation submodule, used to generate a picture vector of the picture original data sample, a picture text vector of the picture text data sample, and a question text vector of the question text data sample;

[0447] A vector combining submodule, used for combining the picture vector and the picture text vector to obtain a picture text vector;

[0448] The set creation submodule is used to create the text positioning data set based on the image-text vector, question text vector and answer text position label.

[0449] Example 30 may include the apparatus of one or more of Examples 24-29, wherein the data determination submodule comprises:

[0450] A confidence obtaining unit, which obtains at least one candidate position of the answer text data and a corresponding candidate confidence according to the original image data, the image text data, and the question text data using the text positioning data set;

[0451] A data comparison unit, used to compare the question text data and the picture text data, and obtain a no-candidate confidence level that the answer text data cannot be located in the picture text data;

[0452] The screening unit is used to determine the answer candidate position corresponding to the candidate confidence greater than the no-candidate confidence, and screen out the answer text data from the answer candidate position.

[0453] Example 31 may include the apparatus of one or more of Examples 24 to 30, wherein the sub-picture providing module comprises:

[0454] A region division submodule, used to detect content-related regions in the image and divide them to obtain at least one image region;

[0455] A region extraction submodule, used for extracting the image region where the answer text data is located as the image region related to the answer text data;

[0456] The cropping submodule is used to crop the image to obtain a sub-image according to the image area.

[0457] Example 32 may include the apparatus of one or more of Examples 24-31, wherein the region division submodule comprises:

[0458] A region detection unit, configured to detect at least one object region and at least one text region in the image;

[0459] a region merging unit, configured to merge at least one relevant region of the at least one object region and the at least one text region;

[0460] The region division unit is used to divide the at least one image region according to the at least one object region, the at least one text region and the at least one related region.

[0461] Example 33 may include the apparatus of one or more of Examples 24-32, wherein the region detection unit comprises:

[0462] An object detection subunit, used to perform object detection on the image to obtain an object target area;

[0463] A superpixel division subunit, used to divide the image into a plurality of superpixel regions according to pixel features of the image;

[0464] The object region merging subunit is used to merge the object target region and a superpixel region related to the object target region to obtain at least one object region.

[0465] Example 34 may include the apparatus of one or more of Examples 24-33, wherein the region detection unit comprises:

[0466] A text detection subunit, used to perform text detection on the image to obtain a text target area;

[0467] The text region merging subunit is used to merge the text target regions whose distance or similarity between regions meets a preset condition to obtain the at least one text region.

[0468] Example 35 may include the apparatus of one or more of Examples 24-34, wherein the apparatus comprises:

[0469] an edge detection module, configured to detect edge information in the image before detecting at least one object region and at least one text region in the image;

[0470] A sub-region division module, used for dividing the image into at least one sub-region according to the edge information;

[0471] The area detection unit comprises:

[0472] The region detection subunit is used to detect at least one object region and at least one text region in each subregion respectively.

[0473] Example 36 may include the apparatus of one or more of Examples 24-35, wherein the apparatus further comprises:

[0474] The processing module is used to perform image processing on the answer text data in the sub-image before providing the sub-image, so that the answer text data has an obvious display effect.

[0475] Example 37 includes an information providing device, comprising:

[0476] Data acquisition module, used to obtain original image data;

[0477] A target data determination module, used to determine the image target data matching the problem data in the original image data;

[0478] The picture providing module is used to extract a picture area related to the picture target data in the picture as an answer picture for the question data, and provide the answer picture.

[0479] Example 38 includes a question-answering processing apparatus, comprising:

[0480] Data acquisition module, used to obtain question data and original image data;

[0481] A target data determination module, used to determine the image target data matching the problem data in the image original data;

[0482] An image extraction module is used to extract an image region in the image that is related to the image target data as an answer image for the question data;

[0483] The picture providing module is used to provide the answer picture.

[0484] Example 39 includes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, one or more methods of Examples 1-19 are implemented.

[0485] Example 40 includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements one or more methods of Examples 1-19.

[0486] Although certain embodiments are for the purpose of illustration and description, various alternatives, and / or, equivalent implementations, or calculations to achieve the same purpose of the embodiment shown and described implementations do not depart from the scope of the present application. This application is intended to cover any modifications or changes of the embodiments discussed herein. Therefore, it is apparent that the embodiments described herein are limited only by the claims and their equivalents.

Claims

1. A method for providing information, characterized in that: include: Get the user's query request; Based on the query request, determining a corresponding query image; Matching the identified text in the query image with the query request to determine a matching query image; Detecting at least one object region and at least one text region in the query image; merging at least one related region among the at least one object region and the at least one text region; Dividing at least one picture region according to the at least one object region, the at least one text region, and the at least one related region; The image region where the answer text data is located is extracted as the image region related to the answer text data; according to the image region, the query image is cut to obtain a sub-image, and the sub-image is provided to the user.

2. A commodity query method, characterized in that: include: Get the user's product query request; Based on the product query request, determine a corresponding product image; Matching the identified text in the product image with the product query request to determine a matching product image; Detecting at least one object region and at least one text region in the product image; merging at least one related region among the at least one object region and the at least one text region; Dividing at least one picture region according to the at least one object region, the at least one text region, and the at least one related region; The image region where the answer text data is located is extracted as the image region related to the answer text data; according to the image region, the product image is cut to obtain a sub-image, and the sub-image is provided to the user.

3. A question-answering processing method, characterized in that: include: Get the user's question and answer request; From the image library, matching the text in the identified query image with the question-answering request to determine a matching query image; Detecting at least one object region and at least one text region in the query image; merging at least one related region among the at least one object region and the at least one text region; Dividing at least one picture area according to the at least one object area, the at least one text area, and the at least one related area; The image region where the answer text data is located is extracted as the image region related to the answer text data; according to the image region, the query image is cut to obtain a sub-image, and the sub-image is provided to the user.

4. A method for providing information, characterized in that: include: Extracting information data of the picture, wherein the information data includes picture text data and picture original data; Determining answer text data matching the question text data in the picture text data according to the information data; Detecting at least one object region and at least one text region in the image; merging at least one related region among the at least one object region and the at least one text region; Dividing at least one picture area according to the at least one object area, the at least one text area, and the at least one related area; The image region where the answer text data is located is extracted as the image region related to the answer text data; according to the image region, the image is cut to obtain a sub-image, and the sub-image is provided.

5. The method according to claim 4, characterized in that The information data of the extracted picture includes: Identifying image text data in the image; The image text data is preprocessed, and the preprocessing includes at least one of word segmentation and filtering.

6. The method according to claim 4, characterized in that Before extracting the information data of the picture, the method further includes: Receiving the question text data; The picture is obtained according to the data object targeted by the question text data.

7. The method according to claim 4, characterized in that Determining answer text data matching the question text data in the picture text data according to the information data comprises: The answer text data is determined based on the original image data, the image text data, and the question text data using a text positioning data set.

8. The method according to claim 7, characterized in that Before determining the answer text data using a text positioning data set based on the original picture data, the picture text data, and the question text data, the method includes: Based on the original image data samples, the image text data samples with answer text position labels, and the question text data samples, a text positioning data set for positioning the answer text data is created.

9. The method according to claim 8, characterized in that The step of creating a text positioning data set for positioning the answer text data based on the original image data sample, the image text data sample with the answer text position label, and the question text data sample comprises: Generate a picture vector of the picture original data sample, a picture text vector of the picture text data sample, and a question text vector of the question text data sample; Combining the picture vector and the picture text vector to obtain a picture text vector; The text positioning data set is created based on the image-text vector, question text vector and answer text position label.

10. The method according to claim 7, characterized in that Determining the answer text data using a text positioning data set according to the original image data, the image text data, and the question text data includes: According to the original image data, the image text data, and the question text data, the text positioning data set is used to obtain at least one candidate position of the answer text data and a corresponding candidate confidence; Comparing the question text data and the picture text data, obtaining a no-candidate confidence level that the answer text data cannot be located in the picture text data; Determine answer candidate positions corresponding to candidate confidences greater than the no-candidate confidence, and filter out the answer text data from the answer candidate positions.

11. The method according to claim 4, characterized in that The detecting at least one object region in the picture comprises: Performing object detection on the image to obtain an object target area; Dividing the image into multiple super-pixel regions according to pixel features of the image; The object target region and a superpixel region related to the object target region are merged to obtain at least one object region.

12. The method according to claim 4, characterized in that The detecting at least one text area in the picture comprises: Performing text detection on the image to obtain a text target area; The text target regions whose distance or similarity between regions meets a preset condition are merged to obtain the at least one text region.

13. The method according to claim 4, characterized in that Before detecting at least one object region and at least one text region in the picture, the method includes: Detecting edge information in the image; Dividing the image into at least one sub-region according to the edge information; The detecting at least one object region and at least one text region in the image comprises: At least one object region and at least one text region in each sub-region are detected respectively.

14. The method according to claim 4, characterized in that Before providing the sub-picture, the method further includes: Image processing is performed on the answer text data in the sub-image so that the answer text data has an obvious display effect.

15. A method for providing information, characterized in that: include: Get the original image data; Determine the image target data matching the problem data in the image original data; Detecting at least one object region and at least one text region in the image; merging at least one related region among the at least one object region and the at least one text region; Dividing at least one picture region according to the at least one object region, the at least one text region, and the at least one related region; A picture region where the picture target data is located is extracted as a picture region related to the picture target data; according to the picture region, the picture is cut to obtain a sub-picture, and the sub-picture is provided.

16. A question-answering processing method, characterized in that, include: Get the original data of the question and the image; Determine the image target data matching the problem data in the original image data; Detecting at least one object region and at least one text region in the image; merging at least one related region among the at least one object region and the at least one text region; Dividing at least one picture region according to the at least one object region, the at least one text region, and the at least one related region; Extracting a picture area where the picture target data is located as a picture area related to the picture target data; and cutting the picture to obtain a sub-picture according to the picture area; The sub-picture is provided.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to one or more of claims 1-16 is implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to one or more of claims 1 to 16 is implemented.

Images