Visual disorder group vertical field multi-modal intelligent question answering method based on neural network

Abstract

The invention relates to the field of intelligent question and answer calculation, in particular to a visual impairment group vertical field multi-modal intelligent question and answer method based on a neural network, which comprises the following steps: S1, knowledge base preprocessing: standardizing a knowledge base, including performing information extraction, semantic understanding and multi-modal dialogue on texts in the knowledge base; s2, performing structured processing on the multi-mode vertical domain knowledge base; s3, generating detailed description information from the knowledge of the picture modality through an image-text big model; s4, forming a document by using the decomposed subject-predicate-object structure and the fixed complementary structure to replace the original characters, and identifying the document by using an ID (Identity); s5, collecting related attributes of the entities according to the query ID, and generating a candidate attribute list; and S6, according to the score, determining an attribute relationship closest to the question, and taking out an attribute value of the closest attribute relationship to complete answering. The method has the advantages that more accurate and personalized multimode vertical domain knowledge introduction question and answer service can be provided for visually impaired people to obtain information.

Description

[Technical field]

[0001] The present invention relates to the field of intelligent question-answering computing, and in particular to a multimodal intelligent question-answering method for a visually impaired group in a vertical field based on a neural network. [Background technology]

[0002] In the field of artificial intelligence, especially in the field of natural language processing (NLP), we often hear the term "vertical domain model". A vertical domain model is a model that focuses on a specific task or domain. These models are often trained using specific data sets to better understand and solve problems in a specific domain. Although generalized models have excellent performance in handling various tasks, vertical domain models have received widespread attention due to their deep understanding of specific domains and excellent performance.

[0003] In many practical applications, vertical domain models have shown significant advantages. For example, fields such as scientific research, medical diagnosis, legal consulting, and financial analysis require deep expertise, which is difficult for generalized models to achieve. Therefore, vertical domain models trained specifically for these fields can provide more accurate and in-depth insights. However, although the current information retrieval system has made progress in universal information acquisition, it still has many inconveniences when dealing with multimodal vertical domain knowledge retrieval. For the visually impaired, the problems they face when using such systems mainly include: the multimodal information retrieval process is complicated, the multimodal information is not intuitive and easy to understand, and there is a lack of effective information filtering and optimization mechanisms. These problems not only slow down the speed of knowledge acquisition, but also affect the accuracy of knowledge acquisition. On the other hand, since the visually impaired cannot quickly screen and evaluate information through vision, they rely more on the relevance and accuracy of the results provided by the information retrieval system.

[0004] Therefore, a neural network-based vertical field multimodal intelligent question-answering method for the visually impaired group is proposed, which can provide more accurate and personalized multimodal vertical field knowledge introduction question-answering services for the visually impaired to obtain information. [Summary of the invention]

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a multimodal intelligent question-answering method in vertical fields for visually impaired groups based on neural networks, which can provide more accurate and personalized multimodal vertical field knowledge introduction question-answering services for visually impaired people to obtain information.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The multimodal intelligent question answering method for vertical fields of visually impaired groups based on neural networks includes the following steps:

[0008] S1, knowledge base preprocessing, standardization of the knowledge base, including information extraction, semantic understanding, and multimodal dialogue of the text in the knowledge base;

[0009] S2, structural processing of the multimodal vertical domain knowledge base, in which the text modality knowledge is decomposed into two parts: subject-verb-object structure and attributive-adverbial-complement structure, which are used for information retrieval and matching;

[0010] S3, the knowledge of the image modality is used through the large picture-text model to generate detailed description information, and the detailed description information is decomposed into the content of the picture, and then matched with the subject-predicate-object sentence;

[0011] S4, after the knowledge of text modality, table modality and picture modality are processed and decomposed, the decomposed subject-verb-object structure and attributive-adverbial-complement structure are formed into a document to replace the original characters, and the document is identified with an ID;

[0012] S5: Collect relevant attributes of the entity according to the query ID and generate a candidate attribute list;

[0013] S6: Determine the attribute relationship that is closest to the question sentence based on the score, and take out the attribute value of the closest attribute relationship to complete the answer.

[0014] As a preferred embodiment, the method specifically includes S21: using a neural network to clean, segment, and stem the text in the knowledge base;

[0015] S22: perform double filtering, retrieve the preprocessed documents, and return N documents;

[0016] S23: merge the preliminary results and select p documents with high similarity;

[0017] S24: Calculate the new ranking and return the final document list.

[0018] Preferably, step S21 specifically includes:

[0019] S211: Using a convolutional neural network to perform word similarity modeling on the text in the knowledge base to obtain word feature vectors;

[0020] S212: Combining the semantic feature vector and the word feature vector, obtaining the score of each candidate attribute through a linear neural network.

[0021] Preferably, in step S6, the contents of the first n documents selected are comprehensively scored, and the scoring is based on the user's historical interaction data, specifically including:

[0022] S61, giving a preliminary score based on the edit distance between the contents of the four documents and the normalized query request;

[0023] S62, combining the document’s relevance metrics, including click-through rate and citation count, to make score adjustments;

[0024] S63, referring to the user's historical query requests and user preferences, the score is adjusted again to obtain a comprehensive score.

[0025] Preferably, the method further comprises S64, sorting the documents using the comprehensive scores to form a list in descending order of relevance;

[0026] S65, presenting the sorted document list to the visually impaired user through a user interaction interface, and providing an information transmission mode of voice reading or Braille output.

[0027] Preferably, the list in descending order of relevance in step 64 is adjusted forward or backward in position according to user selection.

[0028] Preferably, a sentence model is constructed based on a fully connected neural network layer to evaluate the importance of other sentence structures in the sentence domain;

[0029] The parameters of the sentence model are to be optimized to obtain a relation prediction model;

[0030] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups.

[0031] An electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the steps of the above-mentioned neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups.

[0032] The beneficial effects of the present invention are as follows:

[0033] 1. Structural processing of the text modality knowledge in the multimodal vertical domain knowledge base, decomposing it into two parts: subject-predicate-object structure and attributive-adverb-complement structure, and replacing the original characters with documents, using ID to identify the documents, and using neural network algorithm to process them, screen out the most relevant documents for comprehensive scoring and re-ranking, so that non-visually impaired people can conduct retrieval and inspection. Through the above method, a more accurate and personalized multimodal vertical domain knowledge introduction question and answer service can be provided to the visually impaired people, and through the inspection of non-visually impaired users, the question and answer results and sentence comprehension can be optimized, and the words and sentences with high frequency of communication with the masses can be screened out, so as to strengthen the coherence of communication and improve the convenience of communication between the visually impaired and the general public. This improved method can greatly improve the efficiency and accuracy of the visually impaired in acquiring knowledge and enhance the overall query experience.

[0034] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings.

Brief Description of the Drawings

[0035] The present invention will be further described below in conjunction with the accompanying drawings:

[0036] Figure 1 It is a schematic diagram of the process of the present invention;

[0037] Figure 2 Schematic diagram of the NLP model. [Specific implementation method]

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] The following first describes the concepts involved in this application in conjunction with the accompanying drawings. It should be noted that the following description of each concept is only to make the content of this application easier to understand and does not limit the scope of protection of this application.

[0040] Embodiment 1:

[0041] A multimodal intelligent question-answering method for vertical fields of visually impaired groups based on neural networks, such as Figures 1 to 2 As shown, the following steps are included:

[0042] S1, knowledge base preprocessing, standardization of the knowledge base, including information extraction, semantic understanding, and multimodal dialogue of the text in the knowledge base;

[0043] S2, structural processing of the multimodal vertical domain knowledge base, in which the text modality knowledge is decomposed into two parts: subject-verb-object structure and attributive-adverbial-complement structure, which are used for information retrieval and matching;

[0044] Specifically, S21: using neural networks to clean, segment, and stem text in the knowledge base;

[0045] For example: I (subject) helped (predicate) my classmates (object) with their homework (complement) in the newly built (attributive) house (adverbial).

[0046] It evaluates the relevance of documents to queries based on keyword matching. Suppose there is a search query: "How will artificial intelligence affect the development of future cars?" Based on the subject-predicate-object structure, it identifies "artificial intelligence," "influence," and "development," and extends the corresponding semantics to synonyms, such as "artificial intelligence" can be extended to "intelligence," "wisdom," and "wit." "Influence" can be extended to "cause," "promote," and "promote." "Development" can be extended to "change" and "progress." The attributive "car" can be extended to "car" and "sedan." It will return documents that mention related keywords based on word frequency.

[0047] Step S21 specifically includes:

[0048] S211: Using a convolutional neural network to perform word similarity modeling on the text in the knowledge base to obtain word feature vectors;

[0049] Set up a similar vocabulary group, set up an intelligent terminal collection device under the similar vocabulary group, collect user preference data and high-frequency data through the intelligent terminal, and the fully connected convolutional neural network is:

[0050]

[0051] Where b is the deviation, Z l+1 represents the convolution output of the l+1th layer, L is the size of Z, K is the total number of channels of convolution input and output, i is the semantic understanding data of the i-th channel, j is the user bias data of the j-th channel, ω is the correction constant, and T is the time coefficient;

[0052] The existing time series neural network is used to predict and correct multi-level convolution features to ensure data stability. The calculation formula is as follows:

[0053]

[0054] in, is the hidden state of the l-th layer of the temporal neural network at time t, is the weight matrix from the input to the hidden state of the l-th layer of the temporal neural network, is the weight matrix from hidden state to hidden state of the layer of temporal neural network, is the bias term of the l-th layer of the temporal neural network, is the weight matrix from the hidden state to the output of the layer-th temporal neural network, It is

[0055] The output bias term of the layer temporal neural network, is the activation function, is the output data of the rectified layer of the temporal neural network.

[0056] The text modality knowledge in the multimodal vertical domain knowledge base is structured and decomposed into two parts: subject-predicate-object structure and attributive-adverb-complement structure. The original characters are replaced by documents, and the documents are identified by ID. The neural network algorithm is used to process the documents, and the most relevant documents are screened out for comprehensive scoring and re-ranking for retrieval and inspection by non-visually impaired people. Through the above method, a more accurate and personalized multimodal vertical domain knowledge introduction question-and-answer service can be provided to the visually impaired people. Through the inspection of non-visually impaired users, the question-and-answer results and sentence comprehension can be optimized, and the words and sentences with high frequency of communication with the masses can be screened out to strengthen the coherence of communication and improve the convenience of communication between the visually impaired and the general public. This improved method can greatly improve the efficiency and accuracy of the visually impaired in acquiring knowledge and enhance the overall query experience.

[0057] S212: Combining the semantic feature vector and the word feature vector, obtaining the score of each candidate attribute through a linear neural network.

[0058] S22: perform double filtering, retrieve the preprocessed documents, and return N documents;

[0059] The filtered text data is processed by multi-level recursive convolution. The calculation formula is as follows:

[0060]

[0061] in, is the user usage frequency convolution feature of the lth convolution layer at stroke t,

[0062] is the activation function, which is selected based on expert experience. is the convolution kernel, x′ f (t, f) is the text data after filtering, is the bias term of the lth convolutional layer at stroke t, is the feature weight coefficient of the lth convolutional layer at stroke t to the mth convolutional layer at stroke t-1, is the number of convolutional layers, It is the user usage frequency convolution feature of the mth convolution layer at stroke t-1.

[0063] S23: merge the preliminary results and select p documents with high similarity;

[0064] S24: Calculate the new ranking and return the final document list.

[0065] S3, the knowledge of the image modality is used through the large picture-text model to generate detailed description information, and the detailed description information is decomposed into the content of the picture, and then matched with the subject-predicate-object sentence;

[0066] S4, after the knowledge of text modality, table modality and picture modality are processed and decomposed, the decomposed subject-verb-object structure and attributive-adverbial-complement structure are formed into a document to replace the original characters, and the document is identified with an ID;

[0067] S5: Collect relevant attributes of the entity according to the query ID and generate a candidate attribute list;

[0068] S6: Determine the attribute relationship that is closest to the question sentence based on the score, and take out the attribute value of the closest attribute relationship to complete the answer.

[0069] In step S6, the contents of the first n documents selected are comprehensively scored, and the scoring is based on the user's historical interaction data, specifically including:

[0070] S61, give a preliminary score based on the content of the document;

[0071] S62, combining the document’s relevance metrics, including click-through rate and citation count, to make score adjustments;

[0072] S63, referring to the user's historical query requests and user preferences, the score is adjusted again to obtain a comprehensive score.

[0073] Also included is S64, which uses the comprehensive score to sort the documents into a list in descending order of relevance;

[0074] The formula for calculating the score is:

[0075]

[0076] Here, rank1 is the ranking of the retrieved document, rank2 is the ranking of the retrieved document, and k is a constant. Assuming that document A ranks 1st in semantic retrieval and 3rd in stroke retrieval, its RRF score will be 1 / (60·1)+1 / (60·3). Finally, perform the same calculation on all selected documents and add up the scores.

[0077] The descending order list of relevance in step 64 is adjusted forward or backward in position according to the user's selection.

[0078] S65, presenting the sorted document list to the visually impaired user through a user interaction interface, and providing an information transmission mode of voice reading or Braille output.

[0079] A sentence model is constructed based on a fully connected neural network layer to evaluate the importance of other sentence structures in the sentence domain; the parameters of the sentence model are to be optimized to obtain a relationship prediction model;

[0080] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups.

[0081] An electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the steps of the above-mentioned neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups.

[0082] Those of ordinary skill in the art will appreciate that the units (or modules, steps, etc., the same below) of the examples described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0083] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0084] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nly Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A multi-modal intelligent question-answering method for vertical fields of visually impaired groups based on neural networks, characterized by: The following steps are involved: S1, knowledge base preprocessing, standardization of the knowledge base, including information extraction, semantic understanding, and multimodal dialogue of the text in the knowledge base; S2, structuring the multimodal vertical domain knowledge base, wherein the text modality knowledge is decomposed into two parts: subject-predicate-object structure and attributive-adverbial-complement structure, for information retrieval and matching; S3, generating detailed description information by using the image-text model for the knowledge of the image modality, and decomposing the detailed description information into the content of the image, and then matching the subject-predicate-object sentence; S4, after the text modality knowledge, the table modality knowledge and the picture modality knowledge are processed and decomposed, the decomposed subject-verb-object structure and the attributive-adverbial-complement structure are formed into a document to replace the original characters, and the document is identified by using an ID; S5: Collect relevant attributes of the entity according to the query ID and generate a candidate attribute list; S6: Determine the attribute relationship that is closest to the question sentence based on the score, and take out the attribute value of the closest attribute relationship to complete the answer.

2. The neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups according to claim 1, characterized in that: Specifically, S21: using a neural network to clean, segment, and stem the text in the knowledge base; S22: perform double filtering, retrieve the preprocessed documents, and return N documents; S23: merge the preliminary results and select p documents with high similarity; S24: Calculate the new ranking and return the final document list.

3. The neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups as claimed in claim 2, characterized in that: The step S21 specifically includes: S211: Using a convolutional neural network to perform word similarity modeling on the text in the knowledge base to obtain a word feature vector; S212: Combining the semantic feature vector and the word feature vector, obtaining a score for each candidate attribute through a linear neural network.

4. The neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups according to claim 1, characterized in that: In step S6, the contents of the first n documents selected are comprehensively scored, and the scoring is based on the user's historical interaction data, specifically including: S61, giving a preliminary score according to the edit distance between the contents of the four documents and the normalized query request; S62, combining the document’s relevance metrics, including click-through rate and citation count, to make score adjustments; S63, referring to the user's historical query requests and user preferences, the score is adjusted again to obtain a comprehensive score.

5. The neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups as claimed in claim 4, characterized in that: Also included is S64, which uses the comprehensive score to sort the documents into a list in descending order of relevance; S65, presenting the sorted document list to the visually impaired user through a user interaction interface, and providing an information transmission mode of voice reading or Braille output.

6. The neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups as claimed in claim 5, characterized in that: The descending order list of relevance in step 64 is adjusted forward or backward in position according to the user's selection.

7. The neural network-based vertical field multimodal intelligent question-answering method for visually impaired groups according to any one of claims 1 to 6, characterized in that: Build a sentence model based on a fully connected neural network layer to evaluate the importance of other sentence structures in the sentence domain; The parameters of the sentence model are to be optimized to obtain a relationship prediction model.

8. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the steps of the vertical field multimodal intelligent question-answering method for visually impaired groups based on a neural network as described in any one of claims 1-6 are implemented.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the vertical field multimodal intelligent question-answering method for visually impaired groups based on a neural network as described in any one of claims 1-6.

Images