Flowchart-based Question and Answer Processing Method and Device

Abstract

The present invention provides a question-answering processing method and apparatus based on a flowchart, which relates to the field of artificial intelligence technology. The method includes: obtaining a question to be processed and a flowchart to be processed; identifying element information in the flowchart to be processed according to a specified dimension to generate a content description of the flowchart to be processed; calculating a matching degree between the content description and the flowchart to be processed; generating a target description of the flowchart to be processed based on the content description and a specified comparison result; and analyzing and reasoning the question to be processed by using the target description to generate a target answer corresponding to the question to be processed. In the embodiment of the present invention, based on the matching degree and the content description, a target description of the flowchart to be processed can be accurately generated, and finally, the target answer to the question to be processed is analyzed and inferred by using the target description, which can effectively improve the question-answering efficiency and accuracy based on the flowchart.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a method and device for question and answer processing based on a flowchart. Background Art

[0002] As an important information display tool, the flowchart plays a key role in many fields. It can visually display complex work processes, system architectures or algorithm logics, presenting each step, link and their relationships in a graphical way, which is convenient for people to understand and grasp. In the question and answer task based on the flowchart, it is necessary to accurately and comprehensively understand the content of the flowchart in order to give appropriate responses to user questions.

[0003] However, at present, the interpretation of the flowchart mainly relies on manual work. Manual interpretation is not only time-consuming and laborious, but also due to the inconsistent understanding levels of different interpreters, the interpretation quality is uneven, resulting in low execution efficiency and accuracy of the question and answer task based on the flowchart. Summary of the Invention

[0004] Aiming at the above problems, the purpose of the present invention is to provide a method and device for question and answer processing based on a flowchart, which can automatically and accurately interpret the flowchart to improve the efficiency and accuracy of question and answer processing based on the flowchart.

[0005] To solve the above technical problems, the present invention provides the following technical solutions:

[0006] On the one hand, the present invention provides a method for question and answer processing based on a flowchart, including:

[0007] Obtain a question to be processed and a flowchart to be processed;

[0008] Identify the element information in the flowchart to be processed according to a specified dimension to generate a content description of the flowchart to be processed, where the content description includes sub-descriptions corresponding to the specified dimension;

[0009] Calculate a specified comparison result of the content description and the matching degree to generate a target description of the flowchart to be processed, where the specified comparison result is a comparison result of the matching degree and a preset threshold;

[0010] Analyze and reason about the question to be processed by using the target description to generate a target answer corresponding to the question to be processed.

[0011] On the other hand, the present invention also provides a device for question and answer processing based on a flowchart, including:

[0012] An obtaining module, configured to obtain a question to be processed and a flowchart to be processed;

[0013] An identification module, configured to identify the element information in the to-be-processed flow chart according to a specified dimension, so as to generate a content description of the to-be-processed flow chart, where the content description includes a sub-description corresponding to the specified dimension;

[0014] A calculation module, configured to calculate a matching degree between the content description and the to-be-processed flow chart;

[0015] A target generation module, configured to generate a target description of the to-be-processed flow chart based on the content description and a specified comparison result, where the specified comparison result is a comparison result between the matching degree and a preset threshold;

[0016] A question-and-answer module, configured to analyze and reason about the to-be-processed question by using the target description, and generate a target answer corresponding to the to-be-processed question.

[0017] On the other hand, the present invention further provides an electronic device, including a processor and a memory, where the memory stores multiple instructions; the processor loads the instructions from the memory to execute the steps in any one of the flowchart-based question-and-answer processing methods provided by the present invention.

[0018] On the other hand, the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps in any one of the flowchart-based question-and-answer processing methods provided by the present invention.

[0019] On the other hand, an embodiment of the present invention further provides a computer program product, including a computer program / instructions, where when the computer program / instructions are executed by a processor, the steps in any one of the flowchart-based question-and-answer processing methods provided by the present invention are implemented.

[0020] The beneficial effects brought by the technical solution provided by the present invention at least include:

[0021] In an embodiment of the present invention, a to-be-processed question and a to-be-processed flow chart are obtained; the element information in the to-be-processed flow chart is identified according to a specified dimension to generate a content description of the to-be-processed flow chart; then the matching degree between the content description and the to-be-processed flow chart is calculated, and the consistency between the content description and the to-be-processed flow chart is measured by a specified comparison result between the matching degree and a preset threshold. Then, a target description of the to-be-processed flow chart is generated by using the content description and the specified comparison result, which can accurately and comprehensively describe the content in the to-be-processed flow chart. Finally, the target answer to the to-be-processed question is analyzed and inferred by using the target description, which can effectively improve the question-and-answer efficiency and accuracy based on the flow chart. Description of the Drawings

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0023] Figure 1 It is a schematic diagram of an application scenario of the question and answer processing method based on a flowchart provided by an embodiment of the present invention;

[0024] Figure 2 It is a schematic flowchart of the question and answer processing method based on a flowchart provided by an embodiment of the present invention;

[0025] Figure 3 It is a schematic diagram of generating a target description provided by an embodiment of the present invention;

[0026] Figure 4 It is a schematic diagram of correcting a content description to a target description provided by an embodiment of the present invention;

[0027] Figure 5 It is a schematic structural diagram of a question and answer processing device based on a flowchart provided by an embodiment of the present invention;

[0028] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0030] The present invention provides a question and answer processing method and device based on a flowchart, which can quickly and accurately interpret the flowchart and improve the efficiency and accuracy of question and answer.

[0031] It can be understood that in the specific implementation manners of the present invention, data related to user information and the like need to obtain user permission or consent, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards in relevant countries and regions.

[0032] Can refer to Figure 1, which shows a schematic diagram of the application scenario of the question-answering processing method based on a flowchart. Among them, the application scenario may include a terminal 101 and a server 102. Data exchange can be carried out between the terminal 101 and the server 102 through a network, and a corresponding application program can be installed on the terminal 101. Among them, the terminal 101 can be a mobile phone, a tablet computer, a smart Bluetooth device, a computer, a large screen, etc., a robot, etc.; the server 102 can be a single server or a server cluster composed of multiple servers.

[0033] The user can send the question to be processed and the flowchart to be processed to the server 102 through the terminal 101. Thus, the server 102 can identify the element information in the flowchart to be processed according to the specified dimension, and can generate a sub-description corresponding to each specified dimension as the content description; calculate the matching degree between the content description and the flowchart to be processed, and then generate the target description of the flowchart to be processed based on the specified comparison result between the matching degree and the preset threshold and the content description; finally, the target description can be used to analyze and reason about the question to be processed to generate the target answer corresponding to the question to be processed.

[0034] The target answer can be sent from the server 102 to the terminal 101 so that the terminal 101 can display the target answer to the user.

[0035] In this embodiment, a question-answering processing method based on a flowchart is provided, as Figure 2 shown. The specific process of the question-answering processing method based on a flowchart can be as follows:

[0036] S110. Obtain the question to be processed and the flowchart to be processed.

[0037] The question to be processed is the question text input by the user. Among them, the content input by the user can be in text form or voice form. If the input content is in text form, all the content input can be directly used as the text to be processed; if the input content is in voice form, it can be converted into text through speech-to-text technology to obtain the question to be processed.

[0038] The flowchart to be processed is the flowchart required to answer the question to be processed. A flowchart is a graphical tool used to represent the sequence and logical relationships of a process, system, or algorithm. It shows the sequence, branches, loops, and decision points between each step by using standardized symbols and graphics to clearly express complex processes. The flowchart can contain various elements, such as nodes, arrows, text, etc.

[0039] Among them, the flowchart to be processed can be input together with the problem to be processed. For example, the flowchart to be processed and the problem to be processed can be directly input together. Another example is that the position of the flowchart to be processed in the document can be described in words, and the document is provided to the server. The server can locate and extract the flowchart in the document based on the content provided by the user as the flowchart to be processed.

[0040] S120. Identify the element information in the flowchart to be processed according to the specified dimension to generate the content description of the flowchart to be processed.

[0041] The specified dimension refers to the direction for parsing and analyzing the flowchart to be processed. The number and content of the specified dimensions can be set in advance according to actual needs. In the embodiments of the present invention, the specified dimensions may include the node type and text dimension, the arrow and connection dimension, the logical relationship dimension between nodes, and the spatial position dimension of nodes.

[0042] Among them, by identifying the flowchart to be processed according to the specified dimension, the corresponding element information in the flowchart to be processed can be identified. For example, in the node type and text dimension, the type, shape, and text content within each node can be identified; in the arrow and connection dimension, the starting point, ending point, and flow direction of each arrow or line can be identified; in the logical relationship dimension between nodes, the logical relationship between nodes can be identified; in the spatial position dimension of nodes, the relative positions between nodes can be identified.

[0043] In each specified dimension, the corresponding element information can be identified and described in text form as the sub-description corresponding to the specified dimension, and then the content description of the flowchart to be processed can be generated. That is, the content description can include the sub-descriptions corresponding to each specified dimension to comprehensively and completely describe the flowchart to be processed.

[0044] In some embodiments, the optical character recognition (OCR) technology can also be used to identify the text information, graphic element layout and structure, etc. in the flowchart to be processed, and then the large language model is used to integrate the content recognized by OCR according to the specified dimension to obtain the content description.

[0045] In some embodiments, in order to improve the accuracy of content description, when generating the content description of the flowchart to be processed, the image quality of the flowchart to be processed may be optimized to obtain an optimized flowchart; based on the optimized flowchart and the recognition template, a recognition prompt word is generated, and the recognition prompt word includes the recognition requirements and output requirements corresponding to each specified dimension; using the recognition prompt word, guiding the recognition model to recognize the element information corresponding to each specified dimension from the optimized flowchart according to the recognition requirements; for each specified dimension, organizing the element information according to the output requirements to obtain a sub-description corresponding to the specified dimension; and using the sub-descriptions corresponding to all specified dimensions as the content description of the flowchart to be processed.

[0046] The flowchart to be processed is usually in the form of an image, and its image quality can be optimized to improve the accuracy of subsequent recognition. The optimization process may include grayscale processing, denoising processing, enhancement processing, etc. Among them, grayscale processing can convert the color flowchart to be processed into a grayscale image, reducing the interference of color, light, shadow, etc. Denoising processing can remove the noise in the flowchart to be processed through algorithms, reducing interference factors and improving the subsequent recognition effect. Enhancement processing can adjust parameters such as the contrast and brightness of the flowchart to be processed to make the image clearer. That is, the optimized flowchart obtained after the optimization process has higher image quality and is clearer for subsequent recognition.

[0047] As an implementation method, when obtaining the optimized flowchart through optimization processing, it may be to perform grayscale processing on the flowchart to be processed to obtain a grayscale flowchart; perform filtering processing on the grayscale flowchart to obtain a filtered flowchart; perform edge enhancement processing on the edges of the key structures in the filtered flowchart to obtain an edge-enhanced flowchart; perform contrast enhancement on the filtered flowchart to obtain a contrast-enhanced flowchart; and perform superposition processing on the edge-enhanced flowchart and the contrast-enhanced flowchart according to the specified weight to obtain the optimized flowchart.

[0048] The grayscale processing may be to convert the color image into a grayscale image according to the conversion formula, where the conversion formula is: Y = 0.2989 R + 0.5870 G + 0.1140 B, where R is the pixel value of the red channel in the image, G is the pixel value of the green channel in the image; B is the pixel value of the blue channel in the image; and Y is the converted grayscale value.

[0049] Filtering processing is to eliminate the noise in the image. There are various ways to obtain the flowchart to be processed. It may be an image converted from a paper document and may contain scanning noise. In order to retain the edge details while removing the noise, the grayscale flowchart can be processed using bilateral filtering. The filtering process takes into account the spatial distance between each pixel and its surrounding pixels and the similarity of pixel values, and performs weighted averaging on the pixels in the neighborhood. The neighborhood diameter determines the filtering range, and the standard deviations in the color space and coordinate space affect the filtering intensity and the degree of edge retention. Among them, the neighborhood diameter, the standard deviations in the color space and coordinate space can all be set according to actual needs. In the embodiment of the present invention, the neighborhood diameter is set to 9, and the standard deviations in the color space and coordinate space are both set to 75.

[0050] Edge enhancement is to highlight the key structures in the filtered flowchart. The key structures can be set according to actual needs. In the embodiment of the present invention, the key structures may include AND nodes, arrows, connection lines, etc. Among them, when performing edge enhancement, it can be based on the filtered flowchart to calculate the gradient magnitude and direction of each pixel in the image. The magnitude can be used to characterize the intensity of the brightness change at the pixel in the image, and the gradient direction represents the direction of the brightness change. Then non-maximum suppression can be performed, that is, pixels with gradient magnitudes that are not local maxima are set to 0, so as to refine the edges and make them into single-pixel-width thin lines; set high and low thresholds, perform threshold processing on the gradient image to obtain strong edges and weak edges. Track the weak edges, retain the weak edges connected to the strong edges, and remove the isolated weak edges. Use morphological dilation operation to make the edges more prominent. Morphological dilation is based on the structuring element, and choosing an appropriate structuring element can control the degree and shape of the dilation. For example, using a 3×3 square structuring element for dilation can make the edges expand outward by one pixel distance, so that the lines and edges in the filtered flowchart are more prominent.

[0051] Contrast enhancement is to improve the readability of the text and details in the flowchart to be processed. Specifically, the method of contrast-limited adaptive histogram equalization can be used to divide the filtered flowchart into blocks, and equalize the contrast within these blocks respectively, and set the contrast limit to 2.0.

[0052] After obtaining the edge-enhanced flowchart and the contrast-enhanced flowchart, the two flowcharts can be superimposed according to the specified weight to obtain the optimized flowchart. Among them, the specified weight can be set according to actual needs. In the embodiment of the present invention, the weight of the edge-enhanced flowchart is set to 30%, and the weight of the contrast-enhanced flowchart is set to 70%. After superimposing the images according to this weight, an intermediate flowchart is obtained. At this time, the intermediate flowchart is also a grayscale image, and the intermediate flowchart can be converted into an RGB image to obtain the final optimized flowchart.

[0053] The recognition template refers to a pre - set prompt word template for parsing the flow chart, which can be set in advance according to actual needs. Corresponding flow chart slots can be set in the recognition template, as well as the recognition requirements and output requirements corresponding to each specified dimension. The flow chart slots can be used to fill the optimized flow chart. After filling the optimized flow chart into the recognition template, recognition prompt words can be generated. The recognition prompt words can be used to guide the recognition model to parse and recognize the optimized flow chart. The recognition prompt words also contain the recognition requirements and output requirements corresponding to each specified dimension. That is, for each specified dimension, the recognition requirements and the output requirements for the recognized content can be set in the recognition template according to actual needs to ensure that the parsed and recognized content has strong usability.

[0054] In the embodiments of the present invention, the recognition template can be:

[0055] Please analyze the following flow chart, list all the elements in the figure and their relationships in the form of a detailed text description, and output in JSON format. Please include the following information:

[0056] Node type and text: Identify the type of each node (such as start node, end node, processing node, decision node, etc.), describe its shape (such as rectangle, circle, diamond, etc.) and the text content in each node (such as start, processing, end, decision).

[0057] Arrows and connections: Detect the arrows or lines, describe the starting point and ending point of each arrow or line, and their flow direction (for example, from node A to node B).

[0058] Logical relationships between nodes: Describe the logical relationships between nodes, such as "node A points to node B" or "node B points to node C or D according to the decision".

[0059] Spatial position of nodes: Describe the relative positions between nodes (such as node A is on the left of node B).

[0060] The recognition model refers to a model used to parse and recognize the flow chart to be processed. The recognition template can be a vision - language model. Input the recognition prompt words into the recognition model. The recognition model can recognize the corresponding element information from the optimized flow chart according to the recognition requirements of each specified dimension, and organize and output the element information according to the output requirements to obtain the sub - description corresponding to each specified dimension. Use all the sub - descriptions corresponding to the specified dimensions as the content description of the flow chart to be processed.

[0061] S130. Calculate the matching degree between the content description and the flow chart to be processed.

[0062] The content description is the text that describes the information contained in the flowchart to be processed. Subsequently, when answering the questions to be processed input by the user, the answer can be directly based on the content description. It can be seen that the accuracy of the content description directly affects the accuracy of the question and answer. To improve the accuracy of question and answer processing, the consistency between the generated content description and the flowchart to be processed can be verified. For example, the consistency between the two can be verified by calculating the matching degree between the content description and the flowchart to be processed. The higher the matching degree, the stronger the consistency; the lower the matching degree, the weaker the consistency.

[0063] In some embodiments, the content description and the flowchart to be processed can be directly converted into feature vectors, and then the cosine similarity between the feature vectors can be calculated as the matching degree.

[0064] The content description is data in text modality, while the flowchart to be processed is data in image modality. In some embodiments, in order to accurately calculate the matching degree between data of different modalities and better measure the internal structure and relationship of the data, when calculating the matching degree between the content description and the flowchart to be processed, it can be to obtain the text features of the content description in the multi-modal space and the image features of the flowchart to be processed in the multi-modal space; multiply the text features by the transpose of the text features to obtain a text kernel matrix; multiply the image features by the transpose of the image features to obtain an image kernel matrix; calculate the matching degree between the content description and the flowchart to be processed based on the text kernel matrix and the image kernel matrix.

[0065] Among them, the optimized flowchart corresponding to the flowchart to be processed has been calculated previously. The content description and the optimized flowchart can be mapped to the same feature space and then calculated. Among them, the mapping can be realized by means of a neural network model. In the embodiments of the present invention, a multi-modal model can be relied on to project both the content description and the flowchart to be processed into the multi-modal space, and text features and image features can be obtained.

[0066] Among them, the multi-modal model needs to be pre-trained. A large number of sample flowcharts can be collected in advance, and each sample flowchart can be optimized according to the foregoing method. Then, according to the optimized sample flowchart and the recognition template, prompt words are generated, and the prompt words are input into the multi-modal model, and the results of the multi-modal model are manually corrected and sorted. Then, the multi-modal model is fine-tuned using the corresponding data set, and the large language model part in the multi-modal model is fixed, and only the visual encoder and the projection layer are trained. After training, a mapping model can be obtained.

[0067] Encoding the content description using the tokenizer in the mapping model can obtain the text features corresponding to the content description; encoding the optimized flowchart using the visual encoder in the mapping model and projecting it into the multi-modal space through the projection layer can obtain the image features.

[0068] For the text features, the text features can be multiplied by the transpose of the text features to calculate the text kernel matrix. Similarly, for the image features, the image features can be multiplied by the transpose of the image features to calculate the image kernel matrix.

[0069] Using the text kernel matrix and the image kernel matrix, the matching degree between the content description and the flowchart to be processed can be calculated. Optionally, the text kernel matrix can be multiplied by the image kernel matrix to obtain the first matrix; the text kernel matrix can be multiplied by the text kernel matrix to obtain the second matrix; the image kernel matrix can be multiplied by the image kernel matrix to obtain the third matrix; the traces of the first matrix, the second matrix, and the third matrix are calculated respectively. The trace refers to the sum of the diagonal elements of the corresponding matrix, and the corresponding first trace, that is, the sum of the diagonal elements of the first matrix, the second trace, and the third trace, are obtained; calculating the square root of the product of the second trace and the third trace can obtain the first intermediate result; dividing the first trace by the first intermediate result can obtain the matching degree.

[0070] Specifically, the matching degree can be calculated according to the following formula:

[0071] ;

[0072] where C(v,t) represents the matching degree between the content description and the flowchart to be processed; Kv represents the text kernel matrix; Kt represents the image kernel matrix; Tr(KvKt) represents the first trace; Tr(KtKt) represents the second trace; Tr(KvKv) represents the third trace.

[0073] S140. Generate the target description of the flowchart to be processed based on the content description and the specified comparison result.

[0074] The specified comparison result is the comparison result between the matching degree and the preset threshold. The preset threshold is a pre-set matching degree threshold. Comparing the matching degree with the preset threshold can roughly determine the consistency degree between the content description and the flowchart to be processed. Among them, the value range of the matching degree is from 0 to 1. The closer the matching degree is to 1, the more consistent the content description and the flowchart to be processed are. Using the comparison result between the matching degree and the preset threshold and the content description, the target description of the flowchart to be processed can be generated. The target description is the text content obtained by comprehensively and accurately describing the information contained in the flowchart to be processed, and this target description is the basis for generating the answer corresponding to the problem to be processed.

[0075] In some embodiments, the preset threshold may include a first threshold and a second threshold, where the second threshold is less than the first threshold. When generating the target description of the flowchart to be processed, if the matching degree is not less than the first threshold, the content description is used as the target description of the flowchart to be processed; if the matching degree is less than the first threshold and not less than the second threshold, the content description is corrected to the target description based on the sub-matching degree between the sub-description corresponding to each specified dimension and the flowchart to be processed; if the matching degree is less than the second threshold, return to execute the step of identifying the element information in the flowchart to be processed according to the specified dimension to generate the content description of the flowchart to be processed and subsequent steps.

[0076] See Figure 3 , which shows a schematic diagram of generating the target description. Compare the matching degree with the first threshold. If the matching degree is not less than the first threshold, it indicates that the content description is highly consistent with the flowchart to be processed, and the content description has comprehensively and accurately described the content in the flowchart to be processed, and the content description can be directly used as the target description. Among them, the first threshold can be set according to actual needs. In the embodiment of the present invention, the first threshold is 0.8, that is, when the matching degree falls within the range of [0.8, 1], the content description can be directly determined as the target description.

[0077] If the matching degree is less than the first threshold, the matching degree can be compared with the second threshold. If the matching degree is not less than the second threshold, it indicates that the content description has a strong consistency with the flowchart to be processed, but there are still some differences, and some content can be corrected according to the specified dimension.

[0078] Optionally, the content description can be corrected to the target description based on the sub-matching degree between the sub-description of the specified dimension and the flowchart to be processed. See Figure 4 , which shows a schematic diagram of correcting the content description to the target description. For example, for each specified dimension, the sub-description corresponding to the specified dimension can be encoded as a sub-text feature; calculate the sub-matching degree between each sub-text feature and the image feature of the flowchart to be processed; determine the sub-description corresponding to the sub-text feature with the lowest sub-matching degree as the description to be corrected; use the prompt word corresponding to the dimension to be corrected and the flowchart to be processed to generate the corrected description corresponding to the description to be corrected, where the dimension to be corrected is the specified dimension corresponding to the description to be corrected; replace the description to be corrected in the content description with the corrected description to obtain the target description.

[0079] For each specified dimension, sub-descriptions corresponding to each specified dimension can be obtained from the content description. The tokenizer in the mapping model can be used to encode the content description, and sub-text features corresponding to the sub-descriptions can be obtained. Calculate the sub-matching degree between each sub-text feature and the image feature. The calculation of the sub-matching degree is the same as the aforementioned matching degree method. To avoid repetition, it will not be elaborated here and can be directly referred to the corresponding content above.

[0080] By calculation, the sub-matching degree corresponding to each sub-description can be obtained. The lower the value of the sub-matching degree, the lower the consistency between the corresponding sub-description and the flowchart to be processed. As an implementation manner, the sub-description corresponding to the sub-text feature with the lowest sub-matching degree can be directly determined as the description to be corrected. For example Figure 4 in, if the sub-matching degree 1 is the minimum value, then the corresponding sub-description 1 is the description to be corrected. As another implementation manner, a minimum threshold can be set according to actual needs or experience, and the sub-description corresponding to the sub-text feature with a sub-matching degree less than the minimum threshold is determined as the description to be corrected.

[0081] Using the prompt word corresponding to the dimension to be corrected and the flowchart to be processed, a corrected description corresponding to the description to be corrected is generated. Among them, the dimension to be corrected refers to the specified dimension corresponding to the description to be corrected. The prompt word corresponding to the dimension to be corrected can be obtained from the recognized prompt words or can be set in advance. Combine the prompt word corresponding to the dimension to be corrected and the flowchart to be processed Figure 1 and input it into the recognition model for processing, and a corrected description can be obtained.

[0082] For example, if the dimension to be corrected is the logical relationship dimension between nodes, the corresponding prompt word can be: Please analyze the following flowchart, list the logical relationship between the nodes in the figure in the form of a detailed text description, and output it in json format. Please include the following information: Describe the logical relationship between the nodes, such as "Node A points to Node B" or "Node B points to Node C or D according to the decision".

[0083] After obtaining the corrected description, the corresponding sub-description in the content description can be replaced with the corrected description to implement the repair of part of the content in the content description and obtain the target description. For example, Figure 4 in, the content description contains n sub-descriptions, and the matching degree of sub-description 1 is the lowest. Therefore, after correcting sub-description 1 and using the corrected description to replace sub-description 1, and keeping the other sub-descriptions unchanged, the target description can be obtained.

[0084] If the matching degree is less than the second threshold, it indicates that the consistency between the content description and the flowchart to be processed is weak, the content description is not accurate enough, and it needs to be regenerated. Therefore, it can return to execute the steps of identifying the element information in the flowchart to be processed according to the specified dimension to generate the content description in the flowchart to be processed and the steps.

[0085] S150. Analyze and reason about the problem to be processed using the target description to generate a target response corresponding to the problem to be processed.

[0086] The target description can be considered as a complete and accurate description of the flowchart to be processed, that is, the content in the target description is consistent with the content in the flowchart to be processed. Analyzing and reasoning about the problem to be processed using the target description can generate a target response corresponding to the problem to be processed.

[0087] In some embodiments, the target description and the problem to be processed can be directly input into a question - answering model. After the question - answering model understands the target description, it infers a target response corresponding to the problem to be processed. Among them, the question - answering model can be a large - language model, which can be specifically used according to actual needs.

[0088] In some embodiments, there are various questions about the flowchart. The problem to be processed may be a general question about the flowchart to be processed or a detailed question about some nodes in the flowchart to be processed. The scopes of the flowchart to be processed involved in these two types of questions are different. To improve the efficiency and accuracy of question - answering, when generating a target response corresponding to the problem to be processed, the problem to be processed can be subjected to text classification to determine the type of the problem to be processed, and the type of the problem includes at least one of a detailed problem and a general problem; if the type of the problem includes a general problem, transcribe the target description, and call the question - answering model to generate a target response corresponding to the problem to be processed based on the transcribed target description; if the type of the problem includes a detailed problem, control the question - answering model to analyze and reason about the problem to be processed according to a preset thought chain to generate a target response corresponding to the problem to be processed. Classifying the problem to be processed and selecting different reasoning methods according to different types of problems to obtain the target response is more accurate.

[0089] Perform text classification on the problem to be processed to determine the type of the problem to be processed. Among them, the type of the problem can include at least one of a detailed problem and a general problem. A detailed problem focuses on the detailed information of a specific node or a certain step in the flowchart to be processed, and usually requires positioning to a specific part of the flowchart to be processed to answer the question. A general problem usually requires understanding the entire flowchart to be processed from a macroscopic level, and may involve the overall structure, main steps or key processes of the flowchart to be processed.

[0090] Optionally, some common words may be included in both the overall problem and the detailed problem. Keywords common in the corresponding types of problems can be summarized based on actual experience as preset keywords, and the preset keywords and preset types are associated in advance as type association relationships. For example, the preset keywords associated with the overall problem may include overall, main, etc., and the preset keywords associated with the detailed problem may include specific, specific steps, a certain node, etc. When determining the problem type, the preset keywords included in the problem to be processed can be directly determined, and then the problem type can be determined based on the type association relationship.

[0091] Optionally, when determining the problem type of the problem to be processed, it can also be analyzed and determined by means of a large language model. It can be pre-set with corresponding prompt words, and the prompt words are input into the large language model so that the large language model can analyze and reason about the problem to be processed and output the corresponding problem type.

[0092] To more quickly and accurately determine the problem type, optionally, a type mapping relationship can be obtained. The type mapping relationship includes the association relationship between the preset type and the preset keyword. If any preset keyword is included in the problem to be processed, the preset type corresponding to the preset keyword is used as the problem type of the problem to be processed. If no preset keyword is included in the problem to be processed, the problem to be processed and the classification template are merged to obtain a classification prompt word. The classification template is a prompt word template for classifying the problem to be processed. The classification prompt word is input into the classification model so that the classification model can semantically understand and classify the problem to be processed to obtain the problem type of the problem to be processed.

[0093] The keyword matching algorithm can be used to determine all the preset keywords included in the problem to be processed. If the problem to be processed includes a preset keyword, the problem type corresponding to the preset keyword can be used as the problem type. If the problem to be processed does not include a preset keyword, it indicates that the keyword method cannot determine the problem type, and the classification model can be used to determine the problem type.

[0094] The classification template refers to a prompt word template for classifying the problem to be processed. The classification template may include problem slots for filling the problem to be processed. After merging the problem to be processed and the classification template, a classification prompt word can be obtained, and then the classification prompt word is input into the classification model so that the classification model outputs the specific problem type. Optionally, since the overall problem and the detailed problem are artificially defined concepts, the relevant concepts and explanations of the overall problem and the detailed problem can be added to the classification template.

[0095] For example, the classification template in the embodiments of the present invention may be:

[0096] "Overall problem: It requires understanding the entire flowchart from a macroscopic level, involving the overall structure, main steps, or key processes of the flowchart.

[0097] Detail problem: Pay attention to the detailed information of a specific node or a certain step in the flowchart. Usually, it is necessary to locate a specific part of the flowchart.

[0098] Determine whether the following problem is an overall problem or a detail problem for the flowchart: Problem: {user_question} Answer: "

[0099] The classification model can be a large language model. According to the reasoning and analysis of the classification model, the problem type corresponding to the problem to be processed can be determined.

[0100] If the problem type includes an overall problem, the target description can be transcribed, and the question-answering model can be called to generate the target answer corresponding to the problem to be processed based on the written target description. Among them, the target description contains sub-descriptions corresponding to each specified dimension, but it is not a complete and fluent text. The question-answering model can transcribe or rewrite the target description to generate a text describing the main process path. Then, based on the transcribed text, analyze and reason about the problem to be processed, generate the target answer and output it.

[0101] If the problem type includes a detail problem, the question-answering model can be controlled to analyze and reason about the problem to be processed according to the preset thinking chain to generate the target answer corresponding to the problem to be processed. For example, it can be to identify the intention of the problem to be processed to determine the target dimension associated with the problem to be processed from the specified dimensions; locate the target node associated with the problem to be processed; combine the target dimension and the target node to generate a preliminary answer corresponding to the problem to be processed; reflect and verify the preliminary answer to obtain the target answer.

[0102] To ensure the accuracy of the answer to the detail problem, the question-answering model can be guided to reason step by step according to the thinking chain to obtain the final target answer. The specific reasoning steps of the preset thinking chain can be set according to actual needs. In the embodiment of the present invention, it can be to first perform intention recognition, that is, to perform intention recognition on the problem to be processed to identify the user intention and determine the target dimension specifically associated with the problem to be processed. For example, if the problem to be processed is "What is the next step of node A", this problem to be processed involves the flow direction of a certain node, and the target dimension can be the arrow and connection dimension.

[0103] Node positioning can then be performed to locate the target node associated with the problem to be processed, that is, to analyze the specific nodes involved in the problem to be processed. In the problem to be processed in the foregoing example, the target node can be Node A. Through intention recognition and node positioning, the questions of the user can be clearly answered, and the specified dimensions and nodes that need to be focused on can be determined. Then, analyze the input, output, and relationships between each node, focus on the information related to the target node, and combine it with the sub-description corresponding to the target dimension to give a preliminary answer to the problem to be processed. Then, reflect and verify the preliminary answer to check whether there are errors in the generated preliminary answer. When it is determined that there are no errors, the final target answer is obtained.

[0104] The problem to be processed can be decomposed into multiple small problems in the way of a chain of thought, and the question-and-answer model can solve each small problem in turn to obtain the final target answer, which can effectively improve the logic of reasoning and then effectively improve the accuracy of question and answer. It should be noted that if the problem to be processed contains both an overall problem and a detailed problem at the same time, the overall problem part in the problem to be processed can be split out and reasoned according to the reasoning method of the overall problem, and the detailed problem part in the problem to be processed can be split out and reasoned according to the reasoning method of the detailed problem. Finally, the reasoning results of the two parts are combined together as the target answer. Optionally, it can also be that if the problem to be processed contains an overall problem, the target answer can be directly obtained according to the reasoning method of the overall problem; if the problem to be processed only contains a detailed problem, the target answer can be obtained by reasoning in the way of a chain of thought. It can be specifically set according to actual needs and is not specifically limited here.

[0105] The question-and-answer solution based on the flowchart provided by the embodiments of the present invention can be applied to various question-and-answer scenarios. For example, taking the enterprise internal question-and-answer system as an example, enterprise internal management involves multiple processes, such as reimbursement processes, approval processes, etc. These processes can all be drawn with corresponding flowcharts for employees to query. Using the solution provided by the embodiments of the present invention can accurately interpret the flowchart and directly give the answer corresponding to the user's question without the user repeatedly viewing the flowchart.

[0106] Through the method provided by the embodiments of the present invention, the problem to be processed and the flowchart to be processed can be obtained; the element information in the flowchart to be processed is recognized according to the specified dimension to generate the content description of the flowchart to be processed; the matching degree between the content description and the flowchart to be processed is calculated to verify the consistency of the content of the content description and the flowchart to be processed by using the matching degree. When the content is highly consistent, it is directly used as the target description. Otherwise, the content description can be corrected in blocks or as a whole to ensure the accuracy of the target description. Finally, according to different question types, different reasoning methods are adopted on the basis of the target description to answer the problem to be processed, which can effectively improve the efficiency and accuracy of question and answer based on the flowchart.

[0107] To better implement the above method, an embodiment of the present invention further provides a question-and-answer processing device based on a flowchart. The question-and-answer processing device based on a flowchart may be specifically integrated in an electronic device, which may be a terminal, a server, or other devices. Among them, the terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, a personal computer, or other devices; the server may be a single server or a server cluster composed of multiple servers.

[0108] For example, in this embodiment, taking the question-and-answer processing device based on a flowchart being specifically integrated in a server as an example, the method of the embodiment of the present invention will be described in detail.

[0109] For example, as Figure 5 shown, the question-and-answer processing device 200 based on a flowchart may include an acquisition module 210, an identification module 220, a calculation module 230, a target generation module 240, and a question-and-answer module 250.

[0110] The acquisition module 210 is configured to acquire a question to be processed and a flowchart to be processed;

[0111] The identification module 220 is configured to identify the element information in the flowchart to be processed according to a specified dimension to generate a content description of the flowchart to be processed, where the content description includes sub-descriptions corresponding to the specified dimension;

[0112] The calculation module 230 is configured to calculate the matching degree between the content description and the flowchart to be processed;

[0113] The target generation module 240 is configured to generate a target description of the flowchart to be processed based on the content description and a specified comparison result, where the specified comparison result is the comparison result of the matching degree and a preset threshold;

[0114] The question-and-answer module 250 is configured to analyze and reason about the question to be processed by using the target description to generate a target reply corresponding to the question to be processed.

[0115] In some embodiments, the identification module 220 is specifically configured to:

[0116] Optimize the image quality of the flowchart to be processed to obtain an optimized flowchart;

[0117] Generate an identification prompt word based on the optimized flowchart and an identification template, where the identification prompt word includes identification requirements and output requirements corresponding to each specified dimension;

[0118] Use the identification prompt word to guide the identification model to identify the element information corresponding to each specified dimension from the optimized flowchart according to the identification requirements;

[0119] For each specified dimension, organize the element information according to the output requirements to obtain a sub-description corresponding to the specified dimension;

[0120] Use all the sub-descriptions corresponding to the specified dimensions as the content description of the flowchart to be processed.

[0121] In some embodiments, the recognition module 220 is specifically configured to:

[0122] Perform grayscale processing on the flowchart to be processed to obtain a grayscale flowchart;

[0123] Perform filtering processing on the grayscale flowchart to obtain a filtered flowchart;

[0124] Perform edge enhancement processing on the edges of the key structures in the filtered flowchart to obtain an edge-enhanced flowchart;

[0125] Perform contrast enhancement on the filtered flowchart to obtain a contrast-enhanced flowchart;

[0126] Perform superposition processing on the edge-enhanced flowchart and the contrast-enhanced flowchart according to the specified weights to obtain an optimized flowchart.

[0127] In some embodiments, the calculation module 230 is specifically configured to:

[0128] Obtain the text features of the content description in the multi-modal space and the image features of the flowchart to be processed in the multi-modal space;

[0129] Multiply the text features by the transpose of the text features to obtain a text kernel matrix;

[0130] Multiply the image features by the transpose of the image features to obtain an image kernel matrix;

[0131] Calculate the matching degree between the content description and the flowchart to be processed based on the text kernel matrix and the image kernel matrix.

[0132] In some embodiments, the preset threshold includes a first threshold and a second threshold, and the second threshold is less than the first threshold. The target generation module 240 is specifically configured to:

[0133] If the matching degree is not less than the first threshold, use the content description as the target description of the flowchart to be processed;

[0134] If the matching degree is less than the first threshold and not less than the second threshold, correct the content description to the target description based on the sub-matching degree between each sub-description corresponding to the specified dimension and the flowchart to be processed;

[0135] If the matching degree is less than the second threshold, return to execute the step of identifying the element information in the flowchart to be processed according to the specified dimension to generate the content description of the flowchart to be processed and subsequent steps.

[0136] In some embodiments, the target generation module 240 is specifically configured to:

[0137] For each specified dimension, encode the sub-description corresponding to the specified dimension into a sub-text feature;

[0138] Calculate the sub-matching degree between each sub-text feature and the image feature of the flowchart to be processed;

[0139] Determine the sub-description corresponding to the sub-text feature with the lowest sub-matching degree as the description to be corrected;

[0140] Use the prompt word corresponding to the dimension to be corrected and the flowchart to be processed to generate the corrected description corresponding to the description to be corrected, where the dimension to be corrected is the specified dimension corresponding to the description to be corrected;

[0141] Replace the description to be corrected in the content description with the corrected description to obtain the target description.

[0142] In some embodiments, the Q&A module 250 is specifically configured to:

[0143] Perform text classification processing on the question to be processed to determine the question type of the question to be processed, where the question type includes at least one of a detail question and an overall question;

[0144] If the question type includes an overall question, transcribe the target description and call the Q&A model to generate the target answer corresponding to the question to be processed based on the transcribed target description;

[0145] If the question type includes a detail question, control the Q&A model to analyze and reason about the question to be processed according to a preset thinking chain to generate the target answer corresponding to the question to be processed.

[0146] In some embodiments, the Q&A module 250 is specifically configured to:

[0147] Obtain a type mapping relationship, where the type mapping relationship includes the association relationship between a preset type and a preset keyword;

[0148] If any preset keyword is included in the question to be processed, use the preset type corresponding to the preset keyword as the question type of the question to be processed;

[0149] If none of the preset keywords are included in the problem to be processed, the problem to be processed is merged with a classification template to obtain a classification prompt word, where the classification template is a prompt word template for classifying the problem to be processed;

[0150] The classification prompt word is input into a classification model so that the classification model can semantically understand and classify the problem to be processed to obtain the problem type of the problem to be processed.

[0151] In some embodiments, the Q&A module 250 is specifically configured to:

[0152] Identify the intent of the problem to be processed to determine the target dimension associated with the problem to be processed from a specified dimension;

[0153] Locate the target node associated with the problem to be processed;

[0154] Combine the target dimension and the target node to generate a preliminary reply corresponding to the problem to be processed;

[0155] Reflect and verify the preliminary reply to obtain the target reply.

[0156] In specific implementation, each of the above modules can be implemented as an independent entity, or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of each of the above modules, reference can be made to the foregoing method embodiments, which will not be elaborated herein.

[0157] As can be seen from the above, the flowchart-based Q&A processing device in this embodiment can obtain the problem to be processed and the flowchart to be processed; identify the element information in the flowchart to be processed according to the specified dimension to generate a content description of the flowchart to be processed; then calculate the matching degree between the content description and the flowchart to be processed, and measure the consistency between the content description and the flowchart to be processed with the specified comparison result of the matching degree and the preset threshold. Then use the content description and the specified comparison result to generate the target description of the flowchart to be processed, which can accurately and comprehensively describe the content in the flowchart to be processed. Finally, use the target description to analyze and infer the target reply of the problem to be processed, which can effectively improve the efficiency and accuracy of flowchart-based Q&A.

[0158] An embodiment of the present invention further provides an electronic device, which can be a device such as a terminal or a server. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0159] In some embodiments, the flowchart-based Q&A processing apparatus may also be integrated into multiple electronic devices. For example, the flowchart-based Q&A processing apparatus may be integrated into multiple servers, and the multiple servers may implement the flowchart-based Q&A processing method of the present invention.

[0160] In this embodiment, the electronic device in this embodiment will be described in detail by taking the example that the electronic device is a server. For example, as Figure 6 shown, it shows a schematic structural diagram of the electronic device involved in the embodiment of the present invention. Specifically:

[0161] The electronic device may include a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, a communication module 350, and other components. Those skilled in the art can understand that Figure 6 the structure of the electronic device shown in does not constitute a limitation on the electronic device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. Among them:

[0162] The processor 310 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 320, and by calling the data stored in the memory 320, it executes various functions of the electronic device and processes data. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 310 either.

[0163] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area. Among them, the program storage area may store the operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created according to the use of the electronic device. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0164] The electronic device further includes a power supply 330 for powering each component. In some embodiments, the power supply 330 may be logically connected to the processor 310 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 330 may further include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator, etc.

[0165] The electronic device may further include an input module 340, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0166] The electronic device may further include a communication module 350. In some embodiments, the communication module 350 may include a wireless module. The electronic device can perform short-distance wireless transmission through the wireless module of the communication module 350, thereby providing users with wireless broadband Internet access. For example, the communication module 350 can be used to help users send and receive emails, browse web pages, and access streaming media, etc.

[0167] Although not shown, the electronic device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 310 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 320 according to the following instructions, and the processor 310 will run the application programs stored in the memory 320, so as to implement the steps in the methods of the embodiments of the present invention.

[0168] For the specific implementation of the above operations, reference can be made to the previous embodiments, which will not be elaborated here.

[0169] As can be seen from the above, the electronic device provided by the embodiments of the present invention can obtain the problem to be processed and the flowchart to be processed; identify the element information in the flowchart to be processed according to the specified dimension to generate a content description of the flowchart to be processed; then calculate the matching degree between the content description and the flowchart to be processed, and measure the consistency between the content description and the flowchart to be processed with the specified comparison result of the matching degree and the preset threshold. Then use the content description and the specified comparison result to generate a target description of the flowchart to be processed, which can accurately and comprehensively describe the content in the flowchart to be processed. Finally, use the target description to analyze and infer the target answer to the problem to be processed, which can effectively improve the efficiency and accuracy of question and answer based on the flowchart.

[0170] Those of ordinary skill in the art can understand that all or part of the steps in the above methods of the embodiments can be completed by instructions, or by controlling related hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0171] To this end, an embodiment of the present invention provides a computer-readable storage medium, which stores multiple instructions that can be loaded by a processor to execute the steps in any one of the flowchart-based question-answering processing methods provided by the embodiments of the present invention.

[0172] Among them, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0173] According to an aspect of the present invention, there is provided a computer program product or a computer program. The computer program product or the computer program includes computer programs / instructions, and the computer programs / instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer programs / instructions from the computer-readable storage medium, and the processor executes the computer programs / instructions, so that the electronic device executes the methods provided in various alternative implementations in the flowchart parsing aspect or the question-answering processing aspect provided in the above embodiments.

[0174] Since the instructions stored in the storage medium can execute the steps in any one of the flowchart-based question-answering processing methods provided by the embodiments of the present invention, the beneficial effects achievable by any one of the flowchart-based question-answering processing methods provided by the embodiments of the present invention can be realized. For details, see the previous embodiments and will not be repeated here.

[0175] The above has introduced in detail a flowchart-based question-answering processing method and device provided by an embodiment of the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A question and answer processing method based on a flowchart, characterized in that The method includes: Obtaining the problem to be processed and the flowchart to be processed; Identifying the element information in the flowchart to be processed according to the specified dimensions to generate a content description of the flowchart to be processed, where the content description includes sub-descriptions corresponding to the specified dimensions, and the specified dimensions include node type and text dimension, arrow and connection dimension, logical relationship dimension between nodes, and spatial position dimension of nodes; Calculating the matching degree between the content description and the flowchart to be processed; Generating a target description of the flowchart to be processed based on the content description and the specified comparison result, where the specified comparison result is the comparison result of the matching degree and the preset threshold, and the preset threshold includes a first threshold and a second threshold, and the second threshold is less than the first threshold; Analyzing and reasoning the problem to be processed by using the target description to generate a target reply corresponding to the problem to be processed; Among them, generating a target description of the flowchart to be processed based on the content description and the specified comparison result includes: if the matching degree is not less than the first threshold, using the content description as the target description of the flowchart to be processed; if the matching degree is less than the first threshold and not less than the second threshold, for each specified dimension, encoding the sub-description corresponding to the specified dimension as a sub-text feature; calculating the sub-matching degree between each sub-text feature and the image feature of the flowchart to be processed; determining the sub-description corresponding to the sub-text feature with the lowest sub-matching degree as the description to be corrected; generating a corrected description corresponding to the description to be corrected by using the prompt word corresponding to the dimension to be corrected and the flowchart to be processed, where the dimension to be corrected is the specified dimension corresponding to the description to be corrected; replacing the description to be corrected in the content description with the corrected description to obtain the target description; if the matching degree is less than the second threshold, returning to execute the step of identifying the element information in the flowchart to be processed according to the specified dimensions to generate a content description of the flowchart to be processed and subsequent steps.

2. The method according to claim 1, wherein The step of identifying the element information in the flowchart to be processed according to the specified dimensions to generate a content description of the flowchart to be processed includes: Performing an optimization process on the image quality of the flowchart to be processed to obtain an optimized flowchart; Generating an identification prompt word based on the optimized flowchart and the identification template, where the identification prompt word includes the identification requirements and output requirements corresponding to each specified dimension; Guiding the identification model to identify the element information corresponding to each specified dimension from the optimized flowchart according to the identification requirements by using the identification prompt word; Organizing the element information according to the output requirements for each specified dimension to obtain a sub-description corresponding to the specified dimension; Taking the sub-descriptions corresponding to all specified dimensions as the content description of the flowchart to be processed.

3. The method according to claim 2, wherein The step of performing an optimization process on the image quality of the flowchart to be processed to obtain an optimized flowchart includes: Performing grayscale processing on the flowchart to be processed to obtain a grayscale flowchart; Performing filtering processing on the grayscale flowchart to obtain a filtered flowchart; Enhance the edges of the key structures in the filtering flowchart to obtain an edge-enhanced flowchart; Enhance the contrast of the filtering flowchart to obtain a contrast-enhanced flowchart; Superimpose the edge-enhanced flowchart and the contrast-enhanced flowchart according to the specified weights to obtain an optimized flowchart.

4. The method according to claim 1, wherein Calculating the matching degree between the content description and the flowchart to be processed includes: Obtain the text features of the content description in the multimodal space and the image features of the flowchart to be processed in the multimodal space; Multiply the text features by the transpose of the text features to obtain a text kernel matrix; Multiply the image features by the transpose of the image features to obtain an image kernel matrix; Calculate the matching degree between the content description and the flowchart to be processed based on the text kernel matrix and the image kernel matrix.

5. The method according to claim 1, wherein Using the target description to analyze and reason about the problem to be processed and generate a target response corresponding to the problem to be processed includes: Perform text classification on the problem to be processed to determine the problem type of the problem to be processed, where the problem type includes at least one of a detail problem and an overall problem; If the problem type includes an overall problem, transcribe the target description and call a question-answering model to generate a target response corresponding to the problem to be processed based on the transcribed target description; If the problem type includes a detail problem, control the question-answering model to analyze and reason about the problem to be processed according to a preset thinking chain to generate a target response corresponding to the problem to be processed.

6. The method according to claim 5, wherein Performing text classification on the problem to be processed to determine the problem type of the problem to be processed includes: Obtain a type mapping relationship, where the type mapping relationship includes the association relationship between a preset type and a preset keyword; If any preset keyword is included in the problem to be processed, use the preset type corresponding to the preset keyword as the problem type of the problem to be processed; If no preset keyword is included in the problem to be processed, merge the problem to be processed with a classification template to obtain a classification prompt, where the classification template is a prompt template for classifying the problem to be processed; Input the classification prompt into a classification model so that the classification model can perform semantic understanding and classification on the problem to be processed to obtain the problem type of the problem to be processed.

7. The method according to claim 5, characterized in that, Controlling the question-answering model to analyze and reason about the problem to be processed according to a preset thinking chain to generate a target response corresponding to the problem to be processed includes: Perform intention recognition on the problem to be processed to determine the target dimension associated with the problem to be processed from a specified dimension; Locate the target node associated with the problem to be processed; Generate a preliminary response corresponding to the problem to be processed in combination with the target dimension and the target node; Perform reflection and verification on the preliminary response to obtain a target response.

8. A question and answer processing device based on a flowchart, the device being used to implement the method according to any one of claims 1-7, characterized in that, The device includes: An acquisition module for acquiring a problem to be processed and a flowchart to be processed; An identification module, configured to identify the element information in the to-be-processed flow chart according to a specified dimension, so as to generate a content description of the to-be-processed flow chart, where the content description includes sub-descriptions corresponding to the specified dimension, and the specified dimension includes node type and text dimension, arrow and connection dimension, logical relationship dimension between nodes, and spatial position dimension of nodes; A calculation module, configured to calculate the matching degree between the content description and the to-be-processed flow chart; A target generation module, configured to generate a target description of the to-be-processed flow chart based on the content description and a specified comparison result, where the specified comparison result is a comparison result between the matching degree and a preset threshold, and the preset threshold includes a first threshold and a second threshold, and the second threshold is less than the first threshold; A question-and-answer module, configured to analyze and reason about the to-be-processed question by using the target description, and generate a target answer corresponding to the to-be-processed question; Wherein, generating a target description of the to-be-processed flow chart based on the content description and a specified comparison result includes: if the matching degree is not less than the first threshold, using the content description as the target description of the to-be-processed flow chart; if the matching degree is less than the first threshold and not less than the second threshold, for each specified dimension, encoding the sub-description corresponding to the specified dimension into a sub-text feature; calculating the sub-matching degree between each sub-text feature and the image feature of the to-be-processed flow chart; determining the sub-description corresponding to the sub-text feature with the lowest sub-matching degree as the to-be-corrected description; generating a corrected description corresponding to the to-be-corrected description by using a prompt word corresponding to the to-be-corrected dimension and the to-be-processed flow chart, where the to-be-corrected dimension is the specified dimension corresponding to the to-be-corrected description; replacing the to-be-corrected description in the content description with the corrected description to obtain a target description; if the matching degree is less than the second threshold, returning to execute the step of identifying the element information in the to-be-processed flow chart according to the specified dimension to generate a content description of the to-be-processed flow chart and subsequent steps.

Images