Problem-solving method, device, storage medium, and electronic device

Abstract

The present application discloses a method, device, storage medium and electronic device for solving a problem, wherein the method includes: obtaining a target question, wherein the target question includes text and a geometric picture; obtaining M semantic feature vectors corresponding to the text, wherein the semantic feature vector is a feature vector corresponding to each phrase in the text, and M is a positive integer greater than zero; obtaining N sub-graph feature vectors corresponding to the geometric picture, wherein the sub-graph feature vector is a feature vector corresponding to each sub-graph in the geometric picture, wherein the sub-graph is a picture constructed by any two superscripts in the geometric picture, and N is a positive integer greater than zero; obtaining a solution to the target question based on the M semantic feature vectors and the N sub-graph feature vectors. The accuracy of solving the target question is improved by using the embodiments of the present application.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a problem-solving method, device, storage medium, and electronic device. Background Art

[0002] Artificial intelligence technology is developing rapidly and has been extensively applied in various fields. Education is also gradually gaining prominence. Automatic problem-solving technology is a key enabler of intelligent education. In recent years, answering systems and online answering systems for intelligent education services have been put into practical use. The introduction of AI technology has reduced teachers' workload and made it easier for students to solve problems. Summary of the Invention

[0003] In order to solve the above problems, the embodiments of the present application provide a problem-solving method, device, storage medium and electronic device, which improve the accuracy of solving the target problem.

[0004] In a first aspect, an embodiment of the present application provides a problem-solving method, the method comprising:

[0005] Obtaining a target topic, wherein the target topic includes text and a geometric picture;

[0006] Obtaining M semantic feature vectors corresponding to the text, where the semantic feature vectors are feature vectors corresponding to each phrase in the text, and M is a positive integer greater than zero;

[0007] Obtain N sub-image feature vectors corresponding to the geometric image, where the sub-image feature vectors are feature vectors corresponding to each sub-image in the geometric image, and the sub-image is an image constructed by any two subscripts in the geometric image, and N is a positive integer greater than zero;

[0008] A solution to the target question is obtained based on the M semantic feature vectors and the N subgraph feature vectors.

[0009] In a second aspect, an embodiment of the present application provides a problem-solving device, the device comprising:

[0010] A target topic acquisition module is used to acquire a target topic, wherein the target topic includes text and a geometric picture;

[0011] A first vector acquisition module is configured to acquire M semantic feature vectors corresponding to the text, wherein the semantic feature vectors are feature vectors corresponding to each phrase in the text, and M is a positive integer greater than zero;

[0012] A second vector acquisition module is configured to acquire N sub-image feature vectors corresponding to the geometric image, where the sub-image feature vectors are feature vectors corresponding to each sub-image in the geometric image, and the sub-image is an image constructed by any two sub-markers in the geometric image, and N is a positive integer greater than zero;

[0013] A problem-solving result acquisition module is used to obtain a problem-solving result of the target problem based on the M semantic feature vectors and the N subgraph feature vectors.

[0014] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps of the above-mentioned first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps of the above-mentioned first aspect.

[0016] The beneficial effects of the technical solutions provided by some embodiments of the present application include at least:

[0017] In an embodiment of the present application, by obtaining a target question including text and a geometric picture, M semantic feature vectors corresponding to the text are obtained, where the semantic feature vector is a feature vector corresponding to each phrase in the text, and M is a positive integer greater than zero, N sub-graph feature vectors corresponding to the geometric picture are obtained, where the sub-graph feature vector is a feature vector corresponding to each sub-graph in the geometric picture, and a sub-graph is a picture constructed by any two subscripts in the geometric picture, and N is a positive integer greater than zero, and the solution to the target question is obtained based on the M semantic feature vectors and the N sub-graph feature vectors. By making full use of the text and the set of pictures of the target question, the feature vector corresponding to the text in the target question and the feature vector corresponding to each sub-graph in the geometric picture are fused to solve the target question, which fully utilizes the information of the text and the geometric picture and improves the accuracy of solving the target question. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A system architecture diagram of a problem-solving method provided in an embodiment of the present application;

[0020] Figure 2A flowchart of a problem-solving method provided in an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram illustrating an example of a target question input method provided in an embodiment of the present application;

[0022] Figure 4 This is a schematic diagram illustrating an example of a target question input method provided in an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram illustrating an example of a target question input method provided in an embodiment of the present application;

[0024] Figure 6 This is a schematic diagram of an example of a geometric picture provided in an embodiment of the present application;

[0025] Figure 7 A flowchart of a problem-solving method provided in an embodiment of the present application;

[0026] Figure 8 This is a schematic diagram of an example of obtaining a word vector provided in an embodiment of the present application;

[0027] Figure 9 This is a schematic diagram of an example of obtaining a semantic feature vector provided in an embodiment of the present application;

[0028] Figure 10 This is a schematic diagram illustrating an example of identifying a geometric image corner frame provided by an embodiment of the present application;

[0029] Figure 11 This is a schematic diagram illustrating an example of the accuracy of identifying a corner mark frame provided in an embodiment of the present application;

[0030] Figure 12 An example schematic diagram of a construction subgraph provided in an embodiment of the present application;

[0031] Figure 13 This is a schematic diagram of an example of obtaining a subgraph feature vector provided in an embodiment of the present application;

[0032] Figure 14 A flowchart of a problem-solving method provided in an embodiment of the present application;

[0033] Figure 15 This is a schematic diagram of an example of obtaining a fused feature vector provided in an embodiment of the present application;

[0034] Figure 16 This is a schematic diagram illustrating an example of obtaining a decoding vector provided in an embodiment of the present application;

[0035] Figure 17 This is a schematic diagram illustrating an example of obtaining an index vector probability provided in an embodiment of the present application;

[0036] Figure 18 A schematic structural diagram of a problem-solving device provided in an embodiment of the present application;

[0037] Figure 19 A schematic structural diagram of a problem-solving device provided in an embodiment of the present application;

[0038] Figure 20 A schematic structural diagram of a problem-solving device provided in an embodiment of the present application;

[0039] Figure 21 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0041] In the following introduction, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following introduction provides multiple embodiments of the present application. Different embodiments can be replaced or combined, so the present application can also be considered to include all possible combinations of the same and / or different embodiments described. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, even though the embodiment may not be clearly described in the following text.

[0042] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the elements described without departing from the scope of the present application. Various examples may appropriately omit, replace, or add various processes or components. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted, or combined. In addition, features described in some examples may be combined in other examples.

[0043] Artificial intelligence technology is developing rapidly and has been extensively applied in various fields. Its application in education is also gradually emerging. Automatic problem-solving technology is a key technology for intelligent education. In recent years, intelligent education service-oriented answering systems and online answering systems have been gradually put into practical application. Currently, automated geometry problem solving typically involves fusing the entire text of the problem with the entire accompanying image, or fusing the text feature vector of the entire text with the image feature vector of the entire accompanying image. This fusion only solves the problem, making relatively superficial use of the accompanying image information, and not fully integrating the text and image information. Consequently, the accuracy of the problem solution is insufficient.

[0044] Based on this, an embodiment of the present application provides a method for solving a problem, which obtains a target question including text and a geometric picture, obtains M semantic feature vectors corresponding to the text, where the semantic feature vector is a feature vector corresponding to each phrase in the text, and M is a positive integer greater than zero, obtains N sub-graph feature vectors corresponding to the geometric picture, where the sub-graph feature vector is a feature vector corresponding to each sub-graph in the geometric picture, and the sub-graph is a picture constructed by any two subscripts in the geometric picture, and N is a positive integer greater than zero, and obtains the solution to the target question based on the M semantic feature vectors and the N sub-graph feature vectors. By making full use of the text and the set of pictures of the target question, the feature vector corresponding to the text in the target question and the feature vector corresponding to each sub-graph in the geometric picture are fused to solve the target question, thereby improving the accuracy of solving the target question.

[0045] See Figure 1, which is a schematic diagram of the system architecture of a problem-solving method provided in an embodiment of the present application, the system includes a terminal device, and the terminal device may include but is not limited to a smart phone, a smart interactive tablet, a personal computer, a desktop computer, a tablet computer, a PDA, a laptop, an all-in-one computer, a car multimedia, etc. By obtaining a target question including text and a geometric picture; obtaining m word vectors corresponding to each first character in the text, m is a positive integer greater than zero, inputting the m word vectors into the semantic encoder, and outputting M semantic feature vectors corresponding to the text, M is a positive integer greater than zero and M is less than or equal to m; obtaining the superscript box corresponding to the superscript in the geometric picture, based on the superscript box whose horizontal and vertical coordinates of any two center points are different, constructing N subgraphs with the center points of the superscript box as two diagonal vertices as rectangular frames, inputting the N subgraphs into the image feature extraction model, and outputting N subgraph feature vectors; determining the subgraphs corresponding to the M semantic feature vectors. The semantic feature vector with the highest matching degree among the N sub-graph feature vectors is input into the attention mechanism model with the corresponding sub-graph feature vector for alignment and fusion, and the remaining semantic feature vectors are retained to obtain M fused feature vectors. The M fused feature vectors are input into the decoder, and K decoding vectors are output, where K is a positive integer greater than zero. K index probability vectors corresponding to the K decoding vectors are obtained, and the maximum index probability corresponding to each index probability vector is determined in the K index probability vectors. The second character corresponding to each maximum index probability is obtained, and the solution to the target question is generated based on each second character.

[0046] The method inputs the text and geometric image of the target question into the terminal device, obtains the word vector corresponding to each first character in the text, inputs all the word vectors into the semantic encoder, and outputs M semantic feature vectors corresponding to the text. At the same time, by obtaining the superscript box corresponding to each superscript in the geometric image, a rectangular box is constructed with the center points corresponding to the superscript boxes whose horizontal and vertical coordinates are different for two arbitrary center coordinates as two diagonal vertices, and N subgraphs are constructed. The N subgraphs are input into the image feature extractor to obtain N subgraph feature vectors corresponding to the N subgraphs. The subgraph feature vectors are aligned and fused with the semantic feature vector to obtain corresponding M fused feature vectors. The M fused feature vectors are decoded to obtain K decoding vectors. The K decoding vectors are converted to obtain the solution steps of the target question, thereby improving the accuracy of solving the target question.

[0047] exist Figure 1 Based on the system architecture shown below, Figures 2 to 17 , the problem-solving method provided in the embodiment of this application is introduced in detail.

[0048] See Figure 2 , provides a flowchart of a problem-solving method for the embodiment of this application. Figure 2 As shown, the problem-solving method may include the following steps:

[0049] S101, obtaining a target topic, where the target topic includes text and a geometric picture.

[0050] In one embodiment, when a user wants to solve a geometry problem, the user inputs the text portion of the geometry problem and a geometry picture into a terminal device, and the terminal device receives the target problem input by the user.

[0051] The way in which the user inputs the target topic into the terminal device may include but is not limited to voice input, manual input, and terminal device scanning and recognition.

[0052] The text of the target topic may include but is not limited to words, numbers, operation symbols, uppercase letters, lowercase letters, etc.

[0053] The geometric pictures are pictures corresponding to the geometric figures in the target questions.

[0054] like Figure 3 As shown, the user can input the target topic into the terminal device by voice input only; Figure 4 As shown, the user can input the target topic into the terminal device manually; Figure 5 As shown, the user may input the target topic into the terminal device by scanning the target topic through a camera of the terminal device, and the terminal device obtains information such as text and geometric images related to the target topic through recognition.

[0055] S102: Obtain M semantic feature vectors corresponding to the text, where the semantic feature vectors are feature vectors corresponding to each phrase in the text, and M is a positive integer greater than zero.

[0056] In one embodiment, the text of the target topic contains M phrases, and the semantic feature vectors corresponding to each phrase are obtained to obtain M semantic feature vectors corresponding to the M phrases, where M is a positive integer greater than zero.

[0057] The method of obtaining the semantic feature vector of the text may include but is not limited to inputting the text into a semantic encoder, etc.

[0058] Semantic encoders may include but are not limited to Transformer networks, Long Short-Term Memory (LSTM) networks, and the like.

[0059] The Transformer abandons traditional convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Its entire network architecture is composed entirely of an attention mechanism. More precisely, the Transformer consists solely of a self-attention mechanism and a feed-forward neural network.

[0060] LSTM is a time-recurrent neural network designed specifically to address the long-term dependency issues inherent in typical RNNs (recurrent neural networks). All RNNs have a chain-like structure of repeating neural network modules. In a standard RNN, this repeating module consists of a very simple structure, such as a tanh (hyperbolic function) layer.

[0061] S103, obtaining N sub-image feature vectors corresponding to the geometric image, where the sub-image feature vectors are feature vectors corresponding to each sub-image in the geometric image, and the sub-image is a picture constructed by any two sub-markers in the geometric image, and N is a positive integer greater than zero.

[0062] In one embodiment, a geometric image of the target question is input into a neural network model of the target detection task, and the corner marks of the geometric figures in the geometric image are detected. The geometric image of the target question includes multiple corner marks. A rectangle is constructed according to any two corner marks. The rectangle is a sub-image composed of the corresponding two corner marks. Accordingly, the geometric image of the target question constitutes N sub-images. The N sub-images are input into the image feature extraction model to obtain the sub-image feature vector corresponding to each sub-image, where N is a positive integer greater than zero.

[0063] Among them, neural networks that can achieve target detection tasks include but are not limited to deep convolutional neural network models (RetinaNet) and the like.

[0064] RetinaNet is a neural network model currently used for object detection tasks in the field of computer vision and is currently widely used in object detection.

[0065] Before using the model, this solution uses geometric images with manually labeled corner marks as sample data to train the model. RetinaNet learns the data patterns in the sample data through neural network training, thereby gaining the ability to detect corner mark boxes from a new geometric image.

[0066] Image feature extraction models may include but are not limited to deep convolutional neural networks (EfficientNet) and the like.

[0067] EfficientNet is a neural network model currently used in computer vision for image feature extraction and is widely used in tasks such as image recognition and feature encoding. This solution encodes the N extracted subgraphs using a pre-trained EfficientNet. Each subgraph is converted into a feature vector, each of which contains the visual information contained in the subgraph.

[0068] like Figure 6 As shown in the figure, the image is detected by the target detection network, and three corner marks A, B, and C are detected. Then, a sub-image can be constructed from A and B, a sub-image can be constructed from A and C, and a sub-image can be constructed from B and C. The three sub-images are input into the image feature extraction model to obtain three sub-image feature vectors corresponding to the three sub-images.

[0069] S104: Obtain a solution to the target problem based on the M semantic feature vectors and the N subgraph feature vectors.

[0070] In one embodiment, by inputting M semantic feature vectors and N subgraph feature vectors into the attention mechanism model, the information between the semantic feature vectors and the subgraph feature vectors is aligned and fused, thereby solving the target problem and outputting the solution result.

[0071] Among them, the attention mechanism model may include but is not limited to the cross-attention model (CrossAttention), the software attention mechanism model (SoftAttention), and the like.

[0072] The attention mechanism model mainly solves the alignment relationship between different modal information, and represents the two different modal representations as one thing by corresponding fusion.

[0073] The fusion method between the semantic feature vector and the subgraph feature vector may include but is not limited to adding the two vectors, multiplying the two vectors, connecting the two vectors end to end, fusing the values ​​in the two vectors, etc.

[0074] For example, if one of the semantic feature vectors is [0.2, 0.5, 1, 2.5, 1.5] and the subgraph feature vector is [2, 1.2, 0.3, 1, 0.5], and the two vectors are fused by adding them together, the fused vector can be expressed as [2.2, 1.7, 1.3, 3.5, 0.5].

[0075] The problem-solving results may include but are not limited to problem-solving steps, answers to questions, problem-solving methods, etc.

[0076] In an embodiment of the present application, by obtaining a target question including text and a geometric picture, M semantic feature vectors corresponding to the text are obtained, where the semantic feature vector is a feature vector corresponding to each phrase in the text, and M is a positive integer greater than zero, N sub-graph feature vectors corresponding to the geometric picture are obtained, where the sub-graph feature vector is a feature vector corresponding to each sub-graph in the geometric picture, and a sub-graph is a picture constructed by any two subscripts in the geometric picture, and N is a positive integer greater than zero, and the solution to the target question is obtained based on the M semantic feature vectors and the N sub-graph feature vectors. By making full use of the text and the set of pictures of the target question, the feature vector corresponding to the text in the target question and the feature vector corresponding to each sub-graph in the geometric picture are fused to solve the target question, thereby improving the accuracy of solving the target question.

[0077] See Figure 7 , provides a flowchart of a problem-solving method for the embodiment of this application. Figure 7 As shown, the problem-solving method may include the following steps:

[0078] S201, obtaining a target topic, where the target topic includes text and a geometric picture.

[0079] The specific implementation of this step can refer to step S101 and will not be described in detail here.

[0080] S202, obtaining m word vectors corresponding to the text, where the word vectors are word vectors corresponding to the first characters in the text, and m is a positive integer greater than zero.

[0081] In one embodiment, a dictionary is first constructed for all first characters in the entire question bank, which corresponds one-to-one to the first character and index. In the dictionary, each first character has only a unique numeric index. The number of types of first characters in the question bank is the size of the numeric index.

[0082] By inputting the text of the target question into the word vector acquisition neural network, the text of the target question contains m first characters, and then the word vectors corresponding to the m first characters can be obtained through the word vector acquisition neural network, where m is a positive integer greater than zero.

[0083] The word vector acquisition neural network may include but is not limited to a word embedding network, etc.

[0084] The first character is all characters in the text of the target topic, which may include but are not limited to words, numbers, uppercase letters, lowercase letters, operation symbols, etc.

[0085] Assume that the size of the index table is N and the dimension of the feature vector is D. Then Word-Embedding is a trainable matrix of dimension [N, D], and its function is to convert each first character in the index table into a word vector of dimension [1, D].

[0086] For example, if the number of types of first characters in the question bank is 1000, the size of the digital index is 1000, and the dimension of the feature vector is 512, then Word-Embedding is a matrix of 1000 * 512 dimensions. If the number of types of first characters included in the text of the target question is 20, after passing the target question through the Word-Embedding network, 20 word vectors of dimension 1 * 512 corresponding to the 20 first characters are obtained.

[0087] When inputting the text of the target question into the Word-Embedding network, each first character corresponds to a digital index, and the position of each character in the digital index is fixed. When multiplying the digital index of each first character in the text of the target question by the matrix of the Word-Embedding network, only the value at the position of this first character in the digital index is 1, and the values at the other positions are all 0. The purpose is to map each first character in the text of the target question into the high-dimensional space represented by the Word-Embedding network.

[0088] For example, as Figure 8 shown, it is the process of obtaining the corresponding word vector for the first character "angle" through the Word-Embedding network. If the number of types of first characters in the question bank is 7, then the matrix of the Word-Embedding network is a matrix of dimension [7, 512]. The first character "angle" is in the 5th position in the digital index table. When inputting the target text into the Word-Embedding network, only the value at the 5th position of the digital index corresponding to "angle" is 1, and the values at the other positions are all 0. Multiply this digital index by the matrix of the Word-Embedding network, and the result is a vector value of dimension [1, 512] (which is the vector value of the 5th dimension in the matrix of the Word-Embedding network), and this vector value is the word vector corresponding to the first character "angle".

[0089] Here, taking the number of types of first characters in the question bank as 7 as an example is just for illustration and does not limit the number of types of first characters in the question bank.

[0090] S203, Input the m word vectors into the semantic encoder, and output M semantic feature vectors corresponding to the text, where M is less than or equal to m.

[0091] In one embodiment, after the text of the target title is input into the word vector acquisition neural network, m corresponding word vectors are obtained. The m word vectors are input into the semantic encoder, and the self-attention mechanism in the semantic encoder encodes the semantic relationships between different word vectors to obtain M semantic feature vectors corresponding to each phrase in the text. The number of obtained semantic feature vectors is less than or equal to the number of word vectors.

[0092] Among them, the semantic encoder may include, but is not limited to, a Transformer network, a Long Short-Term Memory (LSTM) network, and the like.

[0093] The semantic feature vector may be a semantic feature vector corresponding to a single character, or a semantic feature vector corresponding to a phrase composed of multiple first characters. For example, the three first characters "triangle", "angle", and "shape" obtain three word vectors through step S202. The semantic feature vector may be the three semantic feature vectors corresponding to "triangle", "angle", and "shape" respectively, or a single semantic feature vector corresponding to "triangle", etc.

[0094] The self-attention mechanism of the semantic encoder enables the semantic encoder to effectively encode the semantic relationships between different word vectors. Passing the word vectors through the semantic encoder will obtain a set of feature vectors containing complex semantic relationships such as morphology and grammar. This set of feature vectors contains all the information of the target title text.

[0095] As Figure 9 shown, m word vectors corresponding to the text of the target title are input into the Transformer network. By encoding the semantic relationships between different word vectors, M semantic feature vectors corresponding to the text of the target title are obtained. The number of semantic feature vectors is less than or equal to the number of word vectors.

[0096] For example, in the example of step S202, if the number of types of the first characters in the question bank is 7, then the corresponding generated word vectors are 7 [1, 512]-dimensional word vectors. The 7 word vectors are input into the semantic encoder. According to the encoding of the semantic relationships between different word vectors, the corresponding semantic feature vectors are obtained. The number of obtained semantic feature vectors is 7, or less than 7.

[0097] S204. Obtain the bounding boxes corresponding to each subscript in the geometric picture.

[0098] In one embodiment, the subgraph feature vectors of the geometric picture are subgraph feature vectors corresponding to subgraphs constructed according to the center points of the bounding boxes of the figures in the geometric picture.

[0099] The geometric picture is input into the neural network model of the object detection task. The model detects the superscripts of the figures in the geometric picture and the corresponding superscript frames. After receiving the picture, the model uses the upper left corner of the picture as the coordinate origin by default. The coordinates of the center point corresponding to each superscript frame can be calculated through the position of each superscript frame and the coordinates corresponding to the four fixed points of the superscript frame. The coordinates of the center point represent the position of the superscript, and the size of each superscript frame can be different.

[0100] Among them, neural networks that can achieve target detection tasks include but are not limited to deep convolutional neural network models (RetinaNet) and the like.

[0101] The default coordinate origin of the image is the upper left corner of the geometric image. You can also use the lower left corner, upper right corner, lower right corner, or any point of the image as the coordinate origin of the image.

[0102] like Figure 10 As shown in the figure, the result of the neural network model for the target detection task detecting the corner mark frame in the geometric picture is that the corner mark frames corresponding to corner mark A, corner mark B, corner mark C, corner mark D, corner mark E, and corner mark F are detected. In the process of detecting the corner mark frame, a judgment is also made on whether the detected corner mark frame is correct, as shown in the figure. Figure 11 As shown, the detection of the corner mark C and the probability of detecting the presence of the corner mark at this position are judged. The accuracy rate of detecting the corner mark box at this position in the figure is 1, indicating that the model detects the corner mark box at this position correctly.

[0103] The corner mark box is a rectangular box. The coordinates of the center point of each corner mark box can be calculated using the coordinates of the corner mark box's four vertices. The average of the corner mark box's minimum and maximum horizontal coordinates is used as the horizontal coordinate of the center point, and the average of the corner mark box's minimum and maximum vertical coordinates is used as the vertical coordinate of the center point. For example, if the coordinates of the corner mark box's four vertices are (0, 0) for the upper left corner, (0, 4) for the lower left corner, (5, 0) for the upper right corner, and (5, 4) for the lower right corner, the coordinates of the center point of the corner mark box are calculated to be (2.5, 2).

[0104] Because the coordinates of the superscript frame are the pixels of the geometric image, when determining the coordinates of the superscript frame, the values ​​of the coordinates are larger than those of the normal graphics.

[0105] S205 , draw a rectangular frame with the center points of the sub-marked frame whose horizontal coordinates and vertical coordinates of any two center points are different as two diagonal vertices to obtain N sub-graphs.

[0106] In one embodiment, as in the method of step S204 above, the superscript frame corresponding to the graphic in the picture is obtained, and the coordinates of the center point of the superscript frame are calculated based on the coordinates of the four vertices of each superscript frame, and the coordinates are used to represent the superscript frame at that position. According to the coordinates of the center points of any two superscript frames, it is judged whether the current two superscript frames can construct a subgraph. If the horizontal coordinates and vertical coordinates of the two center point coordinates are different, it is determined that the superscript frames corresponding to the current two center point coordinates can construct a subgraph. If one of the horizontal coordinates and vertical coordinates of the current two center point coordinates is the same, it is determined that the superscript frames corresponding to the current two center point coordinates cannot construct a subgraph. According to the center point coordinates corresponding to any two superscript frames that can construct a subgraph, a rectangle is made for the two diagonal vertices, and the rectangle is the subgraph constructed corresponding to the two superscript frames. Through this method, N subgraphs corresponding to the geometric picture in the target question are obtained.

[0107] like Figure 12 As shown, the left side of the figure is the figure after the neural network model of the target detection task detects the superscript frame of the geometric image. By calculating the coordinates of the center points of each superscript frame, it is determined whether a subgraph can be constructed between any two superscript frames. The right side of the figure is a subgraph constructed with the center points of superscript B and superscript E as diagonal vertices, that is, the shaded part in the figure. The information of the subgraph is the information selected by the dotted part in the figure.

[0108] If the neural network model of the target detection task detects n corner marks in the geometric image, then at most n corner marks can be constructed through this step. 2 Sub-graphs.

[0109] S206: Input the N sub-images into an image feature extraction model, and output N sub-image feature vectors.

[0110] In one embodiment, N subgraphs corresponding to the geometric image in the target question are obtained through the above step S205, and the N subgraphs are input into the image feature extraction model. The N subgraphs are converted into N subgraph feature vectors through tasks such as image recognition and image encoding.

[0111] Among them, the image feature extraction model may include but is not limited to a deep convolutional neural network model (EfficientNet) and the like.

[0112] After N sub-images are input into the image feature extraction model, N [1, D]-dimensional sub-image feature vectors are obtained, such as Figure 12 As shown, the subgraphs in the geometric picture are constructed in the same way, such as Figure 12The geometric picture shown can construct 11 sub-graphs, which are sub-graphs constructed with superscript A and superscript E, superscript A and superscript F, superscript A and superscript C, superscript B and superscript E, superscript B and superscript F, superscript B and superscript D, superscript C and superscript E, superscript C and superscript F, superscript D and superscript E, superscript D and superscript F, and superscript E and superscript F. These 11 sub-graphs are input into the image feature extraction model, and 11 sub-graph feature vectors corresponding to these 11 sub-graphs are output through tasks such as image recognition and feature encoding.

[0113] like Figure 13 As shown in the figure, N subgraphs are passed through a deep convolutional neural network model (EfficientNet) to obtain corresponding N subgraph feature vectors, each of which is a [1, D]-dimensional vector.

[0114] S207 : Obtain a solution to the target problem based on the M semantic feature vectors and the N subgraph feature vectors.

[0115] The specific implementation of this step can refer to step S104 and will not be described in detail here.

[0116] In an embodiment of the present application, a target question containing text and a geometric picture input by a user is received; m word vectors of each first character in the text are obtained, the m word vectors are input into a semantic encoder, and M semantic feature vectors corresponding to the text are output, where the m Ms are positive integers greater than zero and M is less than or equal to m; at the same time, the superscript box corresponding to each superscript in the geometric picture is obtained, and a rectangle is drawn with the center coordinates of the superscript box whose horizontal coordinates and vertical coordinates are different for any two center coordinates as two diagonal vertices to obtain N sub-graphs, which are input into an image feature extraction model, and N sub-graph feature vectors are output. The solution to the target question is obtained based on the M semantic feature vectors and the N sub-graph feature vectors. The user inputs the text and geometric image of the target question into the terminal device, obtains the word vectors corresponding to the m first characters in the text, and passes the word vectors through the semantic encoder to obtain M semantic feature vectors corresponding to the text. At the same time, the superscript boxes corresponding to each superscript of the geometric image are obtained, and the center point coordinates of the superscript boxes are calculated. The center points of the superscript boxes with different horizontal and vertical coordinates of any two center points are used as two diagonal vertices to construct N sub-graphs. The N sub-graphs are input into the image feature extraction model to obtain N sub-graph feature vectors corresponding to the geometric image. The text feature vector and the sub-graph feature vector are fused to solve the target question. By fusing the text feature vector corresponding to the text in the target question and the sub-graph feature vector corresponding to each sub-graph in the geometric image to solve the target question, the accuracy of solving the target question is improved.

[0117] See Figure 14, provides a flowchart of a problem-solving method for the embodiment of this application. Figure 14 As shown, the problem-solving method may include the following steps:

[0118] S301, obtaining a target topic, wherein the target topic includes text and a geometric picture.

[0119] The specific implementation of this step can refer to step S101 and will not be described in detail here.

[0120] S302: Obtain M semantic feature vectors corresponding to the text, where the semantic feature vectors are feature vectors corresponding to each phrase in the text, and M is a positive integer greater than zero.

[0121] The specific implementation of this step can refer to step S102 and will not be described in detail here.

[0122] S303, obtaining N sub-image feature vectors corresponding to the geometric image, where the sub-image feature vectors are feature vectors corresponding to each sub-image in the geometric image, and the sub-image is a picture constructed by any two sub-markers in the geometric image, and N is a positive integer greater than zero.

[0123] The specific implementation of this step can refer to step S103 and will not be described in detail here.

[0124] S304 : Determine, from the M semantic feature vectors, a semantic feature vector that has the highest matching degree with each sub-graph feature vector in the N sub-graph feature vectors.

[0125] In one embodiment, M semantic feature vectors and N subgraph feature vectors are input into the attention mechanism model, each subgraph feature vector is searched in the M semantic feature vectors, each subgraph feature vector is matched with each semantic feature vector, and the semantic feature vector with the highest matching degree among the subgraph feature vectors is determined from the M semantic feature vectors.

[0126] It can be understood that each sub-image is constructed based on the superscript box corresponding to each superscript in the geometric image. Therefore, when matching each sub-image in M ​​semantic feature vectors, it may be possible that the corresponding semantic feature vector cannot be matched. In other words, the information corresponding to the sub-image is not reflected in the text of the target question. At this time, the sub-image feature vector is discarded, because when solving a target question, the main focus is on the descriptive information corresponding to the text, and the information corresponding to the geometric image is mainly used to assist in solving the target question.

[0127] Methods for determining the degree of matching between a semantic feature vector and a subgraph feature vector may include, but are not limited to, whether the text and the subgraph are different modalities of information used to represent the same thing, whether the text and the subgraph represent a correlation between the things represented by the text and the subgraph, whether the information contained in the subgraph is part of the information contained in the text, and the like.

[0128] Attention mechanism models include, but are not limited to, CrossAttention and SoftAttention. Attention mechanism models primarily address the alignment between information in different modalities, integrating and representing information from different modalities as a single entity.

[0129] S305 , fusing each of the semantic feature vectors with the highest matching degree with the corresponding subgraph feature vector, and retaining the remaining semantic feature vectors to obtain M fused feature vectors.

[0130] In one embodiment, the semantic feature vector with the highest matching degree of each sub-graph feature vector obtained in the above step S304 is aligned and fused with the corresponding sub-graph feature vector to obtain a corresponding fused feature vector. When matching the sub-graph feature vector from the M semantic feature vectors, there may be a semantic feature vector that does not match the sub-graph feature vector. Then, after the semantic feature vector is fused with the sub-graph feature vector, there will be a semantic feature vector that has not been fused with the sub-graph feature vector, and a sub-graph feature vector that has not been matched with it. For this purpose, each semantic feature vector that has not been fused is used as the corresponding fused feature vector, and each sub-graph feature vector that has not been matched is discarded. Thus, M fused feature vectors can be obtained, which include fused feature vectors that are less than or equal to N semantic feature vectors that are aligned and fused with the sub-graph feature vector, and semantic feature vectors that have not been fused with the sub-graph feature vector.

[0131] It can be understood that when N sub-graph feature vectors determine the semantic feature vector with the highest matching degree from M semantic feature vectors, there may be two or more sub-graph feature vectors that have the highest matching degree with the same semantic feature vector. At this time, the same semantic feature vector will be aligned and fused with multiple sub-graph feature vectors to obtain multiple fused feature vectors. The number of fused feature vectors finally obtained may be greater than the number of semantic feature vectors.

[0132] For example, when humans see an image of a puppy and the word "puppy," they both refer to dogs in nature, but one is an image representation, and the other is a linguistic representation. The attention mechanism model, through the attention network layer, can fuse these two different modal representations into a single representation. In this solution, the subgraph feature vectors are matched and fused by comparing the information implicit in the semantic feature vectors with the subgraph feature vectors.

[0133] The fusion method between the semantic feature vector and the subgraph feature vector may include but is not limited to adding the two vectors, multiplying the two vectors, connecting the two vectors end to end, fusing the values ​​in the two vectors, etc.

[0134] For example, if one of the semantic feature vectors is [0.2, 0.5, 1, 2.5, 1.5] and the subgraph feature vector is [2, 1.2, 0.3, 1, 0.5], and the two vectors are fused by adding them together, the fused vector can be expressed as [2.2, 1.7, 1.3, 3.5, 0.5].

[0135] like Figure 15 As shown, M semantic feature vectors and N subgraph feature vectors are input into the attention mechanism Cross-Attention. M fused feature vectors are obtained by aligning and fusing the semantic features and subgraph feature vectors. After the subgraph feature vectors are aligned and fused with the semantic feature vectors, the number of fused feature vectors is equal to the number of semantic feature vectors. After the semantic feature vector with the highest matching degree with the subgraph feature vector is fused with the corresponding subgraph feature vector, the semantic feature vector that has not been fused with the subgraph feature vector is retained. Therefore, the number of fused feature vectors is equal to the number of semantic feature vectors, ensuring that the information corresponding to the question text is complete during decoding, so as to solve the target question.

[0136] S306: Input the M fused feature vectors into a decoder, and output K decoded vectors, where K is a positive integer greater than zero.

[0137] In one embodiment, in the above step S305, the semantic feature vectors with the highest matching degree are aligned and fused with the corresponding sub-graph feature vectors, and the remaining semantic feature vectors are retained to obtain M fused feature vectors, and the M fused feature vectors are input into the decoder, and K decoding vectors are obtained by decoding the information of the fused feature vectors.

[0138] The decoder may include but is not limited to a Transformer network, a Long Short-Term Memory (LSTM) network, and the like.

[0139] K decoding vectors are the number of decoding vectors required to solve the longest problem in the problem bank. When solving the target problem, the number of decoding vectors depends on the needs of the corresponding target problem, but does not exceed K decoding vectors.

[0140] In the process of decoding the same feature vector into a decoding vector, the model defines the starting and ending symbols according to the solution requirements of the target problem. During the decoding process, if the ending symbol is detected, the decoding of the current fused feature vector is stopped. If the ending symbol is not detected during the decoding process, the decoding of the fused feature vector will only stop when K decoding vectors are generated.

[0141] The number of decoding vectors indicates the type of second character required to generate a solution to the target problem.

[0142] The second character may include but is not limited to each binary operator in the binary operation serialization, uppercase letters, lowercase letters, operation symbols, numbers, words, etc.

[0143] like Figure 16 As shown, M fused feature vectors are input into the decoder, and corresponding K decoding vectors are obtained. The information in the fused feature vector is decoded by the decoder, and the information in the M fused feature vectors is decoded. The second characters of the solution corresponding to the target question are K, so K decoding vectors are obtained.

[0144] This solution represents the solution as a sequence of binary operations. For example, if the natural language step is "angle A equals angle B," the corresponding binary operation sequence is represented as: "Ops_Equal", "A", "B." "Ops_Equal" is a binary operator whose scope is the following two words, "A" and "B." In this way, this solution transforms the solution into a sequence generation task.

[0145] S307: Obtain K index probability vectors corresponding to the K decoding vectors.

[0146] S308 : Determine the maximum index probability corresponding to each index probability vector in the K index probability vectors, and obtain the second character corresponding to each maximum index probability.

[0147] In one embodiment, an index table is first constructed for all second characters in the entire question bank, in which second characters correspond one to one to indexes. The length of the index table is equal to the number of types of second characters in all solution results in the question bank. In the index table, each second character has only one unique numerical index.

[0148] Constructing a word embedding neural network is the inverse process of inputting characters into the neural network to obtain corresponding vectors. The word embedding neural network is a trainable matrix constructed from the index table size and the eigenvector dimensions. Its inverse process is a trainable inverse matrix constructed from the index table size and the eigenvector dimensions. However, the index table size in the trainable matrix represents the number of first character types in the question bank, while the index size in the trainable inverse matrix represents the number of second character types required to solve the question.

[0149] The neural network used in this step is the inverse process of the word embedding network and so on.

[0150] Assuming the size of the trainable inverse matrix index table is Y and the dimension of the feature vector is D, Word-Embedding is a trainable [D, Y]-dimensional inverse matrix. Its function is to convert each [1, D]-dimensional decoding vector into a [1, Y]-dimensional index table, where Y is the number of types of second characters contained in all solution results in the problem database.

[0151] In this embodiment, K decoding vectors are input into a trainable [D, Y]-dimensional inverse matrix, and through calculation, the index probability vector corresponding to each decoding vector is obtained, thereby obtaining K index probability vectors corresponding to the K decoding vectors respectively.

[0152] The maximum index probability is determined in each index probability vector. Since each second character has a fixed position in the index table, the second character at the position corresponding to the maximum index probability is used as the target character.

[0153] The second character may include, but is not limited to, each binary operator, uppercase letter, lowercase letter, operation symbol, number, text, etc. in the binary operation serialization corresponding to the problem-solving result.

[0154] like Figure 17 As shown in , if the solution to the target question contains 7 second characters and the dimension of the feature vector is 512, then the inverse matrix of the feature vector is a [512, 7]-dimensional matrix. When the decoder decodes the fused feature vector, 7 decoding vectors will be obtained. Multiply each decoding vector with the trainable inverse matrix to obtain the index table corresponding to the 7 decoding vectors. Figure 17 The figure shows that the first [512, 1]-dimensional decoding vector in the decoding vector is multiplied by the [512, 7]-dimensional matrix to obtain the corresponding digital index. As can be seen in the figure, the index probability of the 5th position in the digital index is the largest, which is 0.86. The second character corresponding to this position is used as the first character in the solution.

[0155] S309: Generate a solution to the target question based on each of the second characters.

[0156] In one embodiment, each binary operator and the second character within the scope of the binary operator are used as the solution result.

[0157] The problem-solving results may include but are not limited to problem-solving steps, answers to questions, problem-solving methods, etc.

[0158] The way to divide the scope of a binary operator may include but is not limited to the second character between two binary operators being the scope of the first binary operator among the two binary operators.

[0159] For example, the natural language steps for solving a problem are: 1. Angle A is equal to angle B; 2. Angle A is 60 degrees; 3. Therefore, angle B is also 60 degrees. The corresponding binary operations are serialized as: "Ops_Equal", "A", "60°", "Ops_Equal", "A", "B", "Ops_Equal", "B", "60°". Ops_Equal indicates that the second characters within the scope are equal.

[0160] When outputting the solution to the user, it can be a direct result, a solution step, a solution method, or each solution step can be displayed separately to the user to serve as a prompt during the user's problem-solving process, etc.

[0161] In an embodiment of the present application, a target title containing text and a geometric image input by a user is received; M semantic feature vectors corresponding to each phrase in the text are obtained, a superscript box corresponding to each superscript in the geometric image is obtained, and a rectangle is constructed with the center points of the superscript box whose horizontal and vertical coordinates of any two center points are different as two diagonal vertices to construct N sub-graphs corresponding to the geometric image, and N sub-graph feature vectors corresponding to the N sub-graphs are obtained. The semantic feature vector with the highest matching degree with each sub-graph feature vector in the N sub-graph feature vectors is determined among the M semantic feature vectors, and each semantic feature vector with the highest matching degree is respectively fused with the corresponding sub-graph feature vector rectangle, and the remaining semantic feature vectors are retained to obtain M fused feature vectors, and the M fused feature vectors are input into a decoder to obtain K decoding vectors, and K index probability vectors corresponding to the K decoding vectors are obtained. The maximum index probability corresponding to each index probability vector is determined in the K index probability vectors, and the second character corresponding to each maximum index probability is obtained, and the result of the target title is generated based on each second character. By inputting the target question's text and geometric image into a terminal device, M semantic feature vectors corresponding to the text are obtained by obtaining the semantic feature vectors corresponding to each phrase in the text. At the same time, N sub-image feature vectors are obtained by obtaining the feature vectors corresponding to each sub-image in the geometric image. The N sub-image feature vectors are matched against the M semantic feature vectors to determine the semantic feature vector with the highest degree of match for each sub-image feature vector. Each semantic feature vector with the highest degree of match is fused with the corresponding sub-image feature vector, and the remaining semantic feature vectors not fused with the sub-image feature vector are retained to ensure the integrity of the target question's text information. M fused feature vectors are then passed through a decoder to obtain K decoded vectors, and the result corresponding to the target question is obtained based on the decoded vectors. By fully fusing the target question's text and geometric image and fully utilizing the relevant information about the target question's text and geometric image to solve the target question, the accuracy of solving the target question is improved.

[0162] The following will be combined Figure 18 , the problem-solving device provided in the embodiment of the present application is introduced in detail. It should be noted that. Figure 18 The problem-solving device in the present application is used to execute the Figures 2 to 17 For the sake of convenience, only the relevant parts of the embodiment of the present application are shown. For the specific technical details, please refer to the present application. Figures 2 to 17 The method embodiment shown.

[0163] See Figure 18 , which provides a structural diagram of a problem-solving device for this application. Figure 18As shown, the problem-solving device 1 of the embodiment of the present application includes: a target question acquisition module 11, a first vector acquisition module 12, a second vector acquisition module 13, and a problem-solving result acquisition module 14.

[0164] A title acquisition module 11 is used to acquire a target title, wherein the target title includes text and a geometric image;

[0165] A first vector acquisition module 12 is configured to acquire M semantic feature vectors corresponding to the text, wherein the semantic feature vectors are feature vectors corresponding to each phrase in the text, and M is a positive integer greater than zero;

[0166] A second vector acquisition module 13 is configured to acquire N sub-image feature vectors corresponding to the geometric image, where the sub-image feature vectors are feature vectors corresponding to each sub-image in the geometric image, and the sub-image is an image constructed by any two sub-markers in the geometric image, and N is a positive integer greater than zero;

[0167] The problem-solving result acquisition module 14 is configured to acquire a problem-solving result of the target problem based on the M semantic feature vectors and the N subgraph feature vectors.

[0168] Optionally, the first vector acquisition module 12 is specifically configured to:

[0169] Obtain m word vectors corresponding to the text, where the word vectors are word vectors corresponding to the first characters in the text, and m is a positive integer greater than zero;

[0170] Input the m word vectors into a semantic encoder, and output M semantic feature vectors corresponding to the text, where M is less than or equal to m;

[0171] Optional, such as Figure 19 As shown, the second vector acquisition module 13 includes:

[0172] The corner mark frame obtaining unit 131 is used to obtain the corner mark frame corresponding to each corner mark in the geometric picture;

[0173] A sub-graph construction unit 132 is configured to construct N sub-graphs based on any two corner-marked frames, wherein the horizontal coordinates and the vertical coordinates of the center points of the any two corner-marked frames are different;

[0174] The second vector generating unit 133 is configured to input the N sub-images into an image feature extraction model and output N sub-image feature vectors.

[0175] Optionally, the subgraph construction unit 132 is specifically configured to:

[0176] Draw a rectangular frame with the center points of any two corner-marked frames as the two diagonal vertices to obtain N subgraphs.

[0177] Optional, such as Figure 20 As shown, the problem-solving result acquisition module 14 includes:

[0178] A vector fusion unit 141 is configured to input the M semantic feature vectors and the N subgraph feature vectors into an attention mechanism model for alignment and fusion, thereby obtaining M fused feature vectors;

[0179] A decoding vector generating unit 142 is configured to input the M fused feature vectors into a decoder and output K decoding vectors, where K is a positive integer greater than zero;

[0180] The problem-solving result generating unit 143 is configured to obtain a problem-solving result for the target problem based on each of the decoding vectors.

[0181] Optionally, the vector fusion unit 141 is specifically configured to:

[0182] Determining, from the M semantic feature vectors, a semantic feature vector having the highest matching degree with each sub-graph feature vector in the N sub-graph features;

[0183] The semantic feature vectors with the highest matching degree are respectively fused with the corresponding subgraph feature vectors, and the remaining semantic feature vectors are retained to obtain M fused feature vectors.

[0184] Optionally, the problem-solving result generating unit 143 is specifically configured to:

[0185] Obtain K index probability vectors corresponding to the K decoding vectors;

[0186] Determine the maximum index probability corresponding to each index probability vector in the K index probability vectors, and obtain the second character corresponding to each maximum index probability;

[0187] A solution to the target question is generated based on each of the second characters.

[0188] In an embodiment of the present application, a user inputs a target title containing text and a geometric picture into a terminal device, obtains m word vectors corresponding to each first character in the text, inputs the m word vectors into a semantic encoder, and encodes the semantic relationship between different word vectors to obtain M semantic feature vectors. At the same time, a superscript box corresponding to each superscript in the geometric picture is obtained through a neural network model for a target detection task, and a rectangle is drawn with the center coordinates of the superscript box whose horizontal and vertical coordinates are different for any two center coordinates as diagonal vertices to obtain N sub-graphs corresponding to the geometric picture. The N sub-graphs are input into an image feature extraction model, and N sub-graph feature vectors corresponding to each sub-graph are obtained through tasks such as sub-graph rectangle image recognition and feature encoding. The M semantic features are encoded into a vector. The feature vector and N subgraph feature vectors are input into the attention mechanism model, and the information between the semantic feature vector and the subgraph feature vector is aligned and fused. The semantic feature vector with the highest matching degree with each subgraph feature vector in the N subgraph feature vectors is determined among the M semantic feature vectors, and each semantic feature vector with the highest matching degree is fused with the corresponding subgraph feature vector, and the remaining semantic feature vectors are retained to obtain M fused feature vectors. The M fused feature vectors are input into the decoder, and K decoding vectors are output. K index probability vectors corresponding to the K decoding vectors are obtained, and the maximum index probability corresponding to each index probability vector is determined in the K index probability vectors. The second character corresponding to each maximum index probability is obtained, and the solution to the target question is generated based on each second character. The method inputs the text and geometric image of the target question into the terminal device, obtains the word vector corresponding to each first character in the text, inputs all the word vectors into the semantic encoder, and outputs M semantic feature vectors corresponding to the text. At the same time, by obtaining the superscript box corresponding to each superscript in the geometric image, a rectangular box is constructed with the center points corresponding to the superscript boxes whose horizontal and vertical coordinates are different for two arbitrary center coordinates as two diagonal vertices, and N subgraphs are constructed. The N subgraphs are input into the image feature extractor to obtain N subgraph feature vectors corresponding to the N subgraphs. The subgraph feature vectors are aligned and fused with the semantic feature vector to obtain corresponding M fused feature vectors. The M fused feature vectors are decoded to obtain K decoding vectors. The K decoding vectors are converted to obtain the solution steps of the target question, thereby improving the accuracy of solving the target question.

[0189] The present application also provides a computer storage medium that can store multiple instructions, which are suitable for being loaded and executed by a processor as described above. Figures 2 to 17 The problem-solving method of the embodiment shown in the figure can be found in the specific execution process. Figures 2 to 17 The detailed description of the illustrated embodiment will not be repeated here.

[0190] See Figure 21, is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 21 As shown, the terminal device 1000 may include: at least one processor 1001, at least one network interface 1002, at least one input / output interface 1003, at least one display unit 1004, at least one memory 1005, and at least one communication bus 1006. The processor 1001 may include one or more processing cores. The processor 1001 utilizes various interfaces and circuits to connect the various components within the entire electronic device 1000. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and calling data stored in the memory 1005, the processor 1001 executes various functions of the terminal 1000 and processes data. The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The memory 1005 may optionally be at least one storage device located remote from the processor 1001. The network interface 1002 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface or a Bluetooth interface). The communication bus 1006 is used to enable connection and communication between these components. The display unit 1004 may be a touch panel. like Figure 21 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, an input and output interface module, and a problem-solving program.

[0191] exist Figure 21 In the electronic device 1000 shown, the input and output interface 1003 is mainly used to provide an interface for users and access devices and obtain data input by users and access devices.

[0192] In one embodiment, the processor 1001 may be configured to call a problem-solving program stored in the memory 1005 and specifically perform the following operations:

[0193] Obtaining a target topic, wherein the target topic includes text and a geometric picture;

[0194] Obtaining M semantic feature vectors corresponding to the text, where the semantic feature vectors are feature vectors corresponding to each phrase in the text, and M is a positive integer greater than zero;

[0195] Obtain N sub-image feature vectors corresponding to the geometric image, where the sub-image feature vectors are feature vectors corresponding to each sub-image in the geometric image, and the sub-image is an image constructed by any two sub-markers in the geometric image, and N is a positive integer greater than zero;

[0196] A solution to the target question is obtained based on the M semantic feature vectors and the N subgraph feature vectors.

[0197] In one embodiment, when the processor 1001 obtains the M semantic feature vectors corresponding to the text, it specifically performs the following operations:

[0198] Obtain m word vectors corresponding to the text, where the word vectors are word vectors corresponding to the first characters in the text, and m is a positive integer greater than zero;

[0199] The m word vectors are input into a semantic encoder, and M semantic feature vectors corresponding to the text are output, where M is less than or equal to m.

[0200] In one embodiment, when the processor 1001 obtains the N sub-image feature vectors corresponding to the geometric image, it specifically performs the following operations:

[0201] Obtain the corner mark frame corresponding to each corner mark in the geometric picture;

[0202] Constructing N subgraphs based on any two corner-marked frames, wherein the horizontal coordinates and vertical coordinates of the center points of the any two corner-marked frames are different;

[0203] The N sub-images are input into an image feature extraction model, and N sub-image feature vectors are output.

[0204] In one embodiment, when constructing N sub-images based on any two corner-marked frames, the processor 1001 specifically performs the following operations:

[0205] Draw a rectangular frame with the center points of any two corner-marked frames as the two diagonal vertices to obtain N subgraphs.

[0206] In one embodiment, when the processor 1001 obtains the solution to the target problem based on the M semantic feature vectors and the N subgraph feature vectors, the processor 1001 specifically performs the following operations:

[0207] Input the M semantic feature vectors and the N subgraph feature vectors into the attention mechanism model for alignment and fusion, to obtain M fused feature vectors;

[0208] Input the M fused feature vectors into a decoder, and output K decoded vectors, where K is a positive integer greater than zero;

[0209] A solution to the target problem is obtained based on each of the decoding vectors.

[0210] In one embodiment, when the processor 1001 inputs the M semantic feature vectors and the N subgraph feature vectors into the attention mechanism model for alignment and fusion to obtain M fused feature vectors, the processor 1001 specifically performs the following operations:

[0211] Determining, from the M semantic feature vectors, a semantic feature vector having the highest matching degree with each sub-graph feature vector in the N sub-graph features;

[0212] The semantic feature vectors with the highest matching degree are respectively fused with the corresponding subgraph feature vectors, and the remaining semantic feature vectors are retained to obtain M fused feature vectors.

[0213] In one embodiment, when the processor 1001 obtains the solution to the target problem based on each decoding vector, the processor 1001 specifically performs the following operations:

[0214] Obtain K index probability vectors corresponding to the K decoding vectors;

[0215] Determine the maximum index probability corresponding to each index probability vector in the K index probability vectors, and obtain the second character corresponding to each maximum index probability;

[0216] A solution to the target question is generated based on each of the second characters.

[0217] In an embodiment of the present application, a user inputs a target title containing text and a geometric picture into a terminal device, obtains m word vectors corresponding to each first character in the text, inputs the m word vectors into a semantic encoder, and encodes the semantic relationship between different word vectors to obtain M semantic feature vectors. At the same time, a superscript box corresponding to each superscript in the geometric picture is obtained through a neural network model for target detection tasks, and a rectangle is made with the center coordinates of the superscript box whose horizontal and vertical coordinates are different for any two center coordinates as the diagonal vertices to obtain N sub-graphs corresponding to the geometric picture. The N sub-graphs are input into an image feature extraction model, and N sub-graph feature vectors corresponding to each sub-graph are obtained through tasks such as sub-graph rectangle image recognition and feature encoding. The M semantic features are obtained. The vector and N subgraph feature vectors are input into the attention mechanism model, and the information between the semantic feature vector and the subgraph feature vector is aligned and fused. The semantic feature vector with the highest matching degree with each subgraph feature vector in the N subgraph feature vectors is determined among the M semantic feature vectors, and each semantic feature vector with the highest matching degree is fused with the corresponding subgraph feature vector, and the remaining semantic feature vectors are retained to obtain M fused feature vectors. The M fused feature vectors are input into the decoder, and K decoding vectors are output. K index probability vectors corresponding to the K decoding vectors are obtained, and the maximum index probability corresponding to each index probability vector is determined in the K index probability vectors. The second character corresponding to each maximum index probability is obtained, and the solution to the target question is generated based on each second character. The method inputs the text and geometric image of the target question into the terminal device, obtains the word vector corresponding to each first character in the text, inputs all the word vectors into the semantic encoder, and outputs M semantic feature vectors corresponding to the text. At the same time, by obtaining the superscript box corresponding to each superscript in the geometric image, a rectangular box is constructed with the center points corresponding to the superscript boxes whose horizontal and vertical coordinates are different for two arbitrary center coordinates as two diagonal vertices, and N subgraphs are constructed. The N subgraphs are input into the image feature extractor to obtain N subgraph feature vectors corresponding to the N subgraphs. The subgraph feature vectors are aligned and fused with the semantic feature vector to obtain corresponding M fused feature vectors. The M fused feature vectors are decoded to obtain K decoding vectors. The K decoding vectors are converted to obtain the solution steps of the target question, thereby improving the accuracy of solving the target question.

[0218] Those skilled in the art will clearly understand that the technical solution of the present application can be implemented with the help of software and / or hardware. "Unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.

[0219] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0220] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0222] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0223] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0224] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions for enabling an electronic device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0225] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0226] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure herein, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A problem-solving method, characterized in that: The method comprises: Obtaining a target topic, wherein the target topic includes text and a geometric picture; Obtaining M semantic feature vectors corresponding to the text, where the semantic feature vectors are feature vectors corresponding to each phrase in the text, and M is a positive integer greater than zero; Obtain N sub-image feature vectors corresponding to the geometric image, where the sub-image feature vectors are feature vectors corresponding to each sub-image in the geometric image, and the sub-image is an image constructed by any two subscripts in the geometric image, and N is a positive integer greater than zero; Obtaining a solution to the target problem based on the M semantic feature vectors and the N subgraph feature vectors; The obtaining of N sub-image feature vectors corresponding to the geometric image includes: Obtain the corner mark frame corresponding to each corner mark in the geometric picture; Draw a rectangular frame with the center points of any two corner mark boxes as two diagonal vertices to obtain N subgraphs, where the horizontal coordinates and vertical coordinates of the center points of any two corner mark boxes are different; Inputting the N sub-images into an image feature extraction model and outputting N sub-image feature vectors; The obtaining a solution to the target problem based on the M semantic feature vectors and the N subgraph feature vectors includes: Input the M semantic feature vectors and the N subgraph feature vectors into the attention mechanism model for alignment and fusion, to obtain M fused feature vectors; Inputting the M fused feature vectors into a decoder, and outputting K decoding vectors, where K is a positive integer greater than zero, and the K decoding vectors are used to represent the type of the second character required to solve the target problem to generate a solution; A solution to the target problem is obtained based on each of the decoding vectors.

2. The method according to claim 1, characterized in that The obtaining of M semantic feature vectors corresponding to the text includes: Obtain m word vectors corresponding to the text, where the word vectors are word vectors corresponding to the first characters in the text, and m is a positive integer greater than zero; The m word vectors are input into a semantic encoder, and M semantic feature vectors corresponding to the text are output, where M is less than or equal to m.

3. The method according to claim 1, characterized in that The M semantic feature vectors and the N subgraph feature vectors are input into the attention mechanism model for alignment and fusion to obtain M fused feature vectors, including: Determining, from the M semantic feature vectors, a semantic feature vector having the highest matching degree with each sub-graph feature vector in the N sub-graph feature vectors; The semantic feature vectors with the highest matching degree are respectively fused with the corresponding subgraph feature vectors, and the remaining semantic feature vectors are retained to obtain M fused feature vectors.

4. The method according to claim 1, wherein Obtaining a solution to the target problem based on each of the decoding vectors includes: Obtain K index probability vectors corresponding to the K decoding vectors; Determine the maximum index probability corresponding to each index probability vector in the K index probability vectors, and obtain the second character corresponding to each maximum index probability; A solution to the target question is generated based on each of the second characters.

5. A problem-solving device, characterized in that: The device comprises: A target topic acquisition module is used to acquire a target topic, wherein the target topic includes text and a geometric picture; A first vector acquisition module is configured to acquire M semantic feature vectors corresponding to the text, wherein the semantic feature vectors are feature vectors corresponding to each phrase in the text, and M is a positive integer greater than zero; A second vector acquisition module is configured to acquire N sub-image feature vectors corresponding to the geometric image, where the sub-image feature vectors are feature vectors corresponding to each sub-image in the geometric image, and the sub-image is an image constructed by any two sub-markers in the geometric image, and N is a positive integer greater than zero; A solution result acquisition module, configured to acquire a solution result of the target problem based on the M semantic feature vectors and the N subgraph feature vectors; The second vector acquisition module is specifically configured to: obtain a corner mark frame corresponding to each corner mark in the geometric image; draw a rectangular frame with the center points of any two corner mark frames as two diagonal vertices to obtain N sub-images, wherein the horizontal coordinates and vertical coordinates of the center points of any two corner mark frames are different; input the N sub-images into the image feature extraction model, and output N sub-image feature vectors; The problem-solving result acquisition module is specifically used to: input the M semantic feature vectors and the N subgraph feature vectors into the attention mechanism model for alignment and fusion to obtain M fused feature vectors; input the M fused feature vectors into the decoder and output K decoding vectors, where K is a positive integer greater than zero, and the K decoding vectors are used to represent the type of second character needed to solve the target problem to generate a solution result; and obtain the solution result of the target problem based on each of the decoding vectors.

6. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, which are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 4.

7. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps according to any one of claims 1 to 4.

Images