Method and device for generating image description text

Abstract

The invention provides a method and a device for generating an image description text. The method comprises the following steps: acquiring a question text and an answer text of a geometric question and a geometric image in the geometric question; inputting the geometric image into a pre-trained image description model, and obtaining an original description text which is output by the image description model and is used for describing the geometric image; and correcting the original description text based on the question text and the answer text to obtain a corrected intermediate description text, and determining a target description text for describing the geometric image at least according to the intermediate description text.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a method and device for generating image description text. Background Art

[0002] The geometric problems involved in the field of mathematics usually include geometric images. If comprehensive and accurate descriptive texts can be provided for these geometric images, it can provide significant improvements in many aspects such as mathematics education and geometric problem-solving model training. For example, when training a model that focuses on geometric problems, in order to ensure the effectiveness of the model, a large number of high-quality samples are usually required as training data. For any sample, the descriptive text contained therein should reflect the characteristics of the geometric image it is targeting as comprehensively and accurately as possible.

[0003] However, the samples currently obtained through manual annotation, web page data extraction, and model generation usually cannot guarantee the coverage of the description text on the geometric image features and details, nor can they guarantee the accuracy of the description. Summary of the invention

[0004] In view of this, the present application provides a method and device for generating image description text to solve the defects existing in the related art. The technical solution of the present application is as follows:

[0005] According to an embodiment of the first aspect of the present application, a method for generating image description text is provided, comprising:

[0006] Obtaining a title text and an answer text of a geometry problem, as well as a geometry image in the geometry problem;

[0007] Inputting the geometric image into a pre-trained image description model, and obtaining original description text output by the image description model for describing the geometric image;

[0008] The original description text is corrected based on the question text and the answer text to obtain a corrected intermediate description text, and a target description text for describing the geometric image is determined at least based on the intermediate description text.

[0009] According to an embodiment of the second aspect of the present application, a device for generating image description text is provided, including:

[0010] An acquisition unit, used to acquire the question text and answer text of the geometry problem, and the geometry image in the geometry problem;

[0011] A processing unit, used for inputting the geometric image into a pre-trained image description model, and obtaining an original description text output by the image description model for describing the geometric image;

[0012] A determination unit is used to correct the original description text based on the question text and the answer text to obtain a corrected intermediate description text, and to determine a target description text for describing the geometric image at least based on the intermediate description text.

[0013] According to an embodiment of the third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and executable instructions stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect by executing the executable instructions.

[0014] According to an embodiment of the fourth aspect of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] According to an embodiment of the fifth aspect of the present application, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0016] In the technical solution provided by the present application, the question text, answer text and geometric image of the geometry problem can be obtained at the same time. On this basis, the geometric image is first input into a pre-trained image description model capable of outputting image description text, and the original description text output by the image description model for the geometric image is obtained; then, the obtained original description text is corrected according to the question text and the answer text, and then the corrected intermediate description text is obtained; finally, the target description text used to describe the geometric image is determined based on the intermediate description text.

[0017] By applying the technical solution of the present application, the original description text is generated only for the features of the geometric image itself without referring to the question text and the answer text in the image description model, thereby improving the diversity of the description text and the feature coverage of the geometric image; at the same time, the question text and the answer text are used to correct the original description text generated by the image description model, and the target description text is further determined on the basis of the correction, thereby ensuring the accuracy of the description text, so that the finally determined description text can comprehensively and accurately reflect the geometric features of the geometric image as a whole.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and do not limit the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0020] Figure 1 is a flow chart of a method for generating image description text shown in an exemplary embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a geometric problem shown in an exemplary embodiment of the present application;

[0022] Figure 3 It is an overall flow chart showing an exemplary embodiment of the present application;

[0023] Figure 4 is a diagram of an electronic device shown in an exemplary embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of a device for generating image description text shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present application as detailed in the appended claims.

[0026] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.

[0027] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] The geometric problems involved in the field of mathematics usually include geometric images. If comprehensive and accurate descriptive texts can be provided for these geometric images, it can provide significant improvements in many aspects such as mathematics education and geometric problem-solving model training. For example, when training a model focused on geometric problems, having a large number of high-quality training samples is one of the key factors to ensure model performance. Each sample can be in the form of a "geometric image-descriptive text pair", and for any "geometric image-descriptive text pair", the descriptive text must fully reflect the characteristics and details of the corresponding geometric image so that the model can learn effective patterns and apply them to unknown data.

[0029] In the related art, the methods for obtaining the above training samples are mainly divided into three categories: manual annotation, rule-based web data extraction, and model generation. However, each of these methods has certain limitations. Among them, the manual annotation method is a method in which professionals or trained annotators manually provide description information based on the content of the geometric image. Although this method can provide a relatively accurate description, it is too costly and inefficient, and the number of samples obtained is limited, which is difficult to meet the needs of large-scale model training. The web data extraction method is to extract geometric images and their corresponding titles from web data through simple rules, and directly use the titles as descriptions of the corresponding geometric images. However, since the titles on the web pages may not accurately reflect the content of the geometric images, the quality of the samples cannot be guaranteed. In the last method (the model directly generates description text based on the text and image of the geometric problem), due to the different focus and reference degree of text description of different models, the description text directly generated by the model may ignore some important details in the image, or excessively refer to text information and ignore image information (such as the answer text of the geometric problem includes auxiliary lines, but the actual geometric image does not exist in the auxiliary lines), resulting in inaccurate descriptions, which in turn affects the training effect of subsequent models.

[0030] It can be seen that while pursuing high efficiency and accuracy, the methods in the above-mentioned related technologies often cannot guarantee the coverage and accuracy of the description text for the features of the geometric image. In order to solve the above problems, the present application proposes a method for generating image description text, wherein the original description text is generated only for the features of the geometric image itself when the image description model does not refer to the question text and the answer text, thereby improving the diversity of the description text and the coverage of the features of the geometric image; at the same time, the question text and the answer text are used to correct the original description text generated by the image description model, and the target description text is further determined on the basis of the correction, thereby ensuring the accuracy of the description text, so that the description text finally determined can comprehensively and accurately reflect the features of the geometric image it is targeting. Next, the embodiments of the present application are described in detail.

[0031] Figure 1 This is a method for generating image description text shown in an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0032] Step 102, obtaining the question text and answer text of the geometry problem, as well as the geometry image in the geometry problem.

[0033] In one embodiment, the geometry problem may be a geometry problem with an accompanying geometry image, that is, a geometry problem with a graph, which may be sampled from geometry data sets (such as the geometry multimodal data set Geo170K) and student exercises (such as sample questions or exercises in elementary and middle school textbooks). Figure 2 is a schematic diagram of a geometric problem with a graph shown in an exemplary embodiment of the present application. Figure 2As shown, a geometry problem with a diagram includes a question text 21, an answer text 22, and a geometry image 23 corresponding to the geometry problem. The question text 21 can be a text description of the geometry problem, which usually contains all the information required to solve the geometry problem, such as specific conditions, known data, and the relationship between graphic elements. The answer text 22 can be a textual expression of the answer to the problem raised in the question text, which usually includes the proof principle, reasoning steps, the relationship between the proved graphic elements, and the final result. The geometric image 23 can be a visual representation of the geometric structure or shape mentioned in the question text 22, which intuitively shows multiple graphic elements in the image and their mutual relationship, such as points, lines, surfaces, angles, etc. The above three parts together constitute a complete geometry problem with a diagram. Generally speaking, there is a one-to-one correspondence between the question text, the answer text, and the geometry image, but there may be a situation where a geometry image corresponds to multiple small questions. In this case, the sub-question texts of multiple small questions can be integrated into the question text, and the sub-answer texts corresponding to the multiple small questions can be integrated into the answer text, thereby forming a one-to-one correspondence between the question text, the answer text, and the geometry image. Of course, technicians can also split it according to actual conditions, such as treating the first sub-question text, the first sub-answer text and the geometric image as a geometric problem, and the second sub-question text, the second sub-answer text and the geometric image as another geometric problem, and so on. The specific method can be determined by the sampling method, and this application does not limit this.

[0034] Step 104: input the geometric image into a pre-trained image description model, and obtain the original description text output by the image description model for describing the geometric image.

[0035] In one embodiment, for any acquired geometric problem, a geometric image can be input into an image description model, and the image description model can output a description text for describing the geometric image based on the input geometric image, that is, the original description text. The image description model can be a model pre-trained by a technician according to actual needs. When the image is used as a model input, the image description model can analyze the input image and then output a description text; the image description model can also be a multimodal model in the relevant technology. A multimodal model is a complex neural network model that can process multiple types of data (such as text, images, etc.), and can provide the information required by the user by combining different types of input data, such as different versions of the LLaVA (Large Language and Vision Assistant) model, the GPT4-o model, and the Qwen-VL (Qwen Visual Language Model) model, etc.

[0036] In the case where the image description model is a multimodal model, an image description prompt sentence for a geometric image can be constructed, and the geometric image and the image description prompt sentence are associated and input into the multimodal model. The constructed image description prompt sentence usually contains the URL (Uniform Resource Location) of the geometric image, and the image description prompt sentence containing the URL of the geometric image is input into the multimodal model, that is, the geometric image and the image description prompt sentence are associated and input into the multimodal model.

[0037] Specifically, the image description prompt sentence may include special tags used by the multimodal model and text that guides the multimodal model to describe the image. An exemplary image description prompt sentence may be in the following form:

[0038] ①:<Im_start> (Image URL)<Im_end> <system>Please look at the above picture carefully and describe it in detail from the perspectives of vertices, edges, angles, symbols, etc. The requirement is...

[0039] ②:<Im_start> (Image URL)<Im_end> <system>As a math teacher, please understand the picture above and ask...

[0040] in,"<Im_start> ”、"<Im_end> "as well as" <system>" is a special mark used to divide the fields in the sentence. Different multimodal models may use different special marks; and "Please look at the above picture carefully, and describe it in detail from the perspectives of vertices, edges, angles, symbols, etc., requiring..." and "As a math teacher, please understand the above picture, requiring..." is the text that guides the multimodal model to describe the image. It is understandable that technicians can adjust the special marks and the text that guides the multimodal model to describe the image according to actual conditions and needs (such as the specifications of the special marks of the multimodal model used, the dimensions of the image analysis, etc.), and this application does not limit this.

[0041] In one embodiment, before obtaining a geometric image in a geometric problem and inputting the geometric image into an image description model, the geometric image may be first subjected to data enhancement processing to obtain X enhanced images, where X is a positive integer greater than or equal to 1, which may be specifically set by a technician. The data enhancement processing for an image is used to characterize an information processing method that increases the diversity of data while maintaining the original semantics of the image through a series of image processing operations. The specific image processing operations may be geometric transformations (rotation, scaling, translation, etc.) and noise processing (such as adding noise, reducing noise, etc.). After obtaining X enhanced images, Y images may be randomly selected from the original geometric image (i.e., the geometric image in the geometric problem) and the X enhanced images and input into the image description model. The image description model may output corresponding description texts for each of the Y images, and then all the description texts obtained in this step may be used as the original description texts of the original geometric image. Here, Y is a positive integer greater than or equal to 1, which may be specifically set by a technician.

[0042] It is understandable that the inherent properties of the geometric image do not change with the above processing operations and are less affected by the resolution. Therefore, for multiple enhanced images after data enhancement processing and the original geometric image, these images should have the same description information. For example, a triangle is still a triangle after scaling, rotating and translating, and the angles of each corner do not change. However, since the geometric images input into the image description model may not be exactly the same, the focus of the image description model may change, thereby outputting a variety of description texts, improving the coverage of the features of the geometric image. It should also be noted that in this embodiment, a random selection method is used to select Y images, so these Y images may be the same image. However, it is understandable that even if the same input is provided to the neural network model multiple times, the output obtained each time may not be the same, so in this case, the various description texts output by the image description model can still be obtained.

[0043] In one embodiment, the number of image description models may be one or more.

[0044] In the case of one image description model, such as when only image description model A is used, different processing methods are used for different numbers of images. In the case where the geometric image (such as the image directly corresponding to the geometric image a in the geometric problem) is not enhanced, the geometric image a can be used as all or part of the input and input into the image description model A to obtain the original description text for the geometric image a output by the image description model A. In the case where the geometric image a is enhanced and multiple enhanced images are obtained (such as enhanced image a1, enhanced image a2, and enhanced image a3), Y images can be randomly selected (such as geometric image a and enhanced image a2) and input into the image description model A to obtain the description texts output by the image description model A for the geometric image a and enhanced image a2, respectively, and integrate them into the original description text.

[0045] In the case of multiple image description models, such as when image description model A and image description model B are used at the same time, different processing methods are used for different numbers of images. In the case where the geometric image (such as the image directly corresponding to the geometric image a in the geometric problem) is not enhanced, the geometric image a can be used as all or part of the input, respectively input into the image description model A and the image description model B, and the original description text for the geometric image a output by the image description model A and the image description model A is obtained respectively. In the case where the geometric image a is enhanced and multiple enhanced images are obtained (such as enhanced image a1, enhanced image a2, and enhanced image a3), Y images can be randomly selected (such as geometric image a and enhanced image a2 are selected), and then randomly input into the image description model A or the image description model B, and the description text output by each image description model is obtained respectively, and integrated into the original description text. Here, the number of random selections of images should be the same as the number of random selections of models. It should be emphasized that when the number of input images is multiple and the number of image description models used is also multiple, the selection of images and the selection of models are both random. For example, the images randomly selected three times are all enhanced images a2, and the models randomly selected three times are image description model A, image description model B, and image description model A, respectively. Then, the enhanced image a2 is input into the image description model, image description model B, and image description model A, respectively, and then three description texts output by the image description model can be obtained, which are integrated to form the original description text. It should be noted that, similar to the aforementioned embodiment, although extreme situations may occur in the random selection process, such as multiple randomly selected images are all enhanced images a2 and multiple randomly selected models are all image description model A, even if the same input is provided to the neural network model multiple times, the output obtained each time may not be the same, so in this case, a variety of description texts can still be obtained.

[0046] It can be seen from the above embodiments that the present application enables the image description model to focus on the different features of the geometric image and then output a description text for each feature, thereby improving the coverage of the original description text for the features of the geometric image by providing multiple diverse geometric images that can maintain the original semantics, or by using multiple image description models with different focuses, or by combining the above two.

[0047] It should also be emphasized that in this application, when generating the original description text, only the geometric image is provided to the image description model, but the question text and the answer text are not provided. Therefore, the image description model will not refer to the content of the question text and the answer text, but can better focus on the characteristics of the geometric image itself, and then output a description text that can effectively reflect the characteristics in the image (such as geometric shapes, the relationship between points and edges, the correspondence between symbols and points / lines, etc.).

[0048] Step 106, correcting the original description text based on the question text and the answer text to obtain a corrected intermediate description text, and determining a target description text for describing the geometric image based at least on the intermediate description text.

[0049] In one embodiment, the original description text may be corrected first, and the corrected description text may be used as an intermediate description text. Subsequently, based on the intermediate description text, a target description text that can ultimately fully and accurately reflect the features of the geometric image may be determined.

[0050] Specifically, when correcting the original description text obtained, a text correction prompt sentence can be first constructed based on special marks, the original description text, the question text and the answer text, and the constructed correction prompt sentence can be input into the text language processing model. Here, the text language processing model can be a model that focuses on processing and analyzing text data. By learning a large amount of text data, it masters the grammar, semantics, pragmatics and other knowledge of the language, can better understand user needs, and process and generate text according to user needs. The special mark is the special mark used by the text language processing model, which can be specifically referred to in the aforementioned embodiment and will not be repeated here. It can be understood that since the question text and the answer text are standard texts for geometric images obtained by sampling, they can usually accurately reflect the characteristics of geometric images, so the text language processing model can be guided to correct the original description text based on the question text and the answer text. The constructed exemplary text correction prompt sentence can be in the following form:

[0051] <system>The following is a math geometry question and the corresponding answer <question> (question text) <answer> (answer text). Please analyze the following descriptions and make corrections...

[0052] <Description>Description 1: ...

[0053] <Description>Description 2: ...

[0054] in," <system>", "<title>", "<answer>" and "<description>" are special tags used to divide the fields in the sentence. Different multimodal models may use different special tags; and "The following is a math geometry problem and the corresponding answer... Please analyze the following descriptions and make corrections..." is the text that guides the text language processing model to make description corrections. It is understandable that technicians can adjust the special tags and the text that guides the text language processing model to make description corrections according to actual conditions and needs (such as the specifications of the special tags of the text language processing model used, the dimensions for analyzing and correcting the description, etc.), and the present application does not limit this.

[0055] Afterwards, the text language processing model can correct the prompt sentence based on the input text and output the corrected description text as the intermediate description text. In this embodiment, before determining the final target description text, the original description text can be corrected based on the question text and the answer text, thereby eliminating the information in the original description text that is significantly different from the description of the question and the answer, and obtaining a relatively accurate intermediate description text, which is conducive to improving the subsequent processing efficiency.

[0056] After obtaining the intermediate description text, the target description text can be determined based on the intermediate description text in two ways.

[0057] In one approach, the target description text can be determined by confidence. Specifically, the original description text can be split into M1 original description sub-sentences according to the segmentation symbols in the original description text; at the same time, the intermediate description text can be split into M2 intermediate description sub-sentences according to the segmentation symbols in the intermediate description text. The segmentation symbols can be punctuation marks used to divide the text, such as commas, periods, semicolons, etc., and M1 and M2 are both positive integers.

[0058] Afterwards, for each of the M2 intermediate description sub-sentences obtained by splitting, the semantic similarity between the intermediate description sub-sentence and the M1 original description sub-sentences can be calculated to obtain M1 similarities, and the original description sub-sentences corresponding to the similarity not less than the similarity threshold among the M1 similarities are determined as the original description sub-sentences matching the intermediate description sub-sentences. For any intermediate description sub-sentence, K original description sub-sentences matching the intermediate description sub-sentence can be obtained, where K is a positive integer and is not greater than the total number M1 of original description sub-sentences. For example, after splitting the intermediate description text, three intermediate description sub-sentences are obtained, namely "r0", "s0" and "t0". At the same time, after splitting the original description text, eight original description sub-sentences are obtained, namely "r1", "r2", "r3", "s1", "s2", "t1", "t2" and "t3". When calculating the similarity, the similarity between "r0" and the eight original description sub-sentences "r1", "r2", "r3", "s1", "s2", "t1", "t2" and "t3" will be calculated respectively. If the similarity with "r1", "r2" and "r3" is not less than the similarity threshold, "r1", "r2" and "r3" will be determined as the original description sub-sentences matching the intermediate description sub-sentence "r0". When calculating the similarity, a semantic similarity model can be used for calculation, or similarity can be calculated by dependency syntax analysis, etc., which is not limited in this application. Similar methods are also used for the remaining intermediate description sub-sentences, which will not be repeated here.

[0059] After that, the confidence of the intermediate description sub-sentence can be determined based on the K original description sub-sentences. Specifically, the confidence of the intermediate description sub-sentence can be the ratio of K to the number of times the image description model outputs the description text. For example, if the number of times the image (including the geometric image and the enhanced image) is randomly selected in the above embodiment is the same as the number of times the image description model is randomly selected, both of which are T times, then all the selected image description models will output corresponding description texts respectively, with a total of T pieces. On this basis, the ratio of K to T can reflect the probability of the semantics of the intermediate description sub-sentence appearing in the output of the image description model. It can be understood that the greater the probability, that is, the more times the original description sub-sentences with semantics similar to the intermediate description sub-sentence appear, which further indicates that more image description models believe that the geometric image has the features expressed by the intermediate description sub-sentence, which also indicates that the intermediate description sub-sentence is more credible. In the case of calculating the confidence in this way, if the confidence of an intermediate description sub-sentence exceeds the preset confidence threshold, it means that the intermediate description sub-sentence has reached the credible standard, so it can be determined as the target description sub-sentence.

[0060] Finally, after the above steps are performed for each intermediate description sub-sentence, the obtained target description sub-sentences are integrated to construct a target description text for the geometric image originally contained in the geometric problem.

[0061] In another way, the target description text can be determined by similarity. Specifically, the question text and the answer text can be regarded as an integrated text, and the integrated text can be split into N1 reference description sub-sentences according to the punctuation symbols in the question text and the answer text; at the same time, the intermediate description text can be split into N2 intermediate description sub-sentences according to the punctuation symbols in the intermediate description text. Among them, the punctuation symbols can be punctuation marks used to divide the text, such as commas, periods, semicolons, etc., and N1 and N2 are both positive integers.

[0062] Afterwards, for each of the N2 intermediate description sub-sentences obtained by splitting, the semantic similarity between the intermediate description sub-sentence and the N1 reference description sub-sentences can be calculated to obtain N1 similarities. The method for calculating the similarity between each intermediate description sub-sentence and each reference description sub-sentence is the same as the method for calculating the similarity between each intermediate description sub-sentence and each original description sub-sentence, and will not be repeated here.

[0063] After that, for each intermediate description sub-sentence, if there is at least one similarity that is not less than the similarity threshold value among the determined N1 similarities, the intermediate description sub-sentence can be directly determined as the target description sub-sentence. For example, if for the intermediate reference sub-sentence "s1", the determined similarities with each reference description sub-sentence are 0.3, 0.4, 0.1, 0.1, 0.2, 0.15, 0.25, 0.9 respectively, and the similarity threshold is 0.8, then the calculated similarity 0.9 is greater than 0.8, so it can be considered that the semantics expressed by the intermediate description sub-sentence is the same as that expressed by a sub-sentence in the question text and the answer text, and it can be used as the target description sub-sentence.

[0064] Finally, after the above steps are performed for each intermediate description sub-sentence, the obtained target description sub-sentences are integrated to construct a target description text for the geometric image originally contained in the geometric problem.

[0065] It should be noted that when determining the target description sub-sentence, the above two methods can be used at the same time. In this case, all the target description sub-sentences determined include both the intermediate description sub-sentences screened by the question text and the answer text, ensuring the accuracy of the target description sub-sentences; and the intermediate description sub-sentences screened by the confidence level, so that some descriptions that do not appear in the question text and the answer text are also retained, ensuring the comprehensiveness of the description of the geometric picture.

[0066] It can be seen from the above embodiments that the present application scheme actually divides the entire process into two parts: "generating original description text based on geometric images" and "checking based on question text and answer text". These two parts are independent of each other, avoiding the situation where the original description text generated by the image description model refers too much to questions and answers and ignores the characteristics of the geometric image itself, so that the final description text can comprehensively and accurately reflect the characteristics of the geometric image it is targeting.

[0067] In one embodiment, in the process of determining the target description sub-sentence, a corresponding label can be added to each determined target description sub-sentence, and the label is used to indicate the main information source of the description sub-sentence that is retained. Specifically, if the target description sub-sentence is obtained by screening with confidence (corresponding to the aforementioned method of using confidence to determine the target description text), the label "image" can be added to it, indicating that the intermediate description sub-sentence determined as the target description sub-sentence actually involves the analysis and description of the geometric image by multiple image description models, and the question text and answer text are not referenced during the generation process; if the target description sub-sentence is obtained by screening with the question text and the answer text (corresponding to the aforementioned method of using similarity to determine the target description text), the label "question / answer" can be added to it, indicating that the question or answer of the geometry problem includes a description similar to it. This label mainly acts on the subsequent model training, and the specific usage and reference weight can be set by the technicians during the subsequent model training, and this application does not limit this.

[0068] In one embodiment, after obtaining the target description text, the geometric image in the geometric problem initially obtained and the obtained target description text can be processed to obtain a "geometric image-description text pair" and use it as a training sample. Since the description text for the geometric image generated by the application scheme of the present application can comprehensively and accurately reflect the characteristics of the geometric image, after obtaining a large number of training samples by this method, the relevant image analysis model can be trained so that the image analysis model can learn effective patterns based on accurate training data and apply them to unknown data. Here, the image analysis model can be a geometric problem-solving model, an image description model, an image recognition model, etc. The technician can reorganize the form of the obtained training samples according to the requirements of the model to be trained, and the present application does not limit this.

[0069] The following will be combined Figure 3 The overall process of the method provided in this application is described.

[0070] Step 301, obtaining the question text, answer text and corresponding geometric image of the geometry problem.

[0071] Step 302: Perform data enhancement processing on the geometric image to obtain multiple enhanced images.

[0072] Step 303: randomly select an image from the geometric image and the multiple enhanced images, and randomly input the selected image into the image description model to obtain the original description text output by each image description model for the corresponding image.

[0073] Step 304, based on the question text, the answer text and the obtained original description text, the original description text is modified by a text language processing model to obtain an intermediate description text.

[0074] Step 305: Determine the target description text according to the intermediate description text. Specifically, this includes the method of determining the target description text by using the confidence and the method of determining the target description text by using the similarity as described in the above embodiment, see the above embodiment for details.

[0075] Step 306, based on the target description text obtained from the geometric image included in the geometric problem, construct a "geometric image-description text pair", which can be used as a training sample that can be used to train the image analysis model.

[0076] Figure 4 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4 At the hardware level, the electronic device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include hardware required for other services. The processor 402 reads the corresponding computer program from the non-volatile memory 410 into the memory 408 and then runs it, forming a health monitoring device at the logical level. Of course, in addition to software implementations, this application does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0077] Corresponding to the above method embodiments, the present application also provides device embodiments. Figure 5 , the image description text generation device can be applied to Figure 4 The device shown in the figure is used to implement the technical solution of the present application. The device for generating the image description text may include an acquisition unit 51, a processing unit 52 and a determination unit 53, wherein:

[0078] An acquisition unit 51 is used to acquire the question text and answer text of the geometry problem, and the geometry image in the geometry problem;

[0079] A processing unit 52 is used to input the geometric image into a pre-trained image description model, and obtain an original description text output by the image description model for describing the geometric image;

[0080] The determination unit 53 is used to modify the original description text based on the question text and the answer text to obtain a modified intermediate description text, and determine a target description text for describing the geometric image at least based on the intermediate description text.

[0081] Optionally, the processing unit 52 is specifically configured to:

[0082] An image description prompt sentence for the geometric image is constructed, and the geometric image and the image description prompt sentence are associated and input into the multimodal model.

[0083] Optionally, the processing unit 52 is specifically configured to:

[0084] The geometric image is input into at least one pre-trained image description model, and original description texts output by each image description model after reasoning with respect to the input geometric image are obtained.

[0085] Optionally, the processing unit 52 is specifically configured to:

[0086] Performing data enhancement processing on the geometric image to obtain X enhanced images;

[0087] Y images are randomly selected from the geometric image and the X enhanced images and input into the image description model, and the description text output by the image description model for each of the Y images is used as the original description text of the geometric image.

[0088] Optionally, the determining unit 53 is specifically configured to:

[0089] Constructing a text correction prompt sentence based on the special mark, the original description text, the question text and the answer text, and inputting the text correction prompt sentence into a text language processing model;

[0090] An intermediate description text is obtained which is outputted after the text language processing model corrects the original description text based on the question text and the answer text.

[0091] Optionally, the determining unit 53 is specifically configured to:

[0092] Splitting the original description text into M1 original description sub-sentences according to the sentence segmentation symbols in the original description text; and splitting the intermediate description text into M2 intermediate description sub-sentences according to the sentence segmentation symbols in the intermediate description text, where M1 and M2 are both positive integers;

[0093] For each of the M2 intermediate description sub-sentences, determine K original description sub-sentences matching the intermediate description sub-sentence according to the semantic similarity between the intermediate description sub-sentence and the M1 original description sub-sentences, and determine the confidence of the intermediate description sub-sentence based on the K original description sub-sentences, where K≤M1; and, if the confidence of the intermediate description sub-sentence is not less than a confidence threshold, determine the intermediate description sub-sentence as a target description sub-sentence;

[0094] The target description text of the geometric image is constructed based on the determined target description sub-sentences.

[0095] Optionally, the determining unit 53 is specifically configured to:

[0096] Splitting the question text and the answer text into N1 reference description sub-sentences according to the sentence segmentation symbols in the question text and the answer text; and splitting the intermediate description text into N2 intermediate description sub-sentences according to the sentence segmentation symbols in the intermediate description text, where N1 and N2 are both positive integers;

[0097] For each of the N2 intermediate description sub-sentences, calculate the similarity between the intermediate description sub-sentence and each of the N1 reference description sub-sentences, and if there is at least one similarity that is not less than a similarity threshold, determine the intermediate description sub-sentence as a target description sub-sentence;

[0098] The target description text of the geometric image is constructed based on the determined target description sub-sentences.

[0099] Optionally, the device further comprises a training unit 54 for training an image analysis model using the geometric image and the target description text as training samples.

[0100] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0101] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0102] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0103] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0104] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0105] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0106] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0107] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.< / system> < / system> < / system> < / system> < / system>

Claims

1. A method for generating image description text, characterized in that: include: Obtaining a title text and an answer text of a geometry problem, as well as a geometry image in the geometry problem; Inputting the geometric image into a pre-trained image description model, and obtaining original description text output by the image description model for describing the geometric image; The original description text is corrected based on the question text and the answer text to obtain a corrected intermediate description text, and a target description text for describing the geometric image is determined at least based on the intermediate description text.

2. The method according to claim 1, characterized in that The image description model is a multimodal model, and the step of inputting the geometric image into a pre-trained image description model includes: An image description prompt sentence for the geometric image is constructed, and the geometric image and the image description prompt sentence are associated and input into the multimodal model.

3. The method according to claim 1, characterized in that The step of inputting the geometric image into a pre-trained image description model and obtaining an original description text of the geometric image output by the image description model comprises: The geometric image is input into at least one pre-trained image description model, and original description texts output by each image description model after reasoning with respect to the input geometric image are obtained.

4. The method according to claim 1, characterized in that: The step of inputting the geometric image into a pre-trained image description model and obtaining an original description text of the geometric image output by the image description model comprises: Performing data enhancement processing on the geometric image to obtain X enhanced images; Y images are randomly selected from the geometric image and the X enhanced images and input into the image description model, and the description text output by the image description model for each of the Y images is used as the original description text of the geometric image.

5. The method according to any one of claims 1, characterized in that The step of correcting the original description text based on the question text and the answer text to obtain a corrected intermediate description text includes: Constructing a text correction prompt sentence based on the special mark, the original description text, the question text and the answer text, and inputting the text correction prompt sentence into a text language processing model; An intermediate description text is obtained which is outputted after the text language processing model corrects the original description text based on the question text and the answer text.

6. The method according to claim 1, characterized in that The step of determining the target description text for describing the geometric image at least according to the corrected intermediate description text comprises: Splitting the original description text into M1 original description sub-sentences according to the sentence segmentation symbols in the original description text; and splitting the intermediate description text into M2 intermediate description sub-sentences according to the sentence segmentation symbols in the intermediate description text, where M1 and M2 are both positive integers; For each of the M2 intermediate description sub-sentences, determine K original description sub-sentences matching the intermediate description sub-sentence according to the semantic similarity between the intermediate description sub-sentence and the M1 original description sub-sentences, and determine the confidence of the intermediate description sub-sentence based on the K original description sub-sentences, where K≤M1; and, if the confidence of the intermediate description sub-sentence is not less than a confidence threshold, determine the intermediate description sub-sentence as a target description sub-sentence; The target description text of the geometric image is constructed based on the determined target description sub-sentences.

7. The method according to claim 1, characterized in that The step of determining the target description text for describing the geometric image at least according to the corrected intermediate description text comprises: Splitting the question text and the answer text into N1 reference description sub-sentences according to the sentence segmentation symbols in the question text and the answer text; and splitting the intermediate description text into N2 intermediate description sub-sentences according to the sentence segmentation symbols in the intermediate description text, where N1 and N2 are both positive integers; For each of the N2 intermediate description sub-sentences, calculate the similarity between the intermediate description sub-sentence and each of the N1 reference description sub-sentences, and if there is at least one similarity that is not less than a similarity threshold, determine the intermediate description sub-sentence as a target description sub-sentence; The target description text of the geometric image is constructed based on the determined target description sub-sentences.

8. The method according to claim 1, characterized in that Also includes: The geometric image and the target description text are used as training samples to train the image analysis model.

9. A device for generating image description text, characterized in that: include: An acquisition unit, used to acquire the question text and answer text of the geometry problem, and the geometry image in the geometry problem; A processing unit, used for inputting the geometric image into a pre-trained image description model, and obtaining an original description text output by the image description model for describing the geometric image; A determination unit is used to correct the original description text based on the question text and the answer text to obtain a corrected intermediate description text, and to determine a target description text for describing the geometric image at least based on the intermediate description text.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and executable instructions stored in the memory and executable to be executed on the processor, wherein the processor implements the method according to any one of claims 1 to 9 by executing the executable instructions.

11. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images