AI-based code generation methods, devices, and electronic equipment
By acquiring block text data and question text from research papers, selecting target block text and generating code, the problem of code generation not addressed in existing technologies is solved, achieving automated, accurate, and consistent code generation.
Patent Information
- Application Number
- CN202511376838.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-24
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-09-24
AI Technical Summary
There is no existing technology that addresses code generation based on research papers.
By acquiring block text data and question text from multimedia text, selecting target block text using formula information, and combining it with a large model for code generation processing, including auxiliary element extraction and code element matrix construction to improve accuracy and logical consistency.
It achieves automated conversion from research papers to runnable code, improving the accuracy and logical consistency of code generation, and reducing the time and error of manual reproduction.
Smart Images

Figure CN121349418B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of deep learning, cloud computing, natural language processing, large models, intelligent agents, coding agents, and assisted programming, and especially to a code generation method, apparatus, and electronic device based on artificial intelligence. Background Technology
[0002] Currently, research papers can be parsed to obtain paragraph-level text; each paragraph-level text can include multiple natural paragraphs. This paragraph-level text is then used for paper comprehension and processing.
[0003] None of the above schemes involve code generation based on research papers. Summary of the Invention
[0004] This disclosure provides a code generation method, apparatus, and electronic device based on artificial intelligence.
[0005] According to one aspect of this disclosure, an artificial intelligence-based code generation method is provided, the method comprising: acquiring block text data of multimedia text and question text to be processed; the block text data including multiple block texts and block information of the block texts; at least one block information including formula information; selecting a target block text from the multiple block texts according to the multiple block texts, the formula information in the block information and the question text; and performing code generation processing according to the target block text, the block information of the target block text and the question text.
[0006] According to another aspect of this disclosure, an artificial intelligence-based code generation apparatus is provided. The apparatus includes: an acquisition module, configured to acquire block text data of multimedia text and question text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one of the block information includes formula information; a selection module, configured to select a target block text from the multiple block texts based on the multiple block texts, the formula information in the block information, and the question text; and a generation module, configured to perform code generation processing based on the target block text, the block information of the target block text, and the question text.
[0007] According to another aspect of this disclosure, an intelligent agent is provided, comprising: an acquisition module, configured to acquire block text data of multimedia text and question text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one of the block information includes formula information; a selection module, configured to select a target block text from the multiple block texts based on the multiple block texts, the formula information in the block information, and the question text; and a generation module, configured to perform code generation processing based on the target block text, the block information of the target block text, and the question text.
[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the artificial intelligence-based code generation method proposed above in this disclosure.
[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to execute the artificial intelligence-based code generation method proposed above in this disclosure.
[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the artificial intelligence-based code generation method proposed above.
[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0013] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;
[0014] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;
[0015] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;
[0016] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;
[0017] Figure 5This is a schematic diagram according to the fifth embodiment of the present disclosure;
[0018] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure;
[0019] Figure 7 This is a block diagram of an electronic device used to implement the AI-based code generation method of the embodiments of this disclosure. Detailed Implementation
[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0021] Currently, research papers can be parsed to obtain paragraph-level text; each paragraph-level text can include multiple natural paragraphs. This paragraph-level text is then used for paper comprehension and processing.
[0022] None of the above schemes involve code generation based on research papers.
[0023] To address the aforementioned issues, this disclosure proposes a code generation method, apparatus, and electronic device based on artificial intelligence.
[0024] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure. It should be noted that the AI-based code generation method of the present disclosure can be applied to an AI-based code generation device, which can be configured in an electronic device so that the electronic device can perform AI-based code generation functions.
[0025] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, smart speakers, servers, server clusters, and other hardware devices with various operating systems, touch screens and / or displays.
[0026] The AI-based code generation device can also be software within an electronic device, such as AI-based code generation software. The following embodiments use an electronic device as an example for illustration.
[0027] like Figure 1As shown, this AI-based code generation method may include the following steps:
[0028] Step 101: Obtain the block text data of the multimedia text and the problem text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one block information includes formula information.
[0029] Multimedia text can include papers, academic articles, research reports, and other materials; multimedia text blocks can refer to multimedia texts that have been segmented according to a set text length; block information can include formula information and block description information, where description information can refer to the block summary information.
[0030] The problem text to be processed can refer to the text obtained by converting the query input by the user in the interactive interface through voice, text, etc.; the query is used to request code generation based on multimedia text.
[0031] For example, the problem text to be processed can be "Please summarize the algorithm steps", "Explain the derivation process of equation (3)", "Please output the code corresponding to equations (1)-(5)", etc.
[0032] Step 102: Select the target block text from the multiple block texts based on the formula information in the block information and the question text.
[0033] Among them, the electronic device can select the target block text associated with the problem text from multiple block texts based on the formula information in the block information.
[0034] For example, if the problem text includes a specified formula, the target block text containing the specified formula can be selected from multiple block texts based on the formula information in the block information.
[0035] In this embodiment of the disclosure, prior to step 102, the electronic device may also perform the following process: extracting range-related information from the problem text; determining a first block of text in the block text data that matches the range-related information based on the range-related information; and filtering other block texts in the block text data except for the first block of text.
[0036] Among them, the scope-related information is used to indicate the boundaries and constraints involved in code generation. For example, the scope-related information may include page range, chapter range, formula range, etc.
[0037] After determining the first block of text that matches the range-related information in the block text data, the target block text can be selected from the first block text based on multiple block texts, formula information in the block information, and question text.
[0038] By extracting the scope information from the question text, relevant text blocks can be accurately located, the first text block can be quickly selected, and other irrelevant text blocks can be filtered out, thereby improving information retrieval efficiency, reducing the processing of irrelevant data, and ensuring that subsequent analysis focuses on text blocks within the specified scope.
[0039] Step 103: Perform code generation processing based on the target block text, the block information of the target block text, and the problem text.
[0040] The target block text, its block information, and the question text can be used as input to a large model or agent, enabling the model or agent to generate code based on the input information. It should be noted that users can engage in multi-turn dialogues with the large model or agent. In this case, the target block text can be selected based on the scope-related information within the multi-turn dialogue content, ensuring the coherence and accuracy of the multi-turn dialogue.
[0041] The AI-based code generation method disclosed herein acquires block text data of multimedia text and question text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one block information includes formula information; a target block text is selected from the multiple block texts based on the multiple block texts, the formula information in the block information, and the question text; code generation processing is performed based on the target block text, the block information of the target block text, and the question text; wherein, the formula is an important manifestation of the mathematical logic of the algorithm, and selecting the target block text from the multiple block texts in combination with the formula information can ensure that the code generation strictly follows the mathematical logic in the multimedia text, thereby improving the accuracy and logical consistency of the code generation.
[0042] To further improve the accuracy of code generation, electronic devices can perform auxiliary element extraction on block text to obtain auxiliary element information, thereby obtaining block information of the block text, and selecting the target block text from multiple block texts based on the fusion score of the block text. For example... Figure 2 As shown, Figure 2 This is a schematic diagram based on the second embodiment of the present disclosure. Figure 2 The illustrated embodiment may include the following steps:
[0043] Step 201: Obtain multimedia text; perform block division processing on the multimedia text to obtain multiple block texts.
[0044] The resulting block text may include at least one natural paragraph, and at least one block text may include at least one of the following: formula, image, and table.
[0045] Step 202: Extract auxiliary elements from the block text to obtain auxiliary element information; auxiliary elements include formulas and at least one of the following: images and tables.
[0046] In this embodiment of the disclosure, the electronic device may perform step 202 as follows: perform auxiliary element extraction processing and auxiliary element sequence number extraction processing on the block text to obtain auxiliary elements and / or auxiliary element sequence numbers in the block text; determine the auxiliary element bound to the auxiliary element sequence number; and determine auxiliary element information based on the auxiliary elements in the block text and / or the auxiliary elements bound to the auxiliary element sequence number.
[0047] Each auxiliary element has a corresponding auxiliary element sequence number. For example, a formula has a formula sequence number, an image has an image sequence number, and a table has a table sequence number. The auxiliary element information may include the descriptive information of the auxiliary element.
[0048] For example, the auxiliary element information corresponding to a formula may include the formula, the meaning of each parameter in the formula, and the function of the formula; the auxiliary element information corresponding to an image may include the image's descriptive text; and the auxiliary element information corresponding to a table may include the table's descriptive text.
[0049] The block text may include auxiliary elements and auxiliary element sequence numbers, or it may only include auxiliary element sequence numbers. For example, the current block text may reference auxiliary elements in other block texts by way of reference, without directly including the auxiliary elements.
[0050] The auxiliary element bound to the auxiliary element sequence number can be located in the current block text or in other block text.
[0051] By extracting auxiliary elements from the block text, or by locating the auxiliary elements associated with the block text through the sequence number of the auxiliary elements in the block text, it is possible to comprehensively and accurately capture the auxiliary elements associated with the block text, thereby obtaining comprehensive and complete auxiliary element information.
[0052] In one embodiment of this disclosure, the auxiliary element includes a formula; the auxiliary element sequence number includes a formula sequence number; the process by which the electronic device obtains the auxiliary element information of the formula can be, for example, by aggregating the formulas in the block text according to the formula sequence number to obtain the processed formula; and by determining the formula information in the auxiliary element information based on the processed formula and / or the formula bound to the formula sequence number.
[0053] The formula bound to the formula sequence number can be located in the current block text or in other block text.
[0054] Among them, regular expressions and heuristic methods can be used to extract formulas of various types from block text, such as explicit formulas (formulas that occupy a single line), inline formulas (formulas embedded in a line of text), and other mathematical expressions.
[0055] The formula information can include the processed formula, its corresponding formula serial number, and the meaning of the parameters in the formula, and / or the formula bound to the formula serial number, its corresponding formula serial number, and the meaning of the parameters in the formula. It should be noted that for the obtained formula, the parameters and their meanings can be extracted from the formula context and aligned with the main text to form a dictionary.
[0056] By aggregating formulas in block text according to their sequence numbers, scattered formula information can be effectively integrated, avoiding formula breaks. Combining the aggregated formulas and / or formulas bound to their sequence numbers to determine formula information allows the formulas to be structured into queryable data, enriching the basis for code generation and ensuring the completeness and accuracy of the formula information.
[0057] In one example of this disclosure, the auxiliary elements include images and / or tables; the auxiliary element serial number includes an image serial number and / or a table serial number; the process by which the electronic device obtains the auxiliary element information of the image and / or table can be, for example, determining the image information of the auxiliary element information based on the image description text in the block text and / or the image description text bound to the image serial number; and / or determining the table information in the auxiliary element information based on the table description text in the block text and / or the table description text bound to the table serial number.
[0058] The image description text can be a summary of the image content; for example, the image description text can be a summary of the image content. The table description text can be a summary of the table content; for example, the table description text can be a summary of the table content.
[0059] Specifically, for images and tables, the descriptive text of images and tables within block text, as well as the descriptive text of images and tables bound to serial numbers, can structure images and tables into queryable data, enriching the basis information for code generation and thus improving the accuracy of code generation.
[0060] Step 203: Determine the block information of the block text based on the auxiliary element information; determine the block text data based on multiple block texts and their block information.
[0061] The block information includes auxiliary element information and block text number, etc.; the block data includes multiple block texts and the block information corresponding to each block text.
[0062] Step 204: Obtain the text of the problem to be processed.
[0063] The process can begin by processing the multimedia text uploaded by the user through steps 201-203, then obtaining the query input by the user, and finally processing the query into question text.
[0064] Step 205: For each block text, if the block information of the block text includes formula information, determine the formula matching score of the block text, and at least one of the following: keyword matching score, vector similarity score, based on the block text, the block information of the block text, and the question text.
[0065] Among them, the formula matching score is used to indicate the degree of matching between the formula information corresponding to the block text and the formula information in the question text; the keyword matching score is used to indicate the degree of matching between the keywords in the question text and the keywords in the block text; and the vector similarity score is used to indicate the degree of similarity between the text vector of the question text and the text vector of the block text.
[0066] In one embodiment of this disclosure, the process by which an electronic device determines the keyword matching score of block text may include, for example, performing keyword extraction processing on the question text to obtain a first keyword in the question text; determining a second keyword in the first keyword; the second keyword being a keyword contained in the block text and / or the block information of the block text; and determining the keyword matching score based on the number of second keywords and the number of first keywords.
[0067] The second keyword can refer to a keyword that is contained in both the question text and the block text and / or the block information of the block text.
[0068] For example, the keyword matching score can be determined based on the ratio of the number of second keywords to the number of first keywords. The more second keywords there are, the higher the keyword matching score; the fewer second keywords there are, the lower the keyword matching score.
[0069] The keyword matching score, calculated by counting the number of primary and secondary keywords, can not only accurately measure the degree of matching between the question text and the block text, but also quantify the degree of keyword matching, providing an objective basis for selecting target block text.
[0070] In one example of this disclosure, the process by which an electronic device determines the vector similarity score of block text may include, for example, determining a first vector of the block text and a second vector of the question text; determining the vector similarity between the first vector and the second vector; and determining the vector similarity as a vector similarity score.
[0071] The first vector and the second vector can be text semantic vectors; vector similarity can refer to the cosine similarity between the first vector and the second vector.
[0072] By converting block text and question text into vectors and calculating their similarity, the semantic relationship between the two can be accurately quantified mathematically, resulting in an accurate vector similarity score.
[0073] In one example of this disclosure, the process by which an electronic device determines the formula matching score of block text may include, for example, determining the formula information of the question text; determining at least one of the following based on the formula information of the block text and the formula information of the question text: formula sequence number matching score, formula structure matching score, and formula variable matching score; and determining the formula matching score based on at least one of the formula sequence number matching score, formula structure matching score, and formula variable matching score.
[0074] Among them, the formula sequence number matching score is used to indicate the degree of matching between the formula sequence number associated with the question text and the formula sequence number associated with the block text; the formula structure matching score is used to indicate the degree of matching between the structure of the formula associated with the question text and the structure of the formula associated with the block text; and the formula variable matching score is used to indicate the degree of matching between the variables in the formula associated with the question text and the variables in the formula associated with the block text.
[0075] For example, an electronic device can calculate a formula serial number matching score using the following formula:
[0076] S eqno (d,q)=1{eqno(q)=eqno(d)}
[0077] Among them, S eqno (d,q) represents the formula sequence number matching score, eqno(q) represents the formula sequence number associated with the question text q, and eqno(d) represents the formula sequence number associated with the block text d. Wherein, if the formula sequence number associated with the question text is the same as the formula sequence number associated with the block text, then S... eqno If (d,q) is 1, and the formula sequence number associated with the question text is different from the formula sequence number associated with the block text, then S eqno (d,q) is 0.
[0078] For example, an electronic device can calculate a formula structure matching score using the following formula:
[0079]
[0080] TED(T q ,T d ) = min edit seq ∑ i ω(op i )
[0081] Among them, S subexpr (d,q) represents the formula structure matching score, TED(T) q ,T d ) represents two operator trees T q and T d The weighted tree edit distance between them, T q T represents the operator tree obtained by parsing the formula associated with the question text. d This represents the operator tree obtained by parsing the formula associated with the block text; Z represents the normalization constant, which can be the average tree size or the maximum tree depth; edit seq represents the edit sequence, which refers to the sequence of operations performed on the tree T. q Convert to tree T d A series of editing operations required; ω(op i ) represents the i-th edit operation op in the edit sequence. i The corresponding weights.
[0082] Where the structure of the formula associated with the question text is completely identical to the structure of the formula associated with the block text, TED(T) q ,T d ) = 0, S subexpr (d,q)=1; If the structure of the formula associated with the question text differs significantly from the structure of the formula associated with the block text, then TED(T) q ,T d ) rise, S subexpr (d,q) decay.
[0083] For example, an electronic device can calculate the formula variable matching score using the following formula:
[0084]
[0085] Among them, S var (d,q) represents the matching score of the formula variables, V q V represents the set of variables associated with the formula linked to the question text. d The set of variables representing the formula associated with the block of text. V represents q With V d The number of variables in the intersection of |V q | indicates the number of variables in the variable set of the formula associated with the question text.
[0086] For example, an electronic device can calculate a formula matching score using the following formula:
[0087] M(d,q)=λ1S eqno (d,q)+λ2S subexpr (d,q)+λ3S var(d,q)
[0088] Where M(d,q) represents the formula matching score, and λ1, λ2, and λ3 are the weights corresponding to the formula sequence number matching score, formula structure matching score, and formula variable matching score, respectively. As an example, λ1 = 0.3, λ2 = 0.3, and λ3 = 0.4.
[0089] By extracting formula information from the question text and block text, and calculating matching scores for both in terms of formula sequence number, structure, and variables, the formula matching degree can be accurately measured in multiple dimensions. By combining these scores to determine the formula matching score, the degree of matching between the formulas in the question text and block text can be comprehensively and meticulously evaluated.
[0090] Step 206: Determine the fusion score of the block text based on at least one of the keyword matching score, vector similarity score, and formula matching score.
[0091] The fusion score of the block text can be determined based on one or more of the keyword matching score and vector similarity score.
[0092] For example, an electronic device can calculate the fusion score of block text using the following formula:
[0093] score(d,q)=α·BM25(d,q)+β·cos(e(d),e(q))+γ·M(d,q)
[0094] Where score(d,q) represents the fusion score, BM25(d,q) represents the keyword matching score, cos(e(d),e(q)) represents the vector similarity score, e(d) represents the first vector, e(q) represents the second vector, and M(d,q) represents the formula matching score.
[0095] It should be noted that, in the absence of formula information in the block information of the block text, the keyword matching score and / or vector similarity score can be determined, and the fusion score of the block text can be determined based on the keyword matching score and / or vector similarity score.
[0096] Step 207: Select the target block text from the multiple block texts based on the fusion scores of the multiple block texts.
[0097] In one embodiment of this disclosure, the text blocks can be sorted in descending order based on their fusion scores, thereby selecting the top K text blocks from the sorted text blocks as the target text blocks.
[0098] In another example, the electronic device performing step 207 may be as follows: selecting multiple candidate block texts from multiple block texts based on their fusion scores; combining at least one candidate block text from the multiple candidate block texts to obtain a candidate block text combination; determining whether the candidate block text combination meets the constraints based on the fusion score, block text weight, and length weight of each candidate block text in the candidate block text combination; if the candidate block text combination meets the constraints, determining the candidate block text in the candidate block text combination as the target block text; if the candidate block text combination does not meet the constraints, re-acquiring candidate block text combinations until the acquired candidate block text combination meets the constraints.
[0099] Among them, the text blocks can be sorted in descending order based on their fusion scores, and the top K text blocks can be selected as candidate text blocks from the sorted text blocks.
[0100] Specifically, for any candidate block text, the block text weight w(d) can be calculated based on the candidate block text's fusion score, block text weight, and length weight. Based on the block text weights of each candidate block text in the candidate block text combination, it is determined whether the candidate block text combination meets the constraints. It should be noted that the block text weight can be determined based on the chapter in which the block text is located, and the length weight is determined based on the block text length.
[0101] Here, the constraint can refer to a length cost constraint. For example, the process of obtaining candidate block text combinations that meet the constraint can be represented by the following objective function:
[0102]
[0103] Where w(d) represents the weight of the d-th candidate block text in the candidate block text combination, S represents the candidate block text combination, D represents the combination of multiple candidate block texts selected from multiple block texts, c(d) represents the text length of the d-th candidate block text in the candidate block text combination, and B represents the set text length budget.
[0104] One approach is to use a heuristic "greedy algorithm + diversity" to determine candidate block text combinations that meet the constraints.
[0105] It should be noted that the block text weight can also be determined based on the candidate block text's fusion score, block text weight, length weight, and corresponding diversity score, so that the target block text maintains both breadth and criticality within a limited budget. The diversity score can include chapter diversity score and / or page diversity score.
[0106] The process involves filtering candidate block texts based on fusion scores and combining them to obtain candidate block text combinations. By comprehensively considering fusion scores, block text weights, and length weights, it is determined whether the candidate block text combinations meet the constraints. This method can accurately filter out target block texts that are related to the problem text and meet the constraints.
[0107] Step 208: Perform code generation processing based on the target block text, the block information of the target block text, and the problem text.
[0108] It should be noted that for details regarding steps 204 and 208, please refer to [the relevant documentation / reference]. Figure 1 Steps 101 and 103 in the illustrated embodiment will not be described in detail here.
[0109] This disclosure discloses an artificial intelligence-based code generation method that acquires multimedia text; performs block segmentation on the multimedia text to obtain multiple block texts; extracts auxiliary elements from the block texts to obtain auxiliary element information; the auxiliary elements include formulas and at least one of the following: images and tables; determines block information of the block texts based on the auxiliary element information; determines block text data based on the multiple block texts and their block information; acquires the question text to be processed; for each block text, if the block information includes formula information, determines the formula matching score of the block text, its block information, and the question text, and at least one of the following: keyword matching score and vector similarity score; and determines the formula matching score of the block text based on the keyword matching score and vector similarity score. The fusion score of the block text is determined by at least one of the degree scores and the formula matching score. Based on the fusion scores of multiple block texts, a target block text is selected from them. Code generation is then performed based on the target block text, its block information, and the question text. Specifically, block segmentation of multimedia text enables its decomposition, and the extraction of auxiliary elements from the block text allows for the construction of comprehensive and structured block text data based on formulas, images, tables, and other information, providing a basis for subsequent selection of target block texts. Combining the question text with the fusion score of the block text calculated from multiple dimensions and using this score to filter target block texts can identify those that match the question text, thereby improving the accuracy of code generation and making the generated code more aligned with user needs.
[0110] To further improve the standardization and accuracy of code generation, a code element matrix can be determined based on the target block text and its block information to accurately identify key information for code generation. Code constraint information is then extracted from the code element matrix, and code generation is performed based on this constraint information and the code element matrix, thereby improving the standardization and accuracy of code generation. For example... Figure 3 As shown, Figure 3 This is a schematic diagram based on the third embodiment of the present disclosure. Figure 3 The illustrated embodiment may include the following steps:
[0111] Step 301: Obtain the block text data of the multimedia text and the problem text to be processed.
[0112] Step 302: Select the target block text from the multiple block texts based on the formula information in the block information and the question text.
[0113] Step 303: Extract code elements from the target block text and its block information to obtain at least one code element and its element content.
[0114] Among them, code elements refer to the technical points required to generate target code. For example, code elements include problem and motivation, algorithm steps, formula definition, key hyperparameters, etc.
[0115] Specifically, the target block text and its block information can be mapped and reduced to obtain at least one code element and its element content.
[0116] In this embodiment of the disclosure, the electronic device may perform step 303 as follows: perform summary extraction processing on the target block text and the block information of the target block text to obtain the summary content corresponding to the target block text; perform summary processing on the summary content corresponding to each target block text to obtain the summary summary content; and perform code element extraction processing on the summary summary content to obtain at least one code element and the element content of the code element.
[0117] Specifically, a large model can be used to extract summaries of the target block text and its block information, summarize the summary content corresponding to each target block text, and extract code elements from the summarized summary content.
[0118] The process involves first extracting a summary from the target block text and block information, then summarizing the summary content corresponding to each target block text, and finally extracting code elements from the summary content. This avoids directly processing a large amount of raw text, significantly reducing the amount of data processing, lowering computational complexity, and improving the efficiency of code element extraction.
[0119] Step 304: Based on at least one code element and the element content of the code element, determine the code element matrix and the reference source of the code elements in the code element matrix; the reference source is the target block text or the block information of the target block text.
[0120] This involves generating a code element matrix based on the code elements and their corresponding content. It should be noted that the source of the code element's reference can also be included in the code element matrix. For example, the source of the reference can be a block text identifier or a block information identifier.
[0121] The code element matrix may also include the confidence level of the citation source, which is determined based on the fusion score of the block text.
[0122] Step 305: Extract code constraint information from the code element matrix.
[0123] The code constraint information is used to limit the model's degrees of freedom and avoid implementation details in the code that are inconsistent with the multimedia text. For example, the code constraint information includes input type, input shape (dimensional constraints of input data), output type, output shape (dimensional constraints of output data), variable domain and unit, exception conditions, preconditions and postconditions, etc.
[0124] In this embodiment of the disclosure, before step 305, the electronic device may further perform the following process: determining whether there are missing code elements in the code element matrix that have no element content set; if there are missing code elements in the code element matrix, determining the reference knowledge text corresponding to the problem text; extracting the element content of the missing code elements according to the reference knowledge text; and updating the code element matrix according to the element content of the missing code elements.
[0125] If there are missing code elements in the code element matrix, it indicates that the multimedia text does not describe the code element, and the resulting code element matrix is insufficient to generate complete code.
[0126] Among them, reference knowledge text can refer to knowledge text retrieved from other multimedia texts or other databases.
[0127] This includes extracting summaries from reference knowledge texts to obtain corresponding summary content; and extracting missing code elements from the summary content corresponding to the reference knowledge texts.
[0128] Specifically, by checking for missing content in the code element matrix, incomplete data can be detected in time before code generation; by identifying the reference knowledge text corresponding to the problem text to extract the missing element content and update the matrix, the code element matrix can be supplemented and improved by combining external information, and then code can be generated based on the updated code element matrix, which can improve the completeness of code generation.
[0129] Step 306: Using code constraint information as prompt text, perform code generation processing based on the code element matrix to obtain the target code.
[0130] The prompt text may also include information such as the constraint temperature of the large model, explicit dependencies (such as code database limitations), and output directory structure templates and function comment templates.
[0131] In this embodiment of the disclosure, after step 306, the electronic device may further perform the following process: generating test code corresponding to the target code based on the target code; performing test processing on the target code based on the test code to obtain test results; if the test results indicate that there is an abnormal code segment in the target code, determining the target code elements related to the abnormal code segment in the code element matrix; adjusting the target code according to the reference source of the target code elements until there is no abnormal code segment in the adjusted target code.
[0132] The test code can be used for unit testing / property testing, such as checking for non-negativity of the loss function, monotonicity within the parameter domain, and numerical stability. It can also be used for shape, type, and boundary instance checks, as well as scoring of static analysis rules.
[0133] This includes the ability to re-retrieve the content of code elements based on their source references, and then adjust the target code accordingly.
[0134] Among them, by testing the code, it is possible to achieve automated testing of the target code and accurate location of problematic code segments; by adjusting and processing the target code based on the target code elements related to the abnormal code segments, it is possible to achieve an automatic repair loop for abnormal code, reduce the cost of manual investigation, and improve the quality of code generation.
[0135] In this embodiment of the disclosure, the process of adjusting the target code based on the reference source of the target code element by the electronic device may be as follows: determining the first target block text or first block information related to the abnormal code segment based on the reference source of the target code element; determining the adjustment strategy for the abnormal code segment based on the first target block text or first block information; and adjusting the abnormal code segment in the target code according to the adjustment strategy to obtain the adjusted target code.
[0136] Among these features, targeted retrieval can be performed based on the source of references to target code elements, such as by formula serial number, formula variable, or chapter neighborhood; in addition, adjustment strategies can be generated based on the minimum difference patching strategy to avoid new errors caused by large-scale rewriting.
[0137] Among them, the root cause of the exception is directly related to the original basis of code generation. Therefore, compared with blind repair, accurately locating the root cause of the exception (the first target block text or the first block information) by referencing the source of the target code element, and determining the adjustment strategy in combination with the root cause of the exception, can avoid introducing new problems, make the repair more accurate and thorough, and thus improve the repair effect.
[0138] It should be noted that for details of steps 301 to 302, please refer to [the relevant documentation / reference]. Figure 1 Steps 101 to 102 in the illustrated embodiment will not be described in detail here.
[0139] This disclosure discloses an artificial intelligence-based code generation method that acquires block text data of multimedia text and question text to be processed; selects a target block text from multiple block texts based on formula information in block information and the question text; extracts code elements from the target block text and its block information to obtain at least one code element and its element content; determines a code element matrix and the reference source of the code elements in the code element matrix based on the at least one code element and its element content; the reference source is the target block text or its block information; extracts code constraint information from the code element matrix; and uses the code constraint information as prompt text to perform code generation processing based on the code element matrix to obtain target code. Through code element extraction and matrix construction, the key points of the algorithm in the multimedia text can be systematically integrated, improving the standardization and accuracy of code generation; the reference source of the code elements enables code generation basis backtracking; and by extracting constraint information from the code element matrix to guide code generation, the consistency between the generated code and the algorithm in the multimedia text can be improved.
[0140] Figure 4 This is a schematic diagram based on the fourth embodiment of the present disclosure, as shown below. Figure 4The diagram shows a framework for multimedia text processing. This disclosure can be applied to the "from research paper to runnable code" scenario, and the following explanation uses an academic paper as an example. Specifically, it may include the following steps: Step 401, inputting the academic paper, or inputting the Uniform Resource Locator (URL) of the academic paper, where the academic paper can be in Portable Document Format (PDF), Document (DOC), Document Word Open XML Format (DOCX), etc. Step 402, paper parsing and processing: dividing the academic paper into blocks to obtain multiple block texts and determining the block text data. Step 403, multi-way retrieval and evidence selection: performing keyword retrieval and vector retrieval to determine the fusion score of the block texts, and selecting the target block text from multiple block texts based on the fusion score. Step 404, abstract and key point extraction: extracting code elements and determining the code element matrix based on the code elements and their content. Step 405, Code Generation and Interface Contract: Extract code constraint information (interface contract) from the code element matrix. Using the code constraint information as prompt text, perform code generation processing based on the code element matrix to obtain the target code. Step 406, Automated Testing and Execution: Generate test code and perform testing processing on the target code based on the test code. Step 407, Error Backtracking and Repair: If abnormal code segments exist in the target code, adjust the target code based on the reference sources of the target code elements related to the abnormal code segments until no abnormal code segments exist in the adjusted target code. Step 408, Output: Repaired code, test cases, and verification report.
[0141] This disclosure (1) can strictly convert the key points of the algorithm, mathematical formulas and verification steps in the paper into executable code and perform automated testing. This process not only saves a lot of manual reproduction time, but also ensures that the code is consistent with the algorithm in the paper, reducing the errors that may occur during manual reproduction and the errors and deviations of manual debugging; (2) the generated code has clear evidence sources for each part of the paper, making the reproduction process more transparent and verifiable; (3) through the automated error backtracking and repair mechanism, a closed loop of test-failure-evidence backtracking-minimum repair is realized, avoiding the complexity of manual debugging.
[0142] To implement the above embodiments, this disclosure also provides a code generation apparatus based on artificial intelligence. For example... Figure 5 As shown, Figure 5This is a schematic diagram according to the fifth embodiment of the present disclosure. The AI-based code generation device 50 may include: an acquisition module 501, a selection module 502, and a generation module 503.
[0143] The acquisition module 501 is used to acquire block text data of multimedia text and question text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one of the block information includes formula information; the selection module 502 is used to select a target block text from the multiple block texts according to the multiple block texts, the formula information in the block information and the question text; the generation module 503 is used to perform code generation processing according to the target block text, the block information of the target block text and the question text.
[0144] As one possible implementation of this disclosure, the acquisition module 501 is specifically configured to: acquire the multimedia text; perform block division processing on the multimedia text to obtain multiple block texts; perform auxiliary element extraction processing on the block texts to obtain auxiliary element information; the auxiliary elements include formulas and at least one of the following: images and tables; determine the block information of the block texts based on the auxiliary element information; and determine the block text data based on the multiple block texts and the block information of the block texts.
[0145] As one possible implementation of this disclosure, the acquisition module 501 is specifically used to perform auxiliary element extraction processing and auxiliary element sequence number extraction processing on the block text to obtain auxiliary elements and / or auxiliary element sequence numbers in the block text; determine the auxiliary elements bound to the auxiliary element sequence numbers; and determine the auxiliary element information based on the auxiliary elements in the block text and / or the auxiliary elements bound to the auxiliary element sequence numbers.
[0146] As one possible implementation of this disclosure, the auxiliary element includes a formula; the auxiliary element sequence number includes a formula sequence number; the acquisition module 501 is specifically used to aggregate the formulas in the block text according to the formula sequence number to obtain the processed formula; and determine the formula information in the auxiliary element information based on the processed formula and / or the formula bound to the formula sequence number.
[0147] As one possible implementation of this disclosure, the auxiliary element includes an image and / or a table; the auxiliary element sequence number includes an image sequence number and / or a table sequence number; the acquisition module 501 is specifically used to determine the image information in the auxiliary element information based on the image description text in the block text and / or the image description text bound to the image sequence number; and / or to determine the table information in the auxiliary element information based on the table description text in the block text and / or the table description text bound to the table sequence number.
[0148] As one possible implementation of this disclosure, the apparatus further includes: an extraction module for extracting range-related information from the problem text; a first determination module for determining a first block of text in the block text data that matches the range-related information based on the range-related information; and a first processing module for filtering other block texts in the block text data besides the first block of text.
[0149] As one possible implementation of this disclosure, the selection module 502 is specifically configured to: for each block text, if the block information of the block text includes formula information, determine the formula matching score of the block text, and at least one of the following: keyword matching score, vector similarity score, based on the block text, the block information of the block text, and the question text; determine the fusion score of the block text based on at least one of the keyword matching score, the vector similarity score, and the formula matching score; and select the target block text from the plurality of block texts based on the fusion scores of the plurality of block texts.
[0150] As one possible implementation of this disclosure, the selection module 502 is specifically used for: performing keyword extraction processing on the question text to obtain a first keyword in the question text; determining a second keyword in the first keyword; the second keyword being a keyword contained in the block text and / or the block information of the block text; and determining the keyword matching score based on the number of the second keyword and the number of the first keyword.
[0151] As one possible implementation of this disclosure, the selection module 502 is specifically used to: determine a first vector of the block text and a second vector of the question text; determine the vector similarity between the first vector and the second vector; and determine the vector similarity as the vector similarity score.
[0152] As one possible implementation of this disclosure, the selection module 502 is specifically configured to: determine the formula information of the problem text; determine at least one of the following based on the formula information of the block text and the formula information of the problem text: formula sequence number matching score, formula structure matching score, and formula variable matching score; and determine the formula matching score based on at least one of the formula sequence number matching score, the formula structure matching score, and the formula variable matching score.
[0153] As one possible implementation of this disclosure, the selection module 502 is specifically configured to: select multiple candidate block texts from the multiple block texts based on their fusion scores; combine at least one candidate block text from the multiple candidate block texts to obtain a candidate block text combination; determine whether the candidate block text combination meets the limiting conditions based on the fusion score, block text weight, and length weight of each candidate block text in the candidate block text combination; if the candidate block text combination meets the limiting conditions, determine the candidate block text in the candidate block text combination as the target block text; if the candidate block text combination does not meet the limiting conditions, re-acquire candidate block text combinations until the acquired candidate block text combination meets the limiting conditions.
[0154] As one possible implementation of this disclosure, the generation module 503 is specifically configured to: extract code elements from the target block text and its block information to obtain at least one code element and its element content; determine a code element matrix and the reference source of the code elements in the code element matrix based on the at least one code element and its element content; the reference source is the target block text or its block information; extract code constraint information from the code element matrix; and use the code constraint information as prompt text to perform code generation processing based on the code element matrix to obtain the target code.
[0155] As one possible implementation of this disclosure, the generation module 503 is specifically used for: performing summary extraction processing on the target block text and the block information of the target block text to obtain summary content corresponding to the target block text; performing summary processing on the summary content corresponding to each target block text to obtain summary summary content; and performing code element extraction processing on the summary summary content to obtain at least one code element and the element content of the code element.
[0156] As one possible implementation of this disclosure, the apparatus further includes: a second determining module, configured to determine whether there are missing code elements in the code element matrix that are not set with element content; a third determining module, configured to determine the reference knowledge text corresponding to the problem text when there are missing code elements in the code element matrix; a second processing module, configured to extract the element content of the missing code elements according to the reference knowledge text; and a third processing module, configured to update the code element matrix according to the element content of the missing code elements.
[0157] As one possible implementation of this disclosure, the apparatus further includes: a generation module 503, further configured to generate test code corresponding to the target code based on the target code; a testing module, configured to perform test processing on the target code based on the test code to obtain test results; a fourth determining module, configured to determine target code elements in the code element matrix related to the abnormal code segment when the test results indicate that there is an abnormal code segment in the target code; and an adjustment module, configured to adjust the target code based on the reference source of the target code elements until there is no abnormal code segment in the adjusted target code.
[0158] As one possible implementation of this disclosure, the adjustment module is specifically configured to: determine a first target block text or a first block information related to the abnormal code segment based on the reference source of the target code element; determine an adjustment strategy for the abnormal code segment based on the first target block text or the first block information; and perform adjustment processing on the abnormal code segment in the target code according to the adjustment strategy to obtain the adjusted target code.
[0159] The AI-based code generation apparatus of this disclosure acquires block text data of multimedia text and question text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one block information includes formula information; a target block text is selected from the multiple block texts based on the multiple block texts, the formula information in the block information, and the question text; code generation processing is performed based on the target block text, the block information of the target block text, and the question text; wherein, the formula is an important manifestation of the mathematical logic of the algorithm, and selecting the target block text from the multiple block texts in combination with the formula information can ensure that the code generation strictly follows the mathematical logic in the multimedia text, thereby improving the accuracy and logical consistency of the code generation.
[0160] To implement the above embodiments, this disclosure also provides an intelligent agent. For example... Figure 6 As shown, Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure.
[0161] like Figure 6 As shown, the intelligent agent 60 may include: an acquisition module 601, a selection module 602, and a generation module 603.
[0162] The acquisition module 601 is used to acquire block text data of multimedia text and question text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one of the block information includes formula information; the selection module 602 is used to select a target block text from the multiple block texts based on the multiple block texts, the formula information in the block information and the question text; the generation module 603 is used to perform code generation processing based on the target block text, the block information of the target block text and the question text.
[0163] The intelligent agent of this disclosure acquires block text data of multimedia text and question text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one block information includes formula information; a target block text is selected from the multiple block texts based on the multiple block texts, the formula information in the block information, and the question text; code generation processing is performed based on the target block text, the block information of the target block text, and the question text; wherein, the formula is an important manifestation of the mathematical logic of the algorithm, and selecting the target block text from the multiple block texts in combination with the formula information can ensure that the code generation strictly follows the mathematical logic in the multimedia text, thereby improving the accuracy and logical consistency of the code generation.
[0164] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information are all carried out with the consent of the users, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.
[0165] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0166] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0167] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.
[0168] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0169] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as AI-based code generation methods. For example, in some embodiments, the AI-based code generation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the AI-based code generation method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform AI-based code generation methods by any other suitable means (e.g., by means of firmware).
[0170] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0171] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0172] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0174] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0175] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0176] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0177] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A code generation method based on artificial intelligence, the method comprising: Obtain the block text data of the multimedia text and the problem text to be processed; The block text data includes multiple block texts and block information of the block texts; at least one of the block information includes formula information; Based on the plurality of block texts, the formula information in the block information, and the question text, select the target block text from the plurality of block texts; Code generation processing is performed based on the target block text, the block information of the target block text, and the question text; The step of generating code based on the target block text, the block information of the target block text, and the question text includes: The target block text and its block information are processed to extract code elements to obtain at least one code element and its element content. Based on at least one of the code elements and the element content of the code elements, determine the code element matrix and the reference source of the code elements in the code element matrix; the reference source is the target block text or the block information of the target block text; Extract code constraint information from the code element matrix, wherein the code constraint information is used to restrict the model's degrees of freedom; Using the code constraint information as prompt text, code generation processing is performed based on the code element matrix to obtain the target code.
2. The method according to claim 1, wherein, The acquisition of block text data of multimedia text includes: Obtain the multimedia text; The multimedia text is divided into blocks to obtain multiple blocks of text; The block text is subjected to auxiliary element extraction processing to obtain auxiliary element information; the auxiliary elements include formulas and at least one of the following: images and tables; Based on the auxiliary element information, determine the block information of the block text; The block text data is determined based on the plurality of block texts and the block information of the block texts.
3. The method according to claim 2, wherein, The auxiliary element extraction process for the block text to obtain auxiliary element information includes: The block text is processed by extracting auxiliary elements and extracting auxiliary element sequence numbers to obtain the auxiliary elements and / or auxiliary element sequence numbers in the block text. Identify the auxiliary element bound to the auxiliary element sequence number; The auxiliary element information is determined based on the auxiliary elements in the block text and / or the auxiliary elements bound to the auxiliary element sequence number.
4. The method according to claim 3, wherein, The auxiliary element includes a formula; the auxiliary element serial number includes a formula serial number; The step of determining the auxiliary element information based on auxiliary elements in the block text and / or auxiliary elements bound to the auxiliary element sequence number includes: The formulas in the block text are aggregated according to their formula sequence numbers to obtain the processed formulas; Based on the processed formula and / or the formula bound to the formula sequence number, determine the formula information in the auxiliary element information.
5. The method according to claim 3 or 4, wherein, The auxiliary elements include images and / or tables; the auxiliary element serial numbers include image serial numbers and / or table serial numbers. The step of determining the auxiliary element information based on auxiliary elements in the block text and / or auxiliary elements bound to the auxiliary element sequence number includes: The image information in the auxiliary element information is determined based on the image description text in the block text and / or the image description text bound to the image sequence number; And / or, The table information in the auxiliary element information is determined based on the table description text in the block text and / or the table description text bound to the table sequence number.
6. The method according to claim 1, wherein, Before selecting the target block text from the plurality of block texts based on the plurality of block texts, the block information of the block texts, and the question text, the method further includes: Extract range-related information from the question text; Based on the range-related information, determine the first block of text in the block text data that matches the range-related information; The other block texts in the block text data, excluding the first block text, are filtered.
7. The method according to claim 1 or 6, wherein, The step of selecting a target block text from the plurality of block texts based on the plurality of block texts, the formula information in the block information, and the question text includes: For each block of text, if the block information of the block of text includes formula information, the formula matching score of the block of text is determined based on the block of text, the block information of the block of text, and the question text, as well as at least one of the following: keyword matching score and vector similarity score; The fusion score of the block text is determined based on at least one of the keyword matching score, the vector similarity score, and the formula matching score. The target block text is selected from the plurality of block texts based on their fusion scores.
8. The method according to claim 7, wherein, The method for determining the keyword matching score of the block text includes: The question text is subjected to keyword extraction processing to obtain the first keyword in the question text; Determine the second keyword from the first keyword; the second keyword is a keyword contained in the block text and / or the block information of the block text; The keyword matching score is determined based on the number of the second keyword and the number of the first keyword.
9. The method according to claim 7, wherein, The method for determining the vector similarity score of the block text includes: Determine the first vector of the block text and the second vector of the question text; Determine the vector similarity between the first vector and the second vector; The vector similarity is determined as the vector similarity score.
10. The method according to claim 7, wherein, The method for determining the formula matching score of the block text includes: Determine the formula information in the question text; Based on the formula information of the block text and the formula information of the question text, determine at least one of the following: formula sequence number matching score, formula structure matching score, and formula variable matching score; The formula matching score is determined based on at least one of the formula sequence number matching score, the formula structure matching score, and the formula variable matching score.
11. The method according to claim 7, wherein, The step of selecting the target block text from the plurality of block texts based on their fusion scores includes: Based on the fusion scores of the multiple block texts, select multiple candidate block texts from the multiple block texts; At least one candidate block text from the plurality of candidate block texts is combined to obtain a candidate block text combination; Based on the fusion score, block text weight, and length weight of each candidate block text in the candidate block text combination, determine whether the candidate block text combination meets the constraints. If the candidate block text combination meets the constraints, the candidate block text in the candidate block text combination is determined as the target block text; If the candidate block text combination does not meet the restriction condition, a new candidate block text combination is obtained until the obtained candidate block text combination meets the restriction condition.
12. The method according to claim 1, wherein, The step of extracting code elements from the target block text and its block information to obtain at least one code element and its element content includes: The target block text and its block information are subjected to a summary extraction process to obtain the summary content corresponding to the target block text; The summary content corresponding to each target block text is summarized to obtain the summary summary content; The summary content is processed by extracting code elements to obtain at least one code element and the element content of the code element.
13. The method according to claim 1, wherein, Before extracting code constraint information from the code element matrix, the method further includes: Determine whether there are any missing code elements in the code element matrix that do not have any element content set; In the case of missing code elements in the code element matrix, the reference knowledge text corresponding to the problem text is determined; Extract the element content of the missing code element based on the reference knowledge text; The code element matrix is updated based on the element content of the missing code elements.
14. The method according to claim 1, wherein, The method further includes: Based on the target code, generate the corresponding test code; The target code is tested according to the test code to obtain the test results; If the test results indicate that there is an abnormal code segment in the target code, determine the target code elements in the code element matrix that are related to the abnormal code segment; The target code is adjusted based on the source of the references to the target code elements until no abnormal code segments exist in the adjusted target code.
15. The method according to claim 14, wherein, The process of adjusting the target code based on the source of references to the target code elements includes: Based on the source of the reference of the target code element, determine the first target block text or the first block information related to the abnormal code segment; Based on the first target block text or the first block information, determine the adjustment strategy for the abnormal code segment; The abnormal code segment in the target code is adjusted according to the adjustment strategy to obtain the adjusted target code.
16. An artificial intelligence-based code generation device, the device comprising: The acquisition module is used to acquire block text data of multimedia text and the problem text to be processed; The block text data includes multiple block texts and block information of the block texts; at least one of the block information includes formula information; The selection module is used to select a target block text from the plurality of block texts based on the plurality of block texts, the formula information in the block information, and the question text; The generation module is used to perform code generation processing based on the target block text, the block information of the target block text, and the problem text; The step of generating code based on the target block text, the block information of the target block text, and the question text includes: The target block text and its block information are processed to extract code elements to obtain at least one code element and its element content. Based on at least one of the code elements and the element content of the code elements, determine the code element matrix and the reference source of the code elements in the code element matrix; the reference source is the target block text or the block information of the target block text; Extract code constraint information from the code element matrix, wherein the code constraint information is used to restrict the model's degrees of freedom; Using the code constraint information as prompt text, code generation processing is performed based on the code element matrix to obtain the target code.
17. An intelligent agent, comprising: The acquisition module is used to acquire block text data of multimedia text and the problem text to be processed; the block text data includes multiple block texts and block information of the block texts; at least one of the block information includes formula information; The selection module is used to select a target block text from the plurality of block texts based on the plurality of block texts, the formula information in the block information, and the question text; The generation module is used to perform code generation processing based on the target block text, the block information of the target block text, and the problem text; The step of generating code based on the target block text, the block information of the target block text, and the question text includes: The target block text and its block information are processed to extract code elements to obtain at least one code element and its element content. Based on at least one of the code elements and the element content of the code elements, determine the code element matrix and the reference source of the code elements in the code element matrix; the reference source is the target block text or the block information of the target block text; Extract code constraint information from the code element matrix, wherein the code constraint information is used to restrict the model's degrees of freedom; Using the code constraint information as prompt text, code generation processing is performed based on the code element matrix to obtain the target code.
18. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 15.
19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 15.
20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 15.
Citation Information
Patent Citations
Code conversion method and device, equipment, medium and computer program product
CN119987732A
Systems and methods for automated code generation for calculation based on associated formal specifications
US20240232188A1
