Intelligent programming assistance method and system based on image recognition

Abstract

The application discloses an intelligent programming auxiliary method and system based on image recognition, which comprises the following steps: acquiring an image of a programming language carrier, and defining the image as an information image to be extracted; performing image preprocessing on the information image to be extracted, and converting the information image to be extracted into a sample image; performing OCR recognition on the sample image, and obtaining target character data; acquiring a user instruction, creating a programming task according to the user instruction, and establishing a corresponding relationship with the specified target character data; when the programming task is executed, acquiring character data input by the user actively, and defining the character data as a previous character set; if the user instruction received in the programming task execution process meets preset function trigger information one, a programming following mode one is performed. The application has the effects of reducing repetitive labor in the programming process and improving the enthusiasm and positivity of relevant personnel.

Description

Technical Field

[0001] This application relates to the field of programming language development technology, and in particular to an intelligent programming assistance method and system based on image recognition. Background Technology

[0002] With the rapid development of computer technology, more and more people are coming into contact with and learning programming languages ​​due to their professions, interests, and other reasons.

[0003] For some non-professionals who are more interested in programming, there is a situation where they imitate the programs in some materials to practice. However, many blog posts and book examples contain a large number of repetitive programming languages. Due to the limitations of the medium (images, paper, etc.), they need to manually input the code. The large amount of repetitive input reduces the enthusiasm and motivation of the relevant personnel. Therefore, this application proposes a new technical solution. Summary of the Invention

[0004] To reduce repetitive work in the programming process and increase the enthusiasm and initiative of relevant personnel, this application provides an intelligent programming assistance method and system based on image recognition.

[0005] In a first aspect, this application provides an intelligent programming assistance method and system based on image recognition, employing the following technical solution:

[0006] An intelligent programming assistance method based on image recognition includes:

[0007] The image of the programming language carrier is obtained and defined as the image from which information is to be extracted.

[0008] The image to be extracted is preprocessed and converted into a sample image.

[0009] Perform OCR recognition on the sample image to obtain the target character data;

[0010] Obtain user instructions, create programming tasks based on user instructions, and establish a correspondence between the task and the specified target character data;

[0011] When the programming task is executed, the character data actively input by the user is obtained and defined as the previous character set;

[0012] If, during the execution of the programming task, the received user instruction matches the preset function trigger information one, then programming follow mode one is executed; wherein, programming follow mode one includes:

[0013] Based on pre-set programming logic, the preceding character set and the target character data are segmented to obtain combined dataset one and combined dataset two, each consisting of lines of code.

[0014] Compare combined dataset 1 and combined dataset 2, determine the next code in combined dataset 1 that matches the latest code in combined dataset 2, retrieve it, generate a code input prompt, and output the corresponding code for input after receiving confirmation feedback from the code input prompt.

[0015] Optionally, when a confirmation feedback with a code input prompt is received, a corresponding call is defined, and the output code is the latest character data actively entered by the user, and the previous character set is updated.

[0016] Optionally, if the received user instruction matches the preset function trigger information two during the execution of the programming task, then programming follow mode two is executed; wherein, programming follow mode two includes:

[0017] The most recently input character in the programming task is defined as the search root;

[0018] Based on the search root, find the target character data, determine the next character of the same character as the character to be filled in, retrieve it, and generate character filling prompts;

[0019] After receiving confirmation feedback with a prompt to fill in characters, output the corresponding characters to fill in the blanks.

[0020] Optionally, if there are multiple characters to be filled in, they are marked with serial numbers to generate a candidate input table;

[0021] Based on user instructions, the sequence number in the candidate input table is selected, and the selected character is determined to be the corresponding item for character filling.

[0022] Optionally, if the received user instruction matches the preset function trigger information three during the execution of the programming task, then programming follow mode three is executed; wherein, programming follow mode three includes:

[0023] Based on pre-set programming logic, the target character data is identified and segmented to obtain lines of code, which are then automatically retrieved and output as the programming content for the programming task in a top-to-bottom and left-to-right order.

[0024] Optionally, the programming follow mode three further includes:

[0025] The program logic is pre-set to identify and classify target character data to obtain various categories.

[0026] Once the programming content is complete, a prompt to delete the output category will appear.

[0027] If the feedback for the category deletion prompt is "OK", then obtain the category composition identification information subsequently entered by the user;

[0028] Based on the category composition identification information, the corresponding category composition in the programming content is deleted and replaced with a space.

[0029] Optionally, document pose correction can be performed before OCR recognition of the sample image.

[0030] Optionally, when the programming task is completed, the content of the current programming task is compared with the corresponding target character;

[0031] If the similarity of the comparison results is higher than the preset sample entry threshold, the content of the current programming task will be named based on the creation / end time of the programming task and recorded as the new target character data.

[0032] Secondly, this application provides an intelligent programming assistance system based on image recognition, which adopts the following technical solution:

[0033] An image recognition-based intelligent programming assistance system includes a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor to implement any of the image recognition-based intelligent programming assistance methods described above.

[0034] In summary, this application includes at least one of the following beneficial technical effects: In programming tasks, as long as the samples are pre-matched, the subsequent code can be automatically filled in based on the recognition results of the samples when the code that meets the conditions is encountered. This can reduce the amount of repetitive and meaningless tasks and ensure the enthusiasm and initiative of relevant personnel. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the programming following mode one of the methods in this application. Detailed Implementation

[0036] The following is in conjunction with the appendix Figure 1 This application will be described in further detail.

[0037] This application discloses an intelligent programming assistance method based on image recognition, which is implemented by loading a corresponding computer program through a processor, and the following behavior occurs during the interaction between the user and devices such as computers.

[0038] Reference Figure 1 Image recognition-based intelligent programming assistance methods include:

[0039] S1. Obtain the image of the programming language carrier, defined as the image from which information is to be extracted.

[0040] The programming language can be on paper or a web page, so the image can be taken by the user's mobile phone or captured by a screenshot tool. The obtained image is transmitted to a designated location to wait for information to be extracted and used.

[0041] S2. Perform image preprocessing on the image to be extracted, and convert it into a sample image.

[0042] In this embodiment, image preprocessing includes grayscale conversion and binarization of the image from which information is to be extracted. Grayscale conversion transforms a color image into a grayscale image, reducing the computational load of subsequent image processing; binarization aims to distinguish between the background and the target, laying the foundation for OCR recognition.

[0043] The image preprocessing methods and OCR recognition described above are existing technologies, and this application aims to apply them, so they will not be described in detail here.

[0044] S3. Perform OCR recognition on the sample image to obtain target character data; in this application, the target character is the programming language content in the image.

[0045] S4. Obtain user instructions, create programming tasks based on user instructions, and establish a correspondence between the task and the specified target character data.

[0046] It is understandable that user instructions refer to instructions generated by devices such as computers and tablets when users interact with them. Programming tasks can be created by users through programming language development tools or by creating tasks from a text file; both types have pre-established trigger mechanisms, meaning that when the creation behavior occurs, it is considered that a programming task has been created in this method.

[0047] It should be noted that the image to be extracted can be prepared by the user in advance or created and then implemented after the programming task is completed. Therefore, the correspondence between the programming task and the target character data can be established as follows:

[0048] 1) After the programming task is created, the image to be extracted is obtained. At this time, by default, the target character data to be extracted from the image is established to correspond with the current programming task.

[0049] 2) Before creating a programming task, information about the image to be extracted is obtained. At this time, after the programming task is created, a function button is provided. When the function button is clicked, the target character data is retrieved from the pre-matched database, displayed, and selected by the user. After the user confirms the selection, the correspondence between the selected target character data and the current programming task is established.

[0050] Based on the above, when the programming task is executed, that is, when the user starts to input programming language, the character data actively input by the user is obtained and defined as the previous character set.

[0051] If the user instruction received during the execution of the programming task matches the preset function trigger information one, for example: the programming task interface is pre-configured with a mode one button, and when the user clicks it, function trigger information one is output; the subsequent function trigger information two is similar, and will not be described in detail.

[0052] After receiving the above-mentioned function trigger information, this method follows the programming in mode one, specifically:

[0053] Based on pre-set programming logic, for example, using ";" as the end symbol of a line of code, the preceding character set and the target character data are split to obtain combined dataset one and combined dataset two, each consisting of a line of code.

[0054] The combined dataset 1 and combined dataset 2 are compared to determine the next code in combined dataset 1 that matches the latest code in combined dataset 2. This code is then retrieved, a code input prompt is generated, and after receiving confirmation feedback from the code input prompt, the corresponding code is output for input. In other words, the code is copied and pasted into the programming task.

[0055] The prompt can be displayed as a pop-up window, indicating whether to automatically fill in the next line of code and providing two buttons: OK and Cancel. When the user clicks OK, the prompt will confirm the confirmation.

[0056] As can be seen from the above, after applying this method, users can use image recognition technology to automatically complete the content they do not want to input during the programming process. Thus, this method can reduce repetitive work in the programming process and improve the enthusiasm and initiative of relevant personnel.

[0057] The above method only allows users to input one line of code, and the next line can be completed using extracted characters. This is insufficient to meet users' requirements for continuous code completion. Therefore, this method is further configured as follows:

[0058] When a confirmation feedback with a code input prompt is received, the corresponding call is defined, and the output code is the latest character data actively entered by the user, and the previous character set is updated.

[0059] That is, after completing a line of code using the extracted character information, the user can continue to choose automatic completion instead of manually entering the next line of code.

[0060] In another embodiment of this method, if the received user instruction during the execution of the programming task matches the preset function trigger information two, then programming follow mode two is performed; wherein, programming follow mode two includes:

[0061] The most recently input character in the programming task is defined as the search root;

[0062] Based on the search root, find the target character data, determine the next character of the same character as the character to be filled in, retrieve it, and generate character filling prompts;

[0063] After receiving confirmation feedback with a prompt to fill in characters, output the corresponding characters to fill in the blanks.

[0064] As can be seen from the above, this method not only allows users to complete the next line of code after writing one line, but also allows the use of a second mode, where users can complete the subsequent code using the extracted information after writing a certain part of the code, thus achieving a relatively better effect.

[0065] It is understandable that the same character may appear more than once in the same programming task. Therefore, this method also makes the following settings to prevent code completion errors, specifically:

[0066] When there are multiple characters to be filled in, they are marked with serial numbers to generate a candidate input table; for example, it can be displayed in the form of a pop-up window of the input method.

[0067] Based on user instructions, the sequence number in the candidate input table is selected, and the selected character is determined to be the corresponding item for character filling.

[0068] In another embodiment of this method, the method is not only used to automatically complete subsequent content after the user inputs preceding characters or code bars, but also provides a regional quick deletion function as an aid, specifically:

[0069] If, during the execution of the programming task, the received user instruction matches the preset function trigger information three, then programming follow mode three is executed; wherein programming follow mode three includes:

[0070] Based on pre-set programming logic, such as recognizing and segmenting target character data based on ";", the code is composed of lines and automatically retrieved and output as the programming content for the programming task according to the order from top to bottom and from left to right.

[0071] Based on the above, after the programming follow mode three is triggered, the content extracted from the image can be added to the programming task all at once, followed by:

[0072] Obtain pre-set programming logic, such as standard templates for if statements, identify (character features), and divide the target character data into various categories;

[0073] Once the programming content is filled in, a prompt to delete the output category will appear, i.e., a pop-up window will display whether to delete the category.

[0074] When the feedback for the category deletion prompt is "OK", the system obtains the category composition identification information subsequently entered by the user, such as the features of the if statement.

[0075] Based on the category composition identification information, the corresponding category composition in the programming content is deleted and replaced with a space.

[0076] The above settings can help users quickly delete regions of code extracted from images, which can be used to quickly generate teaching exercises and help relevant personnel conduct more targeted programming exercises.

[0077] As can be seen from the above, an important basis of this method is image recognition. However, the images provided by users are not necessarily upright, which affects the accuracy of information extraction from the images. Therefore, in this method, document pose correction is performed before performing OCR recognition on the sample images.

[0078] Document pose correction can be performed as follows: after obtaining the grayscale image, edge (contour) detection is performed, the horizontal lines in the image are enhanced, the bounding box of the target in the image is detected based on the Hough transform, the tilt angle is obtained, and the tilt angle is corrected for the target contour.

[0079] To effectively utilize users' usual programming achievements, this method not only extracts character data from user-uploaded images, but also sets it to:

[0080] When the programming task is completed, the content of the current programming task is compared with the corresponding target character;

[0081] If the similarity of the comparison results is higher than the preset sample entry threshold, the content of the current programming task will be named based on the creation / end time of the programming task and recorded as the new target character data.

[0082] That is, as long as the content of the programming task created by the user meets the condition of the difference between the content and the sample, the currently completed programming task will also be recorded and wait for reference when creating the next programming task.

[0083] This application also discloses an intelligent programming assistance system based on image recognition.

[0084] An image recognition-based intelligent programming assistance system includes a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor to implement any of the image recognition-based intelligent programming assistance methods described above.

[0085] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for intelligent programming assistance based on image recognition, characterized in that, include: The image of the programming language carrier is obtained and defined as the image from which information is to be extracted. The image to be extracted is preprocessed and converted into a sample image. Perform OCR recognition on the sample image to obtain the target character data; Obtain user instructions, create programming tasks based on user instructions, and establish a correspondence between the task and the specified target character data; When the programming task is executed, the character data actively input by the user is obtained and defined as the previous character set; If, during the execution of the programming task, the received user instruction matches the preset function trigger information one, then programming follow mode one is executed; wherein, programming follow mode one includes: Based on pre-set programming logic, the preceding character set and the target character data are segmented to obtain combined dataset one and combined dataset two, each consisting of lines of code. Compare combined dataset 1 and combined dataset 2, determine the next code in combined dataset 1 that matches the latest code in combined dataset 2, retrieve it, generate a code input prompt, and output the corresponding code for input after receiving confirmation feedback from the code input prompt.

2. The intelligent programming assistance method based on image recognition according to claim 1, characterized in that: When a confirmation feedback with a code input prompt is received, the corresponding call is defined, and the output code is the latest character data actively entered by the user, and the previous character set is updated.

3. The intelligent programming assistance method based on image recognition according to claim 1, characterized in that: If, during the execution of the programming task, the received user instruction matches the preset function trigger information two, then programming follow mode two is executed; wherein, programming follow mode two includes: The most recently input character in the programming task is defined as the search root; Based on the search root, find the target character data, determine the next character of the same character as the character to be filled in, retrieve it, and generate character filling prompts; After receiving confirmation feedback with a prompt to fill in characters, output the corresponding characters to fill in the blanks.

4. The intelligent programming assistance method based on image recognition according to claim 3, characterized in that: When there are multiple characters to be filled in, they are marked with serial numbers to generate a candidate input table; Based on user instructions, the sequence number in the candidate input table is selected, and the selected character is determined to be the corresponding item for character filling.

5. The intelligent programming assistance method based on image recognition according to claim 1, characterized in that: If, during the execution of the programming task, the received user instruction matches the preset function trigger information three, then programming follow mode three is executed; wherein, programming follow mode three includes: Based on pre-set programming logic, the target character data is identified and segmented to obtain lines of code, which are then automatically retrieved and output as the programming content for the programming task in a top-to-bottom and left-to-right order.

6. The intelligent programming assistance method based on image recognition according to claim 5, characterized in that: The third programming follow mode also includes: The program logic is pre-set to identify and classify target character data to obtain various categories. Once the programming content is complete, a prompt to delete the output category will appear. If the feedback for the category deletion prompt is "OK", then obtain the category composition identification information subsequently entered by the user; Based on the category composition identification information, the corresponding category composition in the programming content is deleted and replaced with a space.

7. The intelligent programming assistance method based on image recognition according to claim 1, characterized in that: Before performing OCR recognition on the sample image, document pose correction is performed first.

8. The intelligent programming assistance method based on image recognition according to claim 1, characterized in that: When the programming task is completed, the content of the current programming task is compared with the corresponding target character; If the similarity of the comparison results is higher than the preset sample entry threshold, the content of the current programming task will be named based on the creation / end time of the programming task and recorded as the new target character data.

9. An intelligent programming assistance system based on image recognition, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor to implement the image recognition-based intelligent programming assistance method as described in any one of claims 1-8.

Images