Method and device for automatically identifying and filling two tickets based on convolutional neural network and OCR (Optical Character Recognition)

Abstract

The invention relates to a method and a device for automatically identifying and filling two tickets based on a convolutional neural network and an OCR (Optical Character Recognition). The method comprises the following steps: acquiring a to-be-processed image; the to-be-processed image comprises a work ticket and / or an operation ticket; determining text information in the to-be-processed image through a text recognition model, and determining flow chart information in the to-be-processed image through a flow chart recognition model; determining key value pair information according to each piece of text information and each piece of flow chart information; the key value pair information represents an association relationship between each piece of text information and each piece of flow chart information; and according to the key value pair information, filling each piece of text information and each piece of flow chart information into a preset template to obtain a filling result. By adopting the method, information such as the text and the flow chart in the image can be accurately identified, the incidence relation between the text and the flow chart is analyzed, contents such as the text and the flow chart in the two tickets can be accurately and efficiently identified and automatically filled based on the paper two-ticket image, and the management efficiency of the two tickets in a power system is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for automatically identifying and filling out two invoices based on convolutional neural networks and OCR. Background Art

[0002] In the power system, the two tickets usually refer to the work ticket and the operation ticket. The work ticket stipulates the written instructions for working on electrical equipment, and the operation ticket records the written instructions for the operator to perform the operating steps according to the established procedures. The two-ticket system is an important measure to ensure safe operation in the power system. Through the strict implementation of the work ticket and the operation ticket, it can effectively ensure personnel safety, standardize operating procedures, improve system reliability, facilitate accident investigation, and ensure compliance and supervision.

[0003] The traditional paper-based two-ticket management method has drawbacks such as inefficiency and prone to errors, which makes it difficult to meet the needs of modern enterprises for efficient operation and refined management. In addition, the efficiency of data statistics and data analysis is not high, which is not conducive to improving the management efficiency of the two tickets in the power system. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for automatic identification and filling of two tickets based on convolutional neural network and OCR, which can improve the management efficiency of two tickets in the power system in response to the above technical problems.

[0005] In the first aspect, the present application provides a method for automatically identifying and filling in two tickets based on convolutional neural networks and OCR, including:

[0006] Acquire an image to be processed; the image to be processed includes a work ticket and / or an operation ticket;

[0007] Determine at least one text information in the image to be processed by a pre-trained text recognition model, and determine at least one flowchart information in the image to be processed by a pre-trained flowchart recognition model;

[0008] Determine key-value pair information according to each of the text information and each of the flowchart information; the key-value pair information represents the association relationship between each of the text information and each of the flowchart information;

[0009] Fill each of the text information and each of the flowchart information into a preset template according to the key-value pair information to obtain a filling result.

[0010] In one embodiment, the text recognition model includes a text region recognition network and a text position recognition network, and determining at least one text information in the image to be processed by using the pre-trained text recognition model includes:

[0011] Determine at least one text region in the image to be processed by the text region recognition network;

[0012] Determining the text position in each of the text regions by means of the text position recognition network;

[0013] Perform text recognition on the text position to obtain at least one text information in the image to be processed.

[0014] In one embodiment, the determining of at least one flowchart information in the image to be processed includes:

[0015] determining at least one flowchart region in the image to be processed;

[0016] According to preset flowchart feature information, the flowchart in each flowchart area is identified to obtain at least one flowchart information in the image to be processed.

[0017] In one embodiment, filling each of the text information and each of the flowchart information into a preset template according to the key-value pair information includes:

[0018] Determine the area to be filled in the preset template;

[0019] Determine text information and flow chart information matching the area to be filled in according to the key-value pair information;

[0020] Fill the text information and flow chart information matching the area to be filled in the area to be filled.

[0021] In one embodiment, obtaining the image to be processed includes:

[0022] Obtain the work ticket image and / or operation ticket image uploaded by the user through a preset interface;

[0023] The brightness and / or contrast of the work ticket image and / or the operation ticket image are adjusted to obtain the image to be processed.

[0024] In one embodiment, after the step of obtaining the filling result, the method further includes:

[0025] Verify the filled result according to the image to be processed and the preset verification rule to obtain a verification result;

[0026] and / or,

[0027] In response to an edit instruction for the filling result, the filling result is edited.

[0028] In the second aspect, the present application also provides a two-ticket automatic recognition and filling device based on convolutional neural network and OCR, including:

[0029] An acquisition module, used for acquiring an image to be processed; the image to be processed includes a work ticket and / or an operation ticket;

[0030] A recognition module, used to determine at least one text information in the image to be processed by a pre-trained text recognition model, and to determine at least one flowchart information in the image to be processed by a pre-trained flowchart recognition model;

[0031] An association module, used to determine key-value pair information according to each of the text information and each of the flowchart information; the key-value pair information represents the association relationship between each of the text information and each of the flowchart information;

[0032] The filling module is used to fill each of the text information and each of the flowchart information into a preset template according to the key-value pair information to obtain a filling result.

[0033] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0035] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0036] The above-mentioned method, device, computer equipment, computer-readable storage medium and computer program product for automatic recognition and filling of two tickets based on convolutional neural network and OCR can accurately extract text information and flowchart information in the image through the text recognition model and the flowchart recognition model by acquiring the image to be processed including the work ticket and / or the operation ticket, and determining at least one text information in the image to be processed through the pre-trained text recognition model, and determining at least one flowchart information in the image to be processed through the pre-trained flowchart recognition model; determine key-value pair information according to each text information and each flowchart information, so as to use the key-value pair information Characterize the association between each text information and each flowchart information; fill in each text information and each flowchart information into the preset template according to the key-value pair information to obtain the filling result, so as to accurately fill in the text, flowchart and other contents of the work ticket and / or operation ticket in the image into the template, and be able to accurately identify the text, flowchart and other information in the image, analyze the association between the text and the flowchart, and accurately fill in the corresponding information into the template, accurately and efficiently identify and automatically fill in the text, flowchart and other contents in the two tickets based on the paper two ticket images, generate accurate two ticket filling results, and improve the management efficiency of the two tickets in the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 This is an application environment diagram of a two-ticket automatic recognition and filling method based on a convolutional neural network and OCR in one embodiment;

[0039] Figure 2 It is a flowchart of a method for automatically identifying and filling in two tickets based on a convolutional neural network and OCR in one embodiment;

[0040] Figure 3 It is a flowchart of a two-ticket automatic recognition and filling solution based on convolutional neural network and OCR in one embodiment;

[0041] Figure 4 It is a structural block diagram of a two-ticket automatic recognition and filling device based on convolutional neural network and OCR in one embodiment;

[0042] Figure 5 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] The two-ticket automatic recognition and filling method based on convolutional neural network and OCR provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the image to be processed; the image to be processed includes a work ticket and / or an operation ticket; the server 104 determines at least one text information in the image to be processed through a pre-trained text recognition model, and determines at least one flowchart information in the image to be processed through a pre-trained flowchart recognition model; the server 104 determines the key-value pair information according to each text information and each flowchart information; the key-value pair information represents the association relationship between each text information and each flowchart information; the server 104 fills each text information and each flowchart information into the preset template according to the key-value pair information to obtain the filling result. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart TVs, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0045] In an exemplary embodiment, Figure 2 As shown, a method for automatically identifying and filling in two tickets based on convolutional neural network and OCR is provided, and the method is applied to a server as an example for explanation, including the following steps S202 to S208. Among them:

[0046] Step S202, obtaining an image to be processed; the image to be processed includes a work ticket and / or an operation ticket.

[0047] Among them, the image to be processed may refer to an image uploaded by a user. In actual applications, the image to be processed may include but is not limited to an image obtained by a user taking pictures of two paper tickets (work tickets and / or operation tickets) using an image acquisition device.

[0048] As an example, in key areas such as electricity and industry, traditional technology has long relied heavily on manual filling and processing of two tickets, which not only consumes a lot of manpower and time resources, but also makes it difficult to avoid filling errors caused by human factors. At the same time, traditional technology requires manual review and recording of relevant information one by one for data statistics and analysis of paper two tickets, and then summarizes and analyzes them. It is impossible to obtain the execution status and data of the two tickets in real time, which is not conducive to timely monitoring and management of work progress and safety conditions. For example, in the power system, it is impossible to timely understand the execution status of the two tickets at each work site, and it is difficult to timely discover potential safety hazards and problems. In order to improve the management efficiency of the two tickets in the power system, the user can first collect images of the paper two tickets and upload the images to the server. The server can receive the images uploaded by the user and pre-process the images to obtain the images to be processed.

[0049] Step S204: determining at least one text information in the image to be processed by using a pre-trained text recognition model, and determining at least one flowchart information in the image to be processed by using a pre-trained flowchart recognition model.

[0050] Among them, the text recognition model may refer to a model used to recognize text / words in an image. In practical applications, the text recognition model may include but is not limited to a semantic segmentation model based on deep learning and a convolutional neural network.

[0051] The text information may refer to information output by a text recognition model after performing text recognition on the text in the image to be processed.

[0052] The flowchart recognition model may refer to a model used to recognize flowcharts in an image. In practical applications, the flowchart recognition model may include but is not limited to a convolutional neural network, etc.

[0053] The flowchart information may refer to the information output by the flowchart recognition model after the flowchart in the image to be processed is recognized.

[0054] As an example, the server can input the image to be processed into a pre-trained text recognition model, which can identify at least one text information in the image to be processed. Specifically, the text recognition model can first segment the text area in the image, then detect the position of the text in the text area, and then use optical character recognition (OCR) technology to recognize the text content in each text area to obtain the text information in the image to be processed. The server can also input the image to be processed into a pre-trained flowchart recognition model, which can identify at least one flowchart information in the image to be processed. Specifically, the flowchart recognition model can first segment the area where the flowchart is located in the image, and then determine the type and characteristics of the flowchart elements contained in the image. The type and characteristics of the flowchart elements can be used as information describing the flowchart in the image to be processed (such as flowchart information).

[0055] Step S206, determining key-value pair information according to each text information and each flowchart information; the key-value pair information represents the association relationship between each text information and each flowchart information.

[0056] The key-value pair information may refer to information representing the association relationship between each text information and each flowchart information. In practical applications, the key-value pair information may include but is not limited to fields in the key-value format.

[0057] As an example, since the text information identified by the text recognition model and the flowchart information identified by the flowchart recognition model are scattered / independent, in order to organize / integrate the contents of the two pieces of content in the image to be processed (such as integrating the text information and the flowchart information), the server can determine the key-value pair information based on each piece of text information and each piece of flowchart information. Specifically, the server can perform text recognition on each piece of flowchart information, extract the key text and data elements in the flowchart in the image to be processed, and then the server can combine each piece of text information and the above-mentioned key text and data elements, and classify and mark the key text and data elements according to preset classification rules to determine the identification information of the key text and data elements. The server can determine the association between each piece of text information and each piece of flowchart information according to the identification information, and combine each piece of text information and the flowchart information associated / matched with each piece of text information into a field in key-value form to generate key-value pair information.

[0058] Step S208, fill in each text information and each flowchart information into the preset template according to the key-value pair information to obtain the filling result.

[0059] The preset template may refer to pre-set information for representing the interface layout of a work ticket and / or an operation ticket.

[0060] The filling result may refer to the information obtained after filling the text information and the flowchart information into the corresponding positions in the preset template.

[0061] As an example, since the key-value pair information can represent the association between each text information and each flowchart information, the server can first determine the flowchart information associated with each text information according to the key-value pair information, and the flowchart information has a corresponding layout area in the preset template. Then the server can fill in each text information and the flowchart information associated with each text information into the corresponding layout area in the preset template to obtain the filling result.

[0062] In the above-mentioned two-ticket automatic recognition and filling method based on convolutional neural network and OCR, by obtaining an image to be processed including a work ticket and / or an operation ticket, and determining at least one text information in the image to be processed through a pre-trained text recognition model, and determining at least one flowchart information in the image to be processed through a pre-trained flowchart recognition model, the text recognition model and the flowchart recognition model can accurately extract the text information and flowchart information in the image; determine the key-value pair information according to each text information and each flowchart information, so as to use the key-value pair information to characterize the association relationship between each text information and each flowchart information; fill each text information and each flowchart information into a preset template according to the key-value pair information to obtain a filling result, so as to accurately fill the text, flowchart and other contents of the work ticket and / or operation ticket in the image into the template, can accurately recognize the text and flowchart and other information in the image, analyze the association relationship between the text and the flowchart, and accurately fill the corresponding information into the template, based on the paper two-ticket image, accurately and efficiently recognize and automatically fill in the text and flowchart and other contents in the two tickets, generate accurate two-ticket filling results, and improve the management efficiency of the two tickets in the power system.

[0063] In an exemplary embodiment, the text recognition model includes a text region recognition network and a text position recognition network. At least one text information in the image to be processed is determined by the pre-trained text recognition model, including: determining at least one text region in the image to be processed by the text region recognition network; determining the text position in each text region by the text position recognition network; performing text recognition on the text position to obtain at least one text information in the image to be processed.

[0064] Among them, the text region recognition network can refer to a semantic segmentation model based on deep learning, such as U-Net convolutional neural network.

[0065] Among them, the text location recognition network can refer to the convolutional neural network Faster R-CNN.

[0066] The text area may refer to an area where text / words exist in the image to be processed.

[0067] The text position may refer to information representing the position of text / words in the text area.

[0068] As an example, the server can input the image to be processed into a text area recognition network. Since the text area recognition network is pre-trained, the text area recognition network learns the characteristic patterns of the text areas in the image during the training process. Therefore, the text area recognition network can determine and divide at least one text area in the image to be processed. Then the server can input each text area into a text position recognition network respectively. The text position recognition network can locate the text position in each text area. After that, the server can use optical character recognition technology to perform text recognition on each text position, analyze and recognize the characters in each text position, and output editable text information as the text information in the image to be processed.

[0069] In this embodiment, at least one text region in the image to be processed is determined by a text region recognition network; the text position in each text region is determined by a text position recognition network; text recognition is performed on the text position to obtain at least one text information in the image to be processed, and the text region in the image can be accurately segmented, the position of the text line can be accurately detected, the text content can be accurately recognized, and the accuracy of the text information can be improved.

[0070] In some embodiments, determining at least one flowchart information in the image to be processed includes: determining at least one flowchart area in the image to be processed; identifying the flowchart in each flowchart area according to preset flowchart feature information, and obtaining at least one flowchart information in the image to be processed.

[0071] The flowchart area may refer to an area where the flowchart exists in the image to be processed.

[0072] Among them, flowchart feature information may refer to information that characterizes the type of each flowchart element in the flowchart. In actual applications, flowchart elements may include lines, graphics, symbols, etc. The types of flowchart elements may include but are not limited to line direction, graphic shape, symbol style, etc.

[0073] As an example, the server can input the image to be processed into the flowchart recognition model. The flowchart recognition model can first identify and divide at least one flowchart area in the image to be processed, and then analyze the flowchart elements and the types of flowchart elements in each flowchart area according to the preset flowchart feature information. The flowchart elements and the types of flowchart elements can be used as the flowchart information in the image to be processed.

[0074] In this embodiment, by determining at least one flowchart area in the image to be processed; identifying the flowchart in each flowchart area according to preset flowchart feature information, and obtaining at least one flowchart information in the image to be processed, it is possible to accurately identify the area where the flowchart in the image is located, analyze the features of the flowchart, obtain accurate flowchart information, and improve the accuracy of the flowchart information.

[0075] In some embodiments, each text information and each flowchart information is filled into a preset template according to key-value pair information, including: determining an area to be filled in the preset template; determining the text information and flowchart information that matches the area to be filled in according to the key-value pair information; and filling the text information and flowchart information that matches the area to be filled in into the area to be filled in.

[0076] The area to be filled in may refer to an area for filling in text information and / or flowchart information.

[0077] As an example, the server can first analyze the areas in the preset template that need to be filled in with text information and / or flowchart information to obtain the areas to be filled in. The server can obtain the preset layout rules, the layout rules include text layout rules and flowchart layout rules, and then the server can determine the flowchart information corresponding to each area to be filled in according to the flowchart layout rules. Since the key-value pair information can represent the association between each text information and each flowchart information, the server can fill in the flowchart information corresponding to each area to be filled in and the text information associated with the flowchart information corresponding to each area to be filled in into the area to be filled in; the server can also determine the text information corresponding to each area to be filled in according to the text layout rules, and fill in the text information corresponding to each area to be filled in into the area to be filled in. At this time, the server can obtain the filling result.

[0078] In the present embodiment, by determining the areas to be filled in the preset template; determining the text information and flowchart information that match the areas to be filled in based on the key-value pair information; and filling the text information and flowchart information that match the areas to be filled in into the areas to be filled in, the text information and flowchart information that match each area to be filled in the preset template can be accurately and automatically filled into the areas to be filled in, thereby improving the accuracy of the filling results.

[0079] In some embodiments, obtaining the image to be processed includes: obtaining the work ticket image and / or operation ticket image uploaded by the user through a preset interface; adjusting the brightness and / or contrast of the work ticket image and / or operation ticket image to obtain the image to be processed.

[0080] The preset interface may refer to a data upload interface. In actual applications, the server may obtain images uploaded by users through the preset interface.

[0081] Here, users may refer to power system staff.

[0082] The work ticket image may refer to an image containing the work ticket. In practical applications, the work ticket image may include an image obtained by a user photographing the work ticket through an image acquisition device.

[0083] The operation ticket image may refer to an image containing the operation ticket. In practical applications, the operation ticket image may include an image obtained by a user taking a picture of the operation ticket using an image acquisition device.

[0084] As an example, the user can use the image acquisition device to photograph the work ticket to obtain a work ticket image. The user can also use the image acquisition device to photograph the operation ticket to obtain an operation ticket image. In a specific implementation, the work ticket in the work ticket image can be a complete work ticket. Similarly, the operation ticket in the operation ticket image can be a complete operation ticket. The user can upload the work ticket image and / or operation ticket image to the server through a preset interface. The server obtains the work ticket image and / or operation ticket image uploaded by the user and pre-processes the work ticket image and / or operation ticket image. In actual applications, the server can adjust the brightness and contrast of the image. The server can also perform denoising on the image. The pre-processed work ticket image and / or operation ticket image can be used as the image to be processed.

[0085] In this embodiment, the work ticket image and / or operation ticket image uploaded by the user is obtained through a preset interface; the brightness and / or contrast of the work ticket image and / or operation ticket image are adjusted to obtain the image to be processed, so that the image uploaded by the user can be pre-processed in time to avoid the image quality affecting the subsequent text recognition and flowchart recognition, thereby improving the accuracy of the two ticket recognitions.

[0086] In some embodiments, after the step of obtaining the filling result, the above method also includes: verifying the filling result according to the image to be processed and the preset verification rules to obtain the verification result; and / or, editing the filling result in response to the editing instruction for the filling result.

[0087] Among them, the preset verification rules may refer to information used to determine whether the filled-in results meet the preset processing requirements. In actual applications, the preset verification rules may include but are not limited to work ticket format / operation ticket format (such as font, font size, layout) and logical judgment rules.

[0088] The verification result may refer to information indicating whether the filled result meets the preset processing requirements. In practical applications, the verification result may include verification passed or verification failed.

[0089] The editing instruction may refer to information used to control the server to edit the filling result.

[0090] As an example, after obtaining the filling result, the server can verify the filling result according to the preset verification rules, analyze whether the format and logic of the filling result meet the corresponding requirements, and obtain the first verification result. The server can also compare the image to be processed and the filling result, analyze whether the information in the image to be processed is consistent with the filling result, and obtain the second verification result. The server can determine the final verification result of the filling result according to the first verification result and the second verification result; it can be understood that the server can also verify the filling result only according to the preset verification rules or only compare the image to be processed and the filling result for verification based on actual needs. In a specific implementation, the server can also edit the filling result. When the user is not satisfied with the filling result or the verification result of the filling result indicates that the filling result verification fails, the user can send an editing instruction for the filling result to the server through the operation terminal. The server can respond to the editing instruction for the filling result, edit the filling result, and obtain the edited filling result. If the user is still not satisfied, the server can continue to respond to the editing instruction for the filling result, edit the filling result, until the user is satisfied with the edited filling result. The server may also verify the edited filling result. If the verification fails, the server may continue to respond to the editing instruction for the filling result and edit the filling result until the edited filling result passes the verification.

[0091] In this embodiment, the verification result is obtained by verifying the filling result according to the image to be processed and the preset verification rules; and / or, in response to the editing instruction for the filling result, the filling result is edited, so that the filling result can be accurately verified, or the filling result can be adaptively edited to improve the accuracy of the filling result.

[0092] In some embodiments, Figure 3 As shown in the figure, a flow chart of the two-ticket automatic recognition and filling scheme based on convolutional neural network and OCR is provided. The user uploads the two-ticket image through a specific interface. After the server receives the image, it immediately starts the preprocessing operation to perform preliminary optimization and adjustment on the original image to eliminate possible noise, improve the image quality, and lay a good foundation for subsequent accurate recognition work. Specifically, the preprocessing operation can include appropriate enhancement of the brightness of the image to make the darker area clearly visible, and reasonable adjustment of the contrast to highlight the text and graphic parts in the image. The preprocessing operation mainly uses basic image processing methods, such as functions in the OpenCV library to adjust the brightness, contrast and clarity of the image.

[0093] On the basis of the pre-processing, the server can conduct in-depth analysis and recognition of the optimized image, such as accurately segmenting the text area in the image, accurately detecting the position of the text line, accurately identifying the text content, and effectively detecting the flowchart part in the image, extracting key features, and converting it to an editable format, thereby converting the information in the image into digital data that can be processed and used. Specifically, the server can use a semantic segmentation model based on deep learning, such as U-Net convolutional neural network for text area segmentation. This model can learn the characteristic patterns of the text area in the image, thereby achieving accurate division of the text area. Then the server can use the convolutional neural network Faster R-CNN to quickly and accurately locate the position of the text line in the image, providing an accurate target area for subsequent text recognition. After that, the server uses OCR technology for text recognition, which is essentially the classic text recognition algorithm model CRNN+CTC of the convolutional neural network model, to achieve in-depth analysis and recognition of the extracted text area and convert the text in the image into editable text information. The server can use the Mask R-CNN convolutional neural network to effectively detect the line flow chart part in the image, providing a basis for subsequent flowchart processing, and introduce convolutional neural networks such as VGG or ResNet architecture to extract key features such as line direction, graphic shape, and symbols in the flowchart. The key features such as line direction, graphic shape, and symbols in the flowchart can be used as flowchart information to provide support for accurate conversion and understanding of the meaning of the flowchart. Among them, the introduction of ResNet can prevent the gradient explosion and gradient disappearance problems that occur when the neural network obtains flowchart features when the network is deepened.

[0094] After identifying the text information and flowchart information, the server can organize and classify the text information and flowchart information according to pre-set rules and logic, thereby combining scattered and independent information fragments into a data set with a clear structure and logical relationship, and storing and managing it in the form of key-value. OCR technology is used to extract key information, combined with simple classification algorithms, such as rule-based classification, to extract fields, providing a convenient and efficient organization method for subsequent automatic filling and data utilization. Specifically, the server can use OCR technology to extract key text and data elements from complex image information. At the same time, combined with rule-based classification algorithms, according to pre-defined semantic and logical rules, the extracted information can be accurately classified and marked, and the identified text and flowchart information can be combined into key-value fields according to preset rules.

[0095] Afterwards, the server can use the organized structured data and pre-designed templates to automatically fill in the relevant information into the corresponding positions, generate preliminary two-ticket texts and route diagrams, thereby reducing the tediousness and possible errors of manual operations and improving work efficiency and accuracy. Specifically, the server can automatically fill in the organized information into the corresponding templates to generate preliminary two-ticket texts and route diagrams. For example, by using pre-written automated data filling algorithms and program logic, according to preset rules and templates, it can intelligently identify the type of data and the applicable location, and accurately fill it into the corresponding fields and areas, thereby quickly generating preliminary two-ticket documents (such as filling results), providing a basis for subsequent review and adjustment.

[0096] The server can also conduct a comprehensive and detailed inspection and verification of the preliminary results generated by automatic filling and conversion to ensure that they meet the requirements of the predetermined format, rules and business logic. At the same time, the terminal can provide an intuitive and convenient editing interface for operators and managers, allowing necessary modifications and adjustments to the generated results to meet the special needs and personalized requirements in actual operations. Specifically, the server can use OCR technology to compare and verify characters based on the fixed format and rules of the two tickets, and automatically review and evaluate the generated results in combination with a series of format checks and logical judgment rules. These rules cover multiple aspects such as fonts, font sizes, typesetting formats, and logical consistency of data to ensure the accuracy and standardization of the results. In the specific implementation, the terminal is also provided with an interactive interface, and users (such as team operators) can send editing instructions for the filling results to the server through intuitive operations such as mouse clicks, keyboard input, tablet handwriting, etc. The server can respond to the editing instructions and modify and adjust the automatically generated filling results. During the editing process, the operation is saved in real time, and functions such as undo and redo are provided to ensure the flexibility and convenience of editing.

[0097] In this embodiment, photos of the two tickets are captured by taking photos, and image adjustment is performed based on image acquisition and preprocessing to accurately optimize the brightness, contrast and clarity of the images, thereby laying a high-quality image foundation for the subsequent automatic recognition process, and effectively overcoming the recognition difficulties caused by the poor image quality of the traditional paper tickets. The text recognition link incorporates regional segmentation, efficient detection and precise recognition technology, and specifically adopts a semantic segmentation model in the field of deep learning and an optimized convolutional neural network model, which can improve the accuracy and processing efficiency of text recognition, and effectively avoid common recognition errors and input deviations under traditional technologies. The circuit flow chart recognition and conversion module integrates efficient graphic detection algorithms, sophisticated feature extraction technologies and conversion mechanisms to ensure accurate capture and efficient conversion of flow chart information, fully meeting the high standards of modern enterprises for complex information processing processes. For data statistics and analysis issues, information integration and automatic filling rely on efficient field intelligent extraction algorithms and automatic filling technologies to achieve rapid integration and seamless filling of information, greatly improving work efficiency and bringing a smoother and more efficient information processing experience to enterprises.

[0098] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a two-ticket automatic recognition and filling device based on a convolutional neural network and OCR for realizing the two-ticket automatic recognition and filling method based on a convolutional neural network and OCR involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the two-ticket automatic recognition and filling device based on a convolutional neural network and OCR provided below can refer to the limitations of the two-ticket automatic recognition and filling method based on a convolutional neural network and OCR above, and will not be repeated here.

[0100] In an exemplary embodiment, Figure 4As shown, a two-ticket automatic recognition and filling device based on convolutional neural network and OCR is provided, including: an acquisition module 402, a recognition module 404, an association module 406 and a filling module 408, wherein:

[0101] The acquisition module 402 is used to acquire the image to be processed; the image to be processed includes a work ticket and / or an operation ticket.

[0102] The recognition module 404 is used to determine at least one text information in the image to be processed by using a pre-trained text recognition model, and to determine at least one flowchart information in the image to be processed by using a pre-trained flowchart recognition model.

[0103] The association module 406 is used to determine key-value pair information according to each of the text information and each of the flowchart information; the key-value pair information represents the association relationship between each of the text information and each of the flowchart information.

[0104] The filling module 408 is used to fill each of the text information and each of the flowchart information into a preset template according to the key-value pair information to obtain a filling result.

[0105] In one of the exemplary embodiments, the text recognition model includes a text region recognition network and a text position recognition network, and the recognition module 404 is specifically used to determine at least one text region in the image to be processed through the text region recognition network; determine the text position in each of the text regions through the text position recognition network; perform text recognition on the text positions to obtain at least one text information in the image to be processed.

[0106] In one of the exemplary embodiments, the identification module 404 is specifically used to determine at least one flowchart area in the image to be processed; based on preset flowchart feature information, the flowchart in each flowchart area is identified to obtain at least one flowchart information in the image to be processed.

[0107] In one of the exemplary embodiments, the filling module 408 is specifically used to determine the area to be filled in the preset template; determine the text information and flowchart information matching the area to be filled in according to the key-value pair information; and fill the text information and flowchart information matching the area to be filled in into the area to be filled in.

[0108] In one of the exemplary embodiments, the acquisition module 402 is specifically used to acquire the work ticket image and / or operation ticket image uploaded by the user through a preset interface; adjust the brightness and / or contrast of the work ticket image and / or the operation ticket image to obtain the image to be processed.

[0109] In one of the exemplary embodiments, the above-mentioned device also includes a processing module, which is specifically used to verify the filling result according to the image to be processed and the preset verification rules to obtain the verification result; and / or, in response to the editing instruction for the filling result, edit the filling result.

[0110] Each module in the above-mentioned two-ticket automatic recognition and filling device based on convolutional neural network and OCR can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0111] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as images to be processed, text information, and flowchart information. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a two-ticket automatic recognition and filling method based on a convolutional neural network and OCR is implemented.

[0112] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0113] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0114] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0115] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0116] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0117] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0118] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0119] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for automatic recognition and filling of two tickets based on convolutional neural network and OCR, characterized in that: The method comprises: Acquire an image to be processed; the image to be processed includes a work ticket and / or an operation ticket; Determine at least one text information in the image to be processed by a pre-trained text recognition model, and determine at least one flowchart information in the image to be processed by a pre-trained flowchart recognition model; Determine key-value pair information according to each of the text information and each of the flowchart information; the key-value pair information represents the association relationship between each of the text information and each of the flowchart information; Fill each of the text information and each of the flowchart information into a preset template according to the key-value pair information to obtain a filling result.

2. The method according to claim 1, characterized in that The text recognition model includes a text region recognition network and a text position recognition network, and the determining of at least one text information in the image to be processed by the pre-trained text recognition model includes: Determine at least one text region in the image to be processed by the text region recognition network; Determining the text position in each of the text regions by means of the text position recognition network; Perform text recognition on the text position to obtain at least one text information in the image to be processed.

3. The method according to claim 1, characterized in that The determining of at least one flowchart information in the image to be processed includes: determining at least one flowchart region in the image to be processed; According to preset flowchart feature information, the flowchart in each flowchart area is identified to obtain at least one flowchart information in the image to be processed.

4. The method according to claim 1, characterized in that: Filling each of the text information and each of the flowchart information into a preset template according to the key-value pair information includes: Determine the area to be filled in the preset template; Determine text information and flow chart information matching the area to be filled in according to the key-value pair information; Fill the text information and flow chart information matching the area to be filled in the area to be filled.

5. The method according to claim 1, characterized in that The step of obtaining the image to be processed comprises: Obtain the work ticket image and / or operation ticket image uploaded by the user through a preset interface; The brightness and / or contrast of the work ticket image and / or the operation ticket image are adjusted to obtain the image to be processed.

6. The method according to claim 1, characterized in that After the step of obtaining the filling result, the method further includes: Verify the filled result according to the image to be processed and the preset verification rule to obtain a verification result; and / or, In response to an edit instruction for the filling result, the filling result is edited.

7. A two-ticket automatic recognition and filling device based on convolutional neural network and OCR, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed; the image to be processed includes a work ticket and / or an operation ticket; A recognition module, used to determine at least one text information in the image to be processed by a pre-trained text recognition model, and to determine at least one flowchart information in the image to be processed by a pre-trained flowchart recognition model; An association module, used to determine key-value pair information according to each of the text information and each of the flowchart information; the key-value pair information represents the association relationship between each of the text information and each of the flowchart information; The filling module is used to fill each of the text information and each of the flowchart information into a preset template according to the key-value pair information to obtain a filling result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images