Image processing method, device, computer readable medium and electronic device

Abstract

The embodiments of the present application provide an image processing method, an image processing device, a computer-readable medium and an electronic device, which relate to the field of image processing technology. The method includes: extracting features from a list image to obtain image features corresponding to the list image; identifying each text box in the list image according to the image features, and detecting the fields to which each text box belongs based on the image features to obtain field detection results; obtaining multi-dimensional attributes of each text box, and determining the positional relationship between each text box and the adjacent text box according to the multi-dimensional attributes of each text box; and correcting the field detection results according to the positional relationship between each text box and the adjacent text box. In this way, the list image can be detected for the fields to which it belongs based on the identified text boxes in the list image, and then the positional relationship between the text boxes can be determined based on the attributes of each text box, and the field detection results can be corrected according to the positional relationship, thereby reducing detection errors and improving detection accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing device, a computer-readable medium, and an electronic device. Background Art

[0002] With the promotion of electronic verification systems, people are no longer limited to manually processing verification lists, but directly upload the list image to the electronic verification system for verification. After receiving the list image, the electronic verification system generally performs text detection on it to convert the image into text data for storage.

[0003] A list image generally consists of fields and field information. Fields are preset format information. Different list images may correspond to the same fields and different field information. The position of the field information usually needs to be adapted to the corresponding field to reflect the corresponding relationship.

[0004] However, the field information in different list images is different, and the length and content of the field information are also different. Performing the same list detection on different list images based on the above list image recognition method is prone to detection errors, which can easily lead to the problem of low detection accuracy.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide an image processing method, an image processing device, a computer-readable medium and an electronic device, which can detect the fields of a list image based on the identified text boxes in the list image, and then determine the positional relationship between the text boxes based on the attributes of each text box, and correct the field detection result according to the positional relationship, thereby reducing detection errors and improving detection accuracy.

[0007] In one aspect of an embodiment of the present application, an image processing method is provided, the method comprising:

[0008] Perform feature extraction on the list image to obtain image features corresponding to the list image;

[0009] Identify each text box in the list image according to the image features, and detect the field to which each text box belongs based on the image features to obtain a field detection result;

[0010] Acquire the multidimensional attributes of each text box, and determine the positional relationship between each text box and adjacent text boxes according to the multidimensional attributes of each text box;

[0011] Correct the field detection results based on the positional relationship between each text box and adjacent text boxes.

[0012] In an exemplary embodiment of the present application, feature extraction is performed on the list image to obtain image features corresponding to the list image, including:

[0013] Perform multi-layer convolution processing on the list image, and perform feature upsampling on the convolution results of each layer to obtain multiple upsampling results;

[0014] Multiple upsampling results are fused to obtain the image features corresponding to the list image.

[0015] In an exemplary embodiment of the present application, the fields to which each text box belongs are detected based on the image features to obtain the field detection results, including:

[0016] Generate a first-class representation diagram and a second-class representation diagram corresponding to the list image according to the image features; wherein the first-class representation diagram and the second-class representation diagram are used to highlight and display each text box from different dimensions;

[0017] The fields to which each text box belongs are detected according to the first representation graph and the second representation graph to obtain a field detection result.

[0018] In an exemplary embodiment of the present application, the multidimensional attributes include text features, position features, and image features, and obtaining the multidimensional attributes of each text box includes:

[0019] Extract features of the text content of each text box to obtain text features of each text box;

[0020] Extracting features from the position information of each text box to obtain position features of each text box; wherein the position information is used to represent the position of the corresponding text box in the list image;

[0021] Feature extraction is performed on the local image corresponding to each text box to obtain the image feature of each text box; wherein the local image corresponding to each text box belongs to the list image.

[0022] In an exemplary embodiment of the present application, determining the positional relationship between each text box and adjacent text boxes according to the multi-dimensional attributes of each text box includes:

[0023] Perform feature fusion of text features, position features and image features on each text box to obtain fusion features of each text box;

[0024] The positional relationship between each text box and adjacent text boxes is determined according to the fusion features of each text box.

[0025] In an exemplary embodiment of the present application, correcting the field detection result according to the positional relationship between each text box and an adjacent text box includes:

[0026] Determine at least one text box to be corrected in the field detection result according to the field arrangement mode and the positional relationship between each text box and an adjacent text box; wherein the at least one text box to be corrected includes a missed detection text box and / or an erroneously detected text box;

[0027] The field detection result is corrected according to the multi-dimensional attributes of the target text box related to at least one text box to be corrected.

[0028] In an exemplary embodiment of the present application, correcting the field detection result according to the multi-dimensional attributes of the target text box related to at least one text box to be corrected includes:

[0029] Determine the field arrangement of the list image; wherein the field arrangement includes a horizontal arrangement or a vertical arrangement;

[0030] Determine target text boxes respectively related to at least one text box to be corrected from the positional relationship between each text box and an adjacent text box according to the field arrangement mode;

[0031] Determine the position information and / or the field to which the corresponding text box to be modified belongs according to the position information and / or the field to which the target text box belongs;

[0032] The field detection result is corrected according to the position information of at least one text box to be corrected and / or the corresponding field.

[0033] In an exemplary embodiment of the present application, wherein:

[0034] If the field arrangement is horizontal, the target text box is the text box that is adjacent to the corresponding text box to be modified in the vertical direction;

[0035] If the field arrangement is vertical arrangement, the target text box is a text box that is adjacent to the corresponding text box to be modified in the horizontal direction.

[0036] In an exemplary embodiment of the present application, the fields to which each text box belongs are detected based on the image features to obtain the field detection results, including:

[0037] Obtain a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs;

[0038] Calculate the loss function corresponding to each field in the sample image according to the target field corresponding to each sample text box and the sample field of each sample text box;

[0039] The field detection model is trained according to the loss function corresponding to each field in the sample image;

[0040] The image features are input into the trained field detection model, so that the trained field detection model detects the fields to which each text box belongs, and obtains the field detection result.

[0041] In an exemplary embodiment of the present application, the fields to which each text box belongs are detected based on the image features to obtain the field detection results, including:

[0042] Obtain a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs and generates a first-class representation map and a second-class representation map corresponding to the sample image;

[0043] The first-class representation graph and the second-class representation graph are segmented at the field level according to the sample fields of each sample text box, so as to obtain the first-class representation partial graph and the second-class representation partial graph corresponding to the field to which each sample belongs;

[0044] Calculate the loss function corresponding to the field to which each sample belongs according to the first-class characterization local graph and the second-class characterization local graph corresponding to the field to which each sample belongs;

[0045] Train the field detection model according to the loss function;

[0046] The image features are input into the trained field detection model, so that the trained field detection model detects the fields to which each text box belongs, and obtains the field detection result.

[0047] In an exemplary embodiment of the present application, the field detection model is trained according to the loss function corresponding to each field in the sample image, including:

[0048] Determine the loss function corresponding to each sample text box belonging to the same field according to the target field corresponding to each sample text box and the sample field of each sample text box;

[0049] The loss functions corresponding to each sample text box belonging to the same field are merged to obtain the loss functions corresponding to each field in the sample image;

[0050] Determine the weighted sum of loss functions according to the preset field weights and the loss functions corresponding to each field;

[0051] The field detection model is trained according to the weighted sum of the loss function.

[0052] In an exemplary embodiment of the present application, the field detection model is trained according to the loss function corresponding to each field in the sample image, including:

[0053] Determine a first-class representation map and a second-class representation map corresponding to the sample image;

[0054] Determine the loss function corresponding to each field according to the first-class representation map and the second-class representation map corresponding to the sample image;

[0055] Fusion loss functions corresponding to each field to obtain the fusion result;

[0056] The field detection model is trained based on the fusion results.

[0057] In an exemplary embodiment of the present application, after correcting the field detection result according to the positional relationship between each text box and the adjacent text box, the method further includes:

[0058] generating structured information corresponding to the inventory image according to the field detection result;

[0059] Display structured information through the user end.

[0060] In one aspect of an embodiment of the present application, an image processing device is provided, the device comprising:

[0061] A feature extraction unit, used to extract features from the list image to obtain image features corresponding to the list image;

[0062] A field detection unit, used to identify each text box in the list image according to the image features, and detect the field to which each text box belongs based on the image features to obtain a field detection result;

[0063] A position relationship determination unit, used to obtain the multi-dimensional attributes of each text box, and determine the position relationship between each text box and the adjacent text boxes according to the multi-dimensional attributes of each text box;

[0064] The detection result correction unit is used to correct the field detection result according to the positional relationship between each text box and the adjacent text boxes.

[0065] In an exemplary embodiment of the present application, the feature extraction unit c extracts features from the list image to obtain image features corresponding to the list image, including:

[0066] Perform multi-layer convolution processing on the list image, and perform feature upsampling on the convolution results of each layer to obtain multiple upsampling results;

[0067] Multiple upsampling results are fused to obtain the image features corresponding to the list image.

[0068] In an exemplary embodiment of the present application, the field detection unit detects the fields to which each text box belongs based on the image features, and obtains the field detection result, including:

[0069] Generate a first-class representation diagram and a second-class representation diagram corresponding to the list image according to the image features; wherein the first-class representation diagram and the second-class representation diagram are used to highlight and display each text box from different dimensions;

[0070] The fields to which each text box belongs are detected according to the first representation graph and the second representation graph to obtain a field detection result.

[0071] In an exemplary embodiment of the present application, the multidimensional attributes include text features, position features, and image features. The position relationship determination unit obtains the multidimensional attributes of each text box, including:

[0072] Extract features of the text content of each text box to obtain text features of each text box;

[0073] Extracting features from the position information of each text box to obtain position features of each text box; wherein the position information is used to represent the position of the corresponding text box in the list image;

[0074] Feature extraction is performed on the local image corresponding to each text box to obtain the image feature of each text box; wherein the local image corresponding to each text box belongs to the list image.

[0075] In an exemplary embodiment of the present application, the position relationship determining unit determines the position relationship between each text box and the adjacent text boxes according to the multi-dimensional attribute of each text box, including:

[0076] Perform feature fusion of text features, position features and image features on each text box to obtain fusion features of each text box;

[0077] The positional relationship between each text box and adjacent text boxes is determined according to the fusion features of each text box.

[0078] In an exemplary embodiment of the present application, the detection result correction unit corrects the field detection result according to the positional relationship between each text box and an adjacent text box, including:

[0079] Determine at least one text box to be corrected in the field detection result according to the field arrangement mode and the positional relationship between each text box and an adjacent text box; wherein the at least one text box to be corrected includes a missed detection text box and / or an erroneously detected text box;

[0080] The field detection result is corrected according to the multi-dimensional attributes of the target text box related to at least one text box to be corrected.

[0081] In an exemplary embodiment of the present application, the detection result correction unit corrects the field detection result according to the multi-dimensional attribute of the target text box related to at least one text box to be corrected, including:

[0082] Determine the field arrangement of the list image; wherein the field arrangement includes a horizontal arrangement or a vertical arrangement;

[0083] Determine target text boxes respectively related to at least one text box to be corrected from the positional relationship between each text box and an adjacent text box according to the field arrangement mode;

[0084] Determine the position information and / or the field to which the corresponding text box to be modified belongs according to the position information and / or the field to which the target text box belongs;

[0085] The field detection result is corrected according to the position information of at least one text box to be corrected and / or the corresponding field.

[0086] In an exemplary embodiment of the present application, wherein:

[0087] If the field arrangement is horizontal, the target text box is the text box that is adjacent to the corresponding text box to be modified in the vertical direction;

[0088] If the field arrangement is vertical arrangement, the target text box is a text box that is adjacent to the corresponding text box to be modified in the horizontal direction.

[0089] In an exemplary embodiment of the present application, the field detection unit detects the fields to which each text box belongs based on the image features, and obtains the field detection result, including:

[0090] Obtain a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs and generates a first-class representation map and a second-class representation map corresponding to the sample image;

[0091] The first-class representation graph and the second-class representation graph are segmented at the field level according to the sample fields of each sample text box, so as to obtain the first-class representation partial graph and the second-class representation partial graph corresponding to the field to which each sample belongs;

[0092] Calculate the loss function corresponding to the field to which each sample belongs according to the first-class characterization local graph and the second-class characterization local graph corresponding to the field to which each sample belongs;

[0093] Train the field detection model according to the loss function;

[0094] The image features are input into the trained field detection model, so that the trained field detection model detects the fields to which each text box belongs, and obtains the field detection result.

[0095] In an exemplary embodiment of the present application, the field detection unit detects the fields to which each text box belongs based on the image features, and obtains the field detection result, including:

[0096] Obtain a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs;

[0097] Calculate the loss function corresponding to each field in the sample image according to the target field corresponding to each sample text box and the sample field of each sample text box;

[0098] The field detection model is trained according to the loss function corresponding to each field in the sample image;

[0099] The image features are input into the trained field detection model, so that the trained field detection model detects the fields to which each text box belongs, and obtains the field detection result.

[0100] In an exemplary embodiment of the present application, the field detection unit trains the field detection model according to the loss function corresponding to each field in the sample image, including:

[0101] Determine the loss function corresponding to each sample text box belonging to the same field according to the target field corresponding to each sample text box and the sample field of each sample text box;

[0102] The loss functions corresponding to each sample text box belonging to the same field are merged to obtain the loss functions corresponding to each field in the sample image;

[0103] Determine the weighted sum of loss functions according to the preset field weights and the loss functions corresponding to each field;

[0104] The field detection model is trained according to the weighted sum of the loss function.

[0105] In an exemplary embodiment of the present application, the field detection unit trains the field detection model according to the loss function corresponding to each field in the sample image, including:

[0106] Determine a first-class representation map and a second-class representation map corresponding to the sample image;

[0107] Determine the loss function corresponding to each field according to the first-class representation map and the second-class representation map corresponding to the sample image;

[0108] Fusion loss functions corresponding to each field to obtain the fusion result;

[0109] The field detection model is trained based on the fusion results.

[0110] In an exemplary embodiment of the present application, the above-mentioned device further includes:

[0111] a structured information generating unit, configured to generate structured information corresponding to the list image according to the field detection result after the detection result correcting unit corrects the field detection result according to the positional relationship between each text box and the adjacent text box;

[0112] The information display unit is used to display structured information through the user terminal.

[0113] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the image processing method of the first aspect of the above embodiment is implemented.

[0114] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the image processing method of the first aspect of the above embodiment.

[0115] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0116] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0117] The technical solutions provided in some embodiments of the present application specifically include: extracting features from the list image to obtain image features corresponding to the list image; identifying each text box in the list image according to the image features, and detecting the fields to which each text box belongs based on the image features to obtain field detection results; obtaining the multidimensional attributes of each text box, and determining the positional relationship between each text box and the adjacent text box according to the multidimensional attributes of each text box; and correcting the field detection results according to the positional relationship between each text box and the adjacent text box. In implementing the embodiments of the present application, on the one hand, the list image can be detected for the fields to which it belongs based on the identified text boxes in the list image, and then the positional relationship between the text boxes can be determined based on the attributes of each text box, and the detection results of the fields to which it belongs can be corrected according to the positional relationship, thereby reducing the detection error and improving the detection accuracy. On the other hand, since the field detection results can be corrected based on the positional relationship between adjacent text boxes, the error in the determination of the correspondence between the field and the field information can be avoided to a certain extent, thereby improving the determination accuracy of the correspondence between the field and the field information, and thus facilitating the improvement of the extraction accuracy of the structured information of the list image.

[0118] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0120] Figure 1 A schematic diagram schematically shows an exemplary system architecture of an image processing method and an image processing device to which the embodiments of the present application can be applied;

[0121] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown;

[0122] Figure 3 The flowchart of the image processing method according to one embodiment of the present application is schematically shown;

[0123] Figure 4 A schematic diagram of a list image according to an embodiment of the present application is schematically shown;

[0124] Figure 5 The structural diagram of a field detection model according to an embodiment of the present application is schematically shown;

[0125] Figure 6 The structure block diagram of an image processing network according to one embodiment of the present application is schematically shown;

[0126] Figure 7 A schematic diagram of a list image including a text box according to an embodiment of the present application is schematically shown;

[0127] Figure 8 A type of representation diagram according to an embodiment of the present application is schematically shown;

[0128] Fig. 9 Schematically showing two types of representation diagrams in one embodiment of the present application;

[0129] Fig.10 A schematic diagram of a text box in an embodiment of the present application is schematically shown;

[0130] Fig.11 A schematic diagram of the relationship between text box positions in an embodiment of the present application is shown schematically;

[0131] Fig.12 The flowchart of the image processing method according to one embodiment of the present application is schematically shown;

[0132] Fig.13 The structural block diagram of an image processing device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0133] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present application will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical solutions of the present application may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present application.

[0134] In addition, the accompanying drawings are only schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0135] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which an image processing method and an image processing device according to an embodiment of the present application can be applied is shown.

[0136] like Figure 1 As shown, the system architecture 100 may include one or more of user terminals 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the user terminals 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The user terminals 101, 102, 103 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1The number of user terminals, networks and servers in the embodiment is only for illustration. Any number of user terminals, networks and servers may be provided as required. For example, the server 105 may be a server cluster composed of multiple servers.

[0137] For example, the server 105 can perform feature extraction on the list image to obtain image features corresponding to the list image; identify each text box in the list image according to the image features, and detect the field to which each text box belongs based on the image features to obtain a field detection result; obtain the multi-dimensional attributes of each text box, and determine the positional relationship between each text box and adjacent text boxes according to the multi-dimensional attributes of each text box; and correct the field detection result according to the positional relationship between each text box and adjacent text boxes.

[0138] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0139] It should be noted that Figure 2 The computer system 200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0140] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 202 or the program loaded from the storage part 208 to the random access memory (RAM) 203. In the (RAM) 203, various programs and data required for system operation are also stored. The (CPU) 201, the (ROM) 202 and the (RAM) 203 are connected to each other through a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0141] The following components are connected to the (I / O) interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the (I / O) interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0142] In particular, according to an embodiment of the present application, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 209, and / or installed from a removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, various functions defined in the method and apparatus of the present application are executed.

[0143] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0144] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0146] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being installed in the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiment. For example, the electronic device may implement the following method. Figure 3 The various steps shown, etc.

[0147] See also Figure 3 , provides an image processing method, which may include the following steps S310 to S340, specifically:

[0148] Step S310: extracting features from the list image to obtain image features corresponding to the list image.

[0149] Step S320: identifying each text box in the list image according to the image features, and detecting the field to which each text box belongs based on the image features to obtain a field detection result.

[0150] Step S330: Acquire the multi-dimensional attributes of each text box, and determine the positional relationship between each text box and adjacent text boxes according to the multi-dimensional attributes of each text box.

[0151] Step S340: Correct the field detection result according to the positional relationship between each text box and adjacent text boxes.

[0152] It can be seen that implementation Figure 3 The method shown can detect the fields of the list image based on the identified text boxes in the list image, and then determine the positional relationship between the text boxes based on the attributes of each text box, and correct the detection results of the fields according to the positional relationship, thereby reducing the detection error and improving the detection accuracy. In addition, since the field detection results can be corrected based on the positional relationship between adjacent text boxes, it is possible to avoid errors in the determination of the correspondence between the field and the field information to a certain extent, thereby improving the determination accuracy of the correspondence between the field and the field information, and further facilitating the improvement of the extraction accuracy of the structured information of the list image.

[0153] Next, the above steps of this exemplary embodiment are described in more detail.

[0154] In step S310, feature extraction is performed on the list image to obtain image features corresponding to the list image.

[0155] Specifically, the list image may be an image including fields and field information. For example, the list image may be as follows: Figure 4 As shown. In the specific application process, the list image can be a bill image, a prescription image, an insurance policy image, etc., which is not limited in the embodiments of the present disclosure. For example, when an insurance company accepts a claim application, the insured is generally required to provide claim materials, which can be one or more bill images. The prior art generally requires manual upload of the claim information contained in the bill image to the system, while the present application can realize automatic processing of the bill image, thereby improving the degree of automation, reducing labor costs, and improving image processing efficiency.

[0156] Among them, feature extraction is performed on the list image to obtain image features corresponding to the list image, including: performing multi-layer convolution processing on the list image, and performing feature upsampling on the convolution results of each layer to obtain multiple upsampling results; and fusing the multiple upsampling results to obtain image features corresponding to the list image.

[0157] Specifically, multi-layer convolution processing is performed on the list image, and features are upsampled on the convolution results of each layer to obtain multiple upsampling results. Specifically, the method can be: based on the feature extraction model, multi-layer convolution processing is performed on the list image to obtain convolution results of each layer, and the convolution results of each layer are reduced in dimension and input into the upsampling layer for upsampling processing to obtain multiple upsampling results; wherein the output of each upsampling layer is the fusion result of the dimension-upgraded output of the previous upsampling layer and the output of the corresponding convolution layer.

[0158] Based on this, the method of fusing multiple upsampling results to obtain image features corresponding to the list image can be specifically: based on the feature extraction model, convolution processing is performed on multiple upsampling results respectively to obtain the features to be fused corresponding to each upsampling result, and all the features to be fused are fused to obtain the image features corresponding to the list image.

[0159] For the above feature extraction model, see Figure 5 .like Figure 5 The feature extraction model shown can include feature pyramid networks (FPN) and residual neural networks (ResNet), that is, the structure of the feature extraction model can be expressed as ResNet-FPN; wherein, ResNet can improve the degradation phenomenon (Degradation) in the deep learning model training process through shortcut connections (Shortcut connection), and eliminate the problem of difficulty in training neural networks caused by excessive depth; the FPN structure includes three parts: bottom-up, top-down and lateral connections. The FPN structure is used to fuse the features of each level so that it has strong semantic information and strong spatial information at the same time; ResNet-FPN includes 3 connection methods, bottom-up connection, top-down connection and lateral connection.

[0160] like Figure 5 As shown, the list image can be input into the feature extraction model, and the feature extraction model can input the list image into the convolution layer 511. The convolution layer 511 can perform preliminary feature extraction on the list image based on a step size of 2, and the feature extraction result can be represented as a first reference image. The feature extraction model can input the first reference image into the convolution layer 512 after enlarging it by 0.5 times based on a bilinear interpolation algorithm.

[0161] The convolution layer 512 can perform feature extraction on the 0.5* first reference image based on a step size of 4 to obtain a second reference image. The feature extraction model can be input into the convolution layer 513 after enlarging the second reference image by 0.5 times based on a bilinear interpolation algorithm.

[0162] The convolution layer 513 can perform feature extraction on the 0.5* second reference image based on a step size of 8 to obtain a third reference image. The feature extraction model can input the third reference image into the convolution layer 514 after amplifying the third reference image by 0.5 times based on a bilinear interpolation algorithm.

[0163] The convolution layer 514 can perform feature extraction on the 0.5* third reference image based on a step size of 16 to obtain a fourth reference image, and the feature extraction model can input the fourth reference image into the convolution layer 515 after amplifying the fourth reference image by 0.5 times based on a bilinear interpolation algorithm. The convolution layer 515 can perform feature extraction on the 0.5* fourth reference image based on a step size of 32 to obtain a fourth reference image.

[0164] Furthermore, the feature extraction model can perform convolution processing on the fourth reference image based on an N*N convolution kernel to reduce the number of channels of the fourth reference image to a preset number of channels (e.g., 256), obtain a fourth reference image V1 with a preset number of channels, and upsample the fourth reference image V1 to obtain feature M521.

[0165] Similarly, the feature extraction model can perform convolution processing on the third reference image based on an N*N convolution kernel to reduce the number of channels of the third reference image to a preset number of channels (eg, 256) to obtain a third reference image V1 with a preset number of channels.

[0166] Furthermore, the feature extraction model can magnify the feature M521 by 2 times based on a bilinear interpolation algorithm and fuse it with the third reference image V1, and upsample the fusion result to obtain the feature M522.

[0167] Furthermore, the feature extraction model can amplify the feature M522 by 2 times based on the bilinear interpolation algorithm, and perform convolution processing on the second reference image based on the N*N convolution kernel to reduce the number of channels of the second reference image to a preset number of channels (e.g., 256), thereby obtaining a second reference image V1 with a preset number of channels, and fuse the second reference image V1 with the feature M522, and upsample the fusion result to obtain feature M523.

[0168] Furthermore, the feature extraction model can amplify the feature M523 by 2 times based on the bilinear interpolation algorithm, and perform convolution processing on the first reference image based on the N*N convolution kernel to reduce the number of channels of the first reference image to a preset number of channels (e.g., 256), and obtain the first reference image V1 with the preset number of channels, and fuse the first reference image V1 with the feature M523, and upsample the fusion result to obtain feature M524.

[0169] Furthermore, the feature extraction model can also perform convolution processing on feature M521, feature M522, feature M523, and feature M524 based on the N*N convolution kernel to obtain feature P531 corresponding to feature M521, feature P532 corresponding to feature M522, feature P533 corresponding to feature M523, and feature P534 corresponding to feature M524.

[0170] Furthermore, the feature extraction model can also fuse feature P531, feature P532, feature P533, and feature P534 to obtain image features corresponding to the list image; wherein, optionally, the feature extraction model can fuse feature P531, feature P532, feature P533, and feature P534 by feature splicing.

[0171] Among them, the above-mentioned N can be a positive integer. For example, N*N can be 1*1, 3*3, etc.; feature M521, feature M522, feature M523, and feature M524 can be expressed as an image / feature vector / feature matrix.

[0172] It should be noted that the convolutional layers included in the above-mentioned feature extraction model may include any number of convolutional layers / pooling layers / fully connected layers, etc. in the actual application process, and the embodiments of the present application are not limited thereto. In addition, the parameters such as the stride and convolution kernel size corresponding to the above-mentioned layers are not limited thereto.

[0173] It can be seen that the implementation of this optional embodiment can improve the feature extraction accuracy for the list image, thereby improving the information extraction accuracy for the list image.

[0174] In addition, before extracting features from the list image to obtain image features corresponding to the list image, the method may further include the following steps: receiving at least one list image related to a specific item (e.g., a car insurance claim item) input by a user. If the actual number of the at least one list image is multiple, performing feature extraction on each of the multiple list images Figure 3 The steps shown are used to achieve text detection for each list image and improve the automation level of image information extraction.

[0175] In step S320, each text box in the list image is identified according to the image features, and the field to which each text box belongs is detected based on the image features to obtain a field detection result.

[0176] Specifically, the number of text boxes contained in the list image is not limited in the embodiments of the present application. The text box in the list image can be a text box containing a field or a text box containing field information. The list image can contain multiple fields, and each field (such as amount) corresponds to one or more field information (such as 3, 32, 54, 43, etc.); wherein, the number of fields and the number of field information corresponding to each field are not limited in the embodiments of the present application. In addition, the field detection result is used to characterize the field to which each field information belongs.

[0177] As an optional embodiment, the field to which each text box belongs is detected based on image features to obtain a field detection result, including: obtaining a sample image, and inputting the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs and generates a first-category representation map and a second-category representation map corresponding to the sample image; performing field-level segmentation on the first-category representation map and the second-category representation map according to the field to which the sample of each sample text box belongs, to obtain a first-category representation local map and a second-category representation local map corresponding to the field to which each sample belongs; calculating a loss function corresponding to the field to which each sample belongs according to the first-category representation local map and the second-category representation local map corresponding to the field to which each sample belongs; training the field detection model according to the loss function; inputting the image features into the trained field detection model, so that the trained field detection model detects the field to which each text box belongs, and obtains a field detection result.

[0178] Among them, the first type of representation map is used to strengthen (emphasize) the color of the first type of pixels, and the second type of representation map is used to strengthen (emphasize) the color of the second type of pixels. The size of the first type of representation map and the second type of representation map is consistent with the sample image.

[0179] In addition, the first-class representation map and the second-class representation map are segmented at the field level according to the fields to which the samples belong to each sample text box, and the first-class representation local map and the second-class representation local map corresponding to the fields to which each sample belongs are obtained. For example, if the sample image includes the fields "name" and "amount", the field "name" corresponds to the field information "A pill" and "B pill", and the field "amount" corresponds to the field information "25" and "26". Based on this, both the first-class representation map and the second-class representation map can be segmented into two parts. In the first-class representation map, one part can include "name", "A pill" and "B pill", which is represented as the first-class representation local map corresponding to the field "name"; the other part can include "amount", "25" and "26", which is represented as the first-class representation local map corresponding to the field "field". In the second-class representation map, one part can include "name", "A pill" and "B pill", which is represented as the second-class representation local map corresponding to the field "name"; the other part can include "amount", "25" and "26", which is represented as the second-class representation local map corresponding to the field "field".

[0180] Based on this, the loss function corresponding to the field to which each sample belongs is calculated according to the first-class characterization local graph and the second-class characterization local graph corresponding to the field to which each sample belongs, including: calculating the loss function L according to the first-class characterization local graph corresponding to the field to which the sample belongs bn , and calculate the loss function L according to the two-category representation local graph corresponding to the field to which the sample belongs tn ; Wherein, the value range of n is 0 to N, n is used to represent any field in multiple fields of the sample image, and N is the number of fields in the sample image; Si The representation contains a dataset of all pixels in the local image, y i Represents the field to which the predicted pixel i belongs, x i Characterizes the standard field to which the pre-stored pixel i belongs. R d The representation contains two types of data sets representing all pixels in the local image. Then, based on the expression L n =α*L bn +β*L tn Calculate the loss function L corresponding to each field i ; where α and β represent weights, and α and β can be natural numbers. Furthermore, based on the expression Determine the loss function L s ; Expression-based The loss functions corresponding to each field are integrated to obtain the fusion result.

[0181] Based on this, the field detection model is trained according to the loss function, including: training the field detection model based on the above fusion result until the fusion result converges.

[0182] It can be seen that by implementing this optional embodiment, the model can be trained jointly through different loss functions corresponding to multiple fields, which can help improve the detection accuracy of the model.

[0183] As an optional embodiment, the field to which each text box belongs is detected based on image features to obtain a field detection result, including: obtaining a sample image, and inputting the sample image into a field detection model, so that the field detection model detects the sample field to which each sample text box in the sample image belongs; calculating the loss function corresponding to each field in the sample image according to the target field corresponding to each sample text box and the sample field to which each sample text box belongs; training the field detection model according to the loss function corresponding to each field in the sample image; inputting the image features into the trained field detection model, so that the trained field detection model detects the field to which each text box belongs, and obtains a field detection result.

[0184] Specifically, the sample image may also be a list image. The sample image is input into a field detection model so that the field detection model detects the sample fields of each sample text box in the sample image, including: the sample image is input into a feature extraction model, the feature extraction model can extract image features corresponding to the sample image, and then the image features are input into the field detection model, the field detection model can determine a first-class representation map and a second-class representation map of the sample image according to the image features, and the fields to which each text box in the sample image belongs can be predicted according to the first-class representation map and the second-class representation map; wherein the first-class representation map and the second-class representation map are used to describe the sample image through different dimensions.

[0185] In addition, the trained field detection model detects the fields of each text box and obtains the specific implementation method of the field detection result. Figure 6 The image processing network shown in FIG. 1 is used to implement Figure 3 The steps shown. Figure 6 The image processing network shown may include: a feature extraction model 610, a field detection module 620, and a field detection result correction module 630; wherein the field detection module 620 may include a prediction module 621 and a position relationship detection module 622. It should be noted that the specific hierarchical structure of the feature extraction model 610 can be found in Figure 5 , I will not go into details here.

[0186] Specifically, the list image can be input into the feature extraction model 610 so that the feature extraction model 610 can be based on Figure 5 The steps shown determine the image features corresponding to the list image, and then input the image features into the prediction module 621 and the position relationship detection module 622 in the field detection module 620. Among them, the prediction module 621 can determine each text box in the list image based on the image features and generate a first-class representation map and a second-class representation map corresponding to the list image according to the image features. The position relationship detection module 622 can obtain the multi-dimensional attributes of each text box, and determine the position relationship between each text box and the adjacent text box according to the multi-dimensional attributes of each text box. Then, the field detection module 620 can input the position relationship between each text box and the adjacent text box, the first-class representation map and the second-class representation map into the field detection result correction module 630. The field detection result correction module 630 can correct the field detection result according to the position relationship between each text box and the adjacent text box, and generate structured information based on the corrected field detection result and output the structured information.

[0187] The text boxes in the above-mentioned determined list image can refer to Figure 7 , a type of representation diagram can be found in Figure 8 , the second type of representation diagram can be found in Fig. 9 .

[0188] It can be seen that by implementing this optional embodiment, the field detection model can be first trained based on the sample image, and then the trained field detection model can be applied to field detection, which can improve the accuracy of field detection.

[0189] As an optional embodiment, the field detection model is trained according to the loss function corresponding to each field in the sample image, including: determining the loss function corresponding to each sample text box belonging to the same field according to the target field corresponding to each sample text box and the sample field of each sample text box; fusing the loss function corresponding to each sample text box belonging to the same field to obtain the loss function corresponding to each field in the sample image; determining the weighted sum of the loss functions according to the preset field weights and the loss functions corresponding to each field; and training the field detection model according to the weighted sum of the loss functions.

[0190] Specifically, the loss function corresponding to each sample text box belonging to the same field is determined according to the target field corresponding to each sample text box and the sample field of each sample text box, including: determining the target field corresponding to each sample text box, determining the sample field of each sample text box, and determining the loss function corresponding to each sample text box according to the target field (such as "quantity / unit") and the sample field (such as "project name") corresponding to each sample text box (such as S pills). Optionally, the target field can be Figure 4 The field "Project Name", the field "Quantity / Unit", the field "Amount (Yuan)" or the field "Remarks" is displayed.

[0191] In addition, it should be noted that the field "Project Name", the field "Quantity / Unit", the field "Amount (Yuan)" or the field "Remarks" can correspond to different loss functions, such as Loss1, Loss2, Loss3, and Loss4. Among them, the loss function can be a 0-1 loss function, an absolute value loss function, a log loss function, a square loss function, an exponential loss function, a Hinge loss function, a perceptual loss function, or a cross entropy loss function, etc., which is not limited in the embodiments of the present application.

[0192] Based on this, the loss functions corresponding to each sample text box belonging to the same field are fused to obtain the loss functions corresponding to each field in the sample image, including: the loss functions Loss1 i1, Loss2 i2, Loss3 i3, ..., Loss4 i corresponding to each sample text box belonging to the same field n Fusion is performed separately to obtain the loss function corresponding to each field in the sample image Among them, i1, i2, i3, ..., i n They are used to represent the field information text boxes under different fields, i1∈[1,N1], i2∈[1,N2], i3∈[1,N3], i n ∈[1,N4], where N1, N2, N3, …, N4 are all natural numbers, and N1, N2, N3, …, N4 are used to represent the number of field information text boxes corresponding to each field.

[0193] Based on this, the weighted sum of the loss function is determined according to the preset field weights and the loss functions corresponding to each field, including: The corresponding preset field weights are a1, a2, a3, ..., an and Determine the weighted sum of loss functions, weighted sum of loss functions Among them, a1, a2, a3, ..., an are all natural numbers.

[0194] Based on this, the field detection model is trained according to the weighted sum of the loss function, including: adjusting the model parameters in the field detection model according to the weighted sum of the loss function until the weighted sum of the loss function converges.

[0195] It can be seen that by implementing this optional embodiment, the model can be trained jointly through different loss functions corresponding to multiple fields, which can help improve the detection accuracy of the model.

[0196] As an optional embodiment, the fields to which each text box belongs are detected based on image features to obtain field detection results, including: generating a first-category representation diagram and a second-category representation diagram corresponding to the list image according to the image features; wherein the first-category representation diagram and the second-category representation diagram are used to highlight and display each text box from different dimensions; and detecting the fields to which each text box belongs based on the first-category representation diagram and the second-category representation diagram to obtain field detection results.

[0197] Specifically, the first type of representation graph and the second type of representation graph can be images of the same size (e.g., 1080*1080) as the list image. Figure 4 The list image shown in FIG. 1 , a type of representation graph corresponding to the list image can be as follows Figure 8 As shown, the two-category representation map corresponding to the list image can be Fig. 9 As shown in the figure, the first type of representation map can be used to represent the probability that the pixel point x belongs to the text box. The first type of feature map can also be understood as a probability map, and the pixel point x can be any pixel point in the list image. The second type of representation map can be used to represent the probability that the pixel point y is the edge of the text box. The second type of feature map can also be understood as a threshold map. The first type of representation map can be used to distinguish which areas in the list image are text box areas, and the second type of representation map can be used to distinguish which frame lines in the list image are frame lines representing the text box.

[0198] It can be seen that by implementing this optional embodiment, it is possible to generate representation graphs in various representation forms, and the accuracy of field detection results can be improved based on the representation graphs in various representation forms.

[0199] In step S330, the multi-dimensional attributes of each text box are obtained, and the positional relationship between each text box and adjacent text boxes is determined according to the multi-dimensional attributes of each text box.

[0200] Specifically, the multi-dimensional attributes of a text box are used to describe the text box from different dimensions, and the positional relationship can be used to describe the relationship between adjacent text boxes in the list image.

[0201] As an optional embodiment, the multidimensional attributes include text features, position features and image features, and the multidimensional attributes of each text box are obtained, including: extracting features of the text content of each text box to obtain the text features of each text box; extracting features of the position information of each text box to obtain the position features of each text box; wherein the position information is used to characterize the position of the corresponding text box in the list image; extracting features of the local image corresponding to each text box to obtain the image features of each text box; wherein the local image corresponding to each text box belongs to the same list image.

[0202] Specifically, see Fig.10 , Fig.10 Schematically shows Figure 4 , specifically including the following text boxes: “ABC tablets” 1010, “7.00” 1020, “dose” 1030, “5.60” 1040, “S pills” 1050, “42.00” 1060, “dose” 1070, and “21.17” 1080.

[0203] For the above text boxes, we can first determine the text contents corresponding to "ABC tablets" 1010, "7.00" 1020, "dose" 1030, "5.60" 1040, "S pills" 1050, "42.00" 1060, "dose" 1070, and "21.17" 1080, that is, ABC tablets, 7.00, dose, 5.60, S pills, 42.00, dose, 21.17. Then, we extract the text features corresponding to ABC tablets, 7.00, dose, 5.60, S pills, 42.00, dose, and 21.17. Based on this, feature extraction is performed on the text content of each text box, and the specific implementation method of obtaining the text features of each text box can be: determine the text content of each text box, and extract the text features of each text content.

[0204] Furthermore, the position information corresponding to "ABC tablet" 1010, "7.00" 1020, "dose" 1030, "5.60" 1040, "S pill" 1050, "42.00" 1060, "dose" 1070, and "21.17" 1080 can also be determined (e.g., [90.024, 132.15, 429.00, 439.56], [125.06, 167.30, 460.70, 471.26], etc.); wherein, the position information can be used to characterize the absolute position / relative position of the text box. Then, the position features corresponding to each position information are extracted.

[0205] Furthermore, image features corresponding to “ABC tablet” 1010, “7.00” 1020, “dose” 1030, “5.60” 1040, “S pill” 1050, “42.00” 1060, “dose” 1070, and “21.17” 1080 may be extracted respectively.

[0206] It should be noted that the above-mentioned text features, position features and image features may correspond to the same dimension or may correspond to different dimensions, which is not limited in the embodiments of the present application. In addition, the text features, position features and image features may all be represented as feature vectors / feature matrices.

[0207] It can be seen that by implementing this optional embodiment, text features, position features and image features about the text box can be obtained. A multi-dimensional description of a text box can be achieved through the text features, position features and image features. Position relationship determination based on text features, position features and image features is more accurate, which is conducive to improving the correction accuracy of field detection results.

[0208] As an optional embodiment, the positional relationship between each text box and adjacent text boxes is determined based on the multi-dimensional attributes of each text box, including: fusing the text features, position features and image features of each text box to obtain the fused features of each text box; and determining the positional relationship between each text box and adjacent text boxes based on the fused features of each text box.

[0209] Specifically, the text features, position features and image features of each text box are fused to obtain the fused features of each text box, including: each text box corresponds to a group of features (i.e., text features, position features and image features), and the features within the group of each text box are fused to obtain the fused features of each text box. The fusion method for fusing the features within the group of each text box can be element addition or element splicing, which is not limited in the embodiments of the present application.

[0210] Based on this, the positional relationship between each text box and the adjacent text box is determined according to the fusion features of each text box, including: determining whether the adjacent text boxes are in a vertical relationship or a non-vertical relationship according to the fusion features of each text box; or, determining whether the adjacent text boxes are in a horizontal relationship or a non-horizontal relationship according to the fusion features of each text box.

[0211] Please refer to Fig.11 ,like Fig.11 As shown, it schematically includes text box 1111, text box 1112, text box 1113, text box 1114, text box 1115, text box 1116, text box 1117, and text box 1118.

[0212] Among them, the relationship between text box 1111 and text box 1112, between text box 1112 and text box 1113, between text box 1113 and text box 1114, between text box 1115 and text box 1116, between text box 1116 and text box 1117, and between text box 1117 and text box 1118 is a horizontal relationship (i.e., non-vertical relationship), which are represented by horizontal connecting lines 1121, 1122, 1123, 1127, 1126, and 1125 respectively.

[0213] There is a vertical relationship (i.e., non-horizontal relationship) between text box 1111 and text box 1115, between text box 1112 and text box 1116, between text box 1113 and text box 1117, and between text box 1114 and text box 1118, which are represented by vertical connecting lines 1128, 1129, 1130, and 1124, respectively.

[0214] It can be seen that the implementation of this optional embodiment can determine a more accurate positional relationship based on text features, positional features, and image features, which is helpful in improving the correction accuracy of the field detection results based on the positional relationship.

[0215] In step S340, the field detection result is corrected according to the positional relationship between each text box and adjacent text boxes.

[0216] As an optional embodiment, the field detection results are corrected according to the positional relationship between each text box and the adjacent text boxes, including: determining at least one text box to be corrected in the field detection results according to the field arrangement method and the positional relationship between each text box and the adjacent text boxes; wherein the at least one text box to be corrected includes a missed detection text box and / or an erroneous detection text box; and correcting the field detection results according to the multi-dimensional attributes of a target text box related to the at least one text box to be corrected.

[0217] Specifically, the multi-dimensional attributes can describe the target text box in multiple dimensions. A missed text box can be understood as only detecting the text box but not determining the field to which the text box belongs, and a wrongly detected text box can be understood as incorrectly determining the field to which the text box belongs.

[0218] It can be seen that by implementing this optional embodiment, missed detection text boxes and / or wrongly detected text boxes in the field detection results can be corrected, thereby reducing erroneous field determinations in the field detection results, which is beneficial to improving the extraction accuracy of structured information related to the list image.

[0219] As an optional embodiment, correcting field detection results based on multi-dimensional attributes of a target text box related to at least one text box to be corrected includes: determining a field arrangement of a list image; wherein the field arrangement includes a horizontal arrangement or a vertical arrangement; determining target text boxes respectively related to at least one text box to be corrected from the positional relationship between each text box and an adjacent text box according to the field arrangement; determining position information and / or belonging fields of a corresponding text box to be corrected according to the position information and / or belonging fields of the target text box; and correcting field detection results according to the position information and / or belonging fields of at least one text box to be corrected.

[0220] Among them, optionally, if the field arrangement is horizontal arrangement, the target text box is the text box adjacent to the corresponding text box to be modified in the vertical direction; if the field arrangement is vertical arrangement, the target text box is the text box adjacent to the corresponding text box to be modified in the horizontal direction.

[0221] It can be seen that the implementation of this optional embodiment can enable the present application to be applied to field arrangements in horizontal and vertical arrangements, thereby increasing the application scope of the present application and improving usability.

[0222] As an optional embodiment, after correcting the field detection result according to the positional relationship between each text box and the adjacent text boxes, the method further includes: generating structured information corresponding to the list image according to the field detection result; and displaying the structured information through the user terminal.

[0223] Specifically, generating structured information corresponding to the list image according to the field detection result includes: storing the field detection result in key-value format, and generating the key-value format data into structured information. Based on this, displaying the structured information through the user terminal includes: sending the structured information to the user terminal, so that the user terminal displays the structured information in a user interface.

[0224] It can be seen that by implementing this optional embodiment, the field detection results can be displayed as structured information, so that the readability of the field detection results can be improved, which is convenient for users to read and easy to store.

[0225] As an alternative implementation, see Fig.12 , Fig.12 and Figure 3 The steps and embodiments shown correspond to those shown in FIG. Fig.12 As shown, the image processing method may include: steps S1200 to S1224.

[0226] Step S1200: Acquire a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs.

[0227] Step S1202: Calculate the loss function corresponding to each field in the sample image according to the target field corresponding to each sample text box and the sample field of each sample text box.

[0228] Step S1204: Determine the loss function corresponding to each sample text box belonging to the same field according to the target field corresponding to each sample text box and the sample field of each sample text box.

[0229] Step S1206: The loss functions corresponding to each sample text box belonging to the same field are merged to obtain the loss functions corresponding to each field in the sample image.

[0230] Step S1208: Determine the weighted sum of loss functions according to the preset field weights and the loss functions corresponding to each field, and train the field detection model according to the weighted sum of loss functions.

[0231] Step S1210: performing multi-layer convolution processing on the list image, and performing feature upsampling on the convolution results of each layer to obtain multiple upsampling results.

[0232] Step S1212: multiple up-sampling results are merged to obtain image features corresponding to the list image, and each text box in the list image is identified according to the image features.

[0233] Step S1214: input the image features into the trained field detection model, so that the trained field detection model generates a first-category representation diagram and a second-category representation diagram corresponding to the list image according to the image features, and detects the fields to which each text box belongs according to the first-category representation diagram and the second-category representation diagram to obtain a field detection result; wherein the first-category representation diagram and the second-category representation diagram are used to highlight and display each text box from different dimensions.

[0234] Step S1216: extract features from the text content of each text box to obtain text features of each text box; extract features from the position information of each text box to obtain position features of each text box, wherein the position information is used to characterize the position of the corresponding text box in the list image; extract features from the local image corresponding to each text box to obtain image features of each text box, wherein the local images corresponding to each text box belong to the same list image; perform feature fusion of text features, position features and image features on each text box to obtain fusion features of each text box, and determine the positional relationship between each text box and adjacent text boxes based on the fusion features of each text box.

[0235] Step S1218: Determine at least one text box to be corrected in the field detection result according to the field arrangement mode and the positional relationship between each text box and adjacent text boxes; wherein the at least one text box to be corrected includes a missed detection text box and / or an erroneous detection text box.

[0236] Step S1220: Determine the field arrangement mode of the list image, wherein the field arrangement mode includes a horizontal arrangement mode or a vertical arrangement mode, and then determine the target text boxes respectively related to at least one text box to be corrected from the positional relationship between each text box and the adjacent text box according to the field arrangement mode; if the field arrangement mode is a horizontal arrangement, then the target text box is a text box adjacent to the corresponding text box to be corrected in the vertical direction; if the field arrangement mode is a vertical arrangement, then the target text box is a text box adjacent to the corresponding text box to be corrected in the horizontal direction.

[0237] Step S1222: Determine the position information and / or the field to which the corresponding text box to be corrected is based on the position information and / or the field to which the target text box belongs, and correct the field detection result based on the position information and / or the field to which at least one text box to be corrected belongs.

[0238] Step S1224: Generate structured information corresponding to the list image according to the corrected field detection result, and display the structured information through the user terminal.

[0239] It should be noted that steps S1200 to S1224 are Figure 3 The steps and their embodiments shown correspond to each other. For the specific implementation of steps S1200 to S1224, please refer to Figure 3 The steps and embodiments shown are not described in detail here.

[0240] It can be seen that implementation Fig.12 The method shown can detect the fields of the list image based on the identified text boxes in the list image, and then determine the positional relationship between the text boxes based on the attributes of each text box, and correct the detection results of the fields according to the positional relationship, thereby reducing the detection error and improving the detection accuracy. In addition, since the field detection results can be corrected based on the positional relationship between adjacent text boxes, it is possible to avoid errors in the determination of the correspondence between the field and the field information to a certain extent, thereby improving the determination accuracy of the correspondence between the field and the field information, and further facilitating the improvement of the extraction accuracy of the structured information of the list image.

[0241] Furthermore, in this exemplary embodiment, an image processing device is also provided, wherein the execution steps of each unit in the device are the same as Figure 3 Correspondingly, reference Fig.13 As shown, the image processing device 1300 may include:

[0242] The feature extraction unit 1301 is used to extract features from the list image to obtain image features corresponding to the list image;

[0243] The field detection unit 1302 is used to identify each text box in the list image according to the image features, and detect the field to which each text box belongs based on the image features to obtain a field detection result;

[0244] A position relationship determination unit 1303 is used to obtain the multi-dimensional attributes of each text box, and determine the position relationship between each text box and the adjacent text boxes according to the multi-dimensional attributes of each text box;

[0245] The detection result correction unit 1304 is used to correct the field detection result according to the positional relationship between each text box and adjacent text boxes.

[0246] It can be seen that implementation Fig.13 The device shown can detect the fields of the list image based on the identified text boxes in the list image, and then determine the positional relationship between the text boxes based on the attributes of each text box, and correct the detection result of the fields according to the positional relationship, thereby reducing the detection error and improving the detection accuracy. In addition, since the field detection result can be corrected based on the positional relationship between adjacent text boxes, it is possible to avoid errors in the determination of the correspondence between the field and the field information to a certain extent, thereby improving the determination accuracy of the correspondence between the field and the field information, which is conducive to improving the extraction accuracy of the structured information of the list image.

[0247] In an exemplary embodiment of the present application, the feature extraction unit 1301 performs feature extraction on the list image to obtain image features corresponding to the list image, including:

[0248] Perform multi-layer convolution processing on the list image, and perform feature upsampling on the convolution results of each layer to obtain multiple upsampling results;

[0249] Multiple upsampling results are fused to obtain the image features corresponding to the list image.

[0250] It can be seen that the implementation of this optional embodiment can improve the feature extraction accuracy for the list image, thereby improving the information extraction accuracy for the list image.

[0251] In an exemplary embodiment of the present application, the field detection unit 1302 detects the fields to which each text box belongs based on the image features, and obtains field detection results, including:

[0252] Generate a first-class representation diagram and a second-class representation diagram corresponding to the list image according to the image features; wherein the first-class representation diagram and the second-class representation diagram are used to highlight and display each text box from different dimensions;

[0253] The fields to which each text box belongs are detected according to the first representation graph and the second representation graph to obtain a field detection result.

[0254] It can be seen that by implementing this optional embodiment, it is possible to generate representation graphs in various representation forms, and the accuracy of field detection results can be improved based on the representation graphs in various representation forms.

[0255] In an exemplary embodiment of the present application, the multi-dimensional attributes include text features, position features, and image features. The position relationship determination unit 1303 obtains the multi-dimensional attributes of each text box, including:

[0256] Extract features of the text content of each text box to obtain text features of each text box;

[0257] Extracting features from the position information of each text box to obtain position features of each text box; wherein the position information is used to represent the position of the corresponding text box in the list image;

[0258] Feature extraction is performed on the local image corresponding to each text box to obtain the image feature of each text box; wherein the local image corresponding to each text box belongs to the list image.

[0259] It can be seen that by implementing this optional embodiment, text features, position features and image features about the text box can be obtained. A multi-dimensional description of a text box can be achieved through the text features, position features and image features. Position relationship determination based on text features, position features and image features is more accurate, which is conducive to improving the correction accuracy of field detection results.

[0260] In an exemplary embodiment of the present application, the position relationship determining unit 1303 determines the position relationship between each text box and the adjacent text boxes according to the multi-dimensional attribute of each text box, including:

[0261] Perform feature fusion of text features, position features and image features on each text box to obtain fusion features of each text box;

[0262] The positional relationship between each text box and adjacent text boxes is determined according to the fusion features of each text box.

[0263] It can be seen that the implementation of this optional embodiment can determine a more accurate positional relationship based on text features, positional features, and image features, which is helpful in improving the correction accuracy of the field detection results based on the positional relationship.

[0264] In an exemplary embodiment of the present application, the detection result correction unit 1304 corrects the field detection result according to the positional relationship between each text box and the adjacent text box, including:

[0265] Determine at least one text box to be corrected in the field detection result according to the field arrangement mode and the positional relationship between each text box and an adjacent text box; wherein the at least one text box to be corrected includes a missed detection text box and / or an erroneously detected text box;

[0266] The field detection result is corrected according to the multi-dimensional attributes of the target text box related to at least one text box to be corrected.

[0267] It can be seen that by implementing this optional embodiment, missed detection text boxes and / or wrongly detected text boxes in the field detection results can be corrected, thereby reducing erroneous field determinations in the field detection results, which is beneficial to improving the extraction accuracy of structured information related to the list image.

[0268] In an exemplary embodiment of the present application, the detection result correction unit 1304 corrects the field detection result according to the multi-dimensional attribute of the target text box related to at least one text box to be corrected, including:

[0269] Determine the field arrangement of the list image; wherein the field arrangement includes a horizontal arrangement or a vertical arrangement;

[0270] Determine target text boxes respectively related to at least one text box to be corrected from the positional relationship between each text box and an adjacent text box according to the field arrangement mode;

[0271] Determine the position information and / or the field to which the corresponding text box to be modified belongs according to the position information and / or the field to which the target text box belongs;

[0272] The field detection result is corrected according to the position information of at least one text box to be corrected and / or the corresponding field.

[0273] in:

[0274] If the field arrangement is horizontal, the target text box is the text box that is adjacent to the corresponding text box to be modified in the vertical direction;

[0275] If the field arrangement is vertical arrangement, the target text box is a text box that is adjacent to the corresponding text box to be modified in the horizontal direction.

[0276] It can be seen that the implementation of this optional embodiment can enable the present application to be applied to field arrangements in horizontal and vertical arrangements, thereby increasing the application scope of the present application and improving usability.

[0277] In an exemplary embodiment of the present application, the field detection unit 1302 detects the fields to which each text box belongs based on the image features, and obtains field detection results, including:

[0278] Obtain a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs and generates a first-class representation map and a second-class representation map corresponding to the sample image;

[0279] The first-class representation graph and the second-class representation graph are segmented at the field level according to the sample fields of each sample text box, so as to obtain the first-class representation partial graph and the second-class representation partial graph corresponding to the field to which each sample belongs;

[0280] Calculate the loss function corresponding to the field to which each sample belongs according to the first-class characterization local graph and the second-class characterization local graph corresponding to the field to which each sample belongs;

[0281] Train the field detection model according to the loss function;

[0282] The image features are input into the trained field detection model, so that the trained field detection model detects the fields to which each text box belongs, and obtains the field detection result.

[0283] It can be seen that by implementing this optional embodiment, the field detection model can be first trained based on the sample image, and then the trained field detection model can be applied to field detection, which can improve the accuracy of field detection.

[0284] In an exemplary embodiment of the present application, the field detection unit 1302 detects the fields to which each text box belongs based on the image features, and obtains field detection results, including:

[0285] Obtain a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs;

[0286] Calculate the loss function corresponding to each field in the sample image according to the target field corresponding to each sample text box and the sample field of each sample text box;

[0287] The field detection model is trained according to the loss function corresponding to each field in the sample image;

[0288] The image features are input into the trained field detection model, so that the trained field detection model detects the fields to which each text box belongs, and obtains the field detection result.

[0289] It can be seen that by implementing this optional embodiment, the field detection model can be first trained based on the sample image, and then the trained field detection model can be applied to field detection, which can improve the accuracy of field detection.

[0290] In an exemplary embodiment of the present application, the field detection unit 1302 trains the field detection model according to the loss function corresponding to each field in the sample image, including:

[0291] Determine the loss function corresponding to each sample text box belonging to the same field according to the target field corresponding to each sample text box and the sample field of each sample text box;

[0292] The loss functions corresponding to each sample text box belonging to the same field are merged to obtain the loss functions corresponding to each field in the sample image;

[0293] Determine the weighted sum of loss functions according to the preset field weights and the loss functions corresponding to each field;

[0294] The field detection model is trained according to the weighted sum of the loss function.

[0295] It can be seen that by implementing this optional embodiment, the model can be trained jointly through different loss functions corresponding to multiple fields, which can help improve the detection accuracy of the model.

[0296] In an exemplary embodiment of the present application, the field detection unit 1302 trains the field detection model according to the loss function corresponding to each field in the sample image, including:

[0297] Determine a first-class representation map and a second-class representation map corresponding to the sample image;

[0298] Determine the loss function corresponding to each field according to the first-class representation map and the second-class representation map corresponding to the sample image;

[0299] Fusion loss functions corresponding to each field to obtain the fusion result;

[0300] The field detection model is trained based on the fusion results.

[0301] It can be seen that by implementing this optional embodiment, the model can be trained jointly through different loss functions corresponding to multiple fields, which can help improve the detection accuracy of the model.

[0302] In an exemplary embodiment of the present application, the above-mentioned device further includes:

[0303] a structured information generating unit (not shown) configured to generate structured information corresponding to the list image according to the field detection result after the detection result correcting unit 1304 corrects the field detection result according to the positional relationship between each text box and the adjacent text box;

[0304] The information display unit (not shown) is used to display structured information through the user terminal.

[0305] It can be seen that by implementing this optional embodiment, the field detection results can be displayed as structured information, so that the readability of the field detection results can be improved, which is convenient for users to read and easy to store.

[0306] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0307] Since the various functional modules of the image processing device of the example embodiment of the present application correspond to the steps of the example embodiment of the above-mentioned image processing method, for details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the above-mentioned image processing method of the present application.

[0308] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0309] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Performing feature extraction on the list image to obtain image features corresponding to the list image; identifying each text box in the list image according to the image features, and detecting the fields to which each text box belongs based on the image features to obtain a field detection result; Acquire the multi-dimensional attributes of each text box, and determine the positional relationship between each text box and adjacent text boxes according to the multi-dimensional attributes of each text box; Correcting the field detection result according to the positional relationship between each text box and an adjacent text box includes: Determine at least one text box to be corrected in the field detection result according to the field arrangement mode and the positional relationship between each text box and an adjacent text box; wherein the at least one text box to be corrected includes a missed detection text box and / or an erroneously detected text box; the missed detection text box is a text box that is only detected but the field to which the text box belongs is not determined, and the erroneously detected text box is a text box to which the field to which the text box belongs is determined to be erroneous; Modifying the field detection result according to the multi-dimensional attribute of the target text box related to at least one text box to be modified; The step of correcting the field detection result according to the multi-dimensional attribute of the target text box related to at least one text box to be corrected includes: Determining a field arrangement mode of the list image; wherein the field arrangement mode includes a horizontal arrangement mode or a vertical arrangement mode; Determining target text boxes respectively related to the at least one text box to be corrected from the positional relationship between each text box and an adjacent text box according to the field arrangement mode; Determine the position information and / or the field to which the corresponding text box to be modified belongs according to the position information and / or the field to which the target text box belongs; The field detection result is modified according to the position information of the at least one text box to be modified and / or the field to which it belongs.

2. The method according to claim 1, characterized in that Performing feature extraction on the list image to obtain image features corresponding to the list image includes: Performing multi-layer convolution processing on the list image, and performing feature upsampling on the convolution results of each layer to obtain multiple upsampling results; The multiple upsampling results are fused to obtain image features corresponding to the list image.

3. The method according to claim 1, characterized in that: Detecting the fields to which the text boxes belong based on the image features, and obtaining field detection results, including: Generating a first-category representation diagram and a second-category representation diagram corresponding to the list image according to the image features; wherein the first-category representation diagram and the second-category representation diagram are used to highlight and display each text box from different dimensions; The fields to which the text boxes belong are detected according to the first type of representation graph and the second type of representation graph to obtain a field detection result.

4. The method according to claim 1, characterized in that: The multi-dimensional attributes include text features, position features and image features. The multi-dimensional attributes of each text box are obtained, including: Performing feature extraction on the text content of each text box to obtain text features of each text box; Performing feature extraction on the position information of each text box to obtain the position feature of each text box; wherein the position information is used to represent the position of the corresponding text box in the list image; Feature extraction is performed on the local images corresponding to the text boxes to obtain image features of the text boxes; wherein the local images corresponding to the text boxes belong to the list image.

5. The method according to claim 4, characterized in that Determining the positional relationship between each text box and adjacent text boxes according to the multi-dimensional attributes of each text box includes: Performing feature fusion of text features, position features and image features on each of the text boxes to obtain fusion features of each of the text boxes; The positional relationship between each text box and an adjacent text box is determined according to the fusion feature of each text box.

6. The method according to claim 1, characterized in that in: If the field arrangement is horizontal arrangement, the target text box is a text box that is adjacent to the corresponding text box to be modified in the vertical direction; If the field arrangement is vertical arrangement, the target text box is a text box that is adjacent to the corresponding text box to be corrected in the horizontal direction.

7. The method according to claim 1, characterized in that Detecting the fields to which the text boxes belong based on the image features, and obtaining field detection results, including: Acquire a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs and generates a first-class representation graph and a second-class representation graph corresponding to the sample image; The first-category representation graph and the second-category representation graph are segmented at the field level according to the sample fields of each sample text box, so as to obtain the first-category representation partial graph and the second-category representation partial graph corresponding to the field to which each sample belongs; Calculate the loss function corresponding to the field to which each sample belongs according to the first-class characterization local graph and the second-class characterization local graph corresponding to the field to which each sample belongs; Training the field detection model according to the loss function; The image features are input into a trained field detection model, so that the trained field detection model detects the fields to which the text boxes belong, and obtains field detection results.

8. The method according to claim 1, characterized in that Detecting the fields to which the text boxes belong based on the image features, and obtaining field detection results, including: Acquire a sample image, and input the sample image into a field detection model, so that the field detection model detects the field to which the sample of each sample text box in the sample image belongs; Calculate the loss function corresponding to each field in the sample image according to the target field corresponding to each sample text box and the sample field of each sample text box; Training the field detection model according to the loss function corresponding to each field in the sample image; The image features are input into a trained field detection model, so that the trained field detection model detects the fields to which the text boxes belong, and obtains field detection results.

9. The method according to claim 8, characterized in that The field detection model is trained according to the loss function corresponding to each field in the sample image, including: Determine the loss function corresponding to each sample text box belonging to the same field according to the target field corresponding to each sample text box and the sample field of each sample text box; The loss functions corresponding to each sample text box belonging to the same field are merged to obtain the loss functions corresponding to each field in the sample image; Determine a weighted sum of loss functions according to preset field weights and loss functions corresponding to the fields; The field detection model is trained according to the weighted sum of the loss function.

10. The method according to claim 1, characterized in that After correcting the field detection result according to the positional relationship between each text box and an adjacent text box, the method further includes: generating structured information corresponding to the list image according to the field detection result; The structured information is displayed through a user terminal.

11. An image processing device, characterized in that: include: A feature extraction unit, used to extract features from the list image to obtain image features corresponding to the list image; A field detection unit, configured to identify each text box in the list image according to the image features, and detect the fields to which each text box belongs based on the image features to obtain a field detection result; A position relationship determination unit, used for acquiring the multi-dimensional attribute of each text box, and determining the position relationship between each text box and adjacent text boxes according to the multi-dimensional attribute of each text box; A detection result correction unit, used for correcting the field detection result according to the positional relationship between each text box and an adjacent text box, comprising: determining at least one text box to be corrected in the field detection result according to the field arrangement mode and the positional relationship between each text box and an adjacent text box; wherein the at least one text box to be corrected includes a missed detection text box and / or an erroneously detected text box; correcting the field detection result according to the multi-dimensional attributes of a target text box related to the at least one text box to be corrected; wherein the missed detection text box is a text box that is only detected but the field to which the text box belongs is not determined, and the erroneously detected text box is a text box to which the field to which the text box belongs is determined to be erroneous; The method of correcting the field detection result according to the multi-dimensional attributes of the target text box related to at least one text box to be corrected includes: determining the field arrangement of the list image; wherein the field arrangement includes a horizontal arrangement or a vertical arrangement; determining the target text boxes respectively related to the at least one text box to be corrected from the positional relationship between each text box and the adjacent text box according to the field arrangement; determining the position information and / or the belonging field of the corresponding text box to be corrected according to the position information and / or the belonging field of the target text box; and correcting the field detection result according to the position information and / or the belonging field of the at least one text box to be corrected.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

13. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the image processing method according to any one of claims 1 to 10.

Images