Image processing method and device

Abstract

The invention relates to an image processing method and device, and the method comprises the steps: carrying out the visual detection processing of a to-be-recognized image, and obtaining a plurality of candidate information detail items; performing text entity recognition processing on the to-be-recognized image to obtain a plurality of first text units; respectively extracting unit characteristics of each first text unit, and determining an association relationship between every two first text units according to the unit characteristics; and according to the plurality of candidate information detail items and the association relationship between every two first text units, performing aggregation processing on the plurality of first text units to obtain an information detail identification result of the to-be-identified image. Therefore, the accuracy and the reliability of identifying the information details in the image can be obviously improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method and apparatus. Background Technology

[0002] In the field of electronic ticket processing, accurately identifying and extracting detailed information from electronic tickets is a crucial task. Currently, methods for identifying ticket details mainly rely on visual feature analysis techniques. Specifically, convolutional neural networks are used to extract deep features from ticket images, and then the extracted visual features are used to locate and identify the regions containing individual detailed text.

[0003] However, this visual feature-based ticket detail recognition method has significant limitations. Firstly, since the input data only contains image information of the ticket, it struggles to accurately determine whether multiple lines of detail belong to the same item or should be divided into several independent items when faced with complex scenarios. Secondly, visual feature-based methods are susceptible to various visual interferences in the image, such as background noise, lighting variations, and font style differences. These visual interferences can lead to false detections or missed detections when locating and recognizing ticket detail text regions, severely impacting the accurate extraction of ticket detail information. Summary of the Invention

[0004] This application provides an image processing method and apparatus to solve the technical problems of insufficient recognition accuracy and susceptibility to visual interference in the processing of complex ticket images by existing visual feature-based ticket detail recognition methods.

[0005] In a first aspect, this application provides an image processing method, the method comprising:

[0006] Visual detection processing is performed on the image to be recognized to obtain multiple candidate information details; and text entity recognition processing is performed on the image to be recognized to obtain multiple first text units.

[0007] Extract the unit features of each first text unit, and determine the association relationship between every two first text units based on the unit features;

[0008] Based on the multiple candidate information details and the correlation between each pair of first text units, the multiple first text units are aggregated to obtain the information details recognition result of the image to be recognized.

[0009] In one possible implementation, the image to be identified undergoes visual detection processing to obtain multiple candidate information details, including:

[0010] The image to be identified is subjected to target detection processing to obtain multiple detailed bounding boxes;

[0011] Optical character recognition processing is performed on each of the detailed bounding boxes to obtain the text content within each detailed bounding box;

[0012] The text content within each of the detailed bounding boxes is treated as a candidate information detail item.

[0013] In one possible implementation, the text entity recognition processing of the image to be recognized to obtain a plurality of first text units includes:

[0014] The image to be identified is subjected to optical character recognition processing to obtain the text content in the image to be identified;

[0015] The text content in the image to be identified is subjected to semantic unit segmentation to obtain multiple second text units;

[0016] Named entity recognition is performed on each of the second text units, and the second text units whose entity type belongs to a specific information detail field are selected as the first text units.

[0017] In one possible implementation, the extraction of unit features for each of the first text units includes:

[0018] For each of the first text units, the following processing is performed:

[0019] The semantic feature vector of the first text unit is extracted using a text encoder;

[0020] A location encoding vector is generated based on the location information of the first text unit in the image to be recognized;

[0021] The semantic feature vector and the positional encoding vector are fused to obtain the unit features of the first text unit.

[0022] In one possible implementation, determining the association between every two first text units based on the unit features includes:

[0023] The first text units are combined in pairs to obtain text unit combinations;

[0024] The unit features of each of the two first text units in the text unit combination are input into the entity relationship connection prediction model to obtain the association relationship between the two first text units in the text unit combination.

[0025] In one possible implementation, the entity relationship connection prediction model is trained in the following manner:

[0026] Construct a training sample set, wherein each training sample contains the unit features of two training text units and the associated labels of the annotations;

[0027] The initial model is trained under supervision using the training sample set to obtain the entity relationship connection prediction model.

[0028] In one possible implementation, the step of aggregating the multiple first text units based on the multiple candidate information details and the correlation between every two first text units to obtain the information detail recognition result of the image to be recognized includes:

[0029] For two adjacent candidate information detail items, if it is determined that the two candidate information detail items meet the merging conditions, the first text units in the two candidate information detail items are aggregated to form a set of text units corresponding to the same information detail item;

[0030] The merging condition is that there is a correlation between the first text unit in one candidate information detail item that exceeds a set quantity threshold and the first text unit in another candidate information detail item.

[0031] In one possible implementation, the method further includes:

[0032] Candidate information items that do not meet the merging conditions with any other candidate information item are identified as supplementary items and incorporated into the specific information detail recognition result of the image to be recognized.

[0033] In one possible implementation, the method further includes:

[0034] Based on the aforementioned association, the first text units not covered by any of the candidate information details are aggregated, and the aggregated text unit set is incorporated as a supplementary detail item into the specific information detail recognition result of the image to be recognized.

[0035] In one possible implementation, the aggregation process for first text units not covered by any of the candidate information detail items according to the association relationship includes:

[0036] For the first text unit that is not covered by any of the aforementioned candidate information details, the following processing is performed:

[0037] The first text units that meet the set aggregation conditions are aggregated to form a set of text units corresponding to the same specific information detail item; wherein, the set aggregation conditions are: there is a first text unit as a central node, and the remaining first text units are all associated with the central node.

[0038] Secondly, this application provides an image processing apparatus, the apparatus comprising:

[0039] The first detection module is used to perform visual detection processing on the image to be recognized and obtain multiple candidate information details.

[0040] The second detection module is used to perform text entity recognition processing on the image to be recognized to obtain multiple first text units;

[0041] The feature extraction module is used to extract the unit features of each of the first text units respectively;

[0042] The association relationship determination module is used to determine the association relationship between every two first text units based on the unit characteristics;

[0043] The aggregation module is used to aggregate multiple first text units based on the multiple candidate information details and the correlation between every two first text units to obtain the information details recognition result of the image to be recognized.

[0044] In one possible implementation, the first detection module includes:

[0045] The bounding box detection unit is used to perform target detection processing on the image to be identified to obtain multiple detailed bounding boxes;

[0046] The first text recognition unit is used to perform optical character recognition processing on each of the detailed bounding boxes to obtain the text content within each of the detailed bounding boxes;

[0047] The candidate item determination unit is used to treat the text content within each of the detailed bounding boxes as a candidate information detail item.

[0048] In one possible implementation, the second detection module includes:

[0049] The second text recognition unit is used to perform optical character recognition processing on the image to be recognized to obtain the text content in the image to be recognized.

[0050] A semantic segmentation unit is used to perform semantic unit segmentation processing on the text content in the image to be identified, thereby obtaining multiple second text units;

[0051] The named entity recognition unit is used to perform named entity recognition on each of the second text units and filter out the second text units whose entity type belongs to a specific information detail field as the first text unit.

[0052] In one possible implementation, the feature extraction module is specifically used for:

[0053] For each of the first text units, the following processing is performed:

[0054] The semantic feature vector of the first text unit is extracted using a text encoder;

[0055] A location encoding vector is generated based on the location information of the first text unit in the image to be recognized;

[0056] The semantic feature vector and the positional encoding vector are fused to obtain the unit features of the first text unit.

[0057] In one possible implementation, the association determination module is specifically used for:

[0058] The first text units are combined in pairs to obtain text unit combinations;

[0059] The unit features of each of the two first text units in the text unit combination are input into the entity relationship connection prediction model to obtain the association relationship between the two first text units in the text unit combination.

[0060] In one possible implementation, the device further includes:

[0061] The model training module is used to train the entity relationship connection prediction model in the following ways:

[0062] Construct a training sample set, wherein each training sample contains the unit features of two training text units and the associated labels of the annotations;

[0063] The initial model is trained under supervision using the training sample set to obtain the entity relationship connection prediction model.

[0064] In one possible implementation, the aggregation module is specifically used for:

[0065] For two adjacent candidate information detail items, if it is determined that the two candidate information detail items meet the merging conditions, the first text units in the two candidate information detail items are aggregated to form a set of text units corresponding to the same information detail item;

[0066] The merging condition is that there is a correlation between the first text unit in one candidate information detail item that exceeds a set quantity threshold and the first text unit in another candidate information detail item.

[0067] In one possible implementation, the device further includes:

[0068] The first supplementary module is used to determine candidate information details that do not meet the merging conditions with any other candidate information details as supplementary details and incorporate them into the specific information details recognition result of the image to be recognized.

[0069] In one possible implementation, the device further includes:

[0070] The second supplementary module is used to aggregate the first text units that are not covered by any of the candidate information details items according to the association relationship, and to incorporate the aggregated text unit set into the specific information details recognition result of the image to be recognized as a supplementary detail item.

[0071] In one possible implementation, the second supplementary module is specifically used for:

[0072] For the first text unit that is not covered by any of the aforementioned candidate information details, the following processing is performed:

[0073] The first text units that meet the set aggregation conditions are aggregated to form a set of text units corresponding to the same specific information detail item; wherein, the set aggregation conditions are: there is a first text unit as a central node, and the remaining first text units are all associated with the central node.

[0074] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute an image processing program stored in the memory to implement the image processing method described in any one of the first aspects.

[0075] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the image processing method described in any one aspect.

[0076] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application first processes the image to be recognized using two technical paths: visual detection processing to obtain multiple candidate information details, and text entity recognition processing to obtain multiple first text units. Based on this, the unit features of each first text unit are extracted, and the correlation between every two first text units is determined according to the unit features. Finally, based on the multiple candidate information details and the correlation between every two first text units, the multiple first text units are aggregated to obtain the information detail recognition result of the image to be recognized. This technical solution breaks through the limitations of traditional single information recognition methods. By integrating visual detection, text entity recognition, and correlation analysis techniques, it deeply analyzes the text unit features and correlations in the image, achieving accurate parsing and efficient recognition of complex image information, significantly improving the accuracy and reliability of information detail recognition. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0078] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0079] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0080] Figure 1 A flowchart illustrating an embodiment of an image processing method provided in this application;

[0081] Figure 2 Here is an example of an image to be identified;

[0082] Figure 3 To Figure 2 A schematic diagram showing the detailed recognition results of visual detection of the image to be recognized;

[0083] Figure 4 This is a schematic diagram illustrating the determination and aggregation of relationships among multiple first text units;

[0084] Figure 5A flowchart illustrating an embodiment of another image processing method provided in this application;

[0085] Figure 6 A block diagram illustrating an embodiment of an image processing apparatus provided in this application;

[0086] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0087] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0088] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0089] To address the technical problems of insufficient recognition accuracy and susceptibility to visual interference in the processing of complex ticket images by existing visual feature-based ticket detail recognition methods, this application provides an image processing method and apparatus that can significantly improve the accuracy and reliability of information detail recognition in images.

[0090] Figure 1 This is a flowchart illustrating an embodiment of an image processing method provided in this application. Figure 1 As shown, the method includes the following steps:

[0091] Step 101: Perform visual detection processing on the image to be recognized to obtain multiple candidate information details.

[0092] The image to be identified refers to an image containing detailed information, such as document images (covering various forms of documents such as invoices, receipts, and transaction vouchers), report images, etc. These images contain multiple detailed information items, each containing information from multiple fields. The information from these fields combines to form a complete and meaningful piece of information. See also... Figure 2 Here is an example of an image to be identified. Figure 2The image shows an invoice containing six different information details, specifically invoice details. Each invoice detail typically includes information such as product name, quantity, unit price, and amount.

[0093] Visual inspection processing of an image to be recognized refers to using visual inspection technology to locate the detail list area in the image and further identify each information detail item in the detail list area.

[0094] In one embodiment, an exemplary implementation of performing visual detection processing on the image to be recognized to obtain multiple candidate information details includes: performing object detection processing on the image to be recognized to obtain multiple detailed bounding boxes; performing optical character recognition processing on each detailed bounding box to obtain the text content within each detailed bounding box; and treating the text content within each detailed bounding box as a candidate information detail.

[0095] Specifically, in the visual detection processing stage of the image to be recognized, preprocessing operations are first performed on the image, including but not limited to grayscale conversion, binarization, and denoising, to enhance the quality of the image and reduce the impact of interference factors on subsequent detection. After the above preprocessing operations are completed, object detection processing technology is used to analyze the image to be recognized. Object detection technology can accurately locate regions with specific features in the image, and multiple detailed bounding boxes can be obtained through this technology. These bounding boxes define the regions that may contain detailed information items. Subsequently, optical character recognition processing is performed on each detailed bounding box. Optical character recognition processing uses algorithms and models to convert the image text within the bounding box into text content, thereby obtaining the specific text content within each detailed bounding box.

[0096] Furthermore, visual inspection technology has certain limitations when facing complex scenarios, specifically in the error of incorrectly dividing multiple lines of content for the same item into multiple independent items. For example, see... Figure 3 In order to Figure 2 The diagram shows the results of visual inspection of the image to be identified, yielding detailed identification results. From... Figure 3 As can be seen, the two lines of content that originally belonged to the first detail item were incorrectly divided into two independent detail items. Therefore, based on the potential errors in the aforementioned visual detection technology, optical character recognition (OCR) will be performed on each detail bounding box separately to obtain the text content within each detail bounding box as candidate information detail items. This means that these information detail items are not the final results and require further processing and verification to ensure the accuracy and reliability of information detail recognition.

[0097] Step 102: Perform text entity recognition processing on the image to be recognized to obtain multiple first text units.

[0098] The core task of this application embodiment is to accurately identify the various information details contained in the image. Therefore, the identification process in step 102 focuses on identifying the first text unit belonging to the information details list area from the image to be identified. Other text content in the image to be identified, such as buyer information, password area, seller information, and remarks in the ticket image, is not within the scope of this identification. Furthermore, the information details may have diverse presentation formats, for example... Figure 2 The first ticket proof detail item in the image spans two lines of text. In such cases of multi-line display, recognition errors are prone to occur during the visual detection stage, specifically misidentifying a complete ticket proof detail item as multiple independent details. To avoid such errors, in step 102, the first text unit is constrained to consist of a single line of text. This constraint ensures that each identified first text unit accurately corresponds to an independent information detail item, improving the accuracy of information recognition.

[0099] In summary, step 102 involves performing text entity recognition on the image to be recognized, aiming to identify the first text unit, which consists of a single line of text and belongs to the information details item. In other words, the first text unit defined in step 102 refers to the text content consisting of a single line of text and belonging to the ticket details list area. Typically, there are multiple first text units. For example, Figure 2 The phrases “*meat and meat products* fresh pork tenderloin 280g (XX),” “box,” “1,” “17.9,” and “duty-free” are each a first text unit.

[0100] Text entity recognition refers to a technique based on natural language processing to accurately identify text content with specific semantics and entity types. Therefore, the embodiments of this application do not rely entirely on visual detection technology, but rather combine it with text entity recognition technology to more comprehensively and accurately obtain the required information from the image to be recognized, effectively avoiding the limitations that may exist with a single technology and improving the accuracy and reliability of information recognition.

[0101] As for the specific process of text entity recognition processing of the image to obtain multiple first text units, it will be explained below. Figure 5 The illustrated embodiments are explained in detail here, but will not be described in detail here.

[0102] One additional point to note is that in practical applications, there is no strict requirement for the order of execution between steps 101 and 102. These two steps can be carried out sequentially or in parallel. This application embodiment does not restrict the execution order of these two steps.

[0103] Step 103: Extract the unit features of each first text unit, and determine the relationship between each pair of first text units based on the unit features.

[0104] Step 103 aims to conduct an in-depth analysis of the first text unit identified in step 102, and to provide richer and more accurate information for subsequent detailed information identification by extracting the features of each first text unit (referred to as unit features for ease of description).

[0105] In one embodiment, the unit features extracted in step 103 include two aspects: text encoding features and image location encoding features. Text encoding features characterize the semantic information of the first text unit, specifically referring to converting the text content of the first text unit into a computer-processable numerical form. This process typically involves natural language processing techniques, such as word embedding and character encoding. By extracting text encoding features, the semantic information in the first text unit can be captured, including the meaning of words and grammatical structure, which is crucial for accurately understanding the content of the invoice details. For example, when identifying key information such as product name, specifications, quantity, and unit price in an invoice, text encoding features can provide strong semantic support. Image location encoding features refer to the positional information of the first text unit in the image to be identified. This feature typically involves image processing techniques, such as coordinate positioning and region division. By extracting image location encoding features, the specific position of the first text unit in the image to be identified can be determined, thereby inferring the association between the first text unit and the information details. For example, in an invoice image, the product name is usually located in the upper left corner or a specific area, while information such as quantity and unit price may be located below or to the right. In other words, image location coding features help to accurately determine the information details to which the first text unit belongs.

[0106] Therefore, unlike traditional visual detection methods, the embodiments of this application not only focus on the position of the first text unit in the image, but also on the semantic information of its text content. This is because in complex scenarios with multiple lines of detail (e.g., Figure 2 In the case of the first item on a ticket (a detailed item), it is often difficult to accurately determine whether it belongs to a single item or multiple items based solely on image location information. Traditional visual inspection methods, due to their over-reliance on image location information, are easily affected by visual interference, leading to recognition errors.

[0107] In contrast, the embodiments of this application achieve a more comprehensive and in-depth understanding of the first text unit by comprehensively considering text encoding features and image position encoding features. This comprehensive feature extraction method helps to accurately determine the attribution relationship of each line of details and avoids recognition errors caused by visual interference.

[0108] In one embodiment, an exemplary implementation of extracting the unit features of each first text unit includes: performing the following processing for each first text unit: extracting the semantic feature vector of the first text unit by a text encoder; generating a position encoding vector based on the position information of the first text unit in the image to be recognized; and fusing the semantic feature vector and the position encoding vector to obtain the unit features of the first text unit.

[0109] The semantic feature vector is essentially the numerical representation of the text encoding features, accurately reflecting the semantics and grammatical structure of the words in the first text unit. The positional encoding vector is the numerical representation of the image positional encoding features, clearly indicating the specific location of the first text unit in the image. Finally, the extracted semantic feature vector and positional encoding vector are fused. This fusion organically combines semantic and positional information, resulting in a unit feature of the first text unit that contains rich semantic connotations and clearly defines its location in the image. This provides a more comprehensive and accurate description of the first text unit, helping to accurately determine the attribution of each line of details and effectively avoiding recognition errors caused by visual interference.

[0110] As an optional implementation, an exemplary approach to fusing the extracted semantic feature vector and positional encoding vector includes directly concatenating them. Specifically, assuming the semantic feature vector has dimension n and the positional encoding vector has dimension m, the concatenated vector will have dimension n+m. For example, if the semantic feature vector is represented as S = (s1, s2, ..., sn) and the positional encoding vector is represented as P = (p1, p2, ..., pm), then the concatenated unit feature vector will be F = (s1, s2, ..., sn, p1, p2, ..., pm). This fusion method is simple and direct, preserving the original features of both semantic and positional information.

[0111] As another optional implementation, an exemplary approach to fusing the extracted semantic feature vector and positional encoding vector includes assigning a weight coefficient to each of the semantic feature vector and the positional encoding vector, and then summing them in a weighted manner to obtain the unit feature. For example, let the weight of the semantic feature vector be α, the weight of the positional encoding vector be β, and α + β = 1. Then the unit feature vector F = α × S + β × P, where S is the semantic feature vector and P is the positional encoding vector. The weight coefficients can be adjusted according to the specific task and data characteristics, for example, through experiments or based on prior knowledge. This approach allows for flexible adjustment of the proportions of semantic and positional information in the unit features.

[0112] The above implementation is merely an exemplary method for fusing semantic feature vectors and positional encoding vectors. In practical applications, other fusion methods can also be used, such as attention-based fusion. Specifically, the semantic feature vector and positional encoding vector can be used as inputs to the attention mechanism, and attention weights can be obtained by calculating their similarity. Then, the semantic feature vector and positional encoding vector are weighted and summed according to the attention weights. This application does not limit this approach.

[0113] Furthermore, in step 103, the association relationship between any two first text units is determined based on the unit characteristics of the first text units. This association relationship includes two cases: existence of an association relationship and non-existence of an association relationship. An existence of an association relationship means that the two first text units are highly likely to belong to the same information detail item, while the non-existence of an association relationship means that the two first text units are highly likely not to belong to the same information detail item. Belonging to the same information detail item means that these first text units jointly describe or define a specific information point. For example, see... Figure 4 For example, the first text unit “*Meat and Meat Products*Fresh Pork Tenderloin 280g (X Ma)” and the first text unit “Daily Fresh)” belong to the same information detail item.

[0114] In one embodiment, the specific implementation of determining the association relationship between every two first text units based on the unit features of the first text units includes: combining multiple first text units in pairs to obtain multiple text unit combinations; and performing the following processing for each text unit combination: inputting the unit features of each of the two first text units in the text unit combination into the entity relationship connection prediction model to obtain the association relationship between the two first text units in the text unit combination.

[0115] The aforementioned entity relationship prediction model is a carefully designed deep learning model. Optionally, the model architecture includes: an input layer, which receives the unit features of two first text units as input; a feature fusion layer, which fuses the two input unit features to capture the potential relationship between them. This layer can employ various methods such as concatenation, summation, and attention mechanisms; a linear layer, as the core component of the entity relationship prediction model, which performs a linear transformation on the fused features and outputs a relationship score. This score represents the probability that there is a relationship between the two text units; and an output layer, which maps the output of the linear layer to the [0,1] interval through an activation function (such as the sigmoid function) to obtain the final relationship prediction result.

[0116] As an optional implementation, the entity relationship prediction model is trained as follows: a training sample set is constructed, wherein each training sample contains the unit features of two training text units and the labeled relationship tags; the initial model is trained under supervision using the training sample set to obtain the entity relationship prediction model.

[0117] The training samples are drawn from the information detail list region of images containing information detail items, and can be manually selected. When labeling the training samples in the training sample set with association relationships, the following rule is followed: for each text unit within the same information detail item, there is an association relationship between it and the top-leftmost text unit within that item; however, there is no association relationship between any two other text units. Based on this, the entity relationship connection prediction model can accurately determine the association relationship between every two first text units.

[0118] Supervised training refers to optimizing model parameters during the training process by minimizing the difference between the predicted results and the true labels (such as cross-entropy loss) to obtain the final trained entity relationship connection prediction model.

[0119] Furthermore, the model can be evaluated periodically during training to monitor its performance. Based on the evaluation results, the model can be tuned, such as adjusting the learning rate, adding regularization terms, or changing the model architecture. This application does not impose any limitations on these aspects.

[0120] For example, see Figure 4 The first text units include A1, B1, C1, D1, E1, etc. These first text units are paired to obtain text unit combinations such as A1B1, A1C1, A1D1, A1E1, B1C1, etc. (not all are listed here). Then, the unit features of each pair of first text units in these combinations are input into the entity relationship prediction model to obtain the association relationship between the two first text units in each combination. For example, if the association prediction result for text unit combinations A1B1, A1C1, and A1D1 is true, it indicates that there is an association relationship between the two first text units in these combinations; while the association prediction result for text unit combinations A1E1 and B1C1 is false, it indicates that there is no association relationship between the two first text units in these combinations (not all are listed here).

[0121] Step 104: Based on multiple candidate information details and the correlation between every two first text units, aggregate the multiple first text units to obtain the information details recognition result of the image to be recognized.

[0122] As mentioned earlier, information details may be displayed across multiple lines, for example... Figure 2 The first item in the document spans two lines of text. In this situation, relying solely on visual detection technology, while it can obtain some information from the image, makes it difficult to accurately determine whether the cross-line content belongs to the same item. Although text entity recognition technology successfully distinguishes individual first text units, it cannot directly distinguish which of these first text units belong to the same information item.

[0123] Therefore, in step 104, these first text units are further aggregated to group text units belonging to the same information detail item together, forming a complete information detail item. This aggregation process determines the set of text units belonging to the same information detail item, thereby obtaining the information detail recognition result of the image to be recognized. Each set of text units corresponds to a complete information detail item, containing all relevant information for that item, such as product name, specifications, quantity, unit price, and amount.

[0124] Specifically, in step 104, two technical approaches—visual detection and text entity recognition—are combined to complete the information detail identification. Visual detection technology utilizes image processing algorithms to quickly locate the detail list region and candidate information detail items in the image, providing a basic framework for subsequent processing. Text entity recognition, on the other hand, delves into the semantic information within the first text unit, clarifying the inherent connections between different text units. The complementary advantages of these two approaches enable accurate identification of information details from multiple dimensions, effectively avoiding the limitations of a single method and significantly improving the accuracy and reliability of information detail identification.

[0125] The technical solution provided in this application first processes the image to be recognized using two different technical paths: visual detection to obtain multiple candidate information details, and text entity recognition to obtain multiple first text units. Based on these, the unit features of each first text unit are extracted, and the correlation between any two first text units is determined according to these features. Finally, based on the multiple candidate information details and the correlation between any two first text units, the multiple first text units are aggregated to obtain the information detail recognition result of the image to be recognized. This technical solution overcomes the limitations of traditional single-information recognition methods. By integrating visual detection, text entity recognition, and correlation analysis, it deeply analyzes the text unit features and correlations in the image, achieving accurate parsing and efficient recognition of complex image information, significantly improving the accuracy and reliability of information detail recognition.

[0126] See Figure 5This is a flowchart of another image processing method provided in this application. Figure 5 The process shown is in Figure 1 Based on the illustrated process, an exemplary implementation is described for performing text entity recognition processing on the image to be recognized, resulting in multiple first text units. For example... Figure 6 As shown, it includes the following steps:

[0127] Step 501: Perform optical character recognition processing on the image to be recognized to obtain the text content in the image.

[0128] In one embodiment, optical character recognition (OCR) technology is used to detect text in the image to be recognized. As one of the core technologies in image processing and pattern recognition, OCR technology possesses powerful text extraction capabilities. Through detailed pixel-level analysis of the image to be recognized, it can accurately locate text regions within the image and accurately identify and extract characters, numbers, and other information within these regions to obtain the text content of the image to be recognized.

[0129] Step 502: Perform semantic unit segmentation on the text content in the image to be recognized to obtain multiple second text units.

[0130] Step 503: Perform named entity recognition on each second text unit, and select the second text units whose entity type belongs to a specific information detail field as the first text unit.

[0131] For ease of understanding, steps 502 and 503 will be explained uniformly below:

[0132] First, it should be noted that the second text unit and the first text unit defined here are consistent in their structure, both consisting of a single line of text. This single-line text format facilitates subsequent semantic analysis, association matching, and aggregation processing of the text information, helping to improve the efficiency and accuracy of information processing. However, they differ fundamentally in their scope of information. Specifically, as mentioned above, the first text unit is clearly defined as a text unit belonging to a specific information detail item, while the second text unit is not limited to a specific information detail item during the detection process. For example, the second text unit not only involves specific ticket details but also covers information such as the ticket title, number, date, seal, purchaser, and remarks. This comprehensive text detection method provides a rich and comprehensive raw data foundation for subsequent information processing.

[0133] Specifically, as an optional implementation, semantic unit segmentation is performed on the text content in the image to be recognized, resulting in multiple second text units. This semantic unit segmentation can be accomplished using a word segmenter. Based on pre-defined rules and algorithms, the word segmenter scans and analyzes the character sequences in the text, identifies word boundaries, considers collocation relationships between words and semantic integrity, and thus accurately segments the text into independent semantic units, i.e., second text units.

[0134] After semantic unit segmentation, the feature extraction stage begins. In this stage, the second text unit and its positional information within the image to be recognized are first organized into code elements. Then, a structured document understanding backbone network (such as advanced models like LiLT (Language-independent Layout Transformer)) is used to extract the code element features. These backbone networks are based on deep learning architectures and integrate modular designs such as Transformer models (e.g., BERT (Bidirectional Encoder Representations from Transformers)), visual-language cross-modal interaction technologies (e.g., Gated CrossAttention) and geometry-sensitive positional encoding methods (e.g., RPE (Rotation-aware Positional Encoding)). By decoupling text semantics and spatial layout features, they can model code elements from multiple dimensions. They can learn lexical features of code elements (e.g., part-of-speech, word form changes), syntactic features (e.g., grammatical roles, dependency relations), and contextual semantic features (e.g., semantic role labeling, co-occurrence relations), and encode these features into high-dimensional code element feature vectors.

[0135] Subsequently, the symbol feature sequence (a sequence composed of the symbol feature vectors of multiple second text units) is input into the trained named entity recognition model. This model can employ a structure of a feature encoder followed by a linear classification layer. The feature encoder is responsible for capturing the contextual semantic features of the symbol feature sequence, while the linear classification layer directly classifies individual symbols as entity types. During model training, a large amount of text data labeled with detailed information fields is used as the training set. The model parameters are continuously optimized through backpropagation to ensure accurate recognition of entities within the specified detailed information fields, such as product name, specifications, unit, quantity, unit price, amount, invoice date, and invoice code.

[0136] After receiving the symbol feature sequence, the named entity recognition model predicts the entity type of each symbol and selects the second text unit whose entity type belongs to the set information detail field based on the prediction result, and determines it as the first text unit.

[0137] Figure 6 The process shown involves performing text detection on the image of the ticket to be identified, obtaining multiple second text units, performing named entity recognition on each second text unit, and selecting the second text units whose entity type belongs to the set information detail field as the first text unit. This achieves accurate positioning and extraction of the text content of the information detail items in the image to be identified, providing a reliable data foundation for the accurate identification of the information detail items.

[0138] In one embodiment, an exemplary implementation of aggregating multiple first text units based on multiple candidate information detail items and the correlation between every two first text units to obtain the information detail recognition result of the image to be recognized includes: for two adjacent candidate information detail items, if it is determined that the two candidate information detail items meet the merging condition, the first text units in the two candidate information detail items are aggregated to form a set of text units corresponding to the same information detail item; wherein, the merging condition is: there is a correlation between the first text units in one candidate information detail item that exceed a set number threshold and the first text units in another candidate information detail item.

[0139] Specifically, for two adjacent candidate information detail items, if the first text units exceeding a set threshold in one candidate information detail item (e.g., the next candidate information detail item in two adjacent candidate information detail items) are related to the first text units in the other candidate information detail item (e.g., the previous candidate information detail item in two adjacent candidate information detail items), then these two candidate information detail items can be considered to actually belong to different text lines of the same information detail item. The basis for this judgment is that, from a semantic and logical perspective, the existence of a large number of related first text units between two candidate information detail items means that the information in the two candidate information detail items is closely connected, jointly constituting a complete information detail item. Based on this, the first text units in the two candidate information detail items are aggregated to form a set of text units corresponding to the same information detail item.

[0140] In one embodiment, the aforementioned threshold quantity is set based on a ratio (e.g., 50%). That is, when the number of first text units in the next candidate information detail item that are related to the previous candidate information detail item reaches more than 50% of the total number of first text units in the next candidate information detail item, the set merging condition is met. In practical applications, those skilled in the art can flexibly set or adjust this ratio according to actual needs. It is understood that the ratio directly affects the accuracy and aggregation degree of the recognition results. A higher ratio means a stricter requirement for the correlation between the two candidate information detail items, resulting in higher merging accuracy. This is because aggregation is only performed when most of the first text units in the two candidate information detail items are related, effectively avoiding erroneous merging and thus improving the accuracy of information detail recognition.

[0141] See Figure 4 In the example, looking at it from top to bottom, the second candidate information detail item includes only one first text unit D1, and this first text unit D1 is related to the first text unit A1 in the first candidate information detail item. That is, 100% of the first text units in the second candidate information detail item are related to the first text units in the first candidate information detail item. Therefore, it can be considered that the first and second candidate information detail items actually belong to the same information detail item. Thus, the first text units in these two candidate information detail items are aggregated to form a set of text units corresponding to the same information detail item, ultimately obtaining the correct information detail identification result, such as... Figure 2 As shown.

[0142] Furthermore, in one embodiment, candidate information items that do not meet the merging conditions with any other candidate information item are identified as supplementary items and incorporated into the specific information detail recognition result of the image to be recognized.

[0143] In this embodiment, for candidate information details obtained by visual detection, if there is a candidate information detail that does not meet the merging condition with any other candidate information detail, it is directly identified as the final target information detail. This is because these candidate information details have already been identified as independent and complete information units during the visual detection stage, and no other candidate information detail has a sufficiently close semantic and logical association with them to meet the merging condition. Therefore, they can constitute a complete information detail on their own, and their content is based on the visual detection result.

[0144] Through the above embodiments, the technical solution provided in this application further enhances the completeness and accuracy of information detail recognition. When faced with complex and varied images to be recognized, it can not only effectively process text information with cross-line or related relationships, forming complete and accurate information detail items through precise aggregation; but also, for special cases where these items exist independently and do not meet the merging conditions with other candidate information detail items, it can, based on reasonable judgment logic, determine these unmerged candidate information detail items as supplementary detail items and reasonably incorporate them into the specific information detail recognition result of the image to be recognized. This ensures that the final recognition result can comprehensively and truthfully reflect all information details in the image to be recognized, significantly improving the adaptability and reliability of the information detail recognition technology provided in this application embodiment in various practical application scenarios.

[0145] In one embodiment, based on the association relationship, the first text unit that is not covered by any candidate information detail item is aggregated, and the aggregated text unit set is incorporated as a supplementary detail item into the specific information detail recognition result of the image to be recognized. Here, "not covered by any candidate information detail item" means that, during the visual detection process, the first text unit is not visually defined as belonging to the area defined by any candidate information detail item; that is, the first text unit is visually considered not to be part of the overall information represented by any candidate information detail item.

[0146] In actual ticket images, visual detection primarily relies on the visual presentation features of the image to identify candidate information details. However, factors such as image quality issues (e.g., blurriness, occlusion, noise interference) and complex layouts (e.g., irregular text arrangement, text distribution across multiple regions) can cause some first text units to fail to be accurately included in the corresponding candidate information details. The method adopted in this embodiment, by deeply analyzing the relationships between first text units and aggregating those uncovered first text units, effectively compensates for potential omissions in visual detection. This ensures that all information details in the image are accurately identified and integrated, thereby significantly improving the completeness and accuracy of information detail recognition.

[0147] An exemplary implementation of aggregating first text units not covered by any candidate information detail items based on association relationships includes: aggregating first text units that meet set aggregation conditions to form a set of text units corresponding to the same specific information detail item; wherein the set aggregation conditions are: there exists a first text unit as a central node, and the remaining first text units are all associated with the central node. Here, the first text unit as the central node may belong to a candidate information detail item or may not belong to any candidate information detail item.

[0148] The principle behind the above exemplary implementation is that, within the same information detail item, each text unit is associated with the top-left text unit in that item (this depends on the text's order; if the text is arranged from right to left, then it would be the top-right text unit), while no other two text units are associated with each other. Therefore, by finding the first text unit that is associated with the same first text unit and aggregating the found first text unit with that same first text unit, the set of text units belonging to the same information detail item can be accurately defined.

[0149] For example, see Figure 4 For example, the first text units B1, C1, and D1 are all related to the first text unit A1. That is, the first text unit A1 can be considered the central node of the first text units B1, C1, and D1. Therefore, the first text units A1, B1, C1, and D1 can be aggregated to form a set of text units corresponding to the same information detail item. The first text units aggregated in this set form... Figure 4 The first ticket proof detail in the visualization of the aggregated results shown.

[0150] Finally, as can be seen from the above two embodiments, the technical solution provided in this application, combining visual inspection, text entity recognition, and correlation analysis technologies, can significantly improve the comprehensiveness and accuracy of information detail recognition. Visual inspection, with its keen capture of image visual features, quickly locates candidate information details in the image, establishing a preliminary framework for information recognition; while text entity recognition and correlation analysis technologies delve into the text itself, uncovering the semantic and logical connections between the first text units, supplementing and improving information that visual inspection may have missed or failed to accurately integrate. This multi-technology fusion approach makes this solution more adaptable and robust when facing complex and ever-changing image information. Whether processing text information with cross-line or related relationships, or dealing with independently existing text units or those not accurately covered due to image quality issues, it can achieve comprehensive and accurate information recognition and integration based on reasonable judgment logic and precise correlation analysis. This provides a reliable and efficient method for information detail recognition in various practical application scenarios, such as ticket information processing and financial statement analysis.

[0151] Figure 6 This is a block diagram illustrating an embodiment of a ticket verification device provided in this application. Figure 6 As shown, the device includes:

[0152] The first detection module 61 is used to perform visual detection processing on the image to be recognized and obtain multiple candidate information details.

[0153] The second detection module 62 is used to perform text entity recognition processing on the image to be recognized to obtain multiple first text units;

[0154] The feature extraction module 63 is used to extract the unit features of each of the first text units respectively;

[0155] The association relationship determination module 64 is used to determine the association relationship between every two first text units based on the unit characteristics;

[0156] The aggregation module 65 is used to aggregate multiple first text units based on the multiple candidate information details and the correlation between each pair of first text units to obtain the information details recognition result of the image to be recognized.

[0157] In one possible implementation, the first detection module 61 includes:

[0158] The bounding box detection unit is used to perform target detection processing on the image to be identified to obtain multiple detailed bounding boxes;

[0159] The first text recognition unit is used to perform optical character recognition processing on each of the detailed bounding boxes to obtain the text content within each of the detailed bounding boxes;

[0160] The candidate item determination unit is used to treat the text content within each of the detailed bounding boxes as a candidate information detail item.

[0161] In one possible implementation, the second detection module 62 includes:

[0162] The second text recognition unit is used to perform optical character recognition processing on the image to be recognized to obtain the text content in the image to be recognized.

[0163] A semantic segmentation unit is used to perform semantic unit segmentation processing on the text content in the image to be identified, thereby obtaining multiple second text units;

[0164] The named entity recognition unit is used to perform named entity recognition on each of the second text units and filter out the second text units whose entity type belongs to a specific information detail field as the first text unit.

[0165] In one possible implementation, the feature extraction module 63 is specifically used for:

[0166] For each of the first text units, the following processing is performed:

[0167] The semantic feature vector of the first text unit is extracted using a text encoder;

[0168] A location encoding vector is generated based on the location information of the first text unit in the image to be recognized;

[0169] The semantic feature vector and the positional encoding vector are fused to obtain the unit features of the first text unit.

[0170] In one possible implementation, the association determination module 64 is specifically used for:

[0171] The first text units are combined in pairs to obtain text unit combinations;

[0172] The unit features of each of the two first text units in the text unit combination are input into the entity relationship connection prediction model to obtain the association relationship between the two first text units in the text unit combination.

[0173] In one possible implementation, the device further includes:

[0174] The model training module is used to train the entity relationship connection prediction model in the following ways:

[0175] Construct a training sample set, wherein each training sample contains the unit features of two training text units and the associated labels of the annotations;

[0176] The initial model is trained under supervision using the training sample set to obtain the entity relationship connection prediction model.

[0177] In one possible implementation, the aggregation module 65 is specifically used for:

[0178] For two adjacent candidate information detail items, if it is determined that the two candidate information detail items meet the merging conditions, the first text units in the two candidate information detail items are aggregated to form a set of text units corresponding to the same information detail item;

[0179] The merging condition is that there is a correlation between the first text unit in one candidate information detail item that exceeds a set quantity threshold and the first text unit in another candidate information detail item.

[0180] In one possible implementation, the device further includes:

[0181] The first supplementary module is used to determine candidate information details that do not meet the merging conditions with any other candidate information details as supplementary details and incorporate them into the specific information details recognition result of the image to be recognized.

[0182] In one possible implementation, the device further includes:

[0183] The second supplementary module is used to aggregate the first text units that are not covered by any of the candidate information details items according to the association relationship, and to incorporate the aggregated text unit set into the specific information details recognition result of the image to be recognized as a supplementary detail item.

[0184] In one possible implementation, the second supplementary module is specifically used for:

[0185] For the first text unit that is not covered by any of the aforementioned candidate information details, the following processing is performed:

[0186] The first text units that meet the set aggregation conditions are aggregated to form a set of text units corresponding to the same specific information detail item; wherein, the set aggregation conditions are: there is a first text unit as a central node, and the remaining first text units are all associated with the central node.

[0187] like Figure 7 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0188] Memory 113 is used to store computer programs;

[0189] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the image processing method provided in any of the foregoing method embodiments, including:

[0190] Visual detection processing is performed on the image to be recognized to obtain multiple candidate information details; and text entity recognition processing is performed on the image to be recognized to obtain multiple first text units.

[0191] Extract the unit features of each first text unit, and determine the association relationship between every two first text units based on the unit features;

[0192] Based on the multiple candidate information details and the correlation between each pair of first text units, the multiple first text units are aggregated to obtain the information details recognition result of the image to be recognized.

[0193] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the ticket detail identification method provided in any of the foregoing method embodiments.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0195] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0196] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0197] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An image processing method, characterized in that, The method includes: Visual detection processing is performed on the image to be recognized to obtain multiple candidate information details; and text entity recognition processing is performed on the image to be recognized to obtain multiple first text units. Extract the unit features of each first text unit, and determine the association relationship between every two first text units based on the unit features; Based on the multiple candidate information details and the correlation between each pair of first text units, the multiple first text units are aggregated to obtain the information details recognition result of the image to be recognized.

2. The method according to claim 1, characterized in that, The image to be identified undergoes visual detection processing to obtain multiple candidate information details, including: The image to be identified is subjected to target detection processing to obtain multiple detailed bounding boxes; Optical character recognition processing is performed on each of the detailed bounding boxes to obtain the text content within each detailed bounding box; The text content within each of the detailed bounding boxes is treated as a candidate information detail item.

3. The method according to claim 1, characterized in that, The text entity recognition process performed on the image to be recognized yields multiple first text units, including: The image to be identified is subjected to optical character recognition processing to obtain the text content in the image to be identified; The text content in the image to be identified is subjected to semantic unit segmentation to obtain multiple second text units; Named entity recognition is performed on each of the second text units, and the second text units whose entity type belongs to a specific information detail field are selected as the first text units.

4. The method according to claim 1, characterized in that, The step of extracting the unit features of each of the first text units includes: For each of the first text units, the following processing is performed: The semantic feature vector of the first text unit is extracted using a text encoder; A location encoding vector is generated based on the location information of the first text unit in the image to be recognized; The semantic feature vector and the positional encoding vector are fused to obtain the unit features of the first text unit.

5. The method according to claim 1, characterized in that, Determining the association relationship between every two first text units based on the unit features includes: The first text units are combined in pairs to obtain text unit combinations; The unit features of each of the two first text units in the text unit combination are input into the entity relationship connection prediction model to obtain the association relationship between the two first text units in the text unit combination.

6. The method according to claim 5, characterized in that, The entity relationship prediction model is trained in the following way: Construct a training sample set, wherein each training sample contains the unit features of two training text units and the associated labels of the annotations; The initial model is trained under supervision using the training sample set to obtain the entity relationship connection prediction model.

7. The method according to claim 1, characterized in that, The step of aggregating multiple first text units based on the multiple candidate information details and the correlation between every two first text units to obtain the information detail recognition result of the image to be recognized includes: For two adjacent candidate information detail items, if it is determined that the two candidate information detail items meet the merging conditions, the first text units in the two candidate information detail items are aggregated to form a set of text units corresponding to the same information detail item; The merging condition is that there is a correlation between the first text unit in one candidate information detail item that exceeds a set quantity threshold and the first text unit in another candidate information detail item.

8. The method according to claim 7, characterized in that, The method further includes: Candidate information items that do not meet the merging conditions with any other candidate information item are identified as supplementary items and incorporated into the specific information detail recognition result of the image to be recognized.

9. The method according to claim 7, characterized in that, The method further includes: Based on the aforementioned association, the first text units not covered by any of the candidate information details are aggregated, and the aggregated text unit set is incorporated as a supplementary detail item into the specific information detail recognition result of the image to be recognized.

10. The method according to claim 9, characterized in that, The aggregation process for the first text unit not covered by any of the candidate information detail items according to the association relationship includes: For the first text unit that is not covered by any of the aforementioned candidate information details, the following processing is performed: The first text units that meet the set aggregation conditions are aggregated to form a set of text units corresponding to the same specific information detail item; wherein, the set aggregation conditions are: there is a first text unit as a central node, and the remaining first text units are all associated with the central node.

11. An image processing apparatus, characterized in that, The device includes: The first detection module is used to perform visual detection processing on the image to be recognized and obtain multiple candidate information details. The second detection module is used to perform text entity recognition processing on the image to be recognized to obtain multiple first text units; The feature extraction module is used to extract the unit features of each of the first text units respectively; The association relationship determination module is used to determine the association relationship between every two first text units based on the unit characteristics; The aggregation module is used to aggregate multiple first text units based on the multiple candidate information details and the correlation between every two first text units to obtain the information details recognition result of the image to be recognized.

Images