Information generation method, information processing method, device, electronic device, and medium

Abstract

The present disclosure provides an information generation method, an information processing method, an apparatus, an electronic device, and a medium, which relate to the fields of artificial intelligence technology, in particular to the fields of deep learning technology, image processing technology, and computer vision technology, and can be applied to scenarios such as OCR (Optical Character Recognition). The specific implementation scheme is as follows: performing text detection on a text image to obtain detection information, the detection information including category information and position information of each of a plurality of text regions; obtaining text region images corresponding to each of the plurality of text regions based on the position information and the text image; performing text recognition on the text region images to obtain recognition information, the recognition information including text recognition information of each of the plurality of text region images; determining semantic relationship information based on the recognition information, the semantic relationship information including the semantic relationship between a plurality of text recognition information; generating structured information of the text image based on the category information, the semantic relationship information, and the recognition information.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology, particularly deep learning technology, image processing technology, and computer vision technology, and can be applied to scenarios such as optical character recognition (OCR). Specifically, it relates to an information generation method, an information processing method, an apparatus, an electronic device, and a medium. Background Art

[0002] With the development of computer technology, artificial intelligence technology has also been developed. For example, artificial intelligence technology can be used to perform entity recognition and relationship extraction on images containing text data to obtain text structured information in the image. Summary of the Invention

[0003] The present disclosure provides an information generating method, an information processing method, an apparatus, an electronic device, and a medium.

[0004] According to one aspect of the present disclosure, there is provided an information generation method, comprising: performing text detection on a text image to obtain detection information, wherein the detection information includes category information and position information of each of a plurality of text regions; acquiring a text region image corresponding to each of the plurality of text regions based on the position information and the text image; performing text recognition on the text region image to obtain recognition information, wherein the recognition information includes text recognition information of each of the plurality of text region images; determining semantic relationship information based on the recognition information, wherein the semantic relationship information includes semantic relationships between the plurality of text recognition information; and generating structured information of the text image based on the category information, the semantic relationship information and the recognition information.

[0005] According to another aspect of the present disclosure, an information processing method is provided, comprising: processing a text image to be processed according to the above-mentioned information generation method to obtain structured information of the text image to be processed; and performing information processing using the structured information of the text image to be processed.

[0006] According to another aspect of the present disclosure, an information generating device is provided, comprising: a text detection module for performing text detection on a text image to obtain detection information, wherein the detection information includes category information and position information of each of a plurality of text regions; a first acquisition module for obtaining text region images corresponding to each of the plurality of text regions based on the position information and the text image; a text recognition module for performing text recognition on the text region images to obtain recognition information, wherein the recognition information includes text recognition information of each of the plurality of text region images; a determination module for determining semantic relationship information based on the recognition information, wherein the semantic relationship information includes semantic relationships between the plurality of text recognition information; and a generation module for generating structured information of the text image based on the category information, the semantic relationship information and the recognition information.

[0007] According to another aspect of the present disclosure, an information processing device is provided, comprising: a second acquisition module for processing a text image to be processed using the above-mentioned information generation device to obtain structured information of the above-mentioned text image to be processed; and an information processing module for performing information processing using the structured information of the above-mentioned text image to be processed.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to the present disclosure when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1Schematically illustrates an exemplary system architecture to which the information generating method, information processing method, and apparatus according to an embodiment of the present disclosure may be applied;

[0014] Figure 2 The flowchart of the information generation method according to the embodiment of the present disclosure is schematically shown;

[0015] Figure 3 The flowchart of the method for performing text detection on a text image and obtaining detection information according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 4 Schematically illustrates an example of a process of performing text detection on a text image and obtaining detection information according to an embodiment of the present disclosure;

[0017] Figure 5A Schematically shows a flow chart of a method for determining semantic relationship information based on identification information according to an embodiment of the present disclosure;

[0018] Figure 5B Schematically illustrates an example of a process of determining semantic relationship information based on identification information according to an embodiment of the present disclosure;

[0019] Figure 5C Schematically illustrates an example diagram of a process of determining semantic relationship information based on recognition information according to another embodiment of the present disclosure;

[0020] Figure 5D Schematically shows a flow chart of a method for determining semantic relationship information based on identification information according to another embodiment of the present disclosure;

[0021] Figure 5E Schematically illustrates an example diagram of a process of determining semantic relationship information based on recognition information according to another embodiment of the present disclosure;

[0022] Figure 5F Schematically illustrates an example diagram of a process of determining semantic relationship information based on recognition information according to another embodiment of the present disclosure;

[0023] Figure 5G Schematically shows a flow chart of a method for determining semantic relationship information based on identification information according to another embodiment of the present disclosure;

[0024] Figure 6 A flowchart schematically illustrates a method for performing text recognition on a text region image to obtain recognition information according to an embodiment of the present disclosure;

[0025] Figure 7 Schematically illustrates a flow chart of a method for generating structured information of a text image based on category information, semantic relationship information, and recognition information according to an embodiment of the present disclosure;

[0026] Figure 8 The following schematically illustrates an example of an information generation process according to an embodiment of the present disclosure;

[0027] Figure 9 The flowchart of the information processing method according to the embodiment of the present disclosure is schematically shown;

[0028] Figure 10 Schematically shows a block diagram of an information generating device according to an embodiment of the present disclosure;

[0029] Figure 11 A block diagram schematically shows an information processing device according to an embodiment of the present disclosure; and

[0030] Figure 12 A block diagram schematically shows an electronic device suitable for implementing the information generating method and the information processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] In the technical solution disclosed herein, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0034] Figure 1 An exemplary system architecture to which the information generating method, information processing method, and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0035] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not imply that the embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the information generation method, information processing method, and apparatus may be applied may include a terminal device, but the terminal device may implement the information generation method, information processing method, and apparatus provided in the embodiments of the present disclosure without interacting with a server.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as at least one of a wired and wireless communication link. The terminal device may include at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0037] A user can use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. At least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 can be installed with various communication client applications, such as at least one of a knowledge reading application, a web browser application, a search application, an instant messaging tool, an email client, and a social platform software.

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing. For example, the electronic device can include at least one of a smartphone, a tablet computer, a laptop computer, and a desktop computer.

[0039] Server 105 can be a server that provides various services. For example, server 105 can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It solves the problems of traditional physical hosts and VPS services (Virtual Private Servers), such as difficult management and poor business scalability.

[0040] It should be noted that the information generation method and information processing method provided in the embodiments of the present disclosure can generally be executed by one of the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the information generation device and information processing device provided in the embodiments of the present disclosure can also be set in one of the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0041] Alternatively, the information generation method and information processing method provided in the embodiments of the present disclosure may also be generally executed by the server 105. Accordingly, the information generation device and information processing device provided in the embodiments of the present disclosure may generally be set in the server 105. The information generation method and information processing method provided in the embodiments of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Accordingly, the information generation device and information processing device provided in the embodiments of the present disclosure may also be set in a server or server cluster that is different from the server 105 and can communicate with at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105.

[0042] It should be understood that Figure 1 The number of the first terminal device, the second terminal device, the third terminal device, the network and the server is only illustrative. According to the implementation requirements, there can be any number of the first terminal device, the second terminal device, the third terminal device, the network and the server.

[0043] It should be noted that the sequence numbers of the operations in the following method are only used to indicate the operation for the purpose of description, and should not be regarded as indicating the order in which the operations should be performed. Unless explicitly stated, the method does not need to be performed in the order shown.

[0044] Figure 2 The flowchart of the information generating method according to the embodiment of the present disclosure is schematically shown.

[0045] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0046] In operation S210, text detection is performed on the text image to obtain detection information.

[0047] In operation S220 , text region images corresponding to respective ones of the plurality of text regions are acquired based on the position information and the text image.

[0048] In operation S230 , text recognition is performed on the text region image to obtain recognition information.

[0049] In operation S240 , semantic relationship information is determined based on the recognition information.

[0050] In operation S250 , structured information of the text image is generated based on the category information, the semantic relationship information, and the recognition information.

[0051] According to an embodiment of the present disclosure, the detection information may include category information and location information of each of the plurality of text regions, the recognition information may include text recognition information of each of the plurality of text region images, and the semantic relationship information may include semantic relationships between the plurality of text recognition information.

[0052] According to embodiments of the present disclosure, a text image may refer to an image including text content. The text content in a text image is unstructured information, and the unstructured text content in the text image can be extracted using the information generation method provided in embodiments of the present disclosure to generate structured information of the text image.

[0053] According to embodiments of the present disclosure, text images may include various types, for example, medical text images, product list text images, or financial text images. File formats for text images may include JPG (Joint Photographic Experts Group), TIFF (Tag Image File Format), PNG (Portable Network Graphics), PDF (Portable Document Format), and GIF (Graphics Interchange Format). Embodiments of the present disclosure do not limit the file format of text images.

[0054] According to an embodiment of the present disclosure, a text image may be acquired through real-time acquisition. For example, the text image may be acquired by photographing or scanning the physical text. Alternatively, the text image may be pre-stored in a database. For example, for an electronic document containing text information, the text image may be acquired by taking a screenshot of the document. Alternatively, the text image may be received from another terminal device. The embodiments of the present disclosure do not limit the method for acquiring the text image.

[0055] According to an embodiment of the present disclosure, after obtaining a text image, a text detection model can be used to perform text detection on the text image to obtain detection information corresponding to the text image. The text detection model can be obtained by training a first predetermined model using a first training sample set and a first label set. The first predetermined model may include a deep learning model or a traditional model. The deep learning model may include a text detection model based on a candidate box, a text detection model based on segmentation, or a text detection model based on a mixture of the two. The traditional model may include a text detection model based on SWT (Stroke Width Transform) or a text detection model based on EdgeBox (i.e., edge box), etc.

[0056] According to an embodiment of the present disclosure, the detection information may include category information and location information for each of the multiple text regions. The category information may represent the category of the text content included in the text region. The category information may include at least one of the following: a keyword category or a numerical category. The keyword category may represent the category attribute of the text content included in the text region. The numerical category may represent the content attribute of the text content included in the text region.

[0057] For example, if a text area contains the text content "A City Central Hospital", the category information of the text area is a numerical category. If a text area contains the text content "name", the category information of the text area is a keyword category. If a text area contains the text content "Zhang San", the category information of the text area is a numerical category.

[0058] According to embodiments of the present disclosure, location information can represent the location of a text region. The location information can be used as a basis for extracting a text region image corresponding to the text region from a text image. The location information can be represented using a text detection frame. The text detection frame can include a four-corner point frame, meaning the location information can be represented using four coordinates.

[0059] According to an embodiment of the present disclosure, after obtaining the position information of each of the multiple text regions, a predetermined image segmentation method can be used to perform image segmentation processing on the text image based on the position information to obtain text region images corresponding to each of the multiple text regions. The predetermined image segmentation method can include at least one of the following: a threshold-based image segmentation method, a region-based image segmentation method, an edge-based image segmentation method, an image segmentation method based on a specific theory, an image segmentation method based on genetic coding, an image segmentation method based on wavelet transform, and an image segmentation method based on a neural network.

[0060] According to an embodiment of the present disclosure, after obtaining text area images corresponding to each of the multiple text areas, a text recognition model can be used to perform text recognition on the text area images to obtain recognition information. The recognition information may include text recognition information of each of the multiple text area images. The text recognition information can be used to characterize the text content corresponding to a field consisting of continuous text in the text area image. The text recognition model can be obtained by training a second predetermined model using a second training sample set and a second label set. The second predetermined model may include a pattern matching model, a machine learning model, or a deep learning model. The deep learning model may include a text recognition model based on single character recognition or a text recognition model based on overall recognition.

[0061] According to an embodiment of the present disclosure, after obtaining the recognition information, a text classification model can be used to process the text recognition information of each of the multiple text region images to obtain semantic relationship information. For example, the text classification model can be used to extract the semantic features included in the text recognition information of each of the multiple text region images, and based on the semantic features, the semantic relationship between the multiple text recognition information can be determined. The text classification model can be obtained by training a third predetermined model using a third training sample set. The third predetermined model can include a machine learning model or a deep learning model. The machine learning model can include a text classification model based on a naive Bayes algorithm or a text classification model based on a decision tree.

[0062] According to an embodiment of the present disclosure, after obtaining semantic relationship information, structured information of the text image can be generated based on the category information, semantic relationship information, and identification information. The structured information of the text image can include values ​​corresponding to keyword categories and values ​​corresponding to numerical categories. The values ​​corresponding to keyword categories can include semantic relationship information. The values ​​corresponding to numerical categories can include identification information.

[0063] According to an embodiment of the present invention, operations S210 to S250 may be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be Figure 1 The server 105 in the terminal device can be Figure 1 The first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0064] According to an embodiment of the present invention, by performing text detection on a text image, category information and position information for each of the multiple text regions are obtained. Since the text region images corresponding to each of the multiple text regions are obtained based on the position information and the text image, and the recognition information is obtained by performing text recognition on the text region images, the recognition information can include text recognition information for each of the multiple text region images. Furthermore, since the structured information of the text image is generated based on the category information, semantic relationship information, and recognition information, and the semantic relationship information is determined based on the recognition information, and includes the semantic relationship between the multiple text recognition information, the semantic relationship information and visual information are utilized to generate the structured information, thereby improving the accuracy of the structured information.

[0065] This is only an exemplary embodiment, but is not limited thereto. Other information generation methods known in the art may also be included as long as the accuracy of the structured information can be improved.

[0066] Refer to the following figure Figure 3 、 Figure 4 、 Figure 5A 、 Figure 5B 、 Figure 5C 、 Figure 5D、 Figure 5E 、 Figure 5F 、 Figure 5G 、 Figure 6 、 Figure 7 and Figure 8 , combined with specific embodiments Figure 2 The method shown is further explained.

[0067] According to an embodiment of the present disclosure, the text image includes a medical text image.

[0068] According to the embodiments of the present disclosure, medical text is an important way to save information in medical scenarios. It contains a lot of structured information about users. Obtaining this structured information helps to understand the user's health status and then conduct targeted analysis and processing. At the same time, a complete database and user portrait can also be established. Medical text can exist in the form of images. How to extract the required structured information from medical text images is a technical difficulty faced in medical scenarios, which can be achieved using the information generation solution provided by the embodiments of the present disclosure.

[0069] Figure 3 The flowchart of the method for performing text detection on a text image and obtaining detection information according to an embodiment of the present disclosure is schematically shown.

[0070] like Figure 3 As shown, the method 300 is to Figure 2 As a further limitation of operation S210, the method 300 may include operations S311 to S315.

[0071] In operation S311 , feature extraction is performed on the text image to obtain a first feature map of at least one scale.

[0072] In operation S312 , a second feature map is acquired based on the first feature map of at least one scale.

[0073] In operation S313 , a third feature map is acquired based on the first feature map of at least one scale.

[0074] In operation S314 , category information of each of the plurality of text regions is obtained according to the second feature map.

[0075] In operation S315 , position information of each of the plurality of text regions is obtained according to the third feature map.

[0076] According to an embodiment of the present disclosure, the scale may refer to image resolution. Each scale may have at least one first feature map corresponding to the scale.

[0077] According to an embodiment of the present disclosure, a text image can be processed based on a single-stage serial method to obtain a first feature map of at least one scale. Alternatively, a text image can be processed based on a multi-stage serial method to obtain a first feature map of at least one scale. Alternatively, a text image can be processed based on a multi-stage parallel method to obtain a first feature map of at least one scale.

[0078] According to an embodiment of the present disclosure, after obtaining a first feature map at at least one scale, a second feature map can be obtained based on the first feature map at at least one scale. For example, the first feature map at at least one scale can be fused to obtain a first fused feature map. The second feature map can be obtained based on the first fused feature map. For example, the first fused feature map can be determined as the second feature map. Alternatively, the first fused feature map can be processed to obtain the second feature map.

[0079] According to an embodiment of the present disclosure, after obtaining a first feature map at at least one scale, a third feature map can be obtained based on the first feature map at at least one scale. For example, the first feature map at at least one scale can be fused to obtain a second fused feature map. The third feature map can be obtained based on the second fused feature map. For example, the second fused feature map can be determined as the third feature map. Alternatively, the second fused feature map can be processed to obtain the third feature map.

[0080] According to an embodiment of the present disclosure, after obtaining the second feature map, category information for each of the multiple text regions can be obtained based on the second feature map. For example, a heat map for each of the multiple text regions can be obtained based on the second feature map. Category information for each of the multiple text regions can be determined based on the heat maps for each of the multiple text regions. Alternatively, the second feature map can be processed based on a regression position method to obtain category information for each of the multiple text regions.

[0081] According to an embodiment of the present disclosure, after obtaining the third feature map, the position information of each of the multiple text regions can be obtained based on the third feature map. For example, a heat map of each of the multiple text regions can be obtained based on the third feature map. Based on the heat maps of each of the multiple text regions, the position information of each of the multiple text regions can be determined. Alternatively, the third feature map can be processed based on a regression position method to obtain the position information of each of the multiple text regions.

[0082] According to an embodiment of the present invention, operations S311 to S315 may be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be Figure 1 The server 105 in the terminal device can be Figure 1 The first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0083] According to an embodiment of the present disclosure, since the first feature map of at least one scale can provide richer information, the first feature map of at least one scale is used to obtain the category information and position information of each of the multiple text regions, thereby improving the accuracy of the category information and position information of each of the multiple text regions.

[0084] According to an embodiment of the present disclosure, operation S311 may include the following operations.

[0085] Performing M stages of feature extraction on the text image to obtain at least one first feature map corresponding to the M-th stage. Obtaining at least one scale of the first feature map based on the at least one first feature map corresponding to the M-th stage.

[0086] According to an embodiment of the present disclosure, the mth stage has T m The image resolutions of the first feature maps of the same parallel layer are the same, and the image resolutions of the first feature maps of different parallel layers are different.

[0087] According to an embodiment of the present disclosure, M is an integer greater than or equal to 1. m is an integer greater than or equal to 1 and less than or equal to M. m is an integer greater than or equal to 1.

[0088] According to an embodiment of the present disclosure, the M stage may include an input stage, an intermediate stage, and an output stage. The input stage may refer to the 1st stage. The output stage may refer to the Mth stage. The intermediate stage may refer to the 2nd stage to the M-1th stage. The number of parallel levels of each stage may be the same or different. In the 1st stage to the M-1th stage, the current stage may have at least one more parallel level than the previous stage. The Mth stage may have the same number of parallel levels as the M-1th stage. M can be configured according to actual business needs and is not limited here. For example, M=4. In the 1st stage to the 3rd stage, the current stage may have at least one more parallel level than the previous stage. The 1st stage has T1=2 parallel levels. The 2nd stage has T2=3 parallel levels. The 3rd stage has T3=4 parallel levels. The 4th stage has T4=4 parallel levels.

[0089] According to an embodiment of the present disclosure, the image resolutions of the first feature maps of the same parallel level are the same. The image resolutions of the first feature maps of different parallel levels are different. For example, the image resolution of the first feature map of the current parallel level is smaller than the image resolution of the first feature map of the previous parallel level. The image resolution of the first feature map of the current parallel level of the current stage may be determined based on the image resolution of the first feature map of the previous parallel level of the previous stage. For example, the image resolution of the first feature map of the current stage of the current stage may be obtained by downsampling the image resolution of the first feature map of the previous parallel level of the previous stage.

[0090] According to an embodiment of the present disclosure, when M>1, performing feature extraction on a text image in M stages to obtain at least one first feature map corresponding to the Mth stage may include: in response to m = 1, performing feature extraction on the text image to obtain at least one intermediate first feature map of at least one scale corresponding to the first stage. According to the at least one intermediate first feature map of at least one scale corresponding to the first stage, obtaining at least one first feature map of at least one scale corresponding to the first stage. In response to 1 < m ≤ M, performing feature extraction on the at least one first feature map of at least one scale corresponding to the (m - 1)th stage to obtain at least one intermediate first feature map of at least one scale corresponding to the mth stage. According to the at least one intermediate first feature map of at least one scale corresponding to the mth stage, obtaining at least one first feature map of at least one scale corresponding to the mth stage.

[0091] According to an embodiment of the present disclosure, when M = 1, performing feature extraction on a text image in M stages to obtain at least one first feature map corresponding to the Mth stage may include: performing feature extraction on the text image to obtain at least one intermediate first feature map of at least one scale corresponding to the first stage. According to the at least one intermediate first feature map of at least one scale corresponding to the first stage, obtaining at least one first feature map of at least one scale corresponding to the first stage.

[0092] According to an embodiment of the present disclosure, obtaining at least one first feature map of at least one scale according to at least one first feature map corresponding to the Mth stage may include: determining the at least one first feature map corresponding to the Mth stage as the at least one first feature map of at least one scale.

[0093] According to an embodiment of the present disclosure, since the image resolutions of the first feature maps at the same parallel level are the same, and the image resolutions of the first feature maps at different parallel levels are different, therefore, high-resolution feature representations can be maintained throughout the feature extraction process, and the parallel levels from high resolution to low resolution can be gradually increased. Directly extracting deep semantic information on the high-resolution feature representations instead of supplementing the low-level feature information of the image enables it to have sufficient classification ability and avoid the loss of effective spatial resolution. At least one parallel level can take into account the capture of context information and obtain rich global and local information. In addition, by repeatedly exchanging information on the parallel levels to achieve multi-scale fusion of features, more accurate position information can be obtained, thereby improving the accuracy of the category information and position information of each of the multiple text regions.

[0094] According to an embodiment of the present disclosure, when M is an integer greater than 1, performing feature extraction on a text image in M stages to obtain at least one first feature map corresponding to the Mth stage may include the following operations.

[0095] Perform convolution processing on at least one first feature map corresponding to the m-1th stage to obtain at least one intermediate first feature map corresponding to the mth stage. Perform feature fusion on at least one intermediate first feature map corresponding to the mth stage to obtain at least one first feature map corresponding to the mth stage.

[0096] According to an embodiment of the present disclosure, m is an integer greater than 1 and less than or equal to M.

[0097] According to an embodiment of the present disclosure, for the m-1th stage, for the first feature map in at least one first feature map, convolution processing can be performed on the first feature map to obtain an intermediate first feature map of the mth stage, thereby obtaining at least one intermediate first feature map of the mth stage.

[0098] According to an embodiment of the present disclosure, performing feature fusion on at least one intermediate first feature map corresponding to the m-th stage to obtain at least one first feature map corresponding to the m-th stage may include: for an intermediate first feature map in the at least one intermediate first feature map corresponding to the m-th stage, fusing the intermediate first feature map of the m-th stage with intermediate first feature maps of other parallel levels other than the parallel level where the intermediate first feature map is located in the m-th stage, to obtain a first feature map of the m-th stage corresponding to the intermediate first feature map. Other parallel levels may refer to at least some parallel levels other than the parallel level where the intermediate first feature map is located in the m-th stage.

[0099] According to an embodiment of the present disclosure, performing feature fusion on at least one intermediate first feature map corresponding to the m-th stage to obtain at least one first feature map corresponding to the m-th stage may include the following operations.

[0100] Targeting T m For the i-th parallel level in the parallel levels, a first feature map corresponding to the i-th parallel level is obtained according to other intermediate first feature maps corresponding to the i-th parallel level and the intermediate first feature map corresponding to the i-th parallel level.

[0101] According to an embodiment of the present disclosure, the other intermediate first feature maps corresponding to the i-th parallel level can be m The intermediate first characteristic graphs corresponding to at least some of the parallel levels except the i-th parallel level in the parallel levels. i can be greater than or equal to 1 and less than or equal to T m An integer.

[0102] According to an embodiment of the present disclosure, when 1<i<I, at least one first other intermediate first feature map is upsampled to obtain an upsampled first feature map corresponding to the at least one first other intermediate first feature map. At least one second other intermediate first feature map is downsampled to obtain a downsampled first feature map corresponding to the at least one second other intermediate first feature map. The first other intermediate first feature map may refer to T m The second other intermediate first feature map may refer to T m The image resolution of the upsampled first feature map is the same as the resolution of the intermediate first feature map of the i-th parallel layer. The resolution of the downsampled first feature map is the same as the resolution of the intermediate first feature map of the i-th parallel layer.

[0103] According to an embodiment of the present disclosure, when i=1, at least one second other intermediate first feature map is upsampled to obtain a downsampled first feature map corresponding to at least one first other intermediate first feature map. The first other intermediate first feature map may refer to T m The other intermediate first feature maps in the first parallel level are larger than the first parallel level. The image resolution of the upsampled first feature map is the same as the resolution of the intermediate first feature map of the first parallel level.

[0104] According to an embodiment of the present disclosure, in the case of i=I, at least one second other intermediate first feature map is downsampled to obtain a downsampled first feature map corresponding to the at least one second other intermediate first feature map. The second other intermediate first feature map may refer to T m The resolution of the downsampled first feature map is the same as the resolution of the intermediate first feature map of the i-th parallel level.

[0105] According to an embodiment of the present disclosure, a first feature map corresponding to the i-th parallel level is obtained based on an upsampled first feature map corresponding to at least one first other intermediate first feature map, a downsampled first feature map corresponding to at least one second other intermediate first feature map, and an intermediate first feature map of the i-th parallel level. For example, the upsampled first feature map corresponding to the at least one first other intermediate first feature map, the downsampled first feature map corresponding to the at least one second other intermediate first feature map, and the intermediate first feature map of the i-th parallel level can be fused to obtain the first feature map corresponding to the i-th parallel level. The fusion may include at least one of the following: splicing and addition.

[0106] Figure 4An example schematic diagram of a process of performing M stages of feature extraction on a text image to obtain at least one first feature map corresponding to the Mth stage according to an embodiment of the present disclosure is schematically shown.

[0107] like Figure 4 As shown in FIG400, M=4, for example, a first stage 401, a second stage 402, a third stage 403, and a fourth stage 404. The first stage 401 has two parallel stages, for example, a first parallel stage 405 and a second parallel stage 406. The second stage 402 has three parallel stages, for example, a first parallel stage 405, a second parallel stage 406, and a third parallel stage 407. The third stage 403 has four parallel stages, for example, a first parallel stage 405, a second parallel stage 406, a third parallel stage 407, and a fourth parallel stage 408.

[0108] The at least one first feature map corresponding to the fourth stage may include a first feature map 409, a first feature map 410, a first feature map 411, and a first feature map 412. In addition, Figure 4 The "upper right arrow" between the last two columns of each stage in represents "up sampling". The "lower left arrow" represents "down sampling".

[0109] According to an embodiment of the present disclosure, operation S311 may include the following operations.

[0110] Perform N cascade-level feature extraction on the text image to obtain a first feature map of at least one scale.

[0111] According to the embodiment of the present disclosure, N is an integer greater than 1. N can be configured according to actual business needs and is not limited here. For example, N=4.

[0112] According to an embodiment of the present disclosure, feature extraction of N cascade levels can be performed on a text image to obtain at least one first feature map corresponding to the N cascade levels. Based on the at least one first feature map corresponding to the N cascade levels, a first feature map of at least one scale is obtained. For example, for the n-th cascade level among the N cascade levels, based on the first feature maps of the other cascade levels and the first feature map corresponding to the n-th cascade level, a first feature map of the scale corresponding to the n-th cascade level is obtained. The other cascade levels may refer to at least some of the cascade levels among the N cascade levels except the n-th cascade level.

[0113] According to an embodiment of the present disclosure, since the first feature map of at least one scale can provide richer information, subsequently determining the category information and position information of each of the multiple text regions based on the first feature map of at least one scale can improve the accuracy of the category information and position information of each of the multiple text regions.

[0114] Figure 5A The flowchart of the method for determining semantic relationship information based on identification information according to an embodiment of the present disclosure is schematically shown.

[0115] like Figure 5A As shown, the method 500A is to Figure 2 As further defined in operation S240, the method 500A may include operation S541.

[0116] In operation S541 , semantic relationship information is determined based on the auxiliary information and the recognition information.

[0117] According to an embodiment of the present disclosure, the auxiliary information may include at least one of the following: a second feature map and position information.

[0118] According to an embodiment of the present disclosure, the second feature map may be obtained based on the first fused feature map. The first fused feature map may be obtained by fusing the first feature map at at least one scale. The position information may represent the location of the text region.

[0119] According to an embodiment of the present disclosure, for example, when the auxiliary information includes a second feature map, semantic relationship information can be determined based on the second feature map and identification information. Alternatively, when the auxiliary information includes position information, semantic relationship information can be determined based on the position information and identification information. Alternatively, when the auxiliary information includes a second feature map and position information, semantic relationship information can be determined based on the second feature map, position information, and identification information.

[0120] According to an embodiment of the present disclosure, since the semantic relationship information is determined based on auxiliary information and identification information, the auxiliary information includes at least one of the second feature map and position information. By utilizing the auxiliary information and identification information to determine the semantic relationship information, the accuracy of the semantic relationship information is improved.

[0121] According to an embodiment of the present disclosure, in the case where the auxiliary information includes the second feature map, operation S541 may include the following operations.

[0122] The second feature map and the fourth feature map corresponding to the recognition information are fused to obtain a fused feature map. Semantic relationship information is determined based on the fused feature map.

[0123] According to an embodiment of the present disclosure, when the auxiliary information includes a second feature map, feature extraction can be performed on the text recognition information of each of the multiple text region images to obtain a fourth feature map corresponding to the recognition information. After obtaining the fourth feature map, the second feature map and the fourth feature map corresponding to the recognition information can be fused to obtain a fused feature map. After obtaining the fused feature map, the fused feature map can be processed using a text classification model to obtain semantic relationship information.

[0124] According to an embodiment of the present disclosure, the text classification model may include a deep learning model or a machine learning model. A third predetermined model may be trained using a third training sample set and a third label set to obtain a text classification model. The third training sample set may include multiple training texts, and the third label set may include a third label corresponding to each training text.

[0125] According to an embodiment of the present disclosure, training a third predetermined model using a third training sample set and a third label set to obtain a text classification model may include: inputting each of a plurality of training texts into the third predetermined model to obtain a semantic category result corresponding to each training text; inputting the semantic category result corresponding to each training text and the third label into a first loss function to obtain a first output value; adjusting model parameters of the third predetermined model based on the first output value until the first output value converges; and determining the third predetermined model obtained when the first output value converges as the text classification model.

[0126] According to an embodiment of the present disclosure, in the case where the auxiliary information further includes location information, operation S541 may include the following operations.

[0127] According to the fused feature map and position information, the semantic relationship information is determined.

[0128] According to an embodiment of the present disclosure, when the auxiliary information also includes location information, feature extraction can be performed on the text recognition information of each of the multiple text region images to obtain a fourth feature map corresponding to the recognition information. After obtaining the fourth feature map, the second feature map and the fourth feature map corresponding to the recognition information can be fused to obtain a fused feature map. After obtaining the fused feature map, semantic relationship information can be determined based on the fused feature map and the location information.

[0129] Figure 5B The following schematically illustrates an example process of determining semantic relationship information based on identification information according to an embodiment of the present disclosure.

[0130] like Figure 5B As shown in FIG500B, when the auxiliary information 501 includes the second feature map 5011, the fourth feature map 503 corresponding to the identification information 502 can be determined according to the identification information 502.

[0131] After obtaining the fourth feature map 503 corresponding to the identification information 502 , the second feature map 5011 and the fourth feature map 503 corresponding to the identification information 502 may be fused to obtain a fused feature map 504 .

[0132] After obtaining the fused feature map 504 , semantic relationship information 505 may be determined based on the fused feature map 504 .

[0133] Figure 5C The following schematically illustrates an example process of determining semantic relationship information based on identification information according to another embodiment of the present disclosure.

[0134] like Figure 5C As shown in 500C, when the auxiliary information 506 includes the second feature map 5061 and the position information 5062 , a fourth feature map 508 corresponding to the identification information 507 can be determined according to the identification information 507 .

[0135] After obtaining the fourth feature map 508 corresponding to the identification information 507 , the second feature map 5061 and the fourth feature map 508 corresponding to the identification information 507 may be fused to obtain a fused feature map 509 .

[0136] After obtaining the fused feature map 509 , semantic relationship information 510 may be determined based on the fused feature map 509 and the position information 5062 .

[0137] Figure 5D The flowchart of a method for determining semantic relationship information based on identification information according to another embodiment of the present disclosure is schematically shown.

[0138] like Figure 5D As shown, the method 500D is to Figure 2 Further defining operation S240 in the method 500C, the method 500C may include operations S542 to S543.

[0139] In operation S542 , global feature extraction is performed on the recognition information to obtain global feature information.

[0140] In operation S543 , semantic relationship information is determined based on the global feature information.

[0141] According to an embodiment of the present disclosure, the text recognition information of each of the multiple text area images can be input into the global feature extraction model to obtain global feature information. After obtaining the global feature information, the semantic relationship between the multiple text recognition information can be determined based on the global feature information. The global feature extraction model can be obtained by training a fourth predetermined model using a fourth training sample set. The fourth predetermined model may include a Recurrent Neural Networks (RNN) model, a Long Short-Term Memory (LSTM) model, and a Transformer model. The fourth predetermined model can be configured according to actual business needs, and it only needs to be able to implement the function of global feature extraction, and is not limited here.

[0142] For example, the fourth predetermined model may include at least one model structure. The model structure may include at least one model substructure and the connection relationship between each model substructure. The model structure may be a structure obtained by connecting at least one model substructure based on the connection relationship between the model substructures. The at least one model substructure included in the model structure may be a structure from at least one operation layer. For example, the model structure may be a structure obtained by connecting at least one model substructure from at least one operation layer based on the connection relationship between the model substructures. For example, at least one operation layer may include at least one of the following: input layer, convolution layer, hidden layer, transcription layer, pooling layer, depooling layer, deconvolution layer, feedforward neural network layer, attention layer, residual layer, fully connected layer, batch normalization layer, linear embedding (i.e., LinearEmbedding) layer and nonlinear layer, etc.

[0143] According to the embodiments of the present disclosure, because global feature information is obtained by extracting global features from recognition information, it can characterize the global characteristics of the recognition information. Furthermore, because semantic relationship information is determined based on this global feature information, it can more comprehensively characterize the semantic relationships between text recognition information, thereby improving the accuracy of this semantic relationship information.

[0144] According to an embodiment of the present disclosure, operation S542 may include the following operations.

[0145] The recognition information is processed based on the attention strategy to obtain global feature information.

[0146] According to an embodiment of the present disclosure, an attention strategy can be used to achieve focusing on important information with high weight and ignoring unimportant information with low weight, and can exchange information with other information by sharing important information, thereby achieving the transmission of important information. In an embodiment of the present disclosure, the attention strategy can extract information between the text recognition information of multiple text area images to better complete the information generation of the text image. The attention strategy can include one of a self-attention strategy and a mutual attention strategy.

[0147] According to an embodiment of the present disclosure, the text recognition information can be used to determine the first key matrix, the first value matrix, and the first query matrix. For example, when the attention strategy can be a self-attention strategy, the text recognition information can be used as the first key matrix, the first value matrix, and the first query matrix. The key (i.e., Key) matrix, the value (i.e., Value), and the query (i.e., Query) matrix can be matrices in the attention mechanism.

[0148] According to an embodiment of the present disclosure, when the attention strategy can be a self-attention strategy, the text recognition information corresponding to the multiple text area images as the first key matrix, the first value matrix, and the first query matrix can be processed based on the self-attention strategy to obtain global feature information corresponding to each of the multiple text area images. For example, an attention unit can be determined based on the self-attention strategy. The text recognition information corresponding to the multiple text area images as the first key matrix, the first value matrix, and the first query matrix is ​​processed using the attention unit to obtain global feature information corresponding to each of the multiple text area images.

[0149] According to an embodiment of the present disclosure, an attention strategy is used to process text recognition information corresponding to multiple text area images to obtain global feature information. The attention strategy can extract semantic information between the text area image itself and the text area image and other text area images, thereby improving the accuracy of structured information generation of text images.

[0150] According to an embodiment of the present disclosure, processing recognition information based on an attention strategy to obtain global feature information may include the following operations.

[0151] Based on the self-attention strategy, the recognition information is processed at the U level to obtain global feature information.

[0152] According to an embodiment of the present disclosure, U is an integer greater than or equal to 1. U can be configured according to actual business needs and is not limited here. For example, U=4.

[0153] According to an embodiment of the present disclosure, for a text region image in a plurality of text region images, the text recognition information of each of the plurality of text region images can be processed based on an attention strategy to obtain global feature information. For example, the text recognition information of each of the plurality of text region images can be processed at a U-level based on a self-attention strategy to obtain global feature information.

[0154] According to an embodiment of the present disclosure, when U is an integer greater than 1 and 1<u≤U, U-level processing of the recognition information based on the self-attention strategy to obtain global feature information may include the following operations.

[0155] Based on the first intermediate feature information corresponding to each of the multiple text recognition information at the u-th level, the second intermediate feature information corresponding to each of the multiple text recognition information at the u-th level is obtained. Based on the second intermediate feature information corresponding to each of the multiple text recognition information at the u-th level and the first intermediate feature information corresponding to each of the multiple text recognition information at the u-th level, the first intermediate feature information corresponding to the multiple text recognition information at the u-th level is obtained. Based on the first intermediate feature information corresponding to each of the multiple text recognition information at the R-th level, global feature information is obtained.

[0156] According to an embodiment of the present disclosure, the first intermediate feature information may be used to determine a first query matrix, a first key matrix, and a first value matrix.

[0157] According to an embodiment of the present disclosure, u may be an integer greater than or equal to 1 and less than or equal to U. u∈{1, 2, ..., U-1, U}. R may be an integer greater than or equal to 1 and less than or equal to U.

[0158] According to an embodiment of the present disclosure, when 1<u≤U, the first intermediate feature information corresponding to each of the multiple text recognition information at the u-1th level is processed based on the self-attention strategy to obtain the second intermediate feature information corresponding to each of the multiple text recognition information at the u-th level. The first intermediate feature information corresponding to each of the multiple text recognition information at the u-th level can be used as the first key matrix, first value matrix and first query matrix at the u+1th level. The second intermediate feature information corresponding to each of the multiple text recognition information at the u-th level and the first intermediate feature information corresponding to each of the multiple text recognition information at the u-1th level are fused to obtain the fourth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level. Based on the fourth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level, the first intermediate feature information corresponding to each of the multiple text recognition information at the u-th level is obtained. Based on the first intermediate feature information corresponding to each of the multiple text recognition information at the R-th level, the global feature information is obtained. Fusion may include one of the following: addition and concatenation.

[0159] According to an embodiment of the present disclosure, obtaining the first intermediate feature information corresponding to each of the multiple text recognition information at the u-th level based on the fourth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level may include: performing multi-layer perceptron processing on the fourth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level to obtain the fifth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level. According to the fifth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level, obtaining the first intermediate feature information corresponding to each of the multiple text recognition information at the u-th level. For example, the sixth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level may be normalized to obtain the fourth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level. Normalization (i.e., Normalization) may include one of the following: Batch Normalization (BN) and Layer Normalization (LN). For example, the sixth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level may be batch normalized to obtain the fourth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level.

[0160] According to an embodiment of the present disclosure, processing the first intermediate feature information corresponding to each of the multiple text recognition information at the u-1th level based on the self-attention strategy to obtain the second intermediate feature information corresponding to each of the multiple text recognition information at the u-th level may include: obtaining the eighth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level based on the seventh intermediate feature information corresponding to each of the multiple text recognition information at the u-1th level. For example, standardizing the seventh intermediate feature information corresponding to each of the multiple text recognition information at the u-1th level to obtain the eighth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level. Processing the eighth intermediate feature information corresponding to each of the multiple text recognition information at the u-th level based on the self-attention strategy to obtain the first intermediate feature information corresponding to each of the multiple text recognition information at the u-th level.

[0161] According to an embodiment of the present disclosure, obtaining second intermediate feature information corresponding to each of the multiple text recognition information at the u-th level based on the first intermediate feature information corresponding to each of the multiple text recognition information at the u-1th level may include the following operations.

[0162] Based on the first intermediate feature information corresponding to each of the multiple text recognition information at the u-1th level, multiple first matrix sets corresponding to each of the multiple text recognition information at the u-th level are determined. For the text recognition information in the multiple text recognition information at the u-th level, for the first matrix set in the multiple first matrix sets corresponding to the text recognition information, based on the first query matrix corresponding to the text recognition information at the u-th level and the first key matrix corresponding to each of the multiple text recognition information at the u-th level, the first attention matrix corresponding to the text recognition information at the u-th level is obtained. Based on the first attention matrix corresponding to the text recognition information at the u-th level and the first value matrix corresponding to the text recognition information at the u-th level, the third intermediate feature information corresponding to the text recognition information at the u-th level is obtained. Based on the multiple third intermediate feature information corresponding to the text recognition information at the u-th level, the second intermediate feature information corresponding to the text recognition information at the u-th level is obtained.

[0163] According to an embodiment of the present disclosure, the first matrix set may include a first query matrix, a first key matrix, and a first value matrix.

[0164] According to an embodiment of the present disclosure, a self-attention strategy may include a multi-head self-attention strategy. According to the first intermediate feature information corresponding to each of the multiple text recognition information at the u-1th level, determining multiple first matrix sets corresponding to each of the multiple text recognition information at the u-1th level may include: for the text recognition information in the multiple text recognition information at the u-1th level, according to the first intermediate feature information corresponding to the text recognition information at the u-1th level, determining the first matrix set corresponding to the text recognition information at the u-1th level. The first matrix set may include a first key matrix, a first value matrix, and a first query matrix.

[0165] According to an embodiment of the present disclosure, obtaining the second intermediate feature information corresponding to the text recognition information at the u-th level based on the multiple third intermediate feature information corresponding to the text recognition information at the u-th level may include: obtaining the multiple ninth intermediate feature information corresponding to the text recognition information at the u-th level based on the multiple third intermediate feature information corresponding to the text recognition information at the u-th level. For example, the multiple third intermediate feature information corresponding to the text recognition information at the u-th level may be fused to obtain the multiple ninth intermediate feature information corresponding to the text recognition information at the u-th level. The fusion may include at least one of the following: splicing and addition. According to the multiple ninth intermediate feature information corresponding to the text recognition information at the u-th level, obtaining the second intermediate feature information corresponding to the text recognition information at the u-th level. For example, the multiple ninth intermediate feature information corresponding to the text recognition information at the u-th level may be linearly transformed to obtain the second intermediate feature information corresponding to the text recognition information at the u-th level.

[0166] According to an embodiment of the present disclosure, when u=1, performing U-level processing on the recognition information based on the self-attention strategy to obtain global feature information may include the following operations.

[0167] Based on the first-level global feature information corresponding to each of the plurality of text recognition information, second-level intermediate feature information corresponding to each of the plurality of text recognition information is obtained. Based on the second-level intermediate feature information corresponding to each of the plurality of text recognition information and the first-level global feature information corresponding to each of the plurality of text recognition information, first-level intermediate feature information corresponding to each of the plurality of text recognition information is obtained.

[0168] According to an embodiment of the present disclosure, global feature information may be used to determine a second query matrix, a second key matrix, and a second value matrix.

[0169] According to an embodiment of the present disclosure, the global feature information corresponding to each of the multiple text recognition information at the first level is processed based on the self-attention strategy to obtain the second intermediate feature information corresponding to each of the multiple text recognition information at the second level. The global feature information corresponding to each of the multiple text recognition information at the first level can be used as the first query matrix (i.e., the second query matrix), the first key matrix (i.e., the second key matrix), and the first value matrix (i.e., the second value matrix) at the second level. The second intermediate feature information corresponding to each of the multiple text recognition information at the second level and the global feature information corresponding to each of the multiple text recognition information at the first level are fused to obtain the fourth intermediate feature information corresponding to each of the multiple text recognition information at the second level. Based on the fourth intermediate feature information corresponding to each of the multiple text recognition information at the second level, the first intermediate feature information corresponding to each of the multiple text recognition information at the second level is obtained.

[0170] According to an embodiment of the present disclosure, obtaining first intermediate feature information corresponding to each of the multiple text recognition information at the second level based on fourth intermediate feature information corresponding to each of the multiple text recognition information at the second level may include: performing multi-layer perceptron processing on the fourth intermediate feature information corresponding to each of the multiple text recognition information at the second level to obtain fifth intermediate feature information corresponding to each of the multiple text recognition information at the second level. Obtaining first intermediate feature information corresponding to each of the multiple text recognition information at the second level based on fifth intermediate feature information corresponding to each of the multiple text recognition information at the second level. For example, the sixth intermediate feature information corresponding to each of the multiple text recognition information at the second level may be normalized to obtain fourth intermediate feature information corresponding to each of the multiple text recognition information at the second level. Normalization may include one of the following: batch normalization and layer normalization. For example, the sixth intermediate feature information corresponding to each of the multiple text recognition information at the second level may be batch normalized to obtain fourth intermediate feature information corresponding to each of the multiple text recognition information at the second level.

[0171] According to an embodiment of the present disclosure, processing the first intermediate feature information corresponding to each of the multiple text recognition information at the first level based on the self-attention strategy to obtain the second intermediate feature information corresponding to each of the multiple text recognition information at the second level may include: obtaining the eighth intermediate feature information corresponding to each of the multiple text recognition information at the second level based on the seventh intermediate feature information corresponding to each of the multiple text recognition information at the first level. For example, the seventh intermediate feature information corresponding to each of the multiple text recognition information at the first level is standardized to obtain the eighth intermediate feature information corresponding to each of the multiple text recognition information at the second level. Processing the eighth intermediate feature information corresponding to each of the multiple text recognition information at the second level based on the self-attention strategy to obtain the first intermediate feature information corresponding to each of the multiple text recognition information at the second level.

[0172] According to an embodiment of the present disclosure, obtaining second intermediate feature information corresponding to each of the plurality of text recognition information at the second level based on the first-level global feature information corresponding to each of the plurality of text recognition information may include the following operations.

[0173] Based on the global feature information corresponding to each of the multiple text recognition information at the first level, multiple second matrix sets corresponding to each of the multiple text recognition information at the second level are determined. For the text recognition information in the multiple text recognition information at the second level, for the second matrix set in the multiple second matrix sets corresponding to the text recognition information, based on the second query matrix corresponding to the text recognition information at the second level and the second key matrix corresponding to the multiple text recognition information at the second level, a second attention matrix corresponding to the text recognition information at the second level is obtained. Based on the second attention matrix corresponding to the text recognition information at the second level and the second value matrix corresponding to the text recognition information at the second level, third intermediate feature information corresponding to the text recognition information at the second level is obtained. Based on the multiple third intermediate feature information corresponding to the text recognition information at the second level, second intermediate feature information corresponding to the text recognition information at the second level is obtained.

[0174] According to an embodiment of the present disclosure, the second matrix set may include a second query matrix, a second key matrix, and a second value matrix.

[0175] According to an embodiment of the present disclosure, the self-attention strategy may include a multi-head self-attention strategy. According to the first intermediate feature information corresponding to each of the multiple text recognition information at the first level, determining the multiple second matrix sets corresponding to each of the multiple text recognition information at the second level may include: for the text recognition information in the multiple text recognition information at the first level, according to the first intermediate feature information corresponding to the text recognition information at the first level, determining the second matrix set corresponding to the text recognition information at the second level. The second matrix set may include a second key matrix, a second value matrix, and a second query matrix.

[0176] According to an embodiment of the present disclosure, obtaining the second intermediate feature information corresponding to the text recognition information at the second level based on the multiple third intermediate feature information corresponding to the text recognition information at the second level may include: obtaining the multiple ninth intermediate feature information corresponding to the text recognition information at the second level based on the multiple third intermediate feature information corresponding to the text recognition information at the second level. For example, the multiple third intermediate feature information corresponding to the text recognition information at the second level may be fused to obtain the multiple ninth intermediate feature information corresponding to the text recognition information at the second level. The fusion may include at least one of the following: splicing and addition. According to the multiple ninth intermediate feature information corresponding to the text recognition information at the second level, obtaining the second intermediate feature information corresponding to the text recognition information at the second level. For example, the multiple ninth intermediate feature information corresponding to the text recognition information at the second level may be linearly transformed to obtain the second intermediate feature information corresponding to the text recognition information at the second level.

[0177] Figure 5EThe following schematically illustrates an example process of determining semantic relationship information based on identification information according to another embodiment of the present disclosure.

[0178] like Figure 5E As shown in 500E, multiple first matrix sets 512 corresponding to multiple text recognition information at the u-th level can be determined based on the first intermediate feature information 511 corresponding to each of the multiple text recognition information at the u-1th level. The first matrix set 512 can include a first query matrix 512_1, a first key matrix 512_2, and a first value matrix 512_3.

[0179] After obtaining multiple first matrix sets 512 corresponding to multiple text identification information at the u-th level, the first attention matrix 513 corresponding to the text identification information at the u-th level can be obtained based on the first query matrix 512_1 corresponding to the text identification information at the u-th level and the first key matrix 512_2 corresponding to the multiple text identification information at the u-th level.

[0180] After obtaining the first attention matrix 513, the third intermediate feature information 514 corresponding to the text recognition information at the u-th level can be obtained based on the first attention matrix 513 corresponding to the text recognition information at the u-th level and the first value matrix 5123 corresponding to the text recognition information at the u-th level.

[0181] After obtaining the third intermediate feature information 514 , the second intermediate feature information 515 corresponding to the text recognition information at the u-th level can be obtained based on the plurality of third intermediate feature information 514 corresponding to the text recognition information at the u-th level.

[0182] After obtaining the second intermediate feature information 515 corresponding to the text recognition information at the u-th level, the first intermediate feature information 516 corresponding to the multiple text recognition information at the u-th level can be obtained based on the second intermediate feature information 515 corresponding to each of the multiple text recognition information at the u-th level and the first intermediate feature information 511 corresponding to each of the multiple text recognition information at the u-1-th level.

[0183] After obtaining the first intermediate feature information 516 corresponding to the plurality of text recognition information at the u-th level, the global feature information 517 may be obtained based on the first intermediate feature information corresponding to each of the plurality of text recognition information at the R-th level.

[0184] After obtaining the global feature information 517 , semantic relationship information 518 may be determined based on the global feature information 517 .

[0185] Figure 5F The following schematically illustrates an example process of determining semantic relationship information based on identification information according to another embodiment of the present disclosure.

[0186] like Figure 5F As shown in 500F, the first-level global feature information 519 corresponding to each of the plurality of text recognition information can be used to determine a second-level plurality of second matrix sets 520 corresponding to each of the plurality of text recognition information. The second matrix set 520 can include a second query matrix 520_1, a second key matrix 520_2, and a second value matrix 520_3.

[0187] After obtaining multiple second matrix sets 520 at the second level corresponding to multiple text recognition information, the second attention matrix 521 corresponding to the text recognition information at the second level can be obtained based on the second query matrix 520_1 corresponding to the text recognition information at the second level and the second key matrix 520_2 corresponding to the multiple text recognition information at the second level.

[0188] After obtaining the second attention matrix 521, the third intermediate feature information 522 corresponding to the text recognition information at the second level can be obtained based on the second attention matrix 521 corresponding to the text recognition information at the second level and the second value matrix 520_2 corresponding to the text recognition information at the second level.

[0189] After obtaining the third intermediate feature information 522 , the second intermediate feature information 523 corresponding to the text recognition information at the second level can be obtained based on the plurality of third intermediate feature information 522 corresponding to the text recognition information at the second level.

[0190] After obtaining the second intermediate feature information 523 corresponding to the text recognition information at the second level, the first intermediate feature information 524 corresponding to each of the multiple text recognition information at the second level can be obtained based on the second intermediate feature information 523 corresponding to each of the multiple text recognition information at the second level and the global feature information 519 corresponding to each of the multiple text recognition information at the first level.

[0191] After obtaining the first intermediate feature information 524 corresponding to each of the plurality of text recognition information at the second level, the global feature information 525 may be obtained based on the first intermediate feature information 524 corresponding to each of the plurality of text recognition information at the second level.

[0192] After obtaining the global feature information 525 , semantic relationship information 526 may be determined based on the global feature information 525 .

[0193] Figure 5G The flowchart of a method for determining semantic relationship information based on identification information according to another embodiment of the present disclosure is schematically shown.

[0194] like Figure 5G As shown, the method 500G is for Figure 2 As further defined in operation S240 , the method 500E may include operation S544 .

[0195] In operation S544 , the recognition information is processed using a text semantic relationship model to obtain semantic relationship information.

[0196] According to an embodiment of the present disclosure, a text semantic relationship model can be obtained by training a first deep learning model using multiple positive sample pairs and multiple negative sample pairs. The number of positive sample pairs and negative sample pairs can meet a predetermined balance condition. The two sample texts included in the positive sample pair can have a key-value relationship. The two sample texts included in the negative sample pair can have a non-key-value relationship.

[0197] According to embodiments of the present disclosure, a positive sample pair may refer to two sample texts with a key-value relationship. A negative sample pair may refer to two sample texts with a non-key-value relationship. For example, a sample text may include "name," "age," and "gender" whose category information is a keyword category, and "Zhang San," "42 years old," and "male" whose category information is a numerical category.

[0198] In this case, it can be determined that the positive sample pairs include "name" and "Zhang San", "age" and "42 years old", and "gender" and "male". In addition, it can be determined that the candidate negative sample pairs include "name" and "42 years old", "name" and "male", "age" and "Zhang San", "age" and "male", "gender" and "Zhang San", and "gender" and "42 years old".

[0199] According to an embodiment of the present disclosure, after determining the number of positive sample pairs and the number of candidate negative sample pairs, the positive sample pairs and the candidate negative sample pairs can be screened according to a predetermined balance condition so that the number of positive sample pairs and the number of negative sample pairs meet the predetermined balance condition. The predetermined balance condition can be configured according to actual business needs and is not limited here. For example, the predetermined balance condition can be configured so that the number of positive sample pairs and the number of negative sample pairs are equal. Alternatively, the predetermined balance condition can be configured so that the difference between the number of positive sample pairs and the number of negative sample pairs is less than or equal to a predetermined threshold. The predetermined threshold can be set to 1.

[0200] According to an embodiment of the present invention, operations S541, S542-S543, and S544 may be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be Figure 1 The server 105 in the terminal device can be Figure 1 The first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0201] According to the embodiments of the present disclosure, since the two sample texts included in a positive sample pair have a key-value relationship, and the two sample texts included in a negative sample pair have a non-key-value relationship, by training a first deep learning model using multiple positive sample pairs and multiple negative sample pairs, the resulting text semantic relationship model can automatically perform semantic relationship classification. Furthermore, since the semantic relationship information is obtained by processing the recognition information using the text semantic relationship model, the accuracy of the semantic relationship information is improved.

[0202] According to an embodiment of the present disclosure, multiple negative sample pairs are determined from multiple candidate negative sample pairs based on a negative sample pruning strategy.

[0203] According to an embodiment of the present disclosure, a negative sample pruning strategy can be used to characterize the conditions for determining a negative sample pair from a plurality of candidate negative sample pairs. For example, the negative sample pruning strategy may include a positional relationship condition between candidate sample texts. In this case, the negative sample pair can be determined from a plurality of candidate negative sample pairs based on the positional relationship between the plurality of candidate sample texts. Alternatively, the negative sample pruning strategy may include a similarity condition between candidate sample texts. In this case, the negative sample pair can be determined from a plurality of candidate negative sample pairs based on the similarity between the plurality of candidate sample texts.

[0204] According to an embodiment of the present disclosure, multiple negative sample pairs are determined from multiple candidate negative sample pairs based on a negative sample pruning strategy, which may include the following operations.

[0205] The multiple negative sample pairs are determined from the multiple candidate negative sample pairs according to the positional relationship between the multiple candidate sample texts.

[0206] According to an embodiment of the present disclosure, a negative sample pruning strategy may include at least one of the following: a short text position relationship strategy and a long text position relationship strategy. The short text position relationship strategy may be used to characterize the position relationship conditions that need to be met when two sample texts with a non-key value relationship belong to a short text. The long text position relationship strategy may be used to characterize the position relationship conditions that need to be met when two sample texts with a non-key value relationship belong to a long text.

[0207] For example, the short text position relationship strategy may include that sample text belonging to the keyword category is located to the left of sample text belonging to the numerical category. In this case, based on the short text position relationship strategy and the positional relationships between the plurality of candidate sample texts, multiple negative sample pairs can be determined from the plurality of candidate negative sample pairs in which the sample text belonging to the keyword category is located to the left of the sample text belonging to the numerical category.

[0208] For example, the long text position relationship strategy may include that sample text belonging to the keyword category is located above sample text belonging to the numerical category. In this case, based on the short text position relationship strategy and the positional relationships between the multiple candidate sample texts, multiple negative sample pairs that satisfy the requirement that sample text belonging to the keyword category is located above sample text belonging to the numerical category can be determined from the multiple candidate negative sample pairs.

[0209] Figure 6 The flowchart of the method for performing text recognition on a text region image and obtaining recognition information according to an embodiment of the present disclosure is schematically shown.

[0210] like Figure 6 As shown, the method 600 is to Figure 2 Further defining operation S220 in the method 600 , the method 600 may include operations S621 to S622 .

[0211] In operation S621, the position information is converted into target position information using affine transformation.

[0212] In operation S622 , images corresponding to the plurality of text regions are extracted from the text image according to the target position information, thereby obtaining text region images corresponding to the plurality of text regions.

[0213] According to embodiments of the present disclosure, an affine transformation is a linear transformation between two-dimensional coordinates, used to maintain the "flatness" and "parallelism" of two-dimensional graphics. Straightness can be understood as ensuring that straight lines remain straight and arcs remain circular after the transformation. Parallelism can be understood as maintaining the relative positional relationships between different two-dimensional graphics, ensuring that parallel lines remain parallel and the angle between intersecting lines remains unchanged. Affine transformations can be achieved through at least one of translation, scaling, flipping, rotation, and shearing.

[0214] According to an embodiment of the present disclosure, using affine transformation to convert position information corresponding to a text area into target position information may include: using affine transformation to convert a text area in the form of a four-corner point box into a text area in the form of a rectangular box, and determining the position information corresponding to the text area in the form of a rectangular box as target position information, so that text area images corresponding to multiple text areas can be extracted from the text image according to the target position information.

[0215] For example, the text area is a four-corner box, which can be represented by {P1, P2, P3, P4}, where P1 represents the point at the upper left corner of the four-corner box, P2 represents the point at the upper right corner of the four-corner box, P3 represents the point at the lower left corner of the four-corner box, and P4 represents the point at the lower right corner of the four-corner box. The coordinates of P1 can be represented as {x1, y1}, the coordinates of P2 can be represented as {x2, y2}, the coordinates of P3 can be represented as {x3, y3}, and the coordinates of P4 can be represented as {x4, y4}. Using affine transformation, P1→P′1, P2→P′2, P3→P′3, and P4→P′4 are transformed to obtain a rectangular box {P′1, P′2, P′3, P′4}. The coordinates of P1' can be represented as {x′1, y′1}, and the coordinates of P′2 can be represented as {x′2, y ′ 2}, the coordinates of P′3 can be represented as {x′3, y′3}, and the coordinates of P′4 can be represented as {x′4, y′4}.

[0216] According to an embodiment of the present invention, operations S621 to S622 may be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be Figure 1 The server 105 in the terminal device can be Figure 1 The first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0217] According to the embodiments of the present disclosure, since target position information is obtained by converting position information using an affine transformation, the target position information can be automatically determined, thereby improving the efficiency and accuracy of determining the target position information. Furthermore, since text region images are obtained by extracting images corresponding to multiple text regions from the text image based on the target position information, the text region images can be automatically extracted, thereby improving the efficiency and accuracy of obtaining the text region images.

[0218] Figure 7 The flowchart of the method for generating structured information of a text image according to category information, semantic relationship information and recognition information according to an embodiment of the present disclosure is schematically shown.

[0219] like Figure 7 As shown, the method 700 is to Figure 2 Further defining operation S250 in the method 700 , the method 700 may include operations S751 to S752 .

[0220] In operation S751 , a plurality of target text recognition information is determined from a plurality of text recognition information according to semantic relationship information.

[0221] In operation S752 , structured information of the text image is generated based on the category information, the key-value relationship, and the target text recognition information.

[0222] According to an embodiment of the present disclosure, the category information may include one of a keyword category and a value category. The semantic relationship may include one of a key-value relationship and a non-key-value relationship.

[0223] According to an embodiment of the present disclosure, the semantic relationship corresponding to the text recognition information may be a key-value relationship. The structured information may include keyword categories, target text recognition information corresponding to the keyword categories, numerical categories, and target text recognition information corresponding to the numerical categories.

[0224] According to an embodiment of the present disclosure, target text recognition information can be used to represent two pieces of text recognition information having a key-value relationship. Multiple pieces of target text recognition information can be determined from multiple pieces of text recognition information based on the key-value relationship between the multiple pieces of text recognition information. Alternatively, target text recognition information can be used to represent two pieces of text recognition information having a non-key-value relationship. Multiple pieces of target text recognition information can be determined from multiple pieces of text recognition information based on the non-key-value relationship between the multiple pieces of text recognition information.

[0225] For example, the text recognition information may include "name," "Zhang San," "age," "42 years old," "gender," and "male." Based on the key-value relationship between the plurality of text recognition information, the target text recognition information "name," "age," and "gender" whose category information is a keyword category, and the target text recognition information "Zhang San," "42 years old," and "male" whose category information corresponding to the keyword category is a numerical category can be determined.

[0226] According to an embodiment of the present disclosure, after obtaining a plurality of target text recognition information, structured information of a text image may be generated according to keyword categories, target text recognition information corresponding to the keyword categories, numerical categories, and target text recognition information corresponding to the numerical categories.

[0227] According to an embodiment of the present invention, operations S751 to S752 may be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be Figure 1 The server 105 in the terminal device can be Figure 1 The first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0228] According to the embodiments of the present disclosure, since multiple target text recognition information is determined from multiple text recognition information based on semantic relationship information, the multiple target text recognition information can be used to represent two text recognition information having a key-value relationship. On this basis, since the structured information of the text image is generated based on the category information, the key-value relationship, and the target text recognition information, the accuracy of the structured information generation of the text image is improved.

[0229] Figure 8 An example diagram of an information generation process according to an embodiment of the present disclosure is schematically shown.

[0230] like Figure 8 As shown, in the information generation process 800, the text detection model 802 can perform text detection on the text image 801 to obtain category information and position information of multiple text regions corresponding to the text image 801. The multiple text regions can include text region 8031, text region 803_2, text region 803_3, text region 803_4, text region 803_5, text region 803_6, text region 803_7, and text region 803_8.

[0231] For example, the category information may include a keyword category or a numerical category. The category information of text area 803_1, text area 803_3, text area 803_5, and text area 803_7 may be a keyword category. The category information of text area 803_2, text area 803_4, text area 803_6, and text area 803_8 may be a numerical category.

[0232] After obtaining the category information and position information of each of the multiple text regions, text region images corresponding to each of the multiple text regions can be obtained based on the position information of each of the multiple text regions and text image 801. The text region images corresponding to each of the multiple text regions may include text region image 804_1 (i.e., a text region image including the text content "Name:"), text region image 804_2 (i.e., a text region image including the text content "Zhang San"), text region image 804_3 (i.e., a text region image including the text content "Gender:"), text region image 804_4 (i.e., a text region image including the text content "Male"), text region image 804_5 (i.e., a text region image including the text content "Age:"), text region image 804_6 (i.e., a text region image including the text content "42 years old"), text region image 804_7 (i.e., a text region image including the text content "Detection result:"), and text region image 804_8 (i.e., a text region image including the text content "XX").

[0233] For example, based on the position information of text image 801 and text region 803_1, text region image 804_1 corresponding to text region 803_1 can be obtained. Based on the position information of text image 801 and text region 803_2, text region image 804_2 corresponding to text region 803_2 can be obtained. Similarly, based on the position information of text image 801 and text region 803_8, text region image 804_8 corresponding to text region 803_8 can be obtained.

[0234] After obtaining text region images corresponding to each of the multiple text regions, text recognition model 805 can perform text recognition on the multiple text region images to obtain text recognition information for each of the multiple text region images. The text recognition information for each of the multiple text region images may include text recognition information 806_1 (i.e., "Name:"), text recognition information 806_2 (i.e., "Zhang San"), text recognition information 806_3 (i.e., "Gender:"), text recognition information 806_4 (i.e., "Male"), text recognition information 806_5 (i.e., "Age:"), text recognition information 806_6 (i.e., "42 years old"), text recognition information 806_7 (i.e., "Detection Result:"), and text recognition information 806_8 (i.e., "XX").

[0235] For example, text recognition model 805 can be used to perform text recognition on text region image 804_1 to obtain text recognition information 806_1 for text region image 804_1. Text recognition model 805 can also be used to perform text recognition on text region image 804_2 to obtain text recognition information 806_2 for text region image 804_2. Similarly, text recognition model 805 can be used to perform text recognition on text region image 804_8 to obtain text recognition information 806_8 for text region image 804_8.

[0236] After obtaining the text recognition information of each of the multiple text region images, the text classification model 807 may process the text recognition information of each of the multiple text region images to obtain semantic relationship information.

[0237] For example, the text classification model 807 can be used to process the text recognition information of each of the multiple text area images to obtain the semantic relationship between the text area image 804_1 whose category information is the keyword category (i.e., the text area image including the text content "Name:") and the text area image 804_2 whose category information is the numerical category (i.e., the text area image including the text content "Zhang San"), and the text area image 804_3 whose category information is the keyword category (i.e., the text area image including the text content "Gender:") and the text area image 804_4 whose category information is the numerical category (i.e., the text area image including the text content "Male"). ”), the semantic relationship between text area image 804_5 whose category information is keyword category (i.e., the text area image including the text content “age:”) and text area image 804_6 whose category information is numerical category (i.e., the text area image including the text content “42 years old”), and the semantic relationship between text area image 804_7 whose category information is keyword category (i.e., the text area image including the text content “detection result:”) and text area image 804_8 whose category information is numerical category (i.e., the text area image including the text content “XX”).

[0238] After obtaining the semantic relationships between the multiple text recognition information, structured information of the text image can be generated based on the category information, semantic relationship information, and recognition information. The structured information of the text image may include structured information 808_1 (i.e., "Name: Zhang San"), structured information 808_2 (i.e., "Gender: Male"), structured information 808_3 (i.e., "Age: 42"), and structured information 808_4 (i.e., "Detection Result: XX").

[0239] Figure 9 The flowchart of the information processing method according to the embodiment of the present disclosure is schematically shown.

[0240] like Figure 9 As shown, the method includes operations S910 to S920.

[0241] In operation S910 , the text image to be processed is processed using the information generation method 200 to obtain structured information of the text image to be processed.

[0242] In operation S920 , information processing is performed using the structured information of the text image to be processed.

[0243] According to an embodiment of the present disclosure, the structured information of the text image to be processed may be determined using the information generation method according to an embodiment of the present disclosure.

[0244] According to an embodiment of the present disclosure, an information generation method can be used to process a text image to be processed, obtain structured information of the text image to be processed, and perform information processing using the structured information of the text image to be processed.

[0245] According to an embodiment of the present invention, operations S910 to S920 may be performed by an electronic device. The electronic device may include a server or a terminal device. The server may be Figure 1 The server 105 in the terminal device can be Figure 1 The first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0246] According to an embodiment of the present invention, by processing a text image to be processed using an information generation method, structured information of the text image to be processed is obtained. Since semantic relationship information and visual information are used to generate the structured information, the accuracy of the structured information is improved. Furthermore, since the structured information of the text image to be processed is used for information processing, the accuracy of the information processing is improved.

[0247] This is only an exemplary embodiment, but is not limited thereto. Other information processing methods known in the art may also be included as long as the accuracy of information processing can be improved.

[0248] Figure 10 The block diagram schematically shows an information generating device according to an embodiment of the present disclosure.

[0249] like Figure 10 As shown, the information generating device 1000 may include a text detection module 1010 , a first acquisition module 1020 , a text recognition module 1030 , a determination module 1040 and a generation module 1050 .

[0250] The text detection module 1010 is configured to perform text detection on a text image to obtain detection information, wherein the detection information includes category information and position information of each of a plurality of text regions.

[0251] The first acquisition module 1020 is configured to acquire text region images corresponding to respective text regions according to the position information and the text image.

[0252] The text recognition module 1030 is configured to perform text recognition on the text region image to obtain recognition information, wherein the recognition information includes text recognition information of each of the multiple text region images.

[0253] The determination module 1040 is configured to determine semantic relationship information based on the recognition information, wherein the semantic relationship information includes semantic relationships between multiple text recognition information.

[0254] The generating module 1050 is used to generate structured information of the text image according to the category information, semantic relationship information and recognition information.

[0255] According to an embodiment of the present disclosure, the text detection module 1010 may include a feature extraction submodule, a first acquisition submodule, a second acquisition submodule, a third acquisition submodule, and a fourth acquisition submodule.

[0256] The feature extraction submodule is used to extract features from the text image to obtain a first feature map of at least one scale.

[0257] The first acquisition submodule is configured to acquire a second feature map according to the first feature map at at least one scale.

[0258] The second acquisition submodule is configured to acquire a third feature map according to the first feature map at at least one scale.

[0259] The third acquisition submodule is configured to acquire category information of each of the plurality of text regions according to the second feature map.

[0260] The fourth acquisition submodule is used to obtain position information of each of the multiple text regions according to the third feature map.

[0261] According to an embodiment of the present disclosure, the feature extraction submodule may include a first feature extraction unit and a first obtaining unit.

[0262] The first feature extraction unit is used to perform M stages of feature extraction on the text image to obtain at least one first feature map corresponding to the Mth stage.

[0263] The first obtaining unit is configured to obtain a first feature map of at least one scale according to at least one first feature map corresponding to the Mth stage.

[0264] According to an embodiment of the present disclosure, the mth stage has T m The image resolutions of the first feature maps of the same parallel layer are the same, and the image resolutions of the first feature maps of different parallel layers are different.

[0265] According to an embodiment of the present disclosure, M is an integer greater than or equal to 1, m is an integer greater than or equal to 1 and less than or equal to M, T m is an integer greater than or equal to 1.

[0266] According to an embodiment of the present disclosure, where M is an integer greater than 1, the feature extraction unit may include a convolution processing subunit and a feature fusion subunit.

[0267] The convolution processing subunit is used to perform convolution processing on at least one first feature map corresponding to the m-1th stage to obtain at least one intermediate first feature map corresponding to the mth stage.

[0268] The feature fusion subunit is used to perform feature fusion on at least one intermediate first feature map corresponding to the m-th stage to obtain at least one first feature map corresponding to the m-th stage.

[0269] According to an embodiment of the present disclosure, m is an integer greater than 1 and less than or equal to M.

[0270] According to an embodiment of the present disclosure, the feature fusion subunit may include:

[0271] Targeting T m The i-th parallel level among the parallel levels,

[0272] The first feature map corresponding to the i-th parallel level is obtained according to the other intermediate first feature maps corresponding to the i-th parallel level and the intermediate first feature map corresponding to the i-th parallel level.

[0273] According to an embodiment of the present disclosure, the other intermediate first feature maps corresponding to the i-th parallel level are m The intermediate first characteristic graphs corresponding to at least some of the parallel levels except the i-th parallel level in the parallel levels, i is greater than or equal to 1 and less than or equal to T m An integer.

[0274] According to an embodiment of the present disclosure, the feature extraction submodule may include a second feature extraction unit.

[0275] The second feature extraction unit is used to perform N cascade-level feature extraction on the text image to obtain a first feature map of at least one scale, where N is an integer greater than 1.

[0276] According to an embodiment of the present disclosure, the determination module 1040 may include a first determination submodule.

[0277] The first determination submodule is configured to determine semantic relationship information based on auxiliary information and identification information, wherein the auxiliary information includes at least one of the following: a second feature map and position information.

[0278] According to an embodiment of the present disclosure, when the auxiliary information includes the second feature map, the first determination submodule may include a fusion unit and a first determination unit.

[0279] The fusion unit is used to fuse the second feature map with the fourth feature map corresponding to the identification information to obtain a fused feature map.

[0280] The first determining unit is used to determine semantic relationship information based on the fused feature map.

[0281] According to an embodiment of the present disclosure, in a case where the auxiliary information further includes location information, the first determining unit may include a first determining subunit.

[0282] The first determining subunit is used to determine semantic relationship information based on the fused feature map and position information.

[0283] According to an embodiment of the present disclosure, the determination module 1040 may include a global feature extraction submodule and a second determination submodule.

[0284] The global feature extraction submodule is used to extract global features of the recognition information to obtain global feature information.

[0285] The second determining submodule is used to determine semantic relationship information based on global feature information.

[0286] According to an embodiment of the present disclosure, the global feature extraction submodule may include a processing unit.

[0287] The processing unit is used to process the recognition information based on the attention strategy to obtain global feature information.

[0288] According to an embodiment of the present disclosure, the processing unit may include a processing sub-unit.

[0289] The processing subunit is used to perform U-level processing on the recognition information based on the self-attention strategy to obtain global feature information, where U is an integer greater than or equal to 1.

[0290] According to an embodiment of the present disclosure, when U is an integer greater than 1 and 1<u≤U, the processing subunit may include:

[0291] Obtaining second intermediate feature information corresponding to each of the plurality of text recognition information at the u-th level based on the first intermediate feature information corresponding to each of the plurality of text recognition information at the u-th level, wherein the first intermediate feature information is used to determine a first query matrix, a first key matrix, and a first value matrix;

[0292] Obtaining first intermediate feature information corresponding to the plurality of text recognition information at the u-th level based on the second intermediate feature information corresponding to each of the plurality of text recognition information at the u-th level and the first intermediate feature information corresponding to each of the plurality of text recognition information at the u-1-th level; and

[0293] Global feature information is obtained based on the first intermediate feature information corresponding to each of the plurality of text recognition information at the R-th level.

[0294] According to an embodiment of the present disclosure, u is an integer greater than or equal to 1 and less than or equal to U, and R is an integer greater than or equal to 1 and less than or equal to U.

[0295] According to an embodiment of the present disclosure, obtaining second intermediate feature information corresponding to each of the plurality of text recognition information at the u-th level based on the first intermediate feature information corresponding to each of the plurality of text recognition information at the u-1th level may include:

[0296] Determining, based on the first intermediate feature information corresponding to each of the plurality of text recognition information at the u-1th level, a plurality of first matrix sets corresponding to each of the plurality of text recognition information at the u-th level, wherein the first matrix sets include a first query matrix, a first key matrix, and a first value matrix; and

[0297] For the text recognition information in the plurality of text recognition information at the u-th level,

[0298] For a first matrix set among a plurality of first matrix sets corresponding to text recognition information,

[0299] Obtaining a first attention matrix corresponding to the text recognition information at the u-th level according to the first query matrix corresponding to the text recognition information at the u-th level and the first key matrices corresponding to the plurality of text recognition information at the u-th level;

[0300] Obtaining third intermediate feature information corresponding to the text recognition information at the u-th level according to the first attention matrix corresponding to the text recognition information at the u-th level and the first value matrix corresponding to the text recognition information at the u-th level;

[0301] The second intermediate feature information corresponding to the text recognition information at the u-th level is obtained according to the plurality of third intermediate feature information corresponding to the text recognition information at the u-th level.

[0302] According to an embodiment of the present disclosure, when u=1, the processing subunit may include:

[0303] Obtaining second intermediate feature information corresponding to each of the plurality of text recognition information at a second level based on the first-level global feature information corresponding to each of the plurality of text recognition information, wherein the global feature information is used to determine a second query matrix, a second key matrix, and a second value matrix; and

[0304] The first intermediate feature information corresponding to each of the plurality of text recognition information at the second level is obtained based on the second intermediate feature information corresponding to each of the plurality of text recognition information at the first level and the global feature information corresponding to each of the plurality of text recognition information at the first level.

[0305] According to an embodiment of the present disclosure, obtaining second intermediate feature information corresponding to each of the plurality of text recognition information at the second level based on the first-level global feature information corresponding to each of the plurality of text recognition information may include:

[0306] Determining, based on the first-level global feature information corresponding to each of the plurality of text recognition information, a plurality of second matrix sets corresponding to each of the plurality of text recognition information at the second level, wherein the second matrix sets include a second query matrix, a second key matrix, and a second value matrix; and

[0307] For the text recognition information in the plurality of text recognition information at the second level,

[0308] For a second matrix set among a plurality of second matrix sets corresponding to the text recognition information,

[0309] Obtaining a second attention matrix corresponding to the text recognition information at the second level according to the second query matrix corresponding to the text recognition information at the second level and the second key matrix corresponding to the plurality of text recognition information at the second level;

[0310] Obtaining third intermediate feature information corresponding to the text recognition information at the second level according to the second attention matrix corresponding to the text recognition information at the second level and the second value matrix corresponding to the text recognition information at the second level;

[0311] The second intermediate feature information corresponding to the text recognition information at the second level is obtained based on the plurality of third intermediate feature information corresponding to the text recognition information at the second level.

[0312] According to an embodiment of the present disclosure, the semantic relationship includes one of a key-value relationship and a non-key-value relationship.

[0313] According to an embodiment of the present disclosure, the determination module 1040 may include a first processing submodule.

[0314] The first processing submodule is used to process the recognition information using a text semantic relationship model to obtain semantic relationship information, wherein the text semantic relationship model is obtained by training a first deep learning model using multiple positive sample pairs and multiple negative sample pairs, the number of positive sample pairs and negative sample pairs meets a predetermined balance condition, the two sample texts included in the positive sample pair have a key-value relationship, and the two sample texts included in the negative sample pair have a non-key-value relationship.

[0315] According to an embodiment of the present disclosure, multiple negative sample pairs are determined from multiple candidate negative sample pairs based on a negative sample pruning strategy.

[0316] According to an embodiment of the present disclosure, multiple negative sample pairs are determined from multiple candidate negative sample pairs based on a negative sample pruning strategy, and may include:

[0317] The multiple negative sample pairs are determined from the multiple candidate negative sample pairs according to the positional relationship between the multiple candidate sample texts.

[0318] According to an embodiment of the present disclosure, the text recognition module 1030 may include a conversion submodule and an extraction submodule.

[0319] The conversion submodule is used to convert the position information into target position information using affine transformation.

[0320] The extraction submodule is used to extract images corresponding to the multiple text regions from the text image according to the target position information, so as to obtain text region images corresponding to the multiple text regions.

[0321] According to an embodiment of the present disclosure, the category information includes one of a keyword category and a value category, and the semantic relationship includes one of a key-value relationship and a non-key-value relationship.

[0322] According to an embodiment of the present disclosure, the generation module 1050 may include a third determination submodule and a generation submodule.

[0323] The third determining submodule is configured to determine a plurality of target text recognition information from a plurality of text recognition information according to the semantic relationship information, wherein the semantic relationship corresponding to the text recognition information is a key-value relationship.

[0324] The generation submodule is used to generate structured information of the text image based on category information, key-value relationship and target text recognition information, wherein the structured information includes keyword category, target text recognition information corresponding to the keyword category, numerical category and target text recognition information corresponding to the numerical category.

[0325] According to an embodiment of the present disclosure, the text image includes a medical text image.

[0326] Figure 11 The block diagram schematically shows an information processing device according to an embodiment of the present disclosure.

[0327] like Figure 11 As shown, the information processing device 1100 may include a second acquisition module 1110 and an information processing module 1120 .

[0328] The second acquisition module 1110 is configured to process the text image to be processed using the information generating device 1000 to acquire structured information of the text image to be processed.

[0329] The information processing module 1120 is used to perform information processing using the structured information of the text image to be processed.

[0330] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0331] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the present disclosure.

[0332] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in the present disclosure.

[0333] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described in the present disclosure.

[0334] Figure 12 A block diagram of an electronic device suitable for implementing an information generation method and an information processing method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0335] like Figure 12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 12012 into a random access memory (RAM) 1203. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0336] Various components in device 1200 are connected to I / O interface 1205, including an input unit 1206, such as a keyboard and mouse; an output unit 1207, such as various types of displays and speakers; a storage unit 1208, such as a magnetic disk and optical disk; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0337] The computing unit 1201 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as the information generation method and the information processing method. For example, in some embodiments, the information generation method and the information processing method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the information generation method and the information processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the information generating method and the information processing method in any other appropriate manner (eg, by means of firmware).

[0338] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0339] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0340] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0341] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0342] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0343] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0344] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0345] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating information, comprising: Performing text detection on the text image to obtain detection information, wherein the detection information includes category information and position information of each of the plurality of text regions; acquiring text region images corresponding to respective ones of the plurality of text regions according to the position information and the text image; Performing text recognition on the text region image to obtain recognition information, wherein the recognition information includes text recognition information of each of the plurality of text region images; Performing U-level processing on the recognition information based on a self-attention strategy to obtain global feature information, where U is an integer greater than or equal to 1; Determining semantic relationship information based on the global feature information, wherein the semantic relationship information includes semantic relationships between the plurality of text recognition information; and generating structured information of the text image according to the category information, the semantic relationship information, and the identification information; The obtaining of global feature information includes: in the case of 1<u≤U, Determining, based on the first intermediate feature information corresponding to each of the plurality of text recognition information at the u-1th level, a plurality of first matrix sets corresponding to each of the plurality of text recognition information at the u-th level, wherein the first matrix sets include a first query matrix, a first key matrix, and a first value matrix; For the text recognition information at the u-th level, obtain a first attention matrix based on a first query matrix corresponding to the text recognition information and first key matrices corresponding to each of the plurality of text recognition information; obtain third intermediate feature information corresponding to the text recognition information based on the first attention matrix and a first value matrix corresponding to the text recognition information; and obtain second intermediate feature information corresponding to the text recognition information based on the plurality of third intermediate feature information; Obtaining first intermediate feature information corresponding to the plurality of text recognition information at the u-th level based on the second intermediate feature information corresponding to each of the plurality of text recognition information at the u-th level and the first intermediate feature information corresponding to each of the plurality of text recognition information at the u-1-th level; and Obtaining the global feature information according to the first intermediate feature information corresponding to each of the plurality of text recognition information at the R-th level; Here, u is an integer greater than or equal to 1 and less than or equal to U, and R is an integer greater than or equal to 1 and less than or equal to U.

2. The method according to claim 1, wherein The performing text detection on the text image to obtain detection information includes: Performing feature extraction on the text image to obtain a first feature map of at least one scale; Acquire a second feature map according to the first feature map of the at least one scale; Acquire a third feature map according to the first feature map of the at least one scale; acquiring category information of each of the plurality of text regions according to the second feature map; and According to the third feature map, position information of each of the multiple text regions is obtained.

3. The method according to claim 2, wherein: The step of extracting features from the text image to obtain a first feature map at at least one scale includes: Performing M stages of feature extraction on the text image to obtain at least one first feature map corresponding to the M-th stage; and Obtaining the at least one first feature map at the at least one scale according to the at least one first feature map corresponding to the M-th stage; Among them, the mth stage has T m parallel layers, the image resolution of the first feature maps of the same parallel layer is the same, and the image resolution of the first feature maps of different parallel layers is different; Wherein, M is an integer greater than or equal to 1, m is an integer greater than or equal to 1 and less than or equal to M, T m is an integer greater than or equal to 1.

4. The method according to claim 3, wherein: When M is an integer greater than 1, performing M stages of feature extraction on the text image to obtain at least one first feature map corresponding to the M-th stage includes: Performing convolution processing on at least one first feature map corresponding to the (m-1)th stage to obtain at least one intermediate first feature map corresponding to the (m)th stage; and Performing feature fusion on at least one intermediate first feature map corresponding to the m-th stage to obtain at least one first feature map corresponding to the m-th stage; Here, m is an integer greater than 1 and less than or equal to M.

5. The method according to claim 4, wherein The performing feature fusion on the at least one intermediate first feature map corresponding to the m-th stage to obtain the at least one first feature map corresponding to the m-th stage includes: Regarding the T m The i-th parallel level among the parallel levels, Obtaining a first feature map corresponding to the i-th parallel level according to other intermediate first feature maps corresponding to the i-th parallel level and the intermediate first feature map corresponding to the i-th parallel level; Among them, the other intermediate first feature maps corresponding to the i-th parallel level are m The intermediate first characteristic graphs corresponding to at least some of the parallel levels except the i-th parallel level in the parallel levels, i is greater than or equal to 1 and less than or equal to T m An integer.

6. The method according to claim 2, wherein: The step of extracting features from the text image to obtain a first feature map at at least one scale includes: Performing N cascade-level feature extraction on the text image to obtain a first feature map of the at least one scale, where N is an integer greater than 1.

7. The method according to any one of claims 2 to 6, wherein The determining of semantic relationship information includes: The semantic relationship information is determined based on auxiliary information and the identification information, wherein the auxiliary information includes at least one of the following: the second feature map and the position information.

8. The method according to claim 7, wherein: In a case where the auxiliary information includes the second feature map, determining the semantic relationship information according to the auxiliary information and the identification information includes: Fusing the second feature map with a fourth feature map corresponding to the identification information to obtain a fused feature map; and The semantic relationship information is determined according to the fused feature map.

9. The method according to claim 8, wherein In a case where the auxiliary information further includes the position information, determining the semantic relationship information according to the fused feature map includes: The semantic relationship information is determined according to the fused feature map and the position information.

10. The method according to claim 1, further comprising: In the case of u=1, Obtaining second intermediate feature information corresponding to each of the plurality of text recognition information at a second level based on the first-level global feature information corresponding to each of the plurality of text recognition information, wherein the global feature information is used to determine a second query matrix, a second key matrix, and a second value matrix; and The first intermediate feature information of the second level corresponding to each of the plurality of text recognition information is obtained based on the second intermediate feature information of the second level corresponding to each of the plurality of text recognition information and the global feature information of the first level corresponding to each of the plurality of text recognition information.

11. The method according to claim 10, wherein: The obtaining, based on the first-level global feature information corresponding to each of the plurality of text recognition information, second-level intermediate feature information corresponding to each of the plurality of text recognition information comprises: Determining, based on the global feature information of the first level corresponding to each of the plurality of text recognition information, a plurality of second matrix sets of the second level corresponding to each of the plurality of text recognition information, wherein the second matrix sets include the second query matrix, the second key matrix, and the second value matrix; and For the text identification information in the plurality of text identification information in the second level, For a second matrix set among a plurality of second matrix sets corresponding to the text recognition information, Obtaining a second attention matrix corresponding to the text recognition information at the second level according to the second query matrix corresponding to the text recognition information at the second level and the second key matrix corresponding to the plurality of text recognition information at the second level; Obtaining third intermediate feature information corresponding to the text recognition information at the second level according to the second attention matrix corresponding to the text recognition information at the second level and the second value matrix corresponding to the text recognition information at the second level; The second intermediate feature information corresponding to the text recognition information at the second level is obtained according to the plurality of third intermediate feature information corresponding to the text recognition information at the second level.

12. The method according to any one of claims 1 to 6, wherein The semantic relationship includes one of a key-value relationship and a non-key-value relationship; The determining of semantic relationship information includes: The recognition information is processed using a text semantic relationship model to obtain the semantic relationship information, wherein the text semantic relationship model is obtained by training a first deep learning model using multiple positive sample pairs and multiple negative sample pairs, the number of the positive sample pairs and the number of the negative sample pairs meet a predetermined balance condition, the two sample texts included in the positive sample pair have a key-value relationship, and the two sample texts included in the negative sample pair have a non-key-value relationship.

13. The method according to claim 12, wherein: The multiple negative sample pairs are determined from multiple candidate negative sample pairs based on a negative sample pruning strategy.

14. The method according to claim 13, wherein The multiple negative sample pairs are determined from multiple candidate negative sample pairs based on a negative sample pruning strategy, including: The multiple negative sample pairs are determined from the multiple candidate negative sample pairs according to the positional relationship between the multiple candidate sample texts.

15. The method according to claim 1, wherein The acquiring, based on the position information and the text image, text area images corresponding to each of the plurality of text areas includes: Converting the position information into target position information using affine transformation; and Images corresponding to the multiple text regions are extracted from the text image according to the target position information to obtain text region images corresponding to the multiple text regions.

16. The method according to claim 1, wherein The category information includes one of a keyword category and a value category, and the semantic relationship includes one of a key-value relationship and a non-key-value relationship; The step of generating the structured information of the text image according to the category information, the semantic relationship information, and the identification information includes: Determining a plurality of target text recognition information from the plurality of text recognition information according to the semantic relationship information, wherein the semantic relationship corresponding to the text recognition information is the key-value relationship; and Based on the category information, the key-value relationship and the target text recognition information, structured information of the text image is generated, wherein the structured information includes the keyword category, the target text recognition information corresponding to the keyword category, the numerical category and the target text recognition information corresponding to the numerical category.

17. The method according to claim 1, wherein The text image includes a medical text image.

18. An information processing method, comprising: Processing the text image to be processed using the method according to any one of claims 1 to 17 to obtain structural information of the text image to be processed; as well as Information processing is performed using the structured information of the text image to be processed.

19. An information generating device, comprising: a text detection module, configured to perform text detection on a text image to obtain detection information, wherein the detection information includes category information and position information of each of a plurality of text regions; A first acquisition module is configured to acquire a text region image corresponding to each of the plurality of text regions according to the position information and the text image; a text recognition module, configured to perform text recognition on the text region image to obtain recognition information, wherein the recognition information includes text recognition information of each of the plurality of text region images; a global feature information determination module, configured to perform U-level processing on the recognition information based on a self-attention strategy to obtain global feature information, where U is an integer greater than or equal to 1; a semantic relationship determination module, configured to determine semantic relationship information based on the global feature information; and a generating module, configured to generate structured information of the text image based on the category information, the semantic relationship information, and the identification information; The global feature information determination module is used to determine, under the condition of 1<u≤U, a plurality of first matrix sets corresponding to the plurality of text recognition information at the u-1th level according to the first intermediate feature information corresponding to the plurality of text recognition information, wherein the first matrix set includes a first query matrix, a first key matrix and a first value matrix; for the text recognition information at the u-th level, a first attention matrix is ​​obtained according to the first query matrix corresponding to the text recognition information and the first key matrix corresponding to the plurality of text recognition information, and a first value matrix corresponding to the text recognition information is obtained according to the first attention matrix and the first value matrix corresponding to the text recognition information. According to the third intermediate feature information corresponding to the text recognition information, the second intermediate feature information corresponding to the text recognition information is obtained according to the multiple third intermediate feature information; according to the second intermediate feature information corresponding to the multiple text recognition information at the u-th level and the first intermediate feature information corresponding to the multiple text recognition information at the u-1-th level, the first intermediate feature information corresponding to the multiple text recognition information at the u-th level is obtained; and according to the first intermediate feature information corresponding to the multiple text recognition information at the R-th level, the global feature information is obtained; wherein u is an integer greater than or equal to 1 and less than or equal to U, and R is an integer greater than or equal to 1 and less than or equal to U.

20. An information processing device comprising: a second acquisition module, configured to process the text image to be processed using the apparatus according to claim 19, and acquire structural information of the text image to be processed; as well as The information processing module is used to perform information processing using the structured information of the text image to be processed.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 17 or claim 18.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 17 or claim 18.

23. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 17 or claim 18.

Images