Information processing method, apparatus and electronic device
By extracting image and text features, enhancing the receptive field of the dilated convolution module, and fusing features using a fusion model, the problem of inaccurate feature recognition for long input sequences in existing technologies has been solved, and accurate recognition of the target being processed has been achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2023-05-30
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies cannot effectively utilize positional encoding for accurate identification of targets, especially for features of long input sequences, thus affecting the content recognition results.
Features are extracted by the image feature extraction module and the text recognition module respectively. The receptive field is increased by the dilated convolution module to determine the location information of the features. The features are then fused by the fusion model to generate accurate target feature information.
It achieves accurate recognition of features containing long input sequences, improves the accuracy of image and text feature extraction, and can determine the accurate content of the processing target.
Smart Images
Figure CN116721258B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent image and text recognition, and in particular to an information processing method, apparatus, and electronic device. Background Technology
[0002] In the field of intelligent image text recognition, multimodal models based on natural language processing and image understanding are developing rapidly. These models require combining image, text, and tabular information to understand and recognize the target text (such as scanned documents or PDF files). However, current methods typically use fixed-length position embeddings of features within the target text for recognition. This approach cannot recognize features with long input sequences, thus affecting the accuracy of the content recognition results. Summary of the Invention
[0003] An information processing method according to an embodiment of this application includes:
[0004] The image feature extraction module and the text recognition module extract image features and text features from the target object respectively, generating the corresponding first image features and first text features;
[0005] Based on the dilated convolution module, the position information of the features in the first image features and the features in the first text features are determined respectively, and the receptive field corresponding to the first image features and the receptive field corresponding to the first text features are increased. The receptive field is used to characterize the extraction range of the features in the first image features, and the position information includes relative position information and absolute position information.
[0006] Based on the location information, the dilated convolution module is used to extract features from the first image feature and the first text feature with the increased receptive field, respectively, to generate the second image feature and the second text feature.
[0007] The second text feature and the second image feature are fused using a fusion model to form target feature information.
[0008] Optionally, the dilated convolution module includes zero-padding units, and before performing feature extraction on the first image features with increased receptive fields and the first text features, the method further includes:
[0009] Based on the location information, the edge portion of the first image feature is increased using the zero-padding unit to avoid losing effective information in the first image feature when performing convolution operations on the first image feature.
[0010] Optionally, the step of extracting image features and text features from the target object through an image feature extraction module and a text recognition module, respectively, includes:
[0011] The image feature extraction module extracts image features from the target at multiple image scales, generating multiple image sub-features.
[0012] The first image feature is determined based on multiple image sub-features.
[0013] Optionally, the method of determining the positional information of features in the first image features and features in the first text features based on the dilated convolution module includes:
[0014] Transform the first image features into a two-dimensional plane;
[0015] The location information of features in the first image features is determined by using the two-dimensional convolution in the dilated convolution module.
[0016] Optionally, increasing the receptive field corresponding to the first image feature includes:
[0017] Determine the pixel set of features in the first image features;
[0018] Based on the number of holes corresponding to the pixel set, the receptive field corresponding to the feature in the first image feature is expanded.
[0019] Alternatively, the method may further include:
[0020] Based on the demand instructions, a multilayer perceptron is used to perform corresponding demand processing on the target feature information, wherein the demand processing includes feature classification processing and feature recognition processing.
[0021] Alternatively, the method may further include:
[0022] The fusion model is trained using a training dataset, wherein the training dataset includes text feature training data corresponding to the second text feature and image feature training data corresponding to the second image feature.
[0023] Optionally, the dilated convolution module is connected to the image feature extraction module and the text recognition module respectively to receive the first image features and the first text features respectively; the fusion model is connected to the multilayer perceptron and the dilated convolution module respectively, and the fusion model is used to send the target feature information to the multilayer perceptron.
[0024] This application also provides an information processing apparatus, including:
[0025] The extraction module is configured to extract image features and text features from the processing target through the image feature extraction module and the text recognition module, respectively, and generate corresponding first image features and first text features;
[0026] The processing module is configured to, based on a dilated convolution module, determine the positional information of features in the first image features and features in the first text features, respectively, increase the receptive field corresponding to the first image features and the receptive field corresponding to the first text features, wherein the receptive field is used to characterize the extraction range of features in the first image features, and the positional information includes relative positional information and absolute positional information; based on the positional information, the dilated convolution module is used to extract features from the first image features and the first text features with increased receptive fields, respectively, to generate second image features and second text features.
[0027] The fusion module is configured to use a fusion model to fuse the second text features and the second image features to form target feature information.
[0028] This application also provides an electronic device, including a processor and a memory, wherein the memory stores an executable program, and the memory executes the executable program to perform the steps of the method described above.
[0029] The information processing method of this application embodiment can identify processing targets containing relatively complex image and text features separately. It uses dilated convolution to identify the image features and text features of the processing target separately. Even for features with long input sequences, it can accurately identify them and thus determine the accurate content of the processing target. Attached Figure Description
[0030] Figure 1 This is a flowchart of an information processing method according to an embodiment of this application;
[0031] Figure 2 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S100;
[0032] Figure 3 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S200;
[0033] Figure 4 Examples of embodiments of this application Figure 1 A flowchart of another embodiment of step S200;
[0034] Figure 5This is a schematic diagram illustrating the process of feature extraction for the first text feature and the first image feature according to an embodiment of this application;
[0035] Figure 6 This is a schematic diagram illustrating the difference between the dilated convolution method and the general convolution method in this application.
[0036] Figure 7 This is a schematic diagram of the Zero-padding process according to an embodiment of this application;
[0037] Figure 8 This is a flowchart of a specific embodiment of the information processing method according to this application.
[0038] Figure 9 This is a structural block diagram of an information processing apparatus according to an embodiment of this application. Detailed Implementation
[0039] Various embodiments and features of this application are described herein with reference to the accompanying drawings.
[0040] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.
[0041] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0042] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.
[0043] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.
[0044] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.
[0045] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.
[0046] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.
[0047] An information processing method according to an embodiment of this application can be applied to an electronic device. This method can accurately identify the content of a processing target, which can be a file in various forms, such as an electronic file containing images and / or text. The method includes feature extraction of the processing target to extract corresponding first image features and first text features.
[0048] Dilated convolution, also known as dilated convolution or dilated convolution, expands the convolution kernel by adding spaces between its elements. In this embodiment, based on this positional information, the dilated convolution module (model) can increase the receptive field corresponding to the first image feature and the first text feature. This improves the processing accuracy and intelligence of the dilated convolution module. Consequently, it increases the accuracy of semantic recognition and extraction when extracting features from the first image feature and the first text feature.
[0049] Furthermore, this method, based on a dilated convolution module (model), determines the positional information of features in the first image features and features in the first text features, respectively. The positional information can be the relative or absolute position of the feature within the entire first image features or first text features. This allows for accurate feature extraction from the first image features and first text features, which have increased receptive fields, resulting in more accurate second text features and second image features. The second text features and second image features are then fused to form target feature information, which can be output as needed, such as feature classification output. This target feature information is the feature information of the target being processed.
[0050] The method will now be explained in detail with reference to the accompanying drawings. Figure 1 This is a flowchart of the information processing method according to an embodiment of this application, such as... Figure 1 As shown and combined Figure 8 The method includes the following steps:
[0051] S100: The image feature extraction module and the text recognition module extract image features and text features from the target to be processed, respectively, and generate corresponding first image features and first text features.
[0052] For example, the image feature extraction module can be built based on hardware and / or software to extract image features from the processing target and generate corresponding first image features. In one embodiment, the first image feature can be the main image information of the processing target, such as signatures, identification information, anti-counterfeiting images, etc., when the processing target is an electronic invoice. In another embodiment, multiple image features of the processing target at different scales can be obtained, including local features, global features, etc., wherein the scale of the processing target is related to the resolution of the processing target. For example, if the processing target is an electronic scanned document, the image feature extraction module can extract image features of the electronic scanned document for different resolutions.
[0053] The text recognition module can also be built based on hardware and / or software to extract text features from the processing target. These features can be the main text-related content within the processing target, such as core words in a sentence. For example, when the processing target is a scanned electronic document, the text recognition module can extract features from the text-related content (including content appearing as graphics) in the scanned document to generate first text features. This can be achieved by using OCR technology to recognize the text and generate the corresponding first text features. These first text features can be the main text-related features within the processing target, such as core words, the name of the processing target, or date information.
[0054] S200, based on the dilated convolution module, determine the position information of the features in the first image features and the features in the first text features respectively, increase the receptive field corresponding to the first image features and the receptive field corresponding to the first text features, wherein the receptive field is used to characterize the extraction range of the features in the first image features, and the position information includes relative position information and absolute position information.
[0055] For example, in combination Figure 6 The dilated convolution module can process first image features and first text features using dilated convolution. The principle of dilated convolution is to add holes to a standard convolution to increase the receptive field. Compared to normal convolution, it uses the hyperparameter: dilation rate (the number of kernel intervals). Dilated convolution expands the convolution kernel by adding spaces between kernel elements based on the "dilation rate" hyperparameter.
[0056] In this embodiment, a dilated convolution module is used to increase the receptive field corresponding to the first image feature and the receptive field corresponding to the first text feature, respectively. The receptive field is used to characterize the extraction range of features in the first image feature.
[0057] For example, in one embodiment, in a convolutional neural network, the receptive field can be the size of the region mapped onto the input image by the pixels on the feature map output by each layer of the convolutional neural network. This region size can characterize the extraction range of features from the feature map. In another embodiment, adding dilated convolutions can lead to poor capture of long-distance relationships in text sentences. For example, in the sentence "She is a very beautiful girl," "she" and "girl" are the subject and object, respectively, and their relationship is very important. However, ordinary convolutions in the dilated convolution module (model) may not capture such long-distance information. Therefore, we add dilated convolutions instead of standard convolutions, which increases the receptive field of the dilated convolution module without increasing the computational cost, thereby accurately identifying the central main words and further improving processing performance.
[0058] In this embodiment, a dilated convolution module is used to increase the receptive field corresponding to the first image feature and the receptive field corresponding to the first text feature, respectively. This expands the extraction range of features in the first image feature and features in the first text feature, avoiding omissions during feature extraction.
[0059] In some embodiments, based on a dilated convolution module, the positional information of features in the first image features and features in the first text features is determined respectively. This positional information can be the position of a feature within the first image features (as overall data), including relative and absolute positional information. The positional information can be represented by positional encoding; relative positional information can be the position of the feature relative to other features, while absolute positional information can be the overall position of the feature within the entire first image features. Similarly, this positional information can also be the position of a feature within the first text features (as overall data), including relative and absolute positional information. The positional information can be represented by positional encoding; relative positional information can be the position of the feature relative to other features, while absolute positional information can be the overall position of the feature within the entire first text features. This positional information can accurately characterize the location of each feature, thereby enabling accurate extraction of each feature based on this positional information.
[0060] S300, based on the location information, the dilated convolution module is used to extract features from the first image features and the first text features with increased receptive fields, respectively, to generate second image features and second text features.
[0061] For example, by increasing the receptive field of both the first image feature and the first text feature, the extraction range of the corresponding features can be increased when extracting features from the first image feature and the first text feature, thereby avoiding feature omission. Furthermore, the location information can determine the position of each feature in the first image feature among multiple related features, as well as the absolute position of each feature within the entire first image feature, thereby increasing the accuracy of feature extraction and avoiding extraction errors.
[0062] Next, feature extraction is performed on the first image features to generate the second image features. Similarly, feature extraction is performed on the first text features to generate the second text features. The second image features can represent the main image feature dataset of the target being processed, while the second text features can represent the main text feature dataset of the target being processed.
[0063] S400, using a fusion model, the second text features and the second image features are fused to form target feature information.
[0064] For example, the fusion model can be Transformer Layers, which are used to fuse different types of feature data to obtain accurate key features of the processing target, thereby determining the target feature information of the accurate content contained in the processing target.
[0065] For example, the processing target is an electronic invoice, which contains image features and text features. After processing the electronic invoice using the information processing methods described above, the following are obtained: second image features representing the main image features of the electronic invoice, such as signature features and anti-counterfeiting image features; and second text features representing the main text features of the electronic invoice, such as name features and date features. The second text features and the second image features are then fused using Transformer Layers to form target feature information that accurately represents the content of the electronic invoice.
[0066] The information processing method of this application embodiment can identify processing targets containing relatively complex image and text features separately. It uses dilated convolution to identify the image features and text features of the processing target separately. Even for features with long input sequences, it can accurately identify them and thus determine the accurate content of the processing target.
[0067] In one embodiment of this application, the dilated convolution module includes zero-padding units. Before performing feature extraction on the first image features with increased receptive fields and the first text features, the method further includes the following steps:
[0068] Based on the location information, the edge portion of the first image feature is increased using the zero-padding unit to avoid losing effective information in the first image feature when performing convolution operations on the first image feature.
[0069] For example, such as Figure 7 As shown, the first image feature can be represented in the form of multiple pixels (data blocks). By using zero-padding units to add edges to the first image feature, the integrity of the first image feature can be guaranteed, especially during the feature extraction process, where no feature is lost. Furthermore, the zero-padding units themselves do not contain any actual valid information, thus they do not disturb the content of the first image feature.
[0070] For example, combining Figure 5 After extracting image features (visual tokens) and text features (text tokens) from the target object, the Zero-padding algorithm is applied to the first image features to enhance their edges. Zero-padding is a technique that preserves the original input size by adding zeros at the beginning and end of both the horizontal and vertical axes of the original first image features, thus protecting their edge information. This processed first image feature avoids losing valuable information during convolution operations. Similarly, the Zero-padding algorithm can also be used to enhance the edges of the first text features.
[0071] In one embodiment of this application, the image feature extraction module and the text recognition module respectively extract image features and text features from the processing target, such as... Figure 2 As shown, it includes the following steps:
[0072] S110, The image feature extraction module extracts image features from the processing target at multiple image scales to generate multiple image sub-features.
[0073] For example, image scale can represent the resolution of an image. The content of image sub-features may differ at different resolutions, and the emphasis of each sub-feature may also differ. In this embodiment, image features can be extracted from the processing target at multiple image scales. For example, feature extraction is performed on the processing target at a resolution of 24*24 to generate a first image sub-feature; feature extraction is performed on the processing target at a resolution of 128*128 to generate a second image sub-feature. The first and second image sub-features have different emphases in their represented feature content.
[0074] S120, the first image feature is determined based on the plurality of image sub-features.
[0075] For example, a portion of each of the multiple image sub-features can be selected to determine the first image feature; alternatively, a weighted calculation can be performed on each image sub-feature to determine the first image feature. This allows the advantages of each image sub-feature to be utilized, thereby improving the accuracy of the first image feature.
[0076] In one embodiment of this application, the dilated convolution module determines the positional information of features in the first image features and features in the first text features, respectively. Figure 3 As shown, it includes:
[0077] S210, transform the first image features into a two-dimensional plane.
[0078] For example, the first image feature is easily processed in a two-dimensional plane, including determining its location information and increasing its receptive field. In one embodiment, the first image feature is represented by multiple pixels, and the corresponding pixel set is transformed into a preset two-dimensional plane. The feature extraction operation for the first image feature can be specifically implemented through related operations on each pixel.
[0079] S220, using the two-dimensional convolution in the dilated convolution module, the position information of the features in the first image features is determined.
[0080] For example, a two-dimensional convolution (two-dimensional convolutional layer) in a dilated convolution module can extract specific feature information by sliding a small matrix called a convolution kernel or filter onto the input image. In a two-dimensional convolutional layer, the convolution kernel is a two-dimensional matrix that is multiplied and summed element-wise with the input image to obtain the output feature map. In this embodiment, the two-dimensional convolution in the dilated convolution module is used to determine the positional information of features in the first image features, and this positional information can be represented by positional encoding.
[0081] In one embodiment of this application, such as Figure 4 As shown and combined Figure 6 The step of increasing the receptive field corresponding to the first image feature includes the following steps:
[0082] S230, determine the pixel set of features in the first image features;
[0083] S240, based on the number of holes corresponding to the pixel set, expand the receptive field corresponding to the feature in the first image feature.
[0084] For example, the features in the first image feature can be represented based on pixels, and all the pixels corresponding to the features can form a corresponding pixel set. For example, this pixel set is constructed of 3*3 pixels, and this pixel set can represent the 9 features in the first image feature.
[0085] The pixel set has a corresponding number of holes. After determining the pixel set, a certain number of holes can be inserted into the pixel set, for example, one hole can be inserted between every two adjacent pixels in the pixel set, thereby expanding the receptive field corresponding to the feature in the first image feature.
[0086] For example, if the pixel set is constructed from 3*3 pixels, and a hole is inserted between every two adjacent pixels in the pixel set, including both forward and diagonal adjacent pixels, the receptive field is expanded from the original 3*3 to 5*5.
[0087] In one embodiment of this application, the method further includes the following steps: based on the demand instruction, using a multilayer perceptron to perform corresponding demand processing on the target feature information, wherein the demand processing includes feature classification processing and feature recognition processing.
[0088] For example, target feature information can characterize the precise content contained in the processing target. User requirements may not solely determine this target feature information; they may also output content related to this target feature information based on actual needs.
[0089] For example, the first instruction represents feature classification and / or feature recognition of the target being processed. Upon receiving the first instruction, a multilayer perceptron (MLP) is used to process the target feature information in accordance with the requirements of the first instruction, that is, to perform feature classification and / or feature recognition processing on the target feature information, and then output the processed target feature information to meet the user's needs in the actual use scenario.
[0090] In one embodiment of this application, the method further includes the following steps: training the fusion model using a training dataset, wherein the training dataset includes text feature training data corresponding to the second text feature and image feature training data corresponding to the second image feature.
[0091] For example, the fusion model can be Transformer Layers, which can be built on a neural network to fuse multiple different types of data. In one embodiment, Transformer Layers include encoders and decoders, with the encoder implemented by stacking Encoder layers and the decoder implemented by stacking Decoder layers. The number of Encoder and Decoder layers can be set according to the specific task.
[0092] The fusion model can be trained both before and during its use to improve its fusion accuracy. Specifically, this involves training the fusion model using a training dataset to determine its parameters. The training dataset includes training data for text features corresponding to the second text features and training data for image features corresponding to the second image features. This improves processing efficiency and accuracy when fusing the second image and text features using the fusion model.
[0093] In one embodiment of this application, the dilated convolution module is connected to the image feature extraction module and the text recognition module respectively, so as to receive the first image feature and the first text feature respectively; the fusion model is connected to the multilayer perceptron and the dilated convolution module respectively, and the fusion model is used to send the target feature information to the multilayer perceptron.
[0094] For example, relevant functional modules can be pre-built on the information processing device or electronic device corresponding to this information processing method to achieve information processing of the target. The dilated convolution module (model) is connected to the image feature extraction module and the text recognition module, respectively. The image feature extraction module extracts image features from the target, while the text recognition module extracts text features, generating corresponding first image features and first text features. The image feature extraction module sends the first image features to the dilated convolution module, and the text feature extraction module sends the first text features to the dilated convolution module. The dilated convolution module determines the position information of the features in the first image features and the features in the first text features, respectively, increasing the receptive field corresponding to the first image features and the receptive field corresponding to the first text features. Based on the position information, the dilated convolution module extracts features from the first image features and the first text features with increased receptive fields, respectively, generating second image features and second text features, and sends the second image features and second text features to the fusion model. The fusion model performs a fusion operation on the second image features and the second text features to generate target feature information. The fusion model sends the target feature information to a multilayer perceptron (MLP). The MLP processes the target feature information based on user needs and outputs the result, which can then be used by downstream tasks.
[0095] Based on the same inventive concept, embodiments of this application also provide an information processing device, which can be applied to electronic devices, such as... Figure 9 As shown and combined Figure 8 The device includes:
[0096] The extraction module is configured to extract image features and text features from the processing target through the image feature extraction module and the text recognition module, respectively, and generate corresponding first image features and first text features.
[0097] For example, the image feature extraction module can be built based on hardware and / or software to extract image features from the processing target and generate corresponding first image features. In one embodiment, the first image feature can be the main image information of the processing target, such as signatures, identification information, anti-counterfeiting images, etc., when the processing target is an electronic invoice. In another embodiment, the extraction module can acquire multiple image features of the processing target at different scales, including local features, global features, etc., where the scale of the processing target is related to the resolution of the processing target. For example, if the processing target is an electronic scanned document, the image feature extraction module can extract image features of the electronic scanned document for different resolutions.
[0098] The text recognition module can also be built based on hardware and / or software to extract text features from the processing target. These features can be the main text-related content within the processing target, such as core words in a sentence. For example, when the processing target is a scanned electronic document, the text recognition module can extract features from the text-related content (including content appearing as graphics) in the scanned document to generate first text features. This can be achieved by using OCR technology to recognize the text and generate the corresponding first text features. These first text features can be the main text-related features within the processing target, such as core words, the name of the processing target, or date information.
[0099] The processing module is configured to, based on a dilated convolution module, determine the positional information of features in the first image features and features in the first text features, respectively, and increase the receptive field corresponding to the first image features and the receptive field corresponding to the first text features. The receptive field is used to characterize the extraction range of features in the first image features, and the positional information includes relative positional information and absolute positional information. Based on the positional information, the dilated convolution module is used to extract features from the first image features and the first text features with increased receptive fields, respectively, to generate second image features and second text features.
[0100] For example, a dilated convolution module can process first image features and first text features using dilated convolution. The principle of dilated convolution is to add holes to a standard convolution to increase the receptive field. Compared to normal convolution, it uses the hyperparameter: dilation rate (the number of kernel intervals). Dilated convolution expands the convolution kernel by adding spaces between kernel elements based on the "dilation rate" hyperparameter.
[0101] In this embodiment, the processing module is based on a dilated convolution module, which uses dilated convolution to increase the receptive field corresponding to the first image feature and the receptive field corresponding to the first text feature, respectively. The receptive field is used to characterize the extraction range of the features in the first image feature.
[0102] For example, in one embodiment, in a convolutional neural network, the receptive field can be the size of the region mapped onto the input image by the pixels on the feature map output by each layer of the convolutional neural network. This region size can characterize the extraction range of features from the feature map. In another embodiment, adding dilated convolutions can lead to poor capture of long-distance relationships in text sentences. For example, in the sentence "She is a very beautiful girl," "she" and "girl" are the subject and object, respectively, and their relationship is very important. However, ordinary convolutions in the dilated convolution module (model) may not capture such long-distance information. Therefore, we add dilated convolutions instead of standard convolutions, which increases the receptive field of the dilated convolution module without increasing the computational cost, thereby accurately identifying the central main words and further improving processing performance.
[0103] In this embodiment, the processing module uses a dilated convolution module to increase the receptive field corresponding to the first image feature and the receptive field corresponding to the first text feature, respectively. This expands the extraction range of features in the first image feature and features in the first text feature, avoiding omissions during feature extraction.
[0104] In some embodiments, the processing module, based on a dilated convolution module, determines the positional information of features in the first image features and features in the first text features, respectively. This positional information can be the position of a feature within the first image features (as overall data), including relative and absolute positional information. The positional information can be represented by positional encoding; relative positional information can be the position of the feature relative to other features, while absolute positional information can be the overall position of the feature within the entire first image features. Similarly, this positional information can also be the position of a feature within the first text features (as overall data), including relative and absolute positional information. The positional information can be represented by positional encoding; relative positional information can be the position of the feature relative to other features, while absolute positional information can be the overall position of the feature within the entire first text features. This positional information accurately characterizes the location of each feature, thereby enabling accurate extraction of each feature based on this positional information.
[0105] By increasing the receptive field of both the first image feature and the first text feature, the extraction range of the corresponding features can be expanded during feature extraction, thereby avoiding feature omissions. Furthermore, the location information can determine the position of each feature in the first image feature among multiple related features, as well as the absolute position of each feature within the entire first image feature, thus increasing the accuracy of feature extraction and avoiding extraction errors.
[0106] Then, feature extraction is performed on the first image features to generate the second image features. Similarly, feature extraction is performed on the first text features to generate the second text features. The second image features can represent the main image feature dataset of the target being processed, while the second text features can represent the main text feature dataset of the target being processed.
[0107] The fusion module is configured to use a fusion model to fuse the second text features and the second image features to form target feature information.
[0108] For example, the fusion model can be Transformer Layers, which are used to fuse different types of feature data to obtain accurate key features of the processing target, thereby determining the target feature information of the accurate content contained in the processing target.
[0109] For example, the processing target is an electronic invoice, which contains image features and text features. After processing the electronic invoice using the information processing methods described above, the following are obtained: second image features representing the main image features of the electronic invoice, such as signature features and anti-counterfeiting image features; and second text features representing the main text features of the electronic invoice, such as name features and date features. The fusion module fuses the second text features and the second image features through Transformer Layers to form target feature information that can accurately represent the content of the electronic invoice.
[0110] In one embodiment of this application, the dilated convolution module includes zero-padding units. Before performing feature extraction on the first image features with increased receptive fields and the first text features, the processing module is further configured as follows:
[0111] Based on the location information, the edge portion of the first image feature is increased using the zero-padding unit to avoid losing effective information in the first image feature when performing convolution operations on the first image feature.
[0112] In one embodiment of this application, the extraction module is further configured as follows:
[0113] The image feature extraction module extracts image features from the target at multiple image scales, generating multiple image sub-features.
[0114] The first image feature is determined based on multiple image sub-features.
[0115] In one embodiment of this application, the processing module is further configured as follows:
[0116] Transform the first image features into a two-dimensional plane;
[0117] The location information of features in the first image features is determined by using the two-dimensional convolution in the dilated convolution module.
[0118] In one embodiment of this application, the processing module is further configured as follows:
[0119] Determine the pixel set of features in the first image features;
[0120] Based on the number of holes corresponding to the pixel set, the receptive field corresponding to the feature in the first image feature is expanded.
[0121] In one embodiment of this application, the device further includes an output module, which is configured to:
[0122] Based on the demand instructions, a multilayer perceptron is used to perform corresponding demand processing on the target feature information, wherein the demand processing includes feature classification processing and feature recognition processing.
[0123] In one embodiment of this application, the device further includes a training module, which is configured as follows:
[0124] The fusion model is trained using a training dataset, wherein the training dataset includes text feature training data corresponding to the second text feature and image feature training data corresponding to the second image feature.
[0125] In one embodiment of this application, the dilated convolution module is connected to the image feature extraction module and the text recognition module respectively to receive the first image feature and the first text feature respectively; the fusion model is connected to the multilayer perceptron and the dilated convolution module respectively, and the fusion model is used to send the target feature information to the multilayer perceptron.
[0126] This application also provides an electronic device, including a processor and a memory, wherein the memory stores an executable program, and the memory executes the executable program to perform the steps of the method described above.
[0127] The aforementioned processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0128] The aforementioned memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0129] This application also provides a storage medium that carries one or more computer programs, which, when executed by a processor, implement the steps of the method described above.
[0130] The storage medium in this embodiment may be included in an electronic device / system; or it may exist independently and not assembled into an electronic device / system. The storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.
[0131] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0132] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.
Claims
1. An information processing method, comprising: The image feature extraction module and the text recognition module extract image features and text features from the target object respectively, generating the corresponding first image features and first text features; Based on the dilated convolution module, the position information of the features in the first image features and the position information of the features in the first text features are determined respectively, and the receptive field corresponding to the first image features and the receptive field corresponding to the first text features are increased. The receptive field is used to characterize the extraction range of the features in the first image features, and the position information includes relative position information and absolute position information. Based on the positional information of features in the first image features and the positional information of features in the first text features, the dilated convolution module is used to extract features from the first image features and the first text features with increased receptive fields, respectively, to generate second image features and second text features, including extracting features from the first image features based on the relative positional information and / or absolute positional information of each feature in the first image features, and extracting features from the first text features based on the relative positional information and absolute positional information of each feature in the first text features; Using a fusion model, the second text features and the second image features are fused to form target feature information; wherein, The method of determining the position information of features in the first image features and the position information of features in the first text features based on the dilated convolution module includes: converting the first image features into a two-dimensional plane; and using the two-dimensional convolution in the dilated convolution module to determine the position information of features in the first image features.
2. The method according to claim 1, wherein the dilated convolution module includes zero-padding units, and before performing feature extraction on the first image features with increased receptive fields and the first text features respectively, the method further includes: Based on the location information, the edge portion of the first image feature is increased using the zero-padding unit to avoid losing effective information in the first image feature when performing convolution operations on the first image feature.
3. The method according to claim 1, wherein the step of extracting image features and text features from the processing target through the image feature extraction module and the text recognition module respectively includes: The image feature extraction module extracts image features from the target at multiple image scales, generating multiple image sub-features. The first image feature is determined based on multiple image sub-features.
4. The method according to claim 1, wherein increasing the receptive field corresponding to the first image feature comprises: Determine the pixel set of features in the first image features; Based on the number of holes corresponding to the pixel set, the receptive field corresponding to the feature in the first image feature is expanded.
5. The method according to claim 1, further comprising: Based on the demand instructions, a multilayer perceptron is used to perform corresponding demand processing on the target feature information, wherein the demand processing includes feature classification processing and feature recognition processing.
6. The method according to claim 1, further comprising: The fusion model is trained using a training dataset, wherein the training dataset includes text feature training data corresponding to the second text feature and image feature training data corresponding to the second image feature.
7. The method according to claim 5, wherein, The dilated convolution module is connected to the image feature extraction module and the text recognition module respectively, so as to receive the first image feature and the first text feature respectively; The fusion model is connected to the multilayer perceptron and the dilated convolution module respectively, and the fusion model is used to send the target feature information to the multilayer perceptron.
8. An information processing apparatus, comprising: The extraction module is configured to extract image features and text features from the processing target through the image feature extraction module and the text recognition module, respectively, and generate corresponding first image features and first text features; The processing module is configured to, based on a dilated convolution module, determine the positional information of features in the first image feature and the positional information of features in the first text feature, respectively, increase the receptive field corresponding to the first image feature and the receptive field corresponding to the first text feature, wherein the receptive field is used to characterize the extraction range for features in the first image feature, and the positional information includes relative positional information and absolute positional information; based on the positional information of features in the first image feature and the positional information of features in the first text feature, the dilated convolution module is used to extract features from the first image feature and the first text feature with increased receptive fields, respectively, to generate second image features and second text features, respectively. This includes extracting features from the first image feature based on the relative positional information and / or absolute positional information of each feature in the first image feature, and extracting features from the first text feature based on the relative positional information and absolute positional information of each feature in the first text feature. The fusion module is configured to use a fusion model to fuse the second text features and the second image features to form target feature information; wherein, The processing module is further configured to: convert the first image features into a two-dimensional plane; and use the two-dimensional convolution in the dilated convolution module to determine the position information of the features in the first image features.
9. An electronic device comprising a processor and a memory, the memory storing an executable program, the memory executing the executable program to perform the steps of the method as claimed in any one of claims 1 to 7.