Method, device and equipment for dynamically extracting features, and method for recognizing characters

Abstract

The method for dynamically extracting features, the character recognition method and the device provided by the application, the method for dynamically extracting features comprises: extracting image features of a to-be-processed image, and obtaining the width and height of a target bounding box in the to-be-processed image; sampling in the image features based on the width and height of the target bounding box, determining a plurality of image content features of the target bounding box; and determining a region feature of the target bounding box based on the image content features and the corresponding region position codes. The application realizes dynamic extraction of image features based on the information of the width and height of the target bounding box, that is, even if the difference between the width and height of the target bounding box is large, the image features of the target bounding box can be uniformly sampled, the image details are retained, and the accuracy of target recognition based on the extracted image features can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of feature extraction, in particular to a method for dynamically extracting features, a character recognition method and equipment. BACKGROUND

[0002] In feature extraction of a target region in an image, a model is usually required to divide the target region to extract image features of the target region. In the prior art, a fixed-size grid division method is usually adopted, but the width-to-height ratio of the target region is not fixed, for example, some are close to square region frames, some are vertical rectangular region frames, and some are horizontal rectangular region frames. When the number of rows and columns is basically the same in current grid division, when the width and height of the target region are greatly different (for example, the target region is a narrow or narrow region), the image features are not uniformly sampled in the grid region after the target region is divided, and the details of the target object in the image are easily lost, resulting in low accuracy of target recognition based on the extracted image features.

[0003] Therefore, the prior art has defects and needs to be improved and developed. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a method for dynamically extracting features, a character recognition method and equipment aiming at the above-mentioned defects of the prior art, to solve the problem that when the width and height of the target region are greatly different, the image features are not uniformly sampled in the grid region after the target region is divided, the details of the target object in the image are easily lost, and the accuracy of target recognition based on the extracted image features is low.

[0005] The technical solution adopted by the present application to solve the technical problem is as follows:

[0006] A method for dynamically extracting features, wherein the method for dynamically extracting features comprises:

[0007] extracting image features of a to-be-processed image, and obtaining the width and height of a target bounding box in the to-be-processed image;

[0008] sampling in the image features based on the width and height of the target bounding box, to determine a plurality of image content features of the target bounding box;

[0009] determining a region feature of the target bounding box based on the image content features and their corresponding region position encodings.

[0010] In an embodiment of the present application, sampling in the image features based on the width and height of the target bounding box to determine a plurality of image content features of the target bounding box comprises:

[0011] determine a sampling size based on the width and the height of the target bounding box, and perform grid division on the target bounding box based on the sampling size to obtain a preset number of grid parameter information;

[0012] sample in the image feature based on the grid parameter information to determine an image content feature of each grid.

[0013] In an embodiment of the present application, determining the sampling size based on the width and the height of the target bounding box comprises:

[0014] obtain a preset standard size list, wherein the standard size list comprises a plurality of standard sizes;

[0015] calculate an aspect ratio of the target bounding box according to the width and the height of the target bounding box, and calculate a difference value between each standard size and the aspect ratio of the target bounding box, and take the standard size with the minimum difference value as the sampling size.

[0016] In an embodiment of the present application, determining the sampling size based on the width and the height of the target bounding box comprises:

[0017] calculate an aspect ratio of the target bounding box according to the width and the height of the target bounding box, and determine the sampling size based on the aspect ratio of the target bounding box and the preset number of grids.

[0018] In an embodiment of the present application, the step of obtaining the region position code comprises:

[0019] input the preset number of grid parameter information into a pre-constructed learnable position coding module to obtain a region position code corresponding to the image content feature of each grid.

[0020] In an embodiment of the present application, the grid parameter information comprises a grid horizontal coordinate, a grid vertical coordinate, a grid width, a grid height and a grid area.

[0021] In an embodiment of the present application, when the grid corresponding to the image content feature is located in the last column, the region position code further comprises a row separator.

[0022] In an embodiment of the present application, determining the region feature of the target bounding box based on the image content feature and the region position code corresponding thereto comprises:

[0023] add the region position code and the image content feature to obtain the region feature of the target bounding box.

[0024] The present application also provides a character recognition method, wherein the character recognition method comprises:

[0025] Input the region feature as described above into the trained large language model to obtain a character recognition result.

[0026] In an embodiment of the present application, under the guidance of the task token prompt, the large language model outputs a character recognition result based on the region feature.

[0027] In an embodiment of the present application, the training step of the trained large language model comprises:

[0028] Obtain a training data set, which comprises a plurality of training images and character information labeled for each training image;

[0029] Extract training image features of the training images, and obtain the width and height of a target bounding box in the training images;

[0030] Sample in the training image features based on the width and height of the target bounding box to determine a plurality of training image content features;

[0031] Determine the recognition character features of the target bounding box based on the training image content features and their corresponding region position encodings;

[0032] Train a pre-set large language model based on the recognition character features and the character information labeled for the training images to obtain a trained large language model.

[0033] The present application also provides a feature extraction device, wherein the device comprises:

[0034] An image feature extraction module for extracting image features of an image to be processed;

[0035] A target bounding box acquisition module for acquiring the width and height of a target bounding box in an image to be processed;

[0036] A sampling module for sampling in the image features based on the width and height of the target bounding box to determine a plurality of image content features of the target bounding box;

[0037] A feature output module for outputting the region features of the target bounding box based on the image content features and their corresponding region position encodings.

[0038] In an embodiment of the present application, the sampling module comprises:

[0039] A grid division unit for determining a sampling size based on the width and height of the target bounding box, and performing grid division on the target bounding box based on the sampling size to obtain a pre-set number of grid parameter information;

[0040] A sampling unit is configured to sample the image features based on the grid parameter information to determine image content features of each grid.

[0041] The application further provides an electronic device, comprising a memory, a processor, and a feature extraction program stored in the memory and executable on the processor, wherein the feature extraction program, when executed by the processor, implements the method for dynamically extracting features and / or the steps of the method for recognizing characters.

[0042] The application further provides a computer-readable storage medium storing a computer program executable for implementing the method for dynamically extracting features and / or the steps of the method for recognizing characters.

[0043] The method for dynamically extracting features, the method for recognizing characters, and the device provided by the application, the method for dynamically extracting features comprises: extracting image features of a to-be-processed image, and obtaining a width and a height of a target bounding box in the to-be-processed image; sampling the image features based on the width and the height of the target bounding box to determine a plurality of image content features of the target bounding box; and determining a region feature of the target bounding box based on the image content features and corresponding region positions. The application realizes dynamic extraction of image features, that is, even if the width and the height of the target bounding box are greatly different, the image features of the target bounding box can be uniformly sampled, the image details are retained, and the accuracy of target recognition based on the extracted image features can be improved, and in particular, a large language model can better output a character recognition result based on the extracted image features. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 FIG. 1 is a flowchart of a preferred embodiment of the method for dynamically extracting features in the application.

[0045] Figure 2 FIG. 2 is an embodiment of a traditional region extraction method for extracting a region feature.

[0046] Figure 3 FIG. 3 is another embodiment of the traditional region extraction method for extracting a region feature.

[0047] Figure 4 FIG. 4 is an embodiment of the method for dynamically extracting features in the application for extracting a region feature.

[0048] Figure 5 FIG. 5 is another embodiment of the method for dynamically extracting features in the application for extracting a region feature.

[0049] Figure 6 FIG. 6 is a logic architecture diagram of a specific embodiment of the method for recognizing characters in the application.

[0050] Figure 7 is a flow chart of the feature extraction region of the present application.

[0051] Figure 8 is a functional principle block diagram of the preferred embodiment of the feature extraction device in the present application.

[0052] Figure 9 is a functional principle block diagram of the preferred embodiment of the electronic device in the present application. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present application clearer and more explicit, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0054] In view of the above defects of the prior art, the present embodiment realizes dynamic extraction of image features based on the width and height of different target bounding boxes, that is, even if the width and height of the target bounding box differ greatly, the image features in the target bounding box can be uniformly sampled, the image details are retained, and thus the accuracy of target recognition based on the extracted image features can be improved.

[0055] Please refer to Figure 1 , Figure 1 is a flow chart of the method of dynamic feature extraction in the present application. As Figure 1 shown, the method of dynamic feature extraction described in the present embodiment of the present application includes the following steps:

[0056] Step S100, extract the image features of the image to be processed, and obtain the width and height of the target bounding box in the image to be processed.

[0057] Specifically, the feature extraction task is very dependent on the resolution of the image and the retention of the detail information, and has a high requirement for the fine granularity of the picture features. The output size of the traditional feature extraction method is fixed (for example, 5*5), as Figure 2 shown, due to the different values of the width and height of the target bounding box (region box), if a fixed size is used for sampling, it will result in Figure 2 the sampling information of the target "giraffe" in the vertical direction is sparse; as Figure 3 shown, it will result in Figure 3 the sampling information of the target "dog" in the horizontal direction is sparse, therefore, when the values of the width and height differ greatly, the image features sampled by the fixed size are not uniform, and cannot well reflect the detail features in the image.

[0058] The coordinates of the target bounding box of the present application are [x1, y1, w, h], which is a bounding box pulled in the image to be processed, wherein (x1, y1) is the point of the upper left corner of the target bounding box, w is the width of the target bounding box, and h is the height of the target bounding box.

[0059] The present application can solve the problem of uneven sampling of picture information caused by the large difference in the width and height of the target bounding box in the traditional feature extraction task, and can dynamically divide the image according to the different width and height of the target bounding box to uniformly sample the image features, as shown in Figure 4 and Figure 5 .

[0060] As shown in Figure 1 , the method described in the embodiment of the present application further comprises:

[0061] Step S200: sampling in the image features based on the width and height of the target bounding box to determine a plurality of image content features of the target bounding box.

[0062] Specifically, the input resolution of the picture is limited by the input of the backbone network, which is usually 336*336 or 224*224. The input resolution is small, so if the backbone network performs multi-scale feature extraction on the picture, it should retain more original feature information. If the original picture is not divided into grids, it will result in too much extracted information, which will further result in large model parameters and calculation amount, long model inference time and low efficiency.

[0063] In the prior art, a fixed-size grid division method is usually used to divide the target bounding box, but the width-to-height ratio of the target bounding box is not fixed. For example, some are close to square region boxes, some are vertical rectangular region boxes, and some are horizontal rectangular region boxes. When the number of rows and columns is basically the same during the current grid division, the image feature sampling in the grid region after the division of the target bounding box is not uniform when the width-to-height ratio of the target bounding box is large (such as the target bounding box is a narrow or narrow region), which is easy to lose the details of the target object in the image, resulting in low accuracy of target recognition based on the extracted image features.

[0064] The present application uniformly samples the image features based on the different width and height of the target bounding box, realizes dynamic feature extraction, and even if the difference between the width and height of the target bounding box is large, the image features in each grid region are uniformly sampled due to the dynamic grid division of the target bounding box, the image details are retained, and the accuracy of target recognition based on the extracted image features is improved.

[0065] In the embodiment of the present application, the step S200 specifically comprises:

[0066] In step S210, a sampling size is determined based on the width and height of the target bounding box, and the target bounding box is grid-divided based on the sampling size to obtain a preset number of grid parameter information.

[0067] In step S220, the image features are sampled based on the grid parameter information to determine the image content features of each grid.

[0068] Specifically, the feature dimension of the image content features is [C, H, W]. If the sampling size is 32:2, i.e., the height (H) is 2 and the width (W) is 32, the corresponding feature dimension is [C, 2, 32], and C is the number of channels.

[0069] In the embodiment of the present application, the sampling size is determined based on the total number of grids and the width and height of the target bounding box, and then a dynamic grid division method is used for different widths and heights of the target bounding box, so that the image features in the grid region can be uniformly sampled.

[0070] In the embodiment of the present application, "determining the sampling size based on the width and height of the target bounding box" specifically includes: obtaining a preset standard size list, the standard size list including a plurality of standard sizes; calculating the width-height ratio of the target bounding box according to the width and height of the target bounding box, and calculating the difference between each standard size and the width-height ratio of the target bounding box, and taking the standard size with the smallest difference as the sampling size.

[0071] Specifically, the width-height ratio of the target bounding box is ratio = w / h, and the standard size list (roi output_size_list) can be set as [32:2, 16:4, 8:8, 4:16, 2:32]. The standard size closest to the width-height ratio of the target bounding box in the standard size list is extracted as the sampling size. For example, the width-height ratio of the target bounding box is 12:1, and the standard size closest to 32:2 in the standard size list is 32:2, i.e., the target bounding box is divided in the manner of 32 rows and 2 columns.

[0072] The present application pre-sets a plurality of standard sizes for target bounding boxes of different sizes, so as to select a more appropriate standard size according to the current width-height ratio of the target bounding box and the number of grids, so as to achieve dynamic division of different target bounding boxes and achieve uniform sampling of the target bounding box.

[0073] In another embodiment of the present application, "determining the sampling size based on the width and height of the target bounding box" includes: calculating the width-height ratio of the target bounding box according to the width and height of the target bounding box, and determining the sampling size based on the width-height ratio of the target bounding box and the preset number of grids.

[0074] Specifically, in order to improve the convenience of data processing, since subsequent data processing needs to use the model, the sampling size is determined based on the width-height ratio of the target bounding box and a preset number, for example, the preset number is 64, and the standard size list (roi output_size_list) can be set as [32:2, 16:4, 8:8, 4:16, 2:32].

[0075] The present application determines the sampling size based on the width-height ratio of the target bounding box and the preset number, thereby realizing the convenience of subsequent model processing data.

[0076] As shown in Figure 1 The method provided by the embodiment of the present application further includes:

[0077] In step S300, the region feature of the target bounding box is determined based on the image content feature and the corresponding region position encoding.

[0078] Specifically, for the feature extraction task, the accuracy of the position encoding is very important, because the combination of each character is greatly affected by the position, and the position encoding needs to accurately express the relative position of the image content feature.

[0079] In the embodiment of the present application, the step of obtaining the region position encoding includes: inputting the preset number of grid parameter information into the pre-constructed learnable position encoding module to obtain the region position encoding corresponding to the image content feature of each grid.

[0080] Specifically, since the position information corresponding to each divided grid is lost after the image content feature extracted in step S200 is spliced in row-column order, the present application adds position encoding to the image content feature before feature splicing.

[0081] The present application uses a learnable position encoding module to perform region position encoding to obtain more accurate region position encoding, thereby improving the accuracy of the feature extraction task.

[0082] In the embodiment of the present application, the grid parameter information includes grid abscissa, grid ordinate, grid width, grid height and grid area.

[0083] Specifically, the existing technology only performs region position encoding on the image content feature based on the position of the grid, however, the region position encoding on the image content feature based on only the position of the grid cannot well express the position of the image content feature. The present application not only considers the position (x, y) of the grid, but also considers the grid width w, the grid height h and the grid area area, as shown in Figure 6As shown, the five information [grid_xi, grid_yi, grid_w, grid_h, area] is input into the learnable region position coding module to obtain more accurate region position coding, thereby improving the accuracy of feature extraction.

[0084] In the embodiment of the present application, when the grid corresponding to the image content feature is located in the last column, the region position coding further includes a row separator.

[0085] Since the image content features extracted are spliced in row and column order, the spatial information between positions is lost. For example, when the grid is divided into 32*2, a total of 64 grids, and the image content features Token corresponding to each grid are spliced in row and column order, the image content feature positions of the second row will change from the second row to the first row 31st to 64th Token. Even if the original grid position coordinates are retained, the spatial information that the original second row image is below the first row is lost. Therefore, the present application not only adds position coding to the image content features before flattening, but also adds a row separator. The row separator is used to distinguish different grid rows, and therefore the accuracy of the feature extraction task is further improved.

[0086] In the embodiment of the present application, the step S300 specifically includes: adding the region position coding to the image content features to obtain the region feature of the target bounding box.

[0087] As shown in Figure 7 adding the region position coding to the image content features to obtain the region feature of the target bounding box, so that the region feature has more detailed features, thereby improving the accuracy of the feature extraction task.

[0088] The present application dynamically divides the target bounding box, so that the feature sampling of the target object in the target bounding box is more uniform and reasonable, and the obtained image content features retain more detailed information. In addition, the pre-set number of grid parameter information is input into the pre-constructed learnable position coding module, and based on the image content features and the position coding module, more rich image features can be obtained. When target recognition is performed based on the image features extracted by the present application, more accurate recognition results can be obtained. In the text recognition task, since the text recognition in the image is performed in the form of each row or each column, that is, the difference between the width and the height of the target bounding box is large, the method of dynamically extracting features according to the present application can better retain the details of the text in the image.

[0089] The present application also provides a text recognition method, which comprises:

[0090] inputting the region feature as described above into a trained large language model to obtain a text recognition result.

[0091] The present application processes the region feature with region position coding based on a trained large language model. Before input, the region feature is flattened in one dimension, i.e., the original feature is [N, C, H, W], and after flattening, [N, C, H*W] is obtained. Then, it is spliced with the task token [N, C, text_token_num] to obtain [N, C, H*W+text_token_num], which is input into the large language model. For example, the target bounding box is divided into 3*6 dynamic grids to obtain 18 grids, and the original feature of the target bounding box is [10, 256, 3, 6]. The one-dimensional feature (flattening) obtained by splicing in row and column order is [10, 256, 18]. The task token is [10, 256, 3], so after splicing [10, 256, 18] and [10, 256, 3], [10, 256, 21] is obtained, and the one-dimensional feature of [10, 256, 21] is input into the large language model.

[0092] In an embodiment, the text recognition result output by the large language model is subjected to beam search to obtain the final text recognition result. The large language model can adopt the OPT model.

[0093] If the image contains text information, the present application determines the grid division method based on the width and height of the target bounding box, and then samples the image feature based on the grid parameter information after division. This dynamic feature extraction method can better preserve the text details in the image. In addition, the present application inputs the multi-dimensional grid parameter information into the learnable region position coding module to obtain more accurate region position coding, which can obtain the position relationship between texts through the position relationship between each grid. In addition, the grid located in the last column of the divided grid adds a row separator feature in its region position coding, making the region position coding information more rich. When the large language model of the present application performs text recognition based on the region feature with region position coding, more detailed features are preserved in the region feature, thereby improving the accuracy and detection precision of the text recognition task.

[0094] In the embodiment of the present application, under the guidance of the text recognition task token prompt, the large language model outputs the text recognition result based on the region feature.

[0095] Please refer again to Figure 6 , Figure 6 In the LMM in the above embodiment, the large language model, under the guidance of the task token prompt, recognizes the English characters in the image as the cursor character recognition result output by the model, i.e., two lines of characters are output, the first line is “DEPARTMENT OF GOVERNMENT”, and the second line is “COURSES FOR STAFF”.

[0096] In the embodiment of the present application, the training step of the trained large language model comprises:

[0097] Step S10, obtaining a training data set, wherein the training data set comprises a plurality of training images and labeled text information in each training image;

[0098] Step S20, extracting training image features of the training images and obtaining the width and height of the target bounding box in the training images;

[0099] Step S30, sampling in the training image features based on the width and height of the target bounding box to determine a plurality of training image content features;

[0100] Step S40, determining the recognized text features of the target bounding box based on the training image content features and their corresponding region position encodings;

[0101] Step S50, training a pre-set large language model based on the recognized text features and the labeled text information of the training images to obtain a trained large language model.

[0102] Specifically, the output result of the large language model is mapped to the vocabulary dimension, and the cross-entropy loss calculation is performed with the labeled text information in the training image. The calculation of the cross-entropy loss function in the present application helps to ensure the stability and effectiveness of the optimization process.

[0103] When training the large language model, the present application also samples in the training image features based on the width and height of the target bounding box in the training images, so that the training image content features have more detailed features, thereby improving the accuracy and detection precision of the large language model in the feature extraction task.

[0104] Edit accuracy (edit acc) is an index for measuring the performance of a large language model, and the higher the value is, the better. After training the large language model using the traditional feature extraction method and the dynamic feature extraction method of the present application, the results are evaluated on the IC13 data set, respectively. When the task is optical character recognition (OCR), the method of the present application can obtain higher edit accuracy, as shown in the following table:

[0105] Multi-modal model Editing accuracy Traditional method 0.51 Method of the present invention 0.82

[0106] Further, as Figure 8 shown, based on the above-mentioned dynamic feature extraction method, the present application also provides a feature extraction device, comprising:

[0107] An image feature extraction module 100 is configured to extract image features of a to-be-processed image.

[0108] The target bounding box obtaining module 200 is configured to obtain a width and a height of a target bounding box in an image to be processed.

[0109] The sampling module 300 is configured to sample the image features based on the width and the height of the target bounding box, and determine a plurality of image content features of the target bounding box.

[0110] The feature output module 400 is configured to output a region feature of the target bounding box based on the image content features and corresponding region position encodings.

[0111] In an embodiment of the present application, the sampling module 300 comprises:

[0112] The grid dividing unit is configured to determine a sampling size based on the width and the height of the target bounding box, and divide the target bounding box based on the sampling size to obtain a preset number of grid parameter information.

[0113] The sampling unit is configured to sample the image features based on the grid parameter information, and determine a plurality of image content features of each grid.

[0114] Figure 9 The electronic device provided in the embodiment of the present application is shown in a structural schematic diagram. The electronic device can comprise:

[0115] The memory 501, the processor 502, and a computer program stored in the memory 501 and executable on the processor 502.

[0116] The processor 502 implements the method for dynamically extracting features and / or the method for recognizing characters provided in the above embodiments when executing the program.

[0117] Further, the electronic device further comprises:

[0118] The communication interface 503 is configured to communicate between the memory 501 and the processor 502.

[0119] The memory 501 is configured to store the computer program executable on the processor 502.

[0120] The memory 501 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0121] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0122] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete communication between each other through an internal interface.

[0123] The processor 502 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0124] The embodiments also provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the above method for dynamically extracting features and / or the method for recognizing characters.

[0125] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0126] Moreover, the terms "first", "second", "third", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of instances indicated. Thus, features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0127] Any process or method descriptions or blocks in flow charts herein, and elsewhere, can be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of the preferred embodiments of the present application in which additional functionality can be added or some functionality can be removed, by, for example, adding one or more steps performing a similar or reciprocal function, combining two or more steps into a single step, or splitting one step into two or more steps.

[0128] The logic and / or steps represented in flow charts herein, and elsewhere, can be considered as a sequence of executable instructions, for implementing the logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can specifically include a hardware apparatus (e.g., an electrical connection having one or more wires, a portable computer diskette, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM)). Additionally, the computer-readable medium can be paper or other suitable medium upon which the program is printed, as the program can be electronically captured, via an optical scanner, for instance, and then compiled, interpreted, or otherwise processed, using well known techniques, for implementation into a computer system. In this document, the term "computer-readable medium" is used only to facilitate discussion and is not intended to limit the scope of use or functionality of such computer-readable media.

[0129] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0130] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, which can be stored in a computer readable storage medium, and the programs include one or a combination of steps of the method embodiments when executed.

[0131] In addition, the functional units in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of software functional module. The integrated module, if realized in the form of software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0132] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

[0133] In summary, the method for dynamically extracting features, the character recognition method and the device disclosed by the present application, the method for dynamically extracting features comprises: extracting image features of a to-be-processed image, and obtaining the width and height of a target bounding box in the to-be-processed image; sampling in the image features based on the width and height of the target bounding box, and determining a plurality of image content features of the target bounding box; and determining a region feature of the target bounding box based on the image content features and their corresponding region positions. The present application realizes dynamic extraction of image features based on the information of the width and height of different target bounding boxes, that is, even if the difference between the width and height of the target bounding box is large, the image features of the target bounding box can be uniformly sampled, the image details are retained, and the accuracy of target recognition based on the extracted image features can be improved, and especially a large language model can better output a character recognition result based on the extracted image features.

[0134] It is to be understood that the application is not limited to the examples described above, which can be modified or adapted in several ways by those skilled in the art without departing from the scope of the present application, as defined by the appended claims.

Claims

1. A method of dynamically extracting features, characterized by, The method for dynamically extracting features comprises: extracting image features of a to-be-processed image, and obtaining a width and a height of a target bounding box in the to-be-processed image; sampling in the image features based on the width and the height of the target bounding box, and determining a plurality of image content features of the target bounding box; determining a region feature of the target bounding box based on the image content features and corresponding region position encodings; the step of sampling in the image features based on the width and the height of the target bounding box, and determining a plurality of image content features of the target bounding box, comprises: determining a sampling size based on the width and the height of the target bounding box, and performing grid division on the target bounding box based on the sampling size to obtain a preset number of grid parameter information; the step of obtaining the region position encodings comprises: inputting the preset number of grid parameter information into a pre-constructed learnable position encoding module to obtain region position encodings corresponding to the image content features of each grid; the grid parameter information comprises a grid horizontal coordinate, a grid vertical coordinate, a grid width, a grid height, and a grid area.

2. The method of dynamically extracting features according to claim 1, wherein, the step of sampling in the image features based on the width and the height of the target bounding box, and determining a plurality of image content features of the target bounding box, further comprises: sampling in the image features based on the grid parameter information to determine the image content features of each grid.

3. The method of dynamically extracting features according to claim 2, wherein, the step of determining a sampling size based on the width and the height of the target bounding box comprises: obtaining a preset standard size list, wherein the standard size list comprises a plurality of standard sizes; calculating an aspect ratio of the target bounding box according to the width and the height of the target bounding box, and calculating a difference value between each standard size and the aspect ratio of the target bounding box, and taking the standard size with the smallest difference value as the sampling size.

4. The method of dynamically extracting features according to claim 2, wherein, the step of determining a sampling size based on the width and the height of the target bounding box comprises: calculating an aspect ratio of the target bounding box according to the width and the height of the target bounding box, and determining a sampling size based on the aspect ratio of the target bounding box and the preset number of grids.

5. The method of dynamically extracting features according to claim 1, wherein, when the grid corresponding to the image content features is located in the last column, the region position encodings further comprise a row separator.

6. The method of dynamically extracting features according to claim 1, wherein, the step of determining a region feature of the target bounding box based on the image content features and corresponding region position encodings comprises: adding the region position encodings to the image content features to obtain the region feature of the target bounding box.

7. A character recognition method characterized by comprising: the method for recognizing text comprises: inputting the region feature of any one of claims 1 to 6 into a trained large language model to obtain a text recognition result.

8. The character recognition method of claim 7, wherein, under the guidance of a task token prompt, the large language model outputs a text recognition result based on the region feature.

9. The character recognition method of claim 7, wherein, the training step of the trained large language model comprises: obtaining a training data set, wherein the training data set comprises a plurality of training images, and text information annotated for each training image; extracting training image features of the training images, and obtaining a width and a height of a target bounding box in the training images; sampling in the training image features based on the width and the height of the target bounding box, and determining a plurality of training image content features; Based on the image content features and the corresponding region position encodings, determine the recognized text features of the target bounding box; Based on the recognized text features and the text information of the training image annotation, train a pre-set large language model to obtain a trained large language model.

10. A feature extraction apparatus characterized by comprising: The device comprises: An image feature extraction module for extracting image features of an image to be processed; A target bounding box acquisition module for acquiring the width and height of a target bounding box in the image to be processed; A sampling module for sampling in the image features based on the width and height of the target bounding box to determine a plurality of image content features of the target bounding box; A feature output module for outputting the region features of the target bounding box based on the image content features and the corresponding region position encodings; The sampling module comprises: A grid division unit for determining a sampling size based on the width and height of the target bounding box, and performing grid division on the target bounding box based on the sampling size to obtain a preset number of grid parameter information; The acquisition of the region position encoding comprises: Inputting the preset number of grid parameter information into a pre-constructed learnable position encoding module to obtain the region position encoding corresponding to the image content features of each grid; The grid parameter information comprises a grid horizontal coordinate, a grid vertical coordinate, a grid width, a grid height, and a grid area.

11. The feature extraction apparatus according to claim 10, characterized by, The sampling module further comprises: A sampling unit for sampling in the image features based on the grid parameter information to determine the image content features of each grid.

12. An electronic device, comprising: It comprises: A memory, a processor, and a feature extraction program stored on the memory and executable on the processor, wherein the feature extraction program, when executed by the processor, implements the method of dynamically extracting features according to any one of claims 1-6 and / or the steps of the text recognition method according to any one of claims 7-9.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program which can be executed to implement the method of dynamically extracting features according to any one of claims 1-6 and / or the steps of the text recognition method according to any one of claims 7-9.

Images