Dynamic feature extraction method, character recognition method and equipment

Abstract

The invention provides a dynamic feature extraction method, a character recognition method and equipment, and the method comprises the steps: extracting the image features of a to-be-processed image, and obtaining the width and height of a target bounding box in the to-be-processed image; sampling in the image features based on the width and height of the target bounding box, and determining a plurality of image content features of the target bounding box; and based on the image content features and the corresponding regional position codes, determining regional features of the target bounding box. According to the method, the image features are dynamically extracted based on the information of the width and the height of the target bounding box, even if the difference between the width and the height of the target bounding box is large, the image features of the target bounding box can be uniformly sampled, image details are reserved, and then the accuracy of target recognition based on the extracted image features can be improved.

Description

Technical Field

[0001] The present invention relates to the field of feature extraction technology, and in particular to a method for dynamically extracting features, a text recognition method and equipment. Background Art

[0002] When extracting features from a target area in an image, a model is usually required to divide the target area to extract image features from the target area. In the prior art, a fixed-size grid division method is usually used, but the aspect ratio of the target area is not fixed. For example, some are area frames that are close to squares, some are vertical rectangular area frames, and some are horizontal rectangular area frames. The number of rows and columns in the current grid division is basically the same. When the width and height of the target area are very different (such as the target area is a long or narrow area), the image feature sampling in the grid area after the target area is divided is uneven, and the details of the target object in the image are easily lost, resulting in a low accuracy rate for target recognition based on the extracted image features.

[0003] Therefore, the prior art has defects and needs to be improved and developed. Summary of the invention

[0004] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method for dynamic feature extraction, a text recognition method and a device are provided, aiming to solve the problem in the prior art that when the difference between the width and height of the target area is large, the image feature sampling in the grid area after the target area is divided is uneven, which easily loses the details of the target object in the image, resulting in low accuracy of target recognition based on the extracted image features.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] A method for dynamically extracting features, wherein the method for dynamically extracting features comprises:

[0007] Extract image features of the image to be processed, and obtain the width and height of the target bounding box in the image to be processed;

[0008] Determine a plurality of image content features of the target bounding box by sampling the image features based on the width and height of the target bounding box;

[0009] Based on the image content features and their corresponding region position codes, region features of the target bounding box are determined.

[0010] In one embodiment of the present application, sampling in the image features based on the width and height of the target bounding box to determine several image content features of the target bounding box includes:

[0011] Determine a sampling size based on the width and height of the target bounding box, and divide the target bounding box into grids based on the sampling size to obtain a preset number of grid parameter information;

[0012] The image features are sampled based on the grid parameter information to determine the image content features of each grid.

[0013] In one embodiment of the present application, determining the sampling size based on the width and height of the target bounding box includes:

[0014] Obtain a preset standard size list, wherein the standard size list includes several standard sizes;

[0015] The aspect ratio of the target bounding box is calculated according to the width and height of the target bounding box, and the difference between each standard size and the aspect ratio of the target bounding box is calculated, and the standard size with the smallest difference is used as the sampling size.

[0016] In one embodiment of the present application, determining the sampling size based on the width and height of the target bounding box includes:

[0017] The aspect ratio of the target bounding box is calculated according to the width and height of the target bounding box, and a sampling size is determined based on the aspect ratio of the target bounding box and a preset number of the grids.

[0018] In one embodiment of the present application, the step of obtaining the area position code includes:

[0019] The preset number of grid parameter information is input into a pre-built learnable position encoding module to obtain the regional position encoding corresponding to the image content feature of each grid.

[0020] In one embodiment of the present application, the grid parameter information includes a grid abscissa, a grid ordinate, a grid width, a grid height, and a grid area.

[0021] In one embodiment of the present application, when the grid corresponding to the image content feature is located in the last column, the region position code further includes a row separator.

[0022] In one embodiment of the present application, determining the region feature of the target bounding box based on the image content feature and its corresponding region position code includes:

[0023] The region position code is added to the image content feature to obtain the region feature of the target bounding box.

[0024] The present application also provides a text recognition method, wherein the text recognition method comprises:

[0025] The regional features described above are input into the trained large language model to obtain the text recognition results.

[0026] In one embodiment of the present application, under the guidance of the task token prompt, the large language model outputs a text recognition result based on the regional features.

[0027] In one embodiment of the present application, the training step of the trained large language model includes:

[0028] Acquire a training data set, wherein the training data set includes a plurality of training images and text information annotated on each of the training images;

[0029] Extract training image features of the training image and obtain the width and height of the target bounding box in the training image;

[0030] Determine a plurality of training image content features by sampling the training image features based on the width and height of the target bounding box;

[0031] Determining the recognition text features of the target bounding box based on the content features of the training image and the corresponding region position codes;

[0032] Based on the recognized text features and the text information annotated in the training image, a preset large language model is trained to obtain a trained large language model.

[0033] The present application also provides a feature extraction device, wherein the device comprises:

[0034] An image feature extraction module, used to extract image features of the image to be processed;

[0035] A target bounding box acquisition module is used to obtain the width and height of the target bounding box in the image to be processed;

[0036] A sampling module, configured to sample the image features based on the width and height of the target bounding box to determine a plurality of image content features of the target bounding box;

[0037] The feature output module is used to output the regional features of the target bounding box based on the image content features and the corresponding regional position codes.

[0038] In one embodiment of the present application, the sampling module includes:

[0039] A grid division unit, configured to determine a sampling size based on a width and a height of the target bounding box, and to grid the target bounding box based on the sampling size to obtain a preset number of grid parameter information;

[0040] The sampling unit is used to sample the image features based on the grid parameter information to determine the image content features of each grid.

[0041] The present application also provides an electronic device, comprising: a memory, a processor, and a feature extraction program stored in the memory and executable on the processor, wherein the feature extraction program, when executed by the processor, implements the method for dynamic feature extraction as described above and / or the steps of the text recognition method as described above.

[0042] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed to implement the method for dynamically extracting features and / or the steps of the text recognition method as described above.

[0043] The method for dynamic feature extraction, the method for text recognition and the device provided by the present invention include: extracting image features of an image to be processed, and obtaining the width and height of a target bounding box in the image to be processed; sampling the image features based on the width and height of the target bounding box, and determining a number of image content features of the target bounding box; and determining the regional features of the target bounding box based on the image content features and their corresponding regional position codes. The present invention realizes dynamic extraction of image features, and even if the width and height of the target bounding box differ greatly, the image features of the target bounding box can be uniformly sampled, and the image details can be retained, thereby improving the accuracy of target recognition based on the extracted image features, especially the large language model can better output text recognition results based on the extracted image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a preferred embodiment of the method for dynamically extracting features in the present invention.

[0045] Figure 2 It is an embodiment of extracting regional features using a traditional region extraction method.

[0046] Figure 3 It is another embodiment of extracting regional features using the traditional region extraction method.

[0047] Figure 4 This is an embodiment of extracting regional features in the method for dynamically extracting features of the present invention.

[0048] Figure 5 This is another embodiment of extracting regional features in the method for dynamically extracting features of the present invention.

[0049] Figure 6 It is a logical architecture diagram of a specific embodiment of the text recognition method of the present invention.

[0050] Figure 7 It is a flowchart of extracting regional features of the present invention.

[0051] Figure 8 It is a functional principle block diagram of a preferred embodiment of the feature extraction device in the present invention.

[0052] Fig. 9 It is a functional principle block diagram of a preferred embodiment of the electronic device in the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] In view of the above-mentioned defects of the prior art, the present embodiment realizes dynamic extraction of image features based on the width and height of different target bounding boxes. Even if the width and height of the target bounding box are quite different, the image features in the target bounding box can be uniformly sampled, thereby retaining the image details, thereby improving the accuracy of target recognition based on the extracted image features.

[0055] See also Figure 1 , Figure 1 Flowchart of the method for dynamically extracting features in the present invention. Figure 1 As shown, the method for dynamically extracting features according to an embodiment of the present invention comprises the following steps:

[0056] Step S100: extract image features of the image to be processed, and obtain the width and height of the target bounding box in the image to be processed.

[0057] Specifically, the feature extraction task is highly dependent on the resolution and detail information of the image, and has high requirements for the fine-grainedness of the image features. However, the output size of the traditional feature extraction method is fixed (for example, 5*5), such as Figure 2 As shown in the figure, since the width and height of the target bounding box (region box) are different, sampling with a fixed size will result in Figure 2 The target "giraffe" in the vertical sampling information is relatively sparse; Figure 3 As shown, it will lead to Figure 3 The sampling information of the target "dog" in the horizontal direction is sparse. Therefore, when the width and height values ​​are very different, the image features using fixed-size sampling are uneven and cannot reflect the detailed features in the image well.

[0058] The coordinates of the target bounding box of this application are [x1, y1, w, h], which is the bounding box pulled from the image to be processed, where (x1, y1) is the point in the upper left corner of the target bounding box, w is the width of the target bounding box, and h is the height of the target bounding box.

[0059] This application can solve the problem of uneven sampling of image information caused by large differences in the width and height values ​​of the target bounding box (region boxes) in traditional feature extraction tasks. The image can be dynamically divided according to the different widths and heights of the target bounding box to uniformly sample the image features, such as Figure 4 and Figure 5 shown.

[0060] like Figure 1 As shown, the method described in the embodiment of the present invention also includes:

[0061] Step S200: sampling in the image features based on the width and height of the target bounding box to determine a plurality of image content features of the target bounding box.

[0062] Specifically, the input resolution of the image is limited by the input of the backbone network, which is usually 336*336 or 224*224. Therefore, if the backbone network extracts multi-scale features from the image, more original feature information should be retained. If the original image is not gridded, too much information will be extracted, which will lead to large model parameters and calculations, long model inference time and low efficiency.

[0063] In the prior art, a fixed-size grid division method is usually used to divide the target bounding box, but the aspect ratio of the target bounding box is not fixed. For example, some are nearly square area boxes, some are vertical rectangular area boxes, and some are horizontal rectangular area boxes. The number of rows and columns in the current grid division is basically the same. When the aspect ratio of the target bounding box is large (such as the target bounding box is a long and narrow area), the image feature sampling in the grid area after the target bounding box is divided is uneven, which easily loses the details of the target object in the image, resulting in a low accuracy rate of target recognition based on the extracted image features.

[0064] The present application uniformly samples image features based on the width and height of different target bounding boxes, thereby realizing dynamic feature extraction. Even if the width and height of the target bounding box differ greatly, the present application dynamically grids the target bounding box and uniformly samples the image features in each grid area, thereby retaining image details and thereby improving the accuracy of target recognition based on the extracted image features.

[0065] In the embodiment of the present application, the step S200 specifically includes:

[0066] Step S210: determining a sampling size based on the width and height of the target bounding box, and dividing the target bounding box into grids based on the sampling size to obtain a preset number of grid parameter information;

[0067] Step S220: sampling the image features based on the grid parameter information to determine the image content features of each grid.

[0068] Specifically, the feature dimension of the image content feature is [C, H, W]. If the sampling size is 32:2, that is, the height (H) is 2 and the width (W) is 32, then the corresponding feature dimension is [C, 2, 32], where C is the number of channels.

[0069] The embodiment of the present application determines the sampling size based on the total number of grids and the width and height of the target bounding box, and then adopts a dynamic grid division method according to the width and height of different target bounding boxes, so that image features in the grid area can be evenly sampled.

[0070] In an embodiment of the present application, “determining a sampling size based on the width and height of the target bounding box” specifically includes: obtaining a preset standard size list, wherein the standard size list includes several standard sizes; calculating the aspect ratio of the target bounding box according to the width and height of the target bounding box, and calculating the difference between each of the standard sizes and the aspect ratio of the target bounding box, and taking the standard size with the smallest difference as the sampling size.

[0071] Specifically, the aspect ratio of the target bounding box is ratio = w / h, and the standard size list (roi output_size_list) can be set to [32:2, 16:4, 8:8, 4:16, 2:32]. The standard size in the standard size list that is closest to the aspect ratio of the target bounding box is extracted as the sampling size. For example, if the aspect ratio of the target bounding box is 12:1, it is closer to 32:2 in the standard size list, and the corresponding sampling size is 32:2, that is, the target bounding box is divided into 32 rows and 2 columns.

[0072] The present application pre-sets multiple standard sizes for target bounding boxes of different sizes, so as to select a more appropriate standard size according to the current aspect ratio of the target bounding box and the number of grids, so as to realize dynamic division of different target bounding boxes and achieve uniform sampling in the target bounding box.

[0073] In another embodiment of the present application, “determining the sampling size based on the width and height of the target bounding box” includes: calculating the aspect ratio of the target bounding box according to the width and height of the target bounding box, and determining the sampling size based on the aspect ratio of the target bounding box and a preset number of the grids.

[0074] Specifically, since the model is needed for data processing later, in order to improve the convenience of data processing, the sampling size is determined based on the aspect ratio of the target bounding box and the preset number. For example, the preset number is 64, and the standard size list (roi output_size_list) can be set to [32:2, 16:4, 8:8, 4:16, 2:32].

[0075] The present application determines the sampling size based on the aspect ratio of the target bounding box and the preset number, thereby achieving convenience for subsequent model processing of data.

[0076] like Figure 1 As shown, the method described in the embodiment of the present invention also includes:

[0077] Step S300: Determine the region features of the target bounding box based on the image content features and their corresponding region position codes.

[0078] Specifically, for feature extraction tasks, the accuracy of position encoding is very important because the combination of each character is deeply affected by the position, and the position encoding needs to accurately express the relative position of the image content features.

[0079] In an embodiment of the present application, the step of acquiring the regional position code includes: inputting the preset number of grid parameter information into a pre-built learnable position coding module to obtain the regional position code corresponding to the image content features of each grid.

[0080] Specifically, since the position information corresponding to each divided grid will be lost after the image content features extracted in step S200 are spliced ​​in row and column order, the present application adds position coding to the image content features before feature splicing.

[0081] This application uses a learnable position encoding module to perform regional position encoding to obtain more accurate regional position encoding, thereby improving the accuracy of feature extraction tasks.

[0082] In an embodiment of the present application, the grid parameter information includes a grid horizontal coordinate, a grid vertical coordinate, a grid width, a grid height and a grid area.

[0083] Specifically, existing technologies only encode the image content features based on the position of the grid. However, encoding the image content features based on the position of the grid alone cannot express the position of the image content features well. This application not only considers the position (x, y) of the grid, but also considers the grid width w, grid height h and grid area, such as Figure 6As shown in the figure, the five information [grid_xi, grid_yi, grid_w, grid_h, area] are sent to the learnable region position encoding module to obtain a more accurate region position encoding, which can improve the accuracy of feature extraction.

[0084] In an embodiment of the present application, when the grid corresponding to the image content feature is located in the last column, the region position code further includes a row separator.

[0085] Since the spatial information between the positions will be lost after the extracted image content features are spliced ​​in row and column order, for example, the grid is divided into 32*2, a total of 64 grids. When the image content feature Token corresponding to each grid is spliced ​​in row and column order, the image content feature position of the second row will change from the second row to the 31st to 64th Token of the first row. Even if the position coordinates of the original grid are retained, the spatial information that the original second row image is below the first row is lost. Therefore, the present application not only adds position coding to the image content features before flattening, but also adds row separators. The row separators are used to strengthen the distinction between different grid rows. Therefore, the present application further improves the accuracy of the feature extraction task.

[0086] In the embodiment of the present application, the step S300 specifically includes: adding the region position code to the image content feature to obtain the region feature of the target bounding box.

[0087] like Figure 7 As shown, the region position code is added to the image content feature to obtain the region feature of the target bounding box, so that the region feature has more detailed features, thereby improving the accuracy of the feature extraction task.

[0088] The present application dynamically divides the target bounding box so that the feature sampling of the target object in the target bounding box is more uniform and reasonable, and the obtained image content features retain more detail information. In addition, a preset number of grid parameter information is input into a pre-built learnable position coding module, and richer image features can be obtained based on the image content features and the position coding module. When performing target recognition based on the image features extracted by the present application, more accurate recognition results can be obtained. In the text recognition task, because the text recognition in the image is performed in a row or column manner, that is, the width and height of the target bounding box are quite different, the method of dynamically extracting features by the present application can better retain the details of the text in the image.

[0089] The present application also provides a text recognition method, the text recognition method comprising:

[0090] The regional features described above are input into the trained large language model to obtain the text recognition results.

[0091] This application processes regional features with regional position encoding based on a trained large language model. Before input, the regional features are flattened in one dimension, that is, the original features are [N, C, H, W], and after flattening, [N, C, H*W] is obtained. Then it is spliced ​​with the task token (token) [N, C, text_token_num] to obtain [N, C, H*W+text_token_num], which is input into the large language model. For example, the target bounding box is dynamically divided into 3*6 grids to obtain 18 grids, and the original features of the target bounding box are [10, 256, 3, 6], which are spliced ​​into one-dimensional features (flattened) in row and column order to obtain the features [10, 256, 18]. The task token is [10,256,3], so [10,256,18] and [10,256,3] are concatenated to get [10,256,21], and the one-dimensional feature of [10,256,21] is input into the large language model.

[0092] In one embodiment, a beam search is performed on the text recognition result output by the large language model to obtain a final text recognition result. The large language model may adopt an OPT model.

[0093] If the image contains text information, the present application determines the grid division method based on the width and height of the target bounding box, and then samples the image features based on the grid parameter information after division. This method of dynamically extracting features can better retain the text details in the image. In addition, the present application sends the multi-dimensional grid parameter information into a learnable regional position coding module to obtain a more accurate regional position coding, and the positional relationship between the text can be obtained through the positional relationship between each grid. In addition, the grid located in the last column of the divided grid has a row separator feature added to its regional position coding, making the regional position coding information richer. When the large language model of the present application performs text recognition based on regional features with regional position coding, since more detailed features are retained in the regional features, the accuracy and detection precision of the text recognition task can be improved.

[0094] In the embodiment of the present application, under the guidance of the text recognition task token prompt, the large language model outputs the text recognition result based on the regional features.

[0095] Please refer again Figure 6 , Figure 6 The LMM in the figure is a large language model. Under the guidance of the task token prompt, the English characters recognized in the figure are the cursor character recognition results output by the model, that is, two lines of characters are output, the first line is "DEPARTMENT OF GOVERNMENT", and the second line is "COURSES FOR STAFF".

[0096] In an embodiment of the present application, the training steps of the trained large language model include:

[0097] Step S10, obtaining a training data set, wherein the training data set includes a plurality of training images and text information annotated in each of the training images;

[0098] Step S20, extracting training image features of the training image, and obtaining the width and height of the target bounding box in the training image;

[0099] Step S30, sampling the training image features based on the width and height of the target bounding box to determine a number of training image content features;

[0100] Step S40, determining the recognition text features of the target bounding box based on the content features of the training image and the corresponding region position codes;

[0101] Step S50: training a preset large language model based on the recognized text features and the text information annotated in the training image to obtain a trained large language model.

[0102] Specifically, the output results obtained by the large language model are mapped to the vocabulary dimension, and the cross-entropy loss is calculated with the text information annotated in the training image. The cross-entropy loss function calculated in this application helps to ensure the stability and effectiveness of the optimization process.

[0103] When training a large language model, the present application will also sample the training image features based on the width and height of the target bounding box in the training image, so that the training image content features have more detailed features, thereby improving the accuracy and detection precision of the large language model for feature extraction tasks.

[0104] Edit accuracy is an indicator used to measure the performance of a large language model. The higher the value, the better. After training the large language model using the traditional feature extraction method and the method of the present invention for dynamically extracting features, the results are evaluated on the IC13 dataset. If the task is optical character recognition (OCR), the method of the present invention can obtain a higher edit accuracy, as shown in the following table:

[0105] Multimodal Model Editing accuracy Traditional methods 0.51 Method of the present invention 0.82

[0106] Furthermore, if Figure 8 As shown, based on the above-mentioned method for dynamically extracting features, the present invention also provides a feature extraction device, including:

[0107] An image feature extraction module 100 is used to extract image features of an image to be processed;

[0108] The target bounding box acquisition module 200 is used to acquire the width and height of the target bounding box in the image to be processed;

[0109] A sampling module 300, configured to sample the image features based on the width and height of the target bounding box to determine a plurality of image content features of the target bounding box;

[0110] The feature output module 400 is used to output the region feature of the target bounding box based on the image content feature and its corresponding region position code.

[0111] In one embodiment of the present application, the sampling module 300 includes:

[0112] A grid division unit, configured to determine a sampling size based on a width and a height of the target bounding box, and to grid the target bounding box based on the sampling size to obtain a preset number of grid parameter information;

[0113] The sampling unit is used to sample the image features based on the grid parameter information to determine a number of image content features of each grid.

[0114] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0115] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0116] When the processor 502 executes the program, the dynamic feature extraction method and / or the text recognition method provided in the above embodiments are implemented.

[0117] Furthermore, the electronic device further comprises:

[0118] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0119] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0120] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0121] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0122] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0123] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0124] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for dynamically extracting features and / or the method for text recognition as described above is implemented.

[0125] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0126] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0127] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0128] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0129] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0130] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0131] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0132] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0133] In summary, the present invention discloses a method for dynamic feature extraction, a method for text recognition, and a device, wherein the method for dynamic feature extraction includes: extracting image features of an image to be processed, and obtaining the width and height of a target bounding box in the image to be processed; sampling the image features based on the width and height of the target bounding box, and determining a number of image content features of the target bounding box; and determining the regional features of the target bounding box based on the image content features and their corresponding regional position codes. The present invention realizes dynamic extraction of image features based on information of different widths and heights of target bounding boxes, and even if the difference between the width and height of the target bounding box is large, the image features of the target bounding box can be uniformly sampled, and the image details are retained, thereby improving the accuracy of target recognition based on the extracted image features, especially the large language model can better output text recognition results based on the extracted image features.

[0134] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for dynamically extracting features, characterized in that: The method for dynamically extracting features comprises: Extract image features of the image to be processed, and obtain the width and height of the target bounding box in the image to be processed; Determine a plurality of image content features of the target bounding box by sampling the image features based on the width and height of the target bounding box; Based on the image content features and their corresponding region position codes, region features of the target bounding box are determined.

2. The method for dynamic feature extraction according to claim 1, characterized in that: Sampling the image features based on the width and height of the target bounding box to determine a number of image content features of the target bounding box, including: Determine a sampling size based on the width and height of the target bounding box, and divide the target bounding box into grids based on the sampling size to obtain a preset number of grid parameter information; The image features are sampled based on the grid parameter information to determine the image content features of each grid.

3. The method for dynamic feature extraction according to claim 2, characterized in that: The sampling size is determined based on the width and height of the target bounding box, including: Obtain a preset standard size list, wherein the standard size list includes several standard sizes; The aspect ratio of the target bounding box is calculated according to the width and height of the target bounding box, and the difference between each standard size and the aspect ratio of the target bounding box is calculated, and the standard size with the smallest difference is used as the sampling size.

4. The method for dynamic feature extraction according to claim 2, characterized in that: The sampling size is determined based on the width and height of the target bounding box, including: The aspect ratio of the target bounding box is calculated according to the width and height of the target bounding box, and a sampling size is determined based on the aspect ratio of the target bounding box and a preset number of the grids.

5. The method for dynamic feature extraction according to claim 2, characterized in that: The step of obtaining the area position code comprises: The preset number of grid parameter information is input into a pre-built learnable position encoding module to obtain the regional position encoding corresponding to the image content feature of each grid.

6. The method for dynamic feature extraction according to claim 5, characterized in that: The grid parameter information includes a grid abscissa, a grid ordinate, a grid width, a grid height and a grid area.

7. The method for dynamic feature extraction according to claim 5, characterized in that: When the grid corresponding to the image content feature is located in the last column, the region position code further includes a row separator.

8. The method for dynamic feature extraction according to claim 1, characterized in that: Determining the region feature of the target bounding box based on the image content feature and the corresponding region position code includes: The region position code is added to the image content feature to obtain the region feature of the target bounding box.

9. A method for character recognition, characterized in that: The text recognition method comprises: The regional features as described in any one of claims 1 to 8 are input into a trained large language model to obtain a text recognition result.

10. The text recognition method according to claim 9, characterized in that: Under the guidance of the task token prompt, the large language model outputs a text recognition result based on the regional features.

11. The text recognition method according to claim 9, characterized in that: The training steps of the trained large language model include: Acquire a training data set, wherein the training data set includes a plurality of training images and text information annotated on each of the training images; Extract training image features of the training image and obtain the width and height of the target bounding box in the training image; Determine a plurality of training image content features by sampling the training image features based on the width and height of the target bounding box; Determining the recognition text features of the target bounding box based on the content features of the training image and the corresponding region position codes; Based on the recognized text features and the text information annotated in the training image, a preset large language model is trained to obtain a trained large language model.

12. A feature extraction device, characterized in that: The device comprises: An image feature extraction module, used to extract image features of the image to be processed; A target bounding box acquisition module is used to obtain the width and height of the target bounding box in the image to be processed; A sampling module, configured to sample the image features based on the width and height of the target bounding box to determine a plurality of image content features of the target bounding box; The feature output module is used to output the regional features of the target bounding box based on the image content features and the corresponding regional position codes.

13. The feature extraction device according to claim 12, characterized in that: The sampling module comprises: A grid division unit, configured to determine a sampling size based on a width and a height of the target bounding box, and to grid the target bounding box based on the sampling size to obtain a preset number of grid parameter information; The sampling unit is used to sample the image features based on the grid parameter information to determine the image content features of each grid.

14. An electronic device, characterized in that: include: A memory, a processor, and a feature extraction program stored in the memory and executable on the processor, wherein the feature extraction program, when executed by the processor, implements the method for dynamic feature extraction as described in any one of claims 1 to 8 and / or the steps of the text recognition method as described in any one of claims 9 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed to implement the method for dynamically extracting features as described in any one of claims 1 to 8 and / or the steps of the text recognition method as described in any one of claims 9 to 11.

Images