Text recognition method, electronic device and storage medium

By dividing image blocks and integrating feature representations of images to be identified in natural scenes, the problem of low text recognition efficiency in the prior art is solved, and a more efficient text recognition process is achieved.

CN118522019BActive Publication Date: 2024-11-29ZHEJIANG DAHUA TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410996958.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-29
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

The prior art has low text recognition efficiency in natural scenes, especially high-resolution images, resulting in high-resolution images.

Method used

By obtaining the image to be identified, dividing it into multiple image blocks, performing the first downsampling of feature representation, obtaining the initial feature representation, and fusing it with the feature representation of multiple image blocks, obtaining the target feature representation, and then text recognition is performed .

Benefits of technology

The number of feature representations in the text recognition process is reduced, the efficiency of text recognition is improved, and text recognition tasks in natural scenes can be handled more quickly.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118522019B_ABST
    Figure CN118522019B_ABST
Patent Text Reader

Abstract

The present application discloses a text recognition method, an electronic device, and a storage medium. The text recognition method includes: obtaining an image to be recognized; performing a first downsampling on the feature representations of multiple image blocks obtained by dividing the image to be recognized to obtain an initial feature representation, the combination of the feature representations of the multiple image blocks is the feature representation of the image to be recognized, and the initial feature representation carries the feature representations of at least some of the image blocks; fusing the initial feature representation with the feature representations of the multiple image blocks to obtain a target feature representation, the number of feature representations of the image blocks contained in the target feature representation is the same as the number of feature representations of the image blocks contained in the initial feature representation; performing text recognition on the target feature representation to obtain a text recognition result for the image to be recognized. The above scheme, using the target feature representation to perform text recognition can improve the efficiency of text recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a text recognition method, an electronic device, and a storage medium. Background Art

[0002] In the past few decades, text recognition technology has made great progress, but it still faces challenges. In particular, text recognition in natural scenes will be affected by many factors. Among them, text recognition in natural scenes can refer to images to be recognized with text in natural scenes. The low clarity of the image to be recognized with text will affect the text recognition effect. Therefore, it is necessary to use high-resolution images to be recognized in the prior art to make the small target text in the image to be recognized clearer, so that the clearer small target text improves the text recognition effect. However, the high-resolution images to be recognized have a large amount of calculation in the text recognition process, resulting in low text recognition efficiency of general text recognition models or even the inability to complete the text recognition model. The structure of the text recognition model needs to be expanded and retrained before text recognition can be performed. The whole process is relatively cumbersome.

[0003] In view of the existing technical defects, how to provide an effective text recognition solution is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0004] The present application at least provides a text recognition method, an electronic device, and a storage medium.

[0005] The present application provides a text recognition method, comprising: acquiring an image to be recognized; performing a first down-sampling on feature representations of multiple image blocks obtained by dividing the image to be recognized to obtain an initial feature representation, wherein a combination of the feature representations of the multiple image blocks is a feature representation of the image to be recognized, and the initial feature representation carries feature representations of at least some of the image blocks; fusing the initial feature representation with the feature representations of the multiple image blocks to obtain a target feature representation, wherein the number of feature representations of the image blocks contained in the target feature representation is the same as the number of feature representations of the image blocks contained in the initial feature representation; performing text recognition on the target feature representation to obtain a text recognition result on the image to be recognized.

[0006] The present application provides a text recognition device, including: an acquisition module, a sampling module, a fusion module and a recognition module; the acquisition module is used to acquire an image to be recognized; the sampling module is used to perform a first down-sampling on the feature representations of multiple image blocks obtained by dividing the image to be recognized, so as to obtain an initial feature representation, wherein the combination of the feature representations of the multiple image blocks is the feature representation of the image to be recognized, and the initial feature representation carries the feature representations of at least some of the image blocks; the fusion module is used to fuse the initial feature representation with the feature representations of the multiple image blocks, so as to obtain a target feature representation, wherein the number of feature representations of the image blocks contained in the target feature representation is the same as the number of feature representations of the image blocks contained in the initial feature representation; the recognition module is used to perform text recognition on the target feature representation, so as to obtain a text recognition result on the image to be recognized.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned text recognition method.

[0008] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned text recognition method is implemented.

[0009] The above scheme obtains an image to be identified, performs a first down-sampling on the feature representations of multiple image blocks obtained by dividing the image to be identified, obtains an initial feature representation, and fuses the initial feature representation with a smaller number of feature representations with the feature representations of multiple image blocks to obtain a target feature representation, so that the number of feature representations contained in the target feature representation is less than the number of feature representations contained in the image to be identified. By performing text recognition on the target feature representation instead of directly performing text recognition on the image to be identified, the number of feature representations in the text recognition process can be reduced, thereby improving the efficiency of text recognition.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 1 ;

[0013] Figure 2 is a schematic diagram of an image to be recognized in an embodiment of a text recognition method of the present application;

[0014] Figure 3This is a flowchart of an embodiment of the text recognition method of the present application. Figure 2 ;

[0015] Figure 4 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 3 ;

[0016] Figure 5 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 4 ;

[0017] Figure 6 It is a structural diagram of an embodiment of a text recognition device of the present application;

[0018] Figure 7 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0019] Figure 8 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0020] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0021] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0022] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] The present application provides some text recognition methods and text recognition devices. The application scenarios of the text recognition method include but are not limited to text recognition of images with text in natural scenes. The execution subject of the text recognition method can be any device with text recognition function, for example, a text recognition device. The text recognition device can be set in a terminal device or a server or other processing device, wherein the terminal device can be a device for text recognition, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementations, the text recognition method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0024] See also Figure 1 , Figure 1 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 1 Specifically, the text recognition method may include the following steps:

[0025] Step S11: Acquire the image to be recognized.

[0026] The image to be identified may be an image containing text to be identified. Specifically, the image to be identified may be a natural scene picture, a document scan, a handwritten font picture, a screenshot, an emoticon package with text information, a design work, and a medical imaging material. Among them, the natural scene picture may be a picture with text information such as a street sign, billboard, store sign, road sign, menu, product packaging, etc. Among them, the document scan may be a digital scan of a document such as a contract, file, report, invoice, business card, etc., and the text information therein needs to be extracted. Among them, the handwritten font picture may be a handwritten text picture of different styles and qualities including handwritten notes, letters, envelopes, business cards, calligraphy works, etc. Among them, the screenshot may be a screenshot picture containing text information such as an application interface, a webpage screenshot, a mobile phone page screenshot, etc. Among them, the artwork or design work may be text information in artworks or design works such as posters, advertising design drafts, book covers, paintings, etc. Among them, the medical imaging material may be a picture in the medical field such as a medical report, an imaging diagnosis report, and annotated information in a medical image. This is only an example and does not limit the type of image to be identified.

[0027] The method of acquiring the image to be identified may be to acquire the image to be identified in real time through an image acquisition device, or to use the image to be identified stored in a database, or to capture a frame of image in a video file.

[0028] Step S12: performing a first down-sampling on the feature representations of the multiple image blocks obtained by dividing the image to be identified, to obtain an initial feature representation.

[0029] The combination of feature representations of multiple image blocks is the feature representation of the image to be identified, and the initial feature representation carries the feature representation of at least some of the image blocks.

[0030] First, the image to be identified is divided into image blocks to obtain feature representations of multiple image blocks. Then, the feature representations of the multiple image blocks are first downsampled to obtain an initial feature representation. The combination of the feature representations of the multiple image blocks is the feature representation of the image to be identified. It can be understood that the feature representations of the multiple image blocks may refer to the feature representations of all image blocks contained in the entire image to be identified. The process of performing the first downsampling on the feature representations of the multiple image blocks to obtain the initial feature representation may be performing the first downsampling on the feature representation of the image to be identified to obtain the initial feature representation.

[0031] The image to be identified is divided into image blocks to obtain feature representations of multiple image blocks. In some application scenarios, first, the image to be identified is divided according to a preset size to obtain multiple image blocks. Each image block obtained after the division can be represented as a patch. Then, for each image block, the image block is vector mapped to obtain a feature representation of the image block after the vector mapping. The feature representation of the image block can be represented as a token.

[0032] The first downsampling may be filtering the feature representations of multiple image blocks according to a preset standard to obtain feature representations of at least part of the image blocks. It is understandable that the process of obtaining the initial feature representations of the feature representations of multiple image blocks after the first downsampling can reduce the number of feature representations of the image blocks. The preset standard may be a sampling standard related to at least one preset direction in the image to be recognized. In the process of obtaining the initial feature representations of the feature representations of multiple image blocks after the first downsampling, the influence of noise can be filtered out or the features related to the text to be recognized can be enhanced.

[0033] Step S13: Fusing the initial feature representation with the feature representations of multiple image blocks to obtain a target feature representation.

[0034] The number of feature representations of the image patches included in the target feature representation is the same as the number of feature representations of the image patches included in the initial feature representation.

[0035] The initial feature representation carries the feature representation of at least part of the image blocks. It is understandable that the feature representation of at least part of the image blocks constitutes or is aggregated into the initial feature representation. The initial feature representation has local feature information after denoising. The feature representation of multiple image blocks has global feature information of the entire image to be identified. In some application scenarios, the process of fusing the initial feature representation with the feature representation of multiple image blocks to obtain the target feature representation may be to fuse the local feature information after denoising with the global feature information of the entire image to be identified, and the obtained target feature representation can balance the importance of global feature information and local feature information, so as to effectively capture the important information in the input initial feature representation and the feature representation of multiple image blocks. In some application scenarios, the way to fuse the initial feature representation with the feature representation of multiple image blocks may be to perform multi-scale feature fusion of the initial feature representation with the feature representation of multiple image blocks to obtain the target feature representation. In some application scenarios, the way to fuse the initial feature representation with the feature representation of multiple image blocks may also be to use the attention mechanism to dynamically adjust the weights between the initial feature representation and the feature representation of multiple image blocks to obtain the target feature representation. In some application scenarios, the initial feature representation can be fused with the feature representations of multiple image blocks through a cascaded or parallel network structure. Multiple parallel or cascaded network structures are designed, and each network processes the initial feature representation or the feature representations of multiple image blocks respectively. Finally, the output of each network is fused or jointly trained to obtain the target feature representation. The target feature representation obtained in this way can make full use of local feature information and global feature information at different levels.

[0036] Step S14: Perform text recognition on the target feature representation to obtain a text recognition result for the image to be recognized.

[0037] The text recognition result of the image to be recognized may be a prediction result of the text to be recognized in the image to be recognized.

[0038] See also Figure 2 , Figure 2 It is a schematic diagram of an image to be recognized in an embodiment of the text recognition method of the present application.

[0039] The text to be recognized is composed of at least one word. For example, the text to be recognized can be an article, a paragraph, or a sentence, etc. The specific form of the text to be recognized is not specifically limited here. The number of texts to be recognized can be one or more. A word can be a single Chinese character, a word composed of Chinese characters, an English letter or a word composed of multiple English letters, etc. The specific form of the word is not specifically limited here.

[0040] like Figure 2As shown, the image to be recognized can be an image of a commercial sign on the street with text information. For example, Figure 2 as shown, the building on the street has text information including the address information of the building "XX Road: No. 4" and the store information "Gong abcd Store". For example, the word in the text to be recognized at a certain image position in the image to be recognized can be "Gong". The word in the text to be recognized in the image to be recognized can be related to numbers, such as "4". At this time, the text recognition result of the image to be recognized can be that the word at a certain image position is "Gong", or other words, or a blank word. First, the text recognition result can be whether there is a word at this image position. If there is a word at this image position, it indicates that the text recognition at this image position is successful. Further, the text recognition result can also include whether the word at this image position is "Gong". Specifically, the text recognition result is that the word at this image position is "Gong" or other words. If there is no word at this image position, it indicates that the text recognition at this image position fails. In some application scenarios, the text recognition result can be the address information "XX Road: No. 4" in the above text information, or the store information "Gong abcd Store" in the text information.

[0041] The method for text recognition of the target feature representation can be to input the target feature representation into a preset encoder for feature extraction to obtain the target feature representation after feature extraction. Then, input the target feature representation after feature extraction into a preset decoder for recognition processing or text detection processing to obtain the text recognition result of the image to be recognized.

[0042] In the above solution, an image to be recognized is obtained, and the feature representations of multiple image patches obtained by dividing the image to be recognized are subjected to a first downsampling to obtain an initial feature representation. The initial feature representation with a smaller number of feature representations is fused with the feature representations of the multiple image patches to obtain a target feature representation, so that the number of feature representations included in the target feature representation is less than the number of feature representations included in the image to be recognized. By performing text recognition on the target feature representation instead of directly performing text recognition on the image to be recognized, the number of feature representations in the text recognition process can be reduced, thereby improving the efficiency of text recognition.

[0043] In some embodiments, the above step S12 may include the following steps: First, perform feature extraction on the image to be recognized to determine the sampling weights of each image patch in the image to be recognized in at least one preset direction. Then, sample the feature representations of each image patch along at least one preset direction according to the sampling weights to obtain an initial feature representation.

[0044] Perform feature extraction on the image to be recognized to determine the sampling weights of each image patch in the image to be recognized in at least one preset direction.

[0045] The sampling weight is used to characterize the importance or attention of each image block in the image to be identified in at least one preset direction. Specifically, the sampling weight may be a numerical value corresponding to the feature extraction result corresponding to each image block in the image to be identified. In some application scenarios, the sampling weight may be a mapping value of the feature extraction result obtained by normalizing or activating the feature extraction result corresponding to each image block in the image to be identified. The value range of the mapping value is between 0 and 1. The larger the mapping value, the higher the importance of the sampling weight. The smaller the mapping value, the lower the importance of the sampling weight. In other application scenarios, when the image blocks in the image to be identified are arranged in multiple rows and columns, the sampling weight may be a row value table corresponding to all rows, or a column value table corresponding to all columns. For example, the image blocks in the image to be identified are arranged in 7×8. Among them, the image blocks in the image to be identified are arranged in 7 rows and 8 columns. The sampling weight may be a row value table corresponding to 7 rows, and each value in the row value table may be the importance of the image blocks in the same position in the row direction. The sampling weight may be a column value table corresponding to 8 columns, and each value in the column value table may be the importance of the image block corresponding to the same position in the column direction. It is understandable that, for an image block of the image to be identified, the mapping value of the image block in the row direction and the mapping value in the column direction may be different. Specifically, for an image block of the image to be identified, the importance of the image block in the row direction and the importance of the image block in the column direction may be different.

[0046] In some application scenarios, the process of performing feature extraction on the image to be identified and determining the sampling weight of each image block in the image to be identified in at least one preset direction may be as follows: first, the image to be identified is divided according to a preset size to obtain a plurality of image blocks. The image to be identified includes a plurality of image blocks, and sampling feature extraction is performed on each image block in the image to be identified in units of image blocks to obtain feature extraction results of each image block. The feature extraction results of each image block are arranged or combined according to at least one preset direction to obtain a feature extraction result arrangement set in at least one preset direction. Exemplarily, the image blocks contained in the image to be identified are arranged in multiple rows and / or multiple columns. Among them, the feature extraction result arrangement set in a preset direction includes a feature extraction result arrangement subset obtained by arranging or combining the feature extraction results of each image block in each of the preset directions. The feature extraction result arrangement set in a preset direction includes all feature extraction result arrangement subsets in the preset directions. For example, the feature extraction result arrangement set in the row direction includes the feature extraction result arrangement subsets of all rows. For a row of image blocks in the image to be identified, the feature extraction result arrangement subset of the row includes the feature extraction results of all image blocks in the row.

[0047] In other application scenarios, the process of extracting features from an image to be identified and determining the sampling weights of each image block in the image to be identified in at least one preset direction may be as follows: first, each image block in the image to be identified is divided in at least one preset direction according to a preset scale to obtain an image block set in at least one preset direction. The image block set in at least one preset direction may include multiple image block subsets in the same preset direction. Exemplarily, each image block in the image to be identified is divided into image blocks according to a preset scale to obtain multiple image blocks. The multiple image blocks may be a grid division of the image to be identified according to a preset scale, with each grid corresponding to an image block. The preset scale is used to indicate the size of each image block. Multiple image blocks in a preset direction are regarded as an image block subset in a preset direction. An image block subset in a preset direction includes multiple image blocks in the same preset direction. The preset direction includes a row direction or a column direction. The preset direction includes a horizontal direction or a vertical direction. In some application scenarios, the preset direction of the row direction is equivalent to the preset direction of the horizontal direction. In some application scenarios, the preset direction of the column direction is equivalent to the preset direction of the vertical direction. Exemplarily, multiple image blocks corresponding to a row are regarded as an image block subset in the row direction. The image to be identified includes multiple rows or multiple columns. Then, multiple image blocks corresponding to all rows are used as an image block set in the row direction. The image block set in the row direction includes image block subsets in each row direction. Then, in the case of obtaining an image block set in at least one preset direction, taking the image block set in at least one preset direction as a unit, sampling feature extraction is performed on each image block in the image block set in at least one preset direction to obtain a feature extraction result corresponding to the image block set in at least one preset direction. Exemplarily, taking at least the image block set in the row preset direction as a unit, sampling feature extraction is performed on each image block in the image block set in the row direction to obtain a feature extraction result corresponding to the image block set in the row direction. Specifically, sampling feature extraction is performed on the image block subset in the row direction in the image block set in the row direction to obtain a feature extraction result corresponding to each image block subset in the row direction.

[0048] In some application scenarios, each image block in a set of image blocks in at least one preset direction. The feature extraction result corresponding to the set of image blocks in at least one preset direction is used as the sampling feature extraction output. Or, each image block in the image to be identified is used as the sampling feature extraction input. The feature extraction result of each image block is used as the sampling feature extraction output. In some application scenarios, the sampling feature extraction method can be to use the sampling feature extraction input as the input of the preset feature extraction network layer, obtain the output of the preset feature extraction network layer, and use the output of the preset feature extraction network layer as the above-mentioned sampling feature extraction output. In some application scenarios, the preset feature extraction network can be a convolutional neural network (CNN), or a deep learning model such as a recurrent neural network (RNN), or a Transformer.

[0049] In some application scenarios, the above step of dividing the image to be identified into image blocks to obtain feature representations of multiple image blocks and the above step of extracting features from the image to be identified and determining the sampling weights of each image block in the image to be identified in at least one preset direction can be performed in parallel, and there is no order of precedence between the two. Both steps only need to be completed before the above step of sampling the feature representations of each image block along at least one preset direction according to the sampling weights to obtain the initial feature representation is performed.

[0050] It can be considered that by decomposing the feature representation of multiple image blocks in the image to be recognized into at least one preset direction for importance analysis, a more accurate sampling weight is obtained, so that the initial feature representation is determined to have a small data volume and a higher importance, so that the text recognition result determined by the initial feature representation is more accurate.

[0051] In some embodiments, there are multiple preset directions. The above-mentioned step of extracting features from the image to be identified and determining the sampling weight of each image block in the image to be identified in at least one preset direction may include the following steps: first, performing gradient feature extraction and second downsampling processing on the image to be identified to obtain gradient information and downsampled images in each preset direction. Subsequently, each gradient information is fused with the downsampled image to obtain the sampling weight corresponding to each preset direction.

[0052] Performing gradient feature extraction and second downsampling processing on the image to be identified to obtain gradient information in each preset direction and a downsampled image. Performing gradient feature extraction and second downsampling on the image to be identified can be performed serially or in parallel. Gradient feature extraction is used to extract gradient information of the image to be identified in different preset directions. The second downsampling processing can be to reduce the size of the image to be identified. Specifically, the second downsampling processing can reduce the size of the image to be identified by half. In some application scenarios, when the gradient feature extraction and second downsampling processing on the image to be identified are performed in parallel, the image to be identified is subjected to gradient feature extraction to obtain initial gradient information in each preset direction. And the image to be identified is subjected to second downsampling processing synchronously to obtain an initial downsampled image. Each initial gradient information is respectively combined with the first downsampled image to obtain gradient information in each preset direction. The initial downsampled image is used as a downsampled image. In other application scenarios, when the gradient feature extraction and second downsampling processing on the image to be identified are performed serially, the serial execution order can be to first perform the second downsampling processing on the image to be identified to obtain a downsampled image. Performing gradient feature extraction on the downsampled image to obtain gradient information in each preset direction. In other application scenarios, when the gradient feature extraction and the second downsampling process of the image to be identified are performed in series, the serial execution order may also be to first extract the gradient feature of the image to be identified to obtain the gradient information in each preset direction, and then perform the second downsampling process on the gradient information in each preset direction to obtain a downsampled image.

[0053] In other application scenarios, before the above-mentioned step of extracting gradient features and performing the second downsampling process on the image to be recognized to obtain the gradient information in each preset direction and the downsampled image, the text recognition method may include the following steps: preprocessing the image to be recognized to obtain a preprocessed image. Specifically, preprocessing the image to be recognized may be graying and smoothing the image to be recognized. Graying and smoothing the image to be recognized may be performed serially or in parallel. In some application scenarios, when the graying and smoothing of the image to be recognized are performed in parallel, the image to be recognized is grayed to obtain an initial grayed image. And the image to be recognized is smoothed synchronously to obtain a smoothed image. The initial grayed image is combined with the smoothed image to obtain the above-mentioned preprocessed image. In other application scenarios, when the graying and smoothing of the image to be recognized are performed serially, the image to be recognized is first grayed to obtain a grayed image. Then the grayed image is smoothed to obtain a smoothed image. The smoothed image is used as the preprocessed image. In other application scenarios, when grayscale processing and smoothing processing are performed serially on the image to be identified, the image to be identified is first smoothed to obtain a smoothed image, and then the smoothed image is grayscaled to obtain a grayscaled image, which is used as a preprocessed image.

[0054] The above-mentioned step of performing gradient feature extraction and second downsampling processing on the image to be identified to obtain gradient information in each preset direction and a downsampled image may include the following steps: performing gradient feature extraction and second downsampling processing on the preprocessed image to obtain gradient information in each preset direction and a downsampled image. The gradient feature extraction and second downsampling of the preprocessed image may be performed serially or in parallel. Exemplarily, the gradient feature extraction and second downsampling of the preprocessed image are performed serially. The serial execution order may be to first perform the second downsampling processing on the preprocessed image to obtain the downsampled image. The gradient feature extraction is performed on the downsampled image to obtain the gradient information in each preset direction. It can be understood that the above-mentioned step of performing gradient feature extraction and second downsampling on the image to be identified may be performed serially or in parallel, and will not be repeated here.

[0055] In other application scenarios, after performing gradient feature extraction and a second downsampling process on the image to be recognized to obtain the gradient information in each preset direction and the downsampled image, the text recognition method may further include the following steps: normalizing the gradient information and the downsampled image in each preset direction to obtain the normalized gradient information and the normalized downsampled image in each preset direction. Specifically, the gradient information in each preset direction is normalized to obtain the normalized gradient information in each preset direction. The downsampled image is normalized to obtain the normalized downsampled image. The above-mentioned step of fusing each gradient information with the downsampled image to obtain the sampling weight corresponding to each preset direction may further include the following steps: fusing each normalized gradient information with the normalized downsampled image to obtain the sampling weight corresponding to each preset direction.

[0056] At least one preset direction includes horizontal and vertical directions, and each gradient information includes first gradient information in the horizontal direction and second gradient information in the vertical direction. The above-mentioned step of performing gradient feature extraction and second downsampling processing on the image to be identified to obtain gradient information in each preset direction and the downsampled image may include: performing gradient feature extraction and second downsampling processing on the image to be identified to obtain the downsampled image, first gradient information in the horizontal direction and second gradient information in the vertical direction. The above-mentioned step of fusing each gradient information with the downsampled image to obtain the sampling weight corresponding to each preset direction may include: fusing the first gradient information with the downsampled image to obtain the first sampling weight in the horizontal direction; and fusing the second gradient information with the downsampled image to obtain the second sampling weight in the vertical direction. In other application scenarios, the above-mentioned steps of fusing each normalized gradient information with the normalized down-sampled image to obtain the sampling weights corresponding to each preset direction may include the following steps: fusing the normalized first gradient information with the normalized down-sampled image to obtain the first sampling weight in the horizontal direction; and fusing the normalized second gradient information with the normalized down-sampled image to obtain the second sampling weight in the vertical direction.

[0057] Among them, the gradient information in each preset direction can be used to represent the decomposition information of the signal corresponding to the image to be identified in different frequencies and directions. The first gradient information is used to represent the decomposition information of the image to be identified in the horizontal direction. The second gradient information is used to represent the decomposition information of the image to be identified in the vertical direction. The downsampled image contains semantic information representing the image to be identified after the second downsampling process. Exemplarily, the downsampled image can be an image of the image to be identified after the size change. For example, the downsampled image can be an image of the image to be identified after the size is halved. The sampling weights corresponding to each preset direction can represent the importance of each image block in the image to be identified in the preset direction. Specifically, the first sampling weight can represent the importance of each image block in the image to be identified in the horizontal direction. The second sampling weight can represent the importance of each image block in the image to be identified in the vertical direction.

[0058] In some embodiments, at least one preset direction includes horizontal and vertical directions, and each gradient information includes first gradient information in the horizontal direction and second gradient information in the vertical direction. The above-mentioned steps of fusing each gradient information with the downsampled image to obtain the sampling weights corresponding to each preset direction may include the following steps: using the first gradient information as the target gradient information and the sampling weight corresponding to the horizontal direction as the target sampling weight, or using the second gradient information as the target gradient information and the sampling weight corresponding to the vertical direction as the target sampling weight. Then, the target gradient information is subjected to gradient fusion processing with the downsampled image to obtain a first fused image. Then, the first fused image is subjected to initial feature extraction to obtain initial image features. Subsequently, the initial image features are subjected to first dimensional transformation processing to obtain first dimensional features. Then, the first dimensional features are input into the perceptron module for advanced feature extraction to obtain target sampling weights.

[0059] In some application scenarios, the target gradient information is subjected to gradient fusion processing with the downsampled image to obtain the first fused image by splicing the target gradient information with the downsampled image along the target direction to obtain the first fused image in the target direction. The preset direction corresponding to the target gradient information is used as the target direction. The gradient fusion processing may be splicing the target gradient information with the downsampled image along the target direction. The first fused image in the target direction may include the gradient information in the target direction and the semantic information of the downsampled image.

[0060] In some application scenarios, the process of performing initial feature extraction on the first fused image to obtain initial image features may be to input the first fused image in the target direction into a preset feature extraction network, perform initial feature extraction on the first fused image in the target direction, and obtain initial image features in the target direction. The preset feature extraction network may be the preset feature extraction network layer described above. Exemplarily, the preset feature extraction network may be a feature extraction network ConvNet.

[0061] In some application scenarios, the process of performing a first-dimensional transformation on the initial image features to obtain the first-dimensional features may be to input the initial image features in the target direction into the first-dimensional transformation module, perform a first-dimensional transformation on the initial image features in the target direction, and obtain the first-dimensional features in the target direction. Specifically, the first-dimensional transformation enables the first-dimensional features to adapt to the input size of the perceptron module, and the perceptron module outputs normally.

[0062] In some application scenarios, the first dimensional features are input into the perceptron module for advanced feature extraction to obtain the target sampling weight. In some application scenarios, the perceptron module may be an MLP layer. The first dimensional features in the target direction are input into the perceptron module for advanced feature extraction to obtain the advanced dimensional features in the target direction. The advanced dimensional features in the target direction are determined as the target sampling weight. In some application scenarios, the process of determining the advanced dimensional features in the target direction as the target sampling weight may be to perform normalization processing or activation function processing on the advanced dimensional features in the target direction to obtain the mapping value of the feature extraction result. And the mapping value of the feature extraction result is used as the target sampling weight in the target direction.

[0063] In some application scenarios, the first gradient information is subjected to gradient fusion processing with the downsampled image to obtain a first fused image in the horizontal direction. The first fused image in the horizontal direction is subjected to initial feature extraction to obtain initial image features in the horizontal direction. Subsequently, the initial image features in the horizontal direction are subjected to first dimension transformation processing to obtain first dimensional features in the horizontal direction. Then, the first dimensional features are input into the perceptron module for advanced feature extraction to obtain advanced first dimensional features in the horizontal direction. The advanced first dimensional features in the horizontal direction are determined as sampling weights corresponding to the horizontal direction. The sampling weights corresponding to the horizontal direction are first sampling weights. In some application scenarios, the process of determining the advanced first dimensional features in the horizontal direction as sampling weights corresponding to the horizontal direction may be normalizing or activating the advanced first dimensional features in the horizontal direction to obtain a mapping value of the feature extraction result. The mapping value of the feature extraction result is used as the first sampling weight. In other application scenarios, the second gradient information is subjected to gradient fusion processing with the downsampled image to obtain a first fused image in the vertical direction. The first fused image in the vertical direction is subjected to initial feature extraction to obtain initial image features in the vertical direction. Subsequently, the initial image features in the vertical direction are subjected to first dimension transformation processing to obtain first dimensional features in the vertical direction. Then, the first dimensional features are input into the perceptron module for advanced feature extraction to obtain the advanced first dimensional features in the vertical direction. The advanced first dimensional features in the vertical direction are determined as the sampling weights corresponding to the vertical direction. The sampling weights corresponding to the vertical direction are the second sampling weights. In some application scenarios, the process of determining the advanced first dimensional features in the vertical direction as the sampling weights corresponding to the vertical direction may be to perform normalization processing or activation function processing on the advanced dimensional features in the vertical direction to obtain the mapping value of the feature extraction result. And the mapping value of the feature extraction result is used as the second sampling weight.

[0064] It can be considered that since the text in natural scenes often has an obvious base plate, its edge is quite different from the background environment, so the frequency adaptation module is introduced. The base plate contains the background information of the area where the text is located. Figure 2 As shown, the baseboard includes the background color of the billboard in the area where the text is located. The frequency adaptation module can be represented as a Domain Adapte module. The above-mentioned steps of extracting features from the image to be identified and determining the sampling weights of each image block in the image to be identified in at least one preset direction are performed by the frequency adaptation module. Therefore, the introduction of the frequency adaptation module can reduce the interference of the background information in the area where the text is located, further enrich the modal information used by the network, make it easier for the network to pay attention to the text information in the high-resolution image, and simplify the difficulty of training.

[0065] In some embodiments, sampling the feature representation of each image block along at least one preset direction according to the sampling weight to obtain the initial feature representation may include the following steps: sampling the feature representation of each image block horizontally and / or sampling the feature representation of each image block vertically. In some application scenarios, when at least one preset direction is a preset direction, the preset direction may be horizontal or vertical, and sampling the feature representation of each image block along at least one preset direction according to the sampling weight to obtain the initial feature representation may be sampling the feature representation of each image block horizontally or sampling the feature representation of each image block vertically. Wherein, when the preset direction is horizontal, the process of horizontal sampling may be obtaining a first sampling weight, and performing a first screening on the feature representation of each image block in the horizontal direction according to the first sampling weight, to obtain a first selected number of candidate horizontal feature representations. The candidate horizontal feature representation is used as the initial feature representation. Wherein, when the preset direction is vertical, the process of vertical sampling may be obtaining a second sampling weight, and performing a first screening on the feature representation of each image block in the vertical direction according to the second sampling weight, to obtain a second selected number of candidate vertical feature representations. The candidate vertical feature representation is used as the initial feature representation. In other application scenarios, when at least one preset direction is multiple preset directions. Exemplarily, the present application takes multiple preset directions as horizontal and vertical as an example, and the number of multiple preset directions is not limited here. The process of sampling the feature representation of each image block along at least one preset direction according to the sampling weight to obtain the initial feature representation can be a serial execution of horizontal sampling and vertical sampling, or a parallel execution of horizontal sampling and vertical sampling. Serial execution of horizontal sampling and vertical sampling can be first performing horizontal sampling on the feature representation of each image block, and then performing vertical sampling on the feature representation of each image block. Serial execution of horizontal sampling and vertical sampling can be first performing vertical sampling on the feature representation of each image block, and then performing horizontal sampling on the feature representation of each image block. It can be understood that the specific process of horizontal sampling and vertical sampling can refer to the above content, and will not be repeated here.

[0066] In some application scenarios, the initial feature representation may represent at least part of the feature representation of each image block. The above-mentioned method of sampling the feature representation of each image block may be based on the target sampling weight in the target direction, and the feature representation of each image block is sampled and sorted to obtain the importance ranking corresponding to the feature representation of each image block. Based on the importance ranking, the feature representation of each image block in each target direction is screened to obtain a candidate feature representation. Wherein, in the case where at least one preset direction is a preset direction, the preset direction may be horizontal or vertical, and the candidate feature representation is directly used as the initial feature representation. Wherein, in the case where at least one preset direction is a plurality of preset directions, the target direction may be horizontal or vertical, and the initial feature representation is determined based on each candidate feature representation.

[0067] In some embodiments, the sampling weight corresponding to the horizontal direction is used as the first sampling weight, and the sampling weight corresponding to the vertical direction is used as the second sampling weight. The above step of sampling the feature representation of each image block along at least one preset direction according to the sampling weight to obtain the initial feature representation may include the following steps: first, obtaining a first selected number of image blocks in the horizontal direction of the image to be identified and a second selected number of image blocks in the vertical direction of the image to be identified. Then, according to the first sampling weight, the feature representation of each image block in the horizontal direction is first screened to obtain a first selected number of candidate horizontal feature representations; and, according to the second sampling weight, the feature representation of each image block in the vertical direction is first screened to obtain a second selected number of candidate vertical feature representations. Next, each candidate horizontal feature representation and each candidate vertical feature representation is second screened to obtain a preset number of selected feature representations, and the preset number is equal to the product of the first selected number and the second selected number. Subsequently, the initial feature representation is determined based on the selected feature representation.

[0068] The first selection number and the second selection number may be preset. The first selection number and the second selection number may be the same or different. In some application scenarios, the first selection number and the second selection number are determined based on the input size of the image encoder. Specifically, the input size of the image encoder may be the height of the first preset number and the width of the second preset number. In some application scenarios, the height in the input size of the image encoder is composed of the height of the first preset number of image blocks, and the width in the input size of the image encoder is composed of the width of the first preset number of image blocks. The horizontal direction corresponds to the row direction, and the vertical direction corresponds to the column direction. When the target sampling weight is the first sampling weight, the importance ranking may be the mapping value ranking of the feature representation of each image block for each row. In some application scenarios, the first screening may be, for each row, based on the mapping value ranking of the feature representation of each image block, to screen out the feature representation with the top mapping value ranking from the feature representation of each image block, and use the top ranking feature representation as the candidate horizontal feature representation. Among them, the number of candidate horizontal feature representations is the first preset number, and the top ranking feature representation may be the feature representation of the top M1 image blocks in the mapping value ranking in each row. The value of M1 is the same as the value of the first preset number. When the target sampling weight is the second sampling weight, for each column, the mapping values ​​of the feature representations of each image block are sorted. In other application scenarios, the first screening may also be for each column, based on the sorting of the mapping values ​​of the feature representations of each image block, to screen out the feature representations with top mapping value rankings from the feature representations of each image block, and use the top-ranked feature representations as candidate longitudinal feature representations. Among them, the number of candidate longitudinal feature representations is the second preset number, and the top-ranked feature representations may be the feature representations of the top M2 image blocks in the sorting of mapping values ​​in each column. The value of M2 is the same as the value of the second preset number. It can be understood that each candidate horizontal feature representation and each candidate longitudinal feature representation may be a feature representation of the same image block, or may be a feature representation of different image blocks.

[0069] In some application scenarios, the second screening may be to compare whether each candidate horizontal feature representation and each candidate vertical feature representation are feature representations of the same image block. In response to each candidate horizontal feature representation and each candidate vertical feature representation being feature representations of the same image block, the feature representations of the same image block are determined as selected feature representations. The selected feature representations include feature representations of a preset number of image blocks.

[0070] In some other application scenarios, the second screening may also be to overlap each candidate horizontal feature representation and each candidate vertical feature representation to obtain a candidate target feature representation. Because each candidate horizontal feature representation and each candidate vertical feature representation may be a feature representation of at least some different image blocks, the number of image blocks corresponding to the candidate target feature representation is greater than the number of image blocks corresponding to the selected feature representation, that is, the number of image blocks corresponding to the candidate target feature representation is greater than a preset number. Feature representations of a preset number of image blocks are selected from the target candidate feature representation in a manner that the feature representation mapping values ​​of the image blocks are ranked high, and the feature representations of the preset number of image blocks are determined as each selected feature representation. The selected feature representation is a feature representation of a preset number of image blocks. It can be understood that the selected feature representation may include image information of the image block. In some application scenarios, the above-mentioned step of determining the initial feature representation based on each selected feature representation may be to directly combine each selected feature representation to obtain the initial feature representation. The initial feature representation may include a preset number of selected feature representations.

[0071] It can be considered that the initial feature representation at this time is a feature representation with a higher degree of importance in the image to be identified.

[0072] In some embodiments, the above-mentioned step of determining the initial feature representation based on each selected feature representation may include the following steps: first, obtaining the position information corresponding to each selected feature representation, and the position information is used to indicate the positional relationship between the selected feature representation and each image block of the image to be identified.

[0073] Subsequently, each selected feature representation is positionally fused with each position information to obtain an initial feature representation.

[0074] The positional relationship may be the positional information of the image blocks corresponding to each selected feature representation in the image to be identified. In some application scenarios, the positional relationship may be obtained by first acquiring a blank image. The size of the blank image is the same as that of the image to be identified. Then, the blank image is divided into image blocks to obtain each blank image block. Each blank image block is positionally encoded to obtain the positional information of the blank image block. The positional information of the blank image block is a trainable two-dimensional positional encoding. The image blocks corresponding to each selected feature representation are compared with each blank image block. In response to the position of the image block corresponding to a certain selected feature representation in the image to be identified being the same as the position of a certain blank image block in the blank image block, the positional information of the blank image block is used as the positional information corresponding to the selected feature representation. In some application scenarios, each selected feature representation is positionally fused with each positional information to obtain the initial feature representation, which may be splicing each selected feature representation with each positional information to obtain the initial feature representation. Specifically, for each selected feature representation, the positional information corresponding to the selected feature representation is added to the selected feature representation as a feature label to obtain the added selected feature representation. And each added selected feature representation is combined into the initial feature representation. The initial feature representation may include a preset number of added selected feature representations.

[0075] It can be considered that the initial feature representation at this time is a feature representation that is relatively important in the image to be identified and has position information.

[0076] In some embodiments, the above step S13 may include the following steps: First, the initial feature representation and the feature representation of multiple image blocks are input into the cross-attention layer to obtain the feature representation output by the cross-attention layer. The initial feature representation is used as the query vector of the cross-attention layer, and the feature representation of multiple image blocks is used as the key vector and value vector of the cross-attention layer. Next, the initial feature representation and the feature representation output by the cross-attention layer are subjected to feature fusion processing to obtain an advanced feature representation. Then, the advanced feature representation is normalized to obtain a target feature representation.

[0077] The feature representations of the multiple image blocks are the feature representations of all image blocks in the image to be identified. The inputs of the cross attention layer are the initial feature representation and the feature representations of the multiple image blocks, respectively. Specifically, the cross attention layer can be a CrossAttention module. Exemplarily, the cross attention layer can be a Scaled Dot-ProductAttention layer in the cross attention layer.

[0078] The cross-attention layer uses the input initial feature representation as the query vector of the cross-attention layer, and the feature representations of multiple image blocks as the key vector and value vector of the cross-attention layer, so that the initial feature representation and the feature representations of multiple image blocks are information-fused, and the feature representation output by the cross-attention layer is added with the image information extracted from the feature representations of other image blocks except the image block corresponding to the initial feature representation. It can be understood that the initial feature representation can be sampled tokens. The feature representations of multiple image blocks can be the image tokens of the original input. Using the sampled tokens as the query vector Query, the original input image tokens as the key vector Key and the value vector Value, the information fusion of the initial feature representation and the feature representations of multiple image blocks can be self-attention Attention calculation. The feature fusion processing of the initial feature representation and the feature representation output by the cross-attention layer to obtain the advanced feature representation can be cross-layer addition of the initial feature representation and the feature representation output by the cross-attention layer to obtain the advanced feature representation. Then the advanced feature representation is normalized to obtain the target feature representation, which can be layer-normalized to obtain the target feature representation. The target feature representation is the last sampled image tokens.

[0079] It can be considered that the target feature representation has the image information extracted by the feature representation of other image blocks except the image block corresponding to the initial feature representation. Since the number of tokens contained in the query vector Query is , often than The number is much smaller, which can greatly reduce the amount of self-attention calculation. Taking the typical input size of 224×224 of the image encoder CLIP-ViT as an example, assuming that the image to be recognized is scaled to a fixed size of 1600×2048, the size of the image block is represented by patch_size, and the size of the image block is 16×16, then the number of tokens in the query is reduced from the original Became , so that the entire network can maintain high-resolution input in the scene text recognition task. In addition, the target feature representation directly uses the pre-trained image encoder CLIP-ViT model. The feature representations of each image block in the target feature representation do not need to be aligned again, so that visual instruction fine-tuning can be achieved.

[0080] In some embodiments, the above step S14 may include the following steps: first, extract the target feature representation using an image encoder to obtain a target encoding representation corresponding to the target feature representation. Then, perform a second dimension transformation process on the target encoding representation to obtain a second dimension feature. Next, fuse the second dimension feature and the preset prompt word to obtain a target fusion representation. Subsequently, determine the text recognition result based on the target fusion representation.

[0081] The image encoder is a ViT network that has been feature aligned in CLIP, and the image encoder is recorded as a CLIP-ViT model. The process of extracting features from the target feature representation using the image encoder to obtain the target encoding representation corresponding to the target feature representation is the process of feature alignment between the feature representations of each image block in the target feature representation. In some application scenarios, the feature representation of each image block in the target feature representation may include an image vector and a text vector. Each image vector and each text vector are respectively mapped to the same modal space, and the target encoding representation after feature alignment is obtained by similarity calculation. The second dimension transformation process enables the size corresponding to the second dimension feature to meet the size requirement in the text recognition process, so that the output text recognition result has high accuracy. The preset prompt word can be prompt. The preset prompt word can be set to "Recognize the text in the image and give the corresponding center coordinates", that is, "recognize the text in the image and give the corresponding center coordinates". The second dimension feature and the preset prompt word are fused to obtain the target fused representation. The process can be splicing the second dimension feature and the preset prompt word to obtain the target fused representation. Based on the target fusion representation, the process of determining the text recognition result can be to input the target fusion representation into a pre-trained large language model to obtain the text recognition result. Specifically, the large language model can be a large language model LLM. Exemplarily, the large language model LLM can adopt an open source pre-trained large language model such as Llama3, Opt, T5, etc.

[0082] In some application scenarios, the size of the image to be recognized is the image scaled to . It can represent the height of the image to be recognized. is the width of the image to be identified. It can be understood that the image to be identified is a scaled image, and for the image to be identified, the image before scaling can be a high-resolution size. The size of the image to be identified is a preset size, which can be expressed as In order to prevent the small-scale target information in the scene from being lost, the preset size in the image to be identified Use high-resolution dimensions. For example, the preset dimensions Can be used Enter the size.

[0083] In some application scenarios, the patch module can be represented as a Patch Embedding module. The above steps of dividing the image to be recognized into image blocks and obtaining feature representations of multiple image blocks are performed by the patch module. The image block size of the multiple image blocks output by the patch module is the same as the image block size corresponding to the target feature representation input by the image encoder CLIP-ViT model. For an image block in this application, the image block size of the image block can be recorded as . Can represent image blocks. It can represent the height of the image block in the image block size. It can represent the width of the image block in the image block size. The target feature of the image encoder input is represented as a fixed size, which can be expressed as . It can be said that the target feature represents the height of the corresponding image. It can represent the width of the image corresponding to the target feature representation. Exemplarily, the fixed size can be or The size of each image block in the image corresponding to the target feature representation can be expressed as , then the number of feature representations of each image block in the image corresponding to the target feature representation input by the image encoder is .

[0084] It can be considered that the feature representations of each image block in the target feature representation do not need to be aligned again, and fine-tuning the visual instructions can reduce the amount of calculation. On the basis of the image to be recognized being a large-scale image input, the accuracy of the text recognition result can be improved.

[0085] In some embodiments, the above-mentioned steps of extracting features from the image to be identified and determining the sampling weights of each image block in the image to be identified in at least one preset direction are performed by a frequency adaptation module, and the text recognition method may also include the following steps: First, obtain the labeled distribution information of the sample image in at least one preset direction. Secondly, use the frequency adaptation module to extract features from the sample image and determine the sample sampling weights of each image block in the sample image in at least one preset direction. Then, perform Gaussian distribution processing on the sample sampling weights to determine the predicted distribution information. Next, determine the target loss based on the difference between the labeled distribution information and the predicted distribution information. Subsequently, adjust the parameters of the frequency adaptation module according to the target loss.

[0086] The text information contained in the sample image can come from a publicly available scene text dataset. For example, the text dataset can be a CTW dataset, a ShopSign dataset, etc. The original annotation information in the text dataset is formatted into a sample style. The sample style is associated with the above-mentioned preset prompt words. The sample style can be Among them, the sample style Can represent text information. Indicates the center position of the text. When the original annotation information in the text dataset is formatted into a sample style, a semicolon is used as a separator when a sample image contains multiple text targets. The preset prompt words can be set by referring to the above content. Project the text center coordinates at the image scale in the sample image to the feature map scale. Assume that At the image scale, the center coordinates of a text are , then its projection coordinates are .in, is the width of the image block after the image block is divided, is the height of the image block divided by the image block. With this point as the center, Gaussian distribution is applied along the column direction and row direction respectively, that is, , The value range is , get the first target distribution value; similarly , the value range of Y is , as the second target distribution value. In the case where there are multiple text targets in a single image, each first target distribution value and the second target distribution value are superimposed respectively to obtain a fused distribution value in at least one preset direction. Among them, the fused distribution value in at least one preset direction is used as the annotation distribution information in at least one preset direction. The fused distribution value in at least one preset direction may include a first fused distribution value and a second fused distribution value. The first fused distribution value and the second fused distribution value may correspond to different preset directions. In some application scenarios, each first target distribution value is superimposed to obtain a first fused distribution value in the first direction. The first direction may be horizontal or vertical. In other application scenarios, each second target distribution value is superimposed to obtain a second fused distribution value in the second direction. The second direction may be horizontal or vertical.

[0087] It is understandable that the above-mentioned step of extracting features from the sample image using the frequency adaptation module and determining the sample sampling weight of each image block in the sample image in at least one preset direction can refer to the above-mentioned step of extracting features from the image to be identified and determining the sampling weight of each image block in the image to be identified in at least one preset direction. The sample image can be the same image or the same type of image as the image to be identified. The sample image can be the same as the image to be identified, and the sample image can be an image containing text information.

[0088] In some application scenarios, the sampling weights and the labeled distribution information in the same direction are directly compared, and the target loss is determined based on the difference between the sampling weights and the labeled distribution information in each same direction. According to the target loss, the parameters of the frequency adaptation module are adjusted. In other application scenarios, the sample sampling weights are Gaussian distributed, and the predicted distribution information is determined by performing Gaussian distribution processing on the sample sampling weights in at least one preset direction to determine the predicted distribution information in at least one preset direction. Then, based on the difference between the labeled distribution information and the predicted distribution information in each same direction, the target loss is determined. According to the target loss, the parameters of the frequency adaptation module are adjusted. In some application scenarios, the target loss can be expressed as The target loss can be KL divergence loss.

[0089] In some embodiments, the above-mentioned step of dividing the image to be identified into image blocks to obtain feature representations of multiple image blocks is performed by a block division module, the above-mentioned step of extracting features of the image to be identified and determining the sampling weight of each image block in the image to be identified in at least one preset direction is performed by a frequency adaptation module, and the above-mentioned step of sampling the feature representation of each image block along at least one preset direction according to the sampling weight to obtain the initial feature representation is performed by a sampling module. The text recognition method may also include a training step for the block division module, the frequency adaptation module and the sampling module, and the above-mentioned training step may include the following steps: first, obtaining a sample image and text annotation information corresponding to the sample image. Secondly, using the block division module to divide the sample image into image blocks to obtain sample feature representations of multiple sample image blocks. Then, using the frequency adaptation module to extract features of the sample image, determine the sample sampling weight of each sample image block in the sample image in at least one preset direction. Subsequently, using the sampling module to sample the sample feature representation of each image block along at least one preset direction according to the sample sampling weight to obtain the sample initial feature representation. Using the sampling module, the sample initial feature representation is merged with the sample feature representation of all image blocks to obtain the sample target feature representation. Perform text recognition on the sample target feature representation to obtain a sample text recognition result for the sample image. Based on the difference between the sample text recognition result and the text annotation information, adjust the parameters of the segmentation module, the frequency adaptation module and the sampling module. It can be understood that the various steps in the training process can refer to the above-mentioned steps of dividing the image to be identified into image blocks in the application process to obtain the feature representation of multiple image blocks, the above-mentioned steps of extracting features from the image to be identified and determining the sampling weights of each image block in the image to be identified in at least one preset direction, the above-mentioned steps of sampling the feature representation of each image block along at least one preset direction according to the sampling weight to obtain the initial feature representation, step S13 and step S14, which will not be repeated here.

[0090] Among them, the text recognition loss is determined based on the difference between the sample text recognition result and the text annotation information. In some application scenarios, the text recognition loss is directly used to adjust the parameters of the block module, the frequency adaptation module, and the sampling module. In some application scenarios, the text recognition loss and the target loss are weighted and summed to obtain the final loss. Then, based on the final loss, the parameters of the block module, the frequency adaptation module, and the sampling module are adjusted.

[0091] In some embodiments, the above-mentioned steps of performing text recognition on the sample target feature representation and obtaining the sample text recognition result about the sample image can be sequentially performed by the image encoder, the second dimension transformation module and the decoder. Specifically, the above-mentioned step of extracting features from the target feature representation using the image encoder and obtaining the target coding representation corresponding to the target feature representation is performed by the image encoder. The above-mentioned step of performing second dimension transformation on the target coding representation and obtaining the second dimension feature is performed by the second dimension transformation module. After the step of fusing the second dimension feature and the preset prompt word to obtain the target fusion representation, the above-mentioned step of determining the text recognition result based on the target fusion representation is performed by the decoder. When training the parameters of each module, the parameters of the image encoder and the decoder can be frozen in advance, and the parameters of the remaining modules can be adjusted. Among them, the above-mentioned remaining modules can be a blocking module, a frequency adaptation module, a sampling module and a second dimension transformation module. Exemplarily, in the network training stage, the parameters of the decoder LLM large language model and the image encoder CLIP-ViT are frozen. The size of the input sample image is scaled to obtain . The frequency adaptation module can be represented as a Domain Adaptor module. The sampling module can be represented as a Token Resample module. Based on the difference between the sample text recognition result and the text annotation information, the process of determining the text recognition loss can be to compare the sample text recognition result predicted by the decoder with the text annotation information. Specifically, the predicted text token id in the sample text recognition result is compared with the token id of the real text in the text annotation information to obtain the text recognition loss. Based on the text recognition loss gradient back propagation, the parameters of the trainable part of the model are updated until the loss reaches the set threshold, and the training of the trainable part of the model is completed. The trainable part of the model can be the remaining modules mentioned above.

[0092] In some application scenarios, the text recognition loss and the target loss are weighted and summed to obtain the final loss. Specifically, the process of obtaining the final loss can refer to formula (1):

[0093] Formula (1);

[0094] in, Can represent final loss. Can represent target loss. It can represent the text recognition loss. Among them, is the loss corresponding to the LLM model, which is usually determined by the cross entropy loss function. It can represent the loss weight. is the default value, It can be set to 0.1. In some other application scenarios, the loss weight can also be set dynamically. For example, it can be set according to the ratio between the text recognition loss and the target loss. For example, the text recognition loss and the target loss are summed to obtain the loss sum. The loss weight can be equal to the ratio between the target loss and the loss sum.

[0095] The above scheme obtains an image to be identified, performs a first down-sampling on the feature representations of multiple image blocks obtained by dividing the image to be identified, obtains an initial feature representation, and fuses the initial feature representation with a smaller number of feature representations with the feature representations of multiple image blocks to obtain a target feature representation, so that the number of feature representations contained in the target feature representation is less than the number of feature representations contained in the image to be identified. By performing text recognition on the target feature representation instead of directly performing text recognition on the image to be identified, the number of feature representations in the text recognition process can be reduced, thereby improving the efficiency of text recognition.

[0096] See also Figure 3 , Figure 3 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 2 .

[0097] Execute step S301: obtain the image to be identified. After step S301, the above-mentioned step of dividing the image to be identified into image blocks to obtain feature representations of multiple image blocks may also be executed.

[0098] Step S302: Use the frequency adaptation module to extract features from the image to be identified, and determine the sampling weights of each image block in the image to be identified in at least one preset direction. It can be understood that the above steps of dividing the image to be identified into image blocks and obtaining feature representations of multiple image blocks can be performed in parallel with step S302. After step S302 is executed, steps S303, S304, S305, S306, S307 and S308 can be performed in sequence. Step S303: Use the sampling module to sample the feature representations of each image block along at least one preset direction according to the sampling weight to obtain an initial feature representation. Step S304: Use the sampling module to fuse the initial feature representation with the feature representations of multiple image blocks to obtain a target feature representation. Step S305: Use the image encoder to extract features from the target feature representation to obtain a target coded representation corresponding to the target feature representation. Step S306: Use the second dimension transformation module to perform a second dimension transformation on the target coded representation to obtain a second dimension feature. Step S307: Fusing the second dimensional features and the preset prompt words to obtain a target fusion representation.

[0099] Step S308: Determine the text recognition result based on the target fusion representation using the pre-trained large language model. It is understandable that steps S301 to S308 can refer to the above content and will not be repeated here.

[0100] In some application scenarios, the second dimension transformation module can be represented as a Projection module. The second dimension transformation module is used to perform dimension mapping on the image feature representation image tokens in the target encoding representation output by the image encoder to adapt to the input feature dimension required by the large language model LLM. Among them, the second dimension transformation module can use a multi-layer perceptron or a single linear transformation as a connection structure. The second dimension transformation module can concatenate the image feature representation image tokens in the mapped second dimensional features and the text vector text tokens corresponding to the preset prompt word prompt as the input of the large language model LLM to achieve visual instruction fine-tuning. It can be understood that the image feature representation image tokens in the target encoding representation and the image feature representation image tokens in the second dimensional features both contain image information and text information after feature alignment by the image encoder.

[0101] It can be considered that there may be identical or similar text segments in natural scenes. By further presetting the prompt words, the center coordinates of the text can be learned to increase the interpretability of the text recognition results.

[0102] See also Figure 4 , Figure 4 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 3 .

[0103] Steps S401 to S408 are performed in sequence. Step S401: Preprocess the image to be identified to obtain a preprocessed image. Step S402: Perform gradient feature extraction and second downsampling processing on the preprocessed image to obtain gradient information and downsampled images in each preset direction. Step S403: Normalize the gradient information and downsampled images in each preset direction to obtain normalized gradient information and normalized downsampled images in each preset direction. Step S404: Fuse each normalized gradient information with the normalized downsampled image to obtain sampling weights corresponding to each preset direction. Step S405: Perform gradient fusion processing on the target gradient information and the downsampled image to obtain a first fused image. Step S406: Perform initial feature extraction on the first fused image to obtain initial image features. Step S407: Perform first dimensional transformation processing on the initial image features to obtain first dimensional features. Step S408: Input the first dimensional features into the perceptron module for advanced feature extraction to obtain target sampling weights. It is understandable that steps S401 to S408 can refer to the above content and will not be repeated here. In some application scenarios, the frequency adaptation module can sequentially execute steps S401 to S408 to obtain the target sampling weight. The target sampling weight is the sampling weight in at least one preset direction.

[0104] Exemplarily, the above step S401 may be graying and smoothing the image to be identified to obtain a preprocessed image, and mean filtering or Gaussian filtering may be used for smoothing to reduce noise interference.

[0105] The above steps S402 and S403 may be processed by wavelet transform and normalized to obtain a normalized downsampled image and normalized gradient information. It is understandable that the wavelet transform process is equivalent to the wavelet transform dwt process. The wavelet transform process may refer to processing the preprocessed image using the wavelet function Biorthogonal.

[0106] The wavelet transform process includes gradient feature extraction and second downsampling process. The normalized downsampled image can be expressed as the downsampled image The normalized gradient information includes the normalized first gradient information and the normalized second gradient information. The normalized first gradient information can be represented as a horizontal spectrum diagram: The normalized second gradient information can be expressed as a vertical spectrum graph: .like Figure 3 The normalized downsampled image size obtained after Biorthogonal processing and normalization is changed to . In some application scenarios, the process of step S404 to step S408 can be to fuse the normalized first gradient information with the normalized downsampled image to obtain a first sampling weight in the horizontal direction; and fuse the normalized second gradient information with the normalized downsampled image to obtain a second sampling weight in the vertical direction. For the preset direction being the horizontal direction, the normalized downsampled image and the normalized first gradient information are spliced ​​along the channel dimension to obtain a first fused image in the horizontal direction. , >Splicing along the channel dimension to obtain a first fused image in the horizontal direction. For the preset direction being the vertical and horizontal directions, splicing the normalized downsampled image and the normalized second gradient information along the channel dimension to obtain a first fused image in the horizontal direction. , >Splicing is performed along the channel dimension to obtain the first fused image in the horizontal direction and the first fused image in the vertical direction. The first fused image in the horizontal direction and the first fused image in the vertical direction are respectively sent to the feature extraction network ConvNet for initial feature extraction, and then pass through the branches of the two perceptron modules MLP to obtain the advanced first dimensional features in the horizontal direction output by one of the perceptron modules and the advanced first dimensional features in the vertical direction output by the other perceptron module. The advanced first dimensional features in the horizontal direction can be the feature expression in the horizontal direction. The number of features in the advanced first dimensional features in the horizontal direction is The advanced first dimensional features in the longitudinal direction can be the feature expression in the vertical direction. The number of features in the advanced first dimensional features in the longitudinal direction is In the initial feature extraction process, the convolution operations in ConvNet all use large-size convolution kernels kernal_size and stride to expand the receptive field.

[0107] It can be understood that the wavelet transform process analyzes the signal by using wavelet functions of different scales and directions, and the results include coefficients at each scale and direction. These coefficients can be used to represent the changes of the signal at different scales and directions, thereby making the initial feature representation rich in features and improving the accuracy of the sampling weights. The sampling weights with high accuracy further improve the accuracy of the subsequent target feature representations that are screened out.

[0108] It can be considered that when performing natural scene text recognition, a visual multimodal architecture is used. Under the premise of high-resolution input, the target feature representation is obtained by resampling the feature representation token of the image block, so that the target feature representation can directly use the parameters of the CLIP-ViT and LLM pre-trained models for text recognition to obtain text recognition results, ensuring that more accurate recognition and positioning of small target texts can be achieved within an affordable computing range, making the text recognition results more accurate.

[0109] It can be considered that although the resolution of the image to be recognized is different from the resolution that the image encoder can input, after feature sampling, the target feature representation is the same as the resolution that the image encoder can input, which achieves input adaptation and does not require feature alignment of the image and text again. In addition, during the training process, the parameters of the pre-trained CLIP-ViT and LLM models are frozen and only the remaining modules mentioned above are trained, reducing the required training data size.

[0110] It can be considered that by fine-tuning visual instructions, the generalization ability of LLM can be fully utilized to improve the accuracy of text positioning and recognition in natural scenes.

[0111] It can be considered that under the premise that the image to be recognized is a high-resolution image input, the small target text can provide more effective information, but at the same time it also introduces a lot of interference information. Therefore, the frequency domain adaptation module is used to explicitly add the frequency information of at least one preset direction of the image. Specifically, the frequency information in the row and column directions. The problem of background interference under high-resolution input is alleviated, and further attention is paid to the area of ​​interest. In addition, by performing text recognition on the target feature representation instead of directly performing text recognition on the image to be recognized, the number of feature representations in the text recognition process can be reduced, thereby improving the efficiency of text recognition.

[0112] And use target loss to stabilize network training and simplify the difficulty of model training.

[0113] It can be considered that by decomposing the image to be identified into rows and columns for importance analysis, the corresponding selected feature representation is obtained after resampling and combination. The selected feature representation can be represented as Query tokens. It is ensured that the overall computational complexity of the network will not be significantly increased under high-resolution input. At the same time, a blank image that can be position-encoded is added to the sampled selected feature representation. The position information of the blank image block in the blank image can be represented as 2D Position Embedding. The position information of the blank image block is a trainable two-dimensional position encoding. Each selected feature representation is positionally fused with each position information to obtain an initial feature representation. The initial feature representation and the feature representation of multiple image blocks are input into the cross-attention layer to obtain the feature representation output by the cross-attention layer to achieve further fusion of features, so that the target feature representation has a high degree of importance and position information in the image to be identified.

[0114] See also Figure 5 , Figure 5 This is a flowchart of an embodiment of the text recognition method of the present application. Figure 4 .

[0115] Execute steps S501 to S504 in sequence. Step S501: Sample the feature representation of each image block along at least one preset direction according to the sampling weight to obtain an initial feature representation. Step S502: Input the initial feature representation and the feature representations of multiple image blocks into the cross-attention layer to obtain the feature representation output by the cross-attention layer. Step S503: Perform feature fusion processing on the initial feature representation and the feature representation output by the cross-attention layer to obtain an advanced feature representation. Step S504: Normalize the advanced feature representation to obtain a target feature representation. Among them, the initial feature representation in step S501 is a feature representation with a high degree of importance and position information in the image to be identified. In some application scenarios, the sampling module can execute steps S501 to S504 in sequence to obtain the target feature representation.

[0116] In some application scenarios, the sampling module can be represented as a Token Resample module. The sampling module is used to sample the feature representation image tokens of multiple image blocks that have passed through the block module, and use the feature expressions in the horizontal and vertical directions obtained by the frequency adaptation module, that is, the advanced first dimensional features in the horizontal direction and the advanced first dimensional features in the vertical direction as the basis for screening. The advanced first dimensional features in the horizontal direction and the advanced first dimensional features in the vertical direction are respectively normalized or activated to obtain the mapping value of the feature extraction result. The mapping value of the feature extraction result includes the mapping value in the horizontal direction and the mapping value in the vertical direction. The mapping value in the horizontal direction is used as the first sampling weight. The mapping value in the vertical direction is used as the second weight. The advanced first dimensional features in the horizontal direction and the advanced first dimensional features in the vertical direction are respectively normalized or activated to obtain the mapping value of the feature extraction result by performing a Sigmoid operation on the advanced first dimensional features in the horizontal direction and the advanced first dimensional features in the vertical direction to obtain a confidence score. The confidence score can be the mapping value of the feature extraction result. The above step S501 can be to retain the first M1 rows with the highest confidence in the row direction, and retain the first M2 columns with the highest corresponding confidence in the column direction, and finally obtain a preset number of selected feature representations. Among them, M1 can be expressed as M2 can be expressed as The above step of determining the initial feature representation based on each selected feature representation may include the following steps: performing a first screening and a second screening on the trainable two-dimensional position encoding to obtain the position information of each image block after the position encoding. Each selected feature representation is spliced ​​with each position information to obtain the initial feature representation.

[0117] The above scheme obtains an image to be identified, performs a first down-sampling on the feature representations of multiple image blocks obtained by dividing the image to be identified, obtains an initial feature representation, and fuses the initial feature representation with a smaller number of feature representations with the feature representations of multiple image blocks to obtain a target feature representation, so that the number of feature representations contained in the target feature representation is less than the number of feature representations contained in the image to be identified. By performing text recognition on the target feature representation instead of directly performing text recognition on the image to be identified, the number of feature representations in the text recognition process can be reduced, thereby improving the efficiency of text recognition.

[0118] See also Figure 6 , Figure 6It is a structural diagram of an embodiment of the text recognition device of the present application. The text recognition device 60 includes an acquisition module 61, a sampling module 62, a fusion module 63 and a recognition module 64. The acquisition module 61 is used to acquire the image to be recognized; the sampling module 62 is used to perform a first down-sampling on the feature representations of multiple image blocks obtained by dividing the image to be recognized, and obtain an initial feature representation. The combination of the feature representations of multiple image blocks is the feature representation of the image to be recognized, and the initial feature representation carries the feature representations of at least some of the image blocks; the fusion module 63 is used to fuse the initial feature representation with the feature representations of multiple image blocks to obtain a target feature representation, and the number of feature representations of image blocks contained in the target feature representation is the same as the number of feature representations of image blocks contained in the initial feature representation; the recognition module 64 is used to perform text recognition on the target feature representation to obtain a text recognition result on the image to be recognized.

[0119] The above scheme obtains an image to be identified, performs a first down-sampling on the feature representations of multiple image blocks obtained by dividing the image to be identified, obtains an initial feature representation, and fuses the initial feature representation with a smaller number of feature representations with the feature representations of multiple image blocks to obtain a target feature representation, so that the number of feature representations contained in the target feature representation is less than the number of feature representations contained in the image to be identified. By performing text recognition on the target feature representation instead of directly performing text recognition on the image to be identified, the number of feature representations in the text recognition process can be reduced, thereby improving the efficiency of text recognition.

[0120] Please refer to the text recognition method for the functions performed by each module, which will not be repeated here.

[0121] See also Figure 7 , Figure 7 7 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 70 includes a memory 71 and a processor 72, and the processor 72 is used to execute program instructions stored in the memory 71 to implement the steps in the above text recognition method embodiment. In a specific implementation scenario, the electronic device 70 may include but is not limited to: a multi-camera device, a microcomputer, and a server. In addition, the electronic device 70 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0122] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in the above-mentioned text recognition method embodiment. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0123] The above scheme obtains an image to be identified, performs a first down-sampling on the feature representations of multiple image blocks obtained by dividing the image to be identified, obtains an initial feature representation, and fuses the initial feature representation with a smaller number of feature representations with the feature representations of multiple image blocks to obtain a target feature representation, so that the number of feature representations contained in the target feature representation is less than the number of feature representations contained in the image to be identified. By performing text recognition on the target feature representation instead of directly performing text recognition on the image to be identified, the number of feature representations in the text recognition process can be reduced, thereby improving the efficiency of text recognition.

[0124] See also Figure 8 , Figure 8 The computer-readable storage medium 80 stores program instructions 801, which implement the steps in any of the above-mentioned text recognition method embodiments when executed by a processor.

[0125] The above scheme obtains an image to be identified, performs a first down-sampling on the feature representations of multiple image blocks obtained by dividing the image to be identified, obtains an initial feature representation, and fuses the initial feature representation with a smaller number of feature representations with the feature representations of multiple image blocks to obtain a target feature representation, so that the number of feature representations contained in the target feature representation is less than the number of feature representations contained in the image to be identified. By performing text recognition on the target feature representation instead of directly performing text recognition on the image to be identified, the number of feature representations in the text recognition process can be reduced, thereby improving the efficiency of text recognition.

[0126] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0127] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A text recognition method, characterized in that: The method comprises: Obtain an image to be recognized; Performing a first downsampling on the feature representations of the plurality of image blocks obtained by dividing the image to be identified, to obtain an initial feature representation, wherein a combination of the feature representations of the plurality of image blocks is a feature representation of the image to be identified, and the initial feature representation carries the feature representations of at least part of the image blocks; fusing the initial feature representation with the feature representations of the plurality of image blocks to obtain a target feature representation, wherein the number of feature representations of the image blocks included in the target feature representation is the same as the number of feature representations of the image blocks included in the initial feature representation; Performing text recognition on the target feature representation to obtain a text recognition result on the image to be recognized; The first downsampling of the feature representations of the plurality of image blocks obtained by dividing the image to be identified to obtain an initial feature representation includes: Performing feature extraction on the image to be identified, determining a sampling weight of each image block in the image to be identified in at least one preset direction, wherein the sampling weight is a mapping value of the feature extraction result obtained by normalizing or activating a feature extraction result corresponding to each image block in the image to be identified; The feature representation of each image block is sampled along at least one preset direction according to the sampling weight to obtain the initial feature representation, including: sampling and sorting the feature representation of each image block to obtain an importance ranking corresponding to the feature representation of each image block; based on the importance ranking, the feature representation of each image block in each preset direction is screened to obtain a candidate feature representation; based on each candidate feature representation, the initial feature representation is determined.

2. The method according to claim 1, characterized in that There are multiple preset directions, and extracting features from the image to be identified and determining a sampling weight of each image block in the image to be identified in at least one preset direction includes: Performing gradient feature extraction and a second downsampling process on the image to be identified to obtain gradient information in each of the preset directions and a downsampled image; The gradient information is fused with the downsampled image to obtain sampling weights corresponding to the preset directions.

3. The method according to claim 2, characterized in that: The at least one preset direction includes a horizontal direction and a vertical direction, each of the gradient information includes first gradient information in the horizontal direction and second gradient information in the vertical direction, and each of the gradient information is fused with the downsampled image to obtain a sampling weight corresponding to each of the preset directions, including: Using the first gradient information as target gradient information and the sampling weight corresponding to the horizontal direction as the target sampling weight, or using the second gradient information as target gradient information and the sampling weight corresponding to the vertical direction as the target sampling weight; Performing gradient fusion processing on the target gradient information and the downsampled image to obtain a first fused image; Performing initial feature extraction on the first fused image to obtain initial image features; Performing a first dimensional transformation process on the initial image features to obtain first dimensional features; The first dimensional feature is input into the perceptron module for advanced feature extraction to obtain the target sampling weight.

4. The method according to claim 2, characterized in that: The at least one preset direction includes a horizontal direction and a vertical direction, using a sampling weight corresponding to the horizontal direction as a first sampling weight, using a sampling weight corresponding to the vertical direction as a second sampling weight, and sampling the feature representation of each image block along the at least one preset direction according to the sampling weight to obtain the initial feature representation, including: Acquire a first selected number of image blocks of the image to be identified in the horizontal direction and a second selected number of image blocks of the image to be identified in the vertical direction; According to the first sampling weight, a first screening is performed on the feature representation of each of the image blocks in the horizontal direction to obtain the first selected number of candidate horizontal feature representations; and according to the second sampling weight, a first screening is performed on the feature representation of each of the image blocks in the vertical direction to obtain the second selected number of candidate vertical feature representations; Performing a second screening on each of the candidate transverse feature representations and each of the candidate longitudinal feature representations to obtain a preset number of selected feature representations, where the preset number is equal to the product of the first selected number and the second selected number; Based on each of the selected feature representations, the initial feature representation is determined.

5. The method according to claim 4, characterized in that The determining the initial feature representation based on each of the selected feature representations comprises: Acquire position information corresponding to each of the selected feature representations, where the position information is used to indicate a positional relationship between the selected feature representation and each of the image blocks of the image to be identified; Each of the selected feature representations is subjected to position fusion processing with each of the position information to obtain the initial feature representation.

6. The method according to any one of claims 1 to 5, characterized in that The step of extracting features from the image to be identified and determining the sampling weight of each image block in the image to be identified in at least one preset direction is performed by a frequency adaptation module, and the method further includes: Acquire a sample image and annotation distribution information of the sample image in the at least one preset direction; Using the frequency adaptation module to perform feature extraction on the sample image, and determine a sample sampling weight of each image block in the sample image in at least one preset direction; Performing Gaussian distribution processing on the sample sampling weights to determine predicted distribution information; Determining a target loss based on a difference between the labeled distribution information and the predicted distribution information; According to the target loss, the parameters of the frequency adaptation module are adjusted.

7. The method according to any one of claims 1 to 5, characterized in that The fusing the initial feature representation with the feature representations of the multiple image blocks to obtain a target feature representation includes: Inputting the initial feature representation and the feature representations of the multiple image blocks into a cross-attention layer to obtain a feature representation output by the cross-attention layer, wherein the initial feature representation serves as a query vector of the cross-attention layer, and the feature representations of the multiple image blocks serve as a key vector and a value vector of the cross-attention layer; Performing feature fusion processing on the initial feature representation and the feature representation output by the cross attention layer to obtain an advanced feature representation; The advanced feature representation is normalized to obtain the target feature representation.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: include: A program file is stored, and when the program file is executed by a processor, it is used to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Noise image recognition method and related equipment

    CN117011533A

  • Multi-modal text image classification method and device, electronic equipment and storage medium

    CN117893859A