Image processing method and device, equipment and readable medium

Abstract

The embodiment of the invention discloses an image processing method, device and equipment and a readable medium, and the method comprises the steps: carrying out the image rotation of a text image based on a direction difference value between a character arrangement direction of a text in the text image and a plurality of designated arrangement directions, and obtaining a rotation image corresponding to each designated arrangement direction; performing feature extraction on each rotating image to obtain image features of each rotating image; performing feature fusion on the plurality of image features to obtain a fused feature; and based on the positions of the characters in each rotation image, performing classification processing on the fusion features to obtain a recognition text of the text image. According to the technical scheme, the recognition accuracy of the text in the image can be improved.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to image processing methods, image processing apparatus, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] In modern society, text recognition technology has been widely applied in various fields, such as street view recognition, formula recognition, and seal recognition. Traditional text recognition models can accurately identify text in images with fixed character orientations; however, when the character orientation of the text in an image changes, the accuracy of the text recognized by traditional models decreases significantly. Therefore, improving the accuracy of text recognition in images is a pressing issue that needs to be addressed. Summary of the Invention

[0003] The embodiments of this application provide image processing methods, image processing apparatus, electronic devices, computer-readable storage media, and computer program products, which can improve the accuracy of text recognition in images.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, an image processing method is provided, the method comprising:

[0006] Based on the direction difference between the character arrangement direction of the text in the text image and multiple specified arrangement directions, the text image is rotated to obtain rotated images corresponding to each specified arrangement direction;

[0007] Feature extraction is performed on each rotated image to obtain the image features of each rotated image;

[0008] Multiple image features are fused to obtain fused features;

[0009] Based on the position of the characters in each rotated image, the fused features are classified to obtain the recognized text of the text image.

[0010] According to one aspect of the embodiments of this application, an image processing apparatus is provided, the apparatus comprising an image rotation unit, a feature acquisition unit, and a processing unit, wherein:

[0011] The image rotation unit is used to rotate the text image based on the direction difference between the character arrangement direction of the text in the text image and multiple specified arrangement directions, so as to obtain the rotated image corresponding to each specified arrangement direction.

[0012] The feature acquisition unit is used to extract features from each rotated image to obtain the image features of each rotated image.

[0013] The feature acquisition unit is used to fuse multiple image features to obtain fused features;

[0014] The processing unit is used to classify the fused features based on the positions of the characters in each of the rotated images to obtain the recognized text of the text image.

[0015] According to one aspect of the embodiments of this application, an electronic device is provided, including one or more processors; and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the image processing method as described above.

[0016] According to one aspect of the embodiments of this application, the embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the image processing method as described above.

[0017] According to one aspect of the embodiments of this application, this application provides a computer program product, including a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the image processing method as described above.

[0018] In the technical solutions provided by the embodiments of this application, by fusing the image features of multiple rotated images, a fused feature can be obtained that characterizes the features of text in a text image under multiple specified arrangement directions. This allows for comprehensive classification based on the fused feature, taking into account the characteristics of text in multiple arrangement directions, which is beneficial for identifying more accurate text. Furthermore, in the embodiments of this application, the fused feature is classified based on the position of characters in each rotated image. This method makes the classification of the fused feature a character-level prediction, which is more accurate than predicting the entire text, thus facilitating further identification of more accurate text. Additionally, by rotating the images, the embodiments of this application ensure that accurate text can be identified from the text image regardless of the character arrangement direction.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0021] Figure 1 This is a schematic diagram of text with different character arrangement directions provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0023] Figure 3 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a classification process based on fused features provided in an embodiment of this application;

[0025] Figure 5 This is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram illustrating the recognition process of a text recognition model provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of an image rotation process provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of a feature fusion process provided in an embodiment of this application;

[0029] Figure 9 This is a structural block diagram of an image processing apparatus illustrated in an exemplary embodiment of this application;

[0030] Figure 10 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying diagrams are merely illustrative and do not necessarily include all content and operations, nor do they necessarily have to be executed in the described order. For example, some operations may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0034] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0035] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0036] With the continuous development of internet technology, artificial intelligence (AI) technology has also seen significant advancements. AI technology refers to the theories, methods, techniques, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science; it primarily aims to understand the essence of intelligence and produce new intelligent machines that can react in a manner similar to human intelligence, enabling these machines to possess multiple functions such as perception, reasoning, and decision-making.

[0037] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision (CV), speech processing, natural language processing (NLP), and machine learning / deep learning.

[0038] Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in identifying, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0039] Research in multiple areas of computer vision technology relies on machine learning. Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of AI and the fundamental way to endow computer devices with intelligence. Deep learning, on the other hand, is a technique that utilizes deep neural network systems for machine learning. Machine learning / deep learning typically includes artificial neural networks, adversarial learning, reinforcement learning (RL), supervised learning, and unsupervised learning. Supervised learning refers to training models using training samples with known categories (labeled categories), while unsupervised learning refers to training models using training samples with unknown categories (unlabeled categories).

[0040] With the research and advancement of computer vision technology, it has been applied in various fields. For example, text recognition technology within computer vision has been widely used in street view recognition, formula recognition, and seal recognition. Traditional text recognition techniques perform well on images containing text with a fixed character orientation, achieving relatively accurate results. However, for images containing text with non-fixed character orientations, the recognition performance of traditional techniques is significantly reduced, and errors are more likely to occur in the identified text.

[0041] For example, please see the appendix. Figure 1 This diagram illustrates text with different character arrangement directions. Figure 1 In image 101, the text characters are arranged from left to right, while in image 102, the text characters are arranged from top to bottom.

[0042] In practical applications, the character arrangement direction of most text is the same as that in image 101. Therefore, for text with the same character arrangement direction as in image 101, more training images can be found to train a better text recognition model for recognizing text with the same character arrangement direction. However, for text with the same character arrangement direction as in image 102, the number of training images available is limited. Consequently, the accuracy of the trained text recognition model for recognizing text with the corresponding character arrangement direction is not as good as the aforementioned text recognition model. This demonstrates that even if different text recognition models are trained for different character arrangement directions, the limited training data results in the accuracy of some text recognition models still being low.

[0043] Based on this, this application provides an image processing scheme. This scheme is not limited to the orientation of the characters in the text image. After acquiring the text image, the character orientation is rotated into multiple specified orientations to obtain rotated images corresponding to each specified orientation. Then, the image features of each rotated image are fused to obtain fused features. Finally, based on the position of the characters in each rotated image, the fused features are classified to obtain the recognized text of the text image.

[0044] Here, a text image refers to an image containing text; and the character arrangement direction of the text in the rotated images corresponding to various specified arrangement directions obtained by rotating the text image are respectively the specified arrangement directions. Multiple specified arrangement directions are preset. The character arrangement direction of the text in the text image may be the same as one of the specified arrangement directions.

[0045] Furthermore, the classification processing of the fused features based on the position of characters in each rotated image mainly involves finding the feature parts that map each character in the fused features based on the position of characters in each rotated image. Finally, the recognized text is obtained by classifying the feature parts of each character.

[0046] Therefore, this scheme, by fusing image features from multiple rotated images, can obtain fused features that characterize the text in a text image under multiple specified orientations. This allows for comprehensive classification based on the fused features, taking into account the characteristics of the text in multiple orientations, thus leading to more accurate text recognition. Furthermore, this scheme classifies the fused features based on the position of characters in each rotated image. This approach provides character-level classification prediction based on the fused features, which is more accurate than predicting the entire text, further contributing to more precise text recognition.

[0047] Based on the above image processing scheme, this application provides an image processing system, which can be found in [reference needed]. Figure 2 , Figure 2 The image processing system shown may include terminal devices 201 and servers 202. The number of terminal devices 201 and servers 202 may be multiple. A communication connection is established between any terminal device and any server. For example, terminal device 201 may include any one or more of the following: smartphone, tablet, laptop, desktop computer, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, and smart wearable device. Terminal device 201 may have various clients installed, such as live streaming clients, social networking clients, shopping clients, payment clients, and map clients. Server 202 may be a server or server cluster providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Any terminal device 201 and any server 202 may communicate directly or indirectly via wired or wireless communication; this application does not impose any limitations on this.

[0048] In some embodiments, the above image processing method may be performed solely by Figure 2The image processing system shown executes the following process by server 202: First, server 202 rotates the text image based on the directional differences between the character arrangement direction of the text and multiple specified arrangement directions, obtaining rotated images corresponding to each specified arrangement direction. Then, server 202 extracts features from each rotated image, obtaining image features for each rotated image. Furthermore, server 202 fuses multiple image features to obtain fused features. Finally, server 202 classifies the fused features based on the positions of characters in each rotated image to obtain the recognized text of the text image.

[0049] Optionally, the above image processing method can also be achieved solely by... Figure 2 The terminal device 201 in the image processing system shown executes the process, and its specific execution process can be found in the specific execution process of the server 202, which will not be repeated here.

[0050] In other embodiments, the image processing method described above can be run in an image processing system, which may include a terminal device and a server. Specifically, the image processing method described above can be... Figure 2 The image processing system shown is jointly implemented by terminal device 201 and server 202. The specific execution process is as follows: Terminal device 201 acquires a text image and sends it to server 202. Server 202 first rotates the text image based on the directional differences between the character arrangement direction of the text and multiple specified arrangement directions, obtaining rotated images corresponding to each specified arrangement direction. Then, server 202 extracts features from each rotated image to obtain image features for each rotated image. Furthermore, server 202 fuses multiple image features to obtain fused features. Afterward, server 202 classifies the fused features based on the positions of characters in each rotated image to obtain the recognized text of the text image. Finally, server 202 sends the recognized text of the text image to terminal device 201; and after receiving the recognized text of the text image from terminal device 201, terminal device 201 can output the recognized text of the text image.

[0051] It should be noted that the embodiments of this application can be applied to various scenarios that require text recognition, including but not limited to cloud technology, AI (Artificial Intelligence), smart cities, smart transportation, assisted driving and other scenarios. They can also be used in live streaming applications, shopping applications, payment applications and any other applications that may contain text images, without any limitation.

[0052] It should be noted that in the specific embodiments of this application, if the text, images, and other related data or information involve objects, when the embodiments of this application are applied to specific products or technologies, permission or consent from the objects is required, and the collection, use, and processing of related data or information must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0053] The following details the various implementation details of the technical solutions in the embodiments of this application:

[0054] like Figure 3 As shown, Figure 3 This is a schematic flowchart illustrating an embodiment of an image processing method, which can be applied to... Figure 2 The image processing system shown can be executed by a terminal device or a server, or by both a terminal device and a server. In this embodiment, the method is described as being executed by a server. The image processing method may include steps S301 to S304, which are described in detail below:

[0055] S301. Based on the direction difference between the character arrangement direction of the text in the text image and multiple specified arrangement directions, rotate the text image to obtain the rotated image corresponding to each specified arrangement direction.

[0056] In the embodiments of this application, the multiple specified arrangement directions can be set manually or by the terminal device or server in the above-mentioned image processing system, and are not limited here.

[0057] In one embodiment, the frequency of occurrence of each character arrangement direction among all the character arrangement directions that have appeared can be counted, and then a specified arrangement direction can be selected from all the character arrangement directions based on the frequency of occurrence; wherein, the frequency of occurrence of the specified arrangement direction is greater than that of other character arrangement directions.

[0058] In general, a higher frequency of a character arrangement direction indicates that the text in that direction is more readable and easier to recognize and understand. Therefore, a character arrangement direction with a higher frequency of occurrence can be selected as the specified arrangement direction.

[0059] Furthermore, the subsequent text recognition processes such as feature extraction, feature fusion, and classification in the embodiments of this application all require the use of machine learning technology. The higher the frequency of the specified arrangement direction, the more images containing text in the specified arrangement direction there are. In this case, there is more training data available during the model training phase, which is beneficial for training a more accurate model and thus for obtaining more accurate text recognition.

[0060] In one embodiment, the specific process of image rotation may include: performing character orientation recognition on the text image to obtain the character arrangement orientation of the text in the text image; then calculating the orientation difference between the recognized character arrangement orientation and each specified arrangement orientation; finally, rotating the text image in each specified arrangement orientation by the corresponding orientation difference to obtain the rotated image corresponding to each specified arrangement orientation.

[0061] Specifically, the direction difference can be data representing the difference between directions, such as the angle between directions; the direction difference can be positive, negative, or zero. For example, the character arrangement direction in a text image may be the same as a specified arrangement direction, so in this case, the direction difference can be zero. Furthermore, character direction recognition can be achieved by training a corresponding model or neural network based on machine learning techniques, which will not be elaborated upon here.

[0062] S302. Extract features from each rotated image to obtain the image features of each rotated image.

[0063] In this embodiment, a multimodal encoder can be invoked to extract features from each rotated image to obtain image features for each rotated image. The multimodal encoder may include feature extraction modules for features in each specified orientation; the network structures of the feature extraction modules in the multimodal encoder may be the same or different.

[0064] In specific implementations, the network structure of each feature extraction module in the multimodal encoder can be Resnet-50 (a convolutional neural network with a depth of 50 layers), Resnet-50-DC5 (a convolutional neural network with a depth of 50 layers and 16x downsampling), or Resnet-101 (a convolutional neural network with a depth of 101 layers), etc., and is not limited here.

[0065] S303. Perform feature fusion on multiple image features to obtain fused features.

[0066] In this embodiment, the specific process of feature fusion may include: concatenating multiple image features to obtain fused features. Optionally, the feature fusion process may also be: performing attention fusion encoding processing on multiple image features to obtain fused features. In specific implementation, the attention fusion encoding processing is based on a neural network model with a cross-modality attention mechanism or a self-attention mechanism.

[0067] In one embodiment, the feature fusion process can also be as follows: concatenating multiple image features to obtain concatenated features; performing activation mapping processing on each feature value in the concatenated features to obtain activated features; obtaining target features of each rotated image based on the similarity between the activated features and each image feature; and fusing multiple target features to obtain fused features.

[0068] Since the similarity between two features can also be calculated by multiplying them, the target features can be obtained by multiplying the activation features and each image feature to obtain the target features for each rotated image. Optionally, the similarity between the activation features and each image feature can also be calculated in other ways to obtain the target features for each rotated image; this is not limited here.

[0069] Optionally, the specific way to fuse multiple target features to obtain fused features can be to add the multiple target features together to obtain fused features.

[0070] Specifically, by first stitching together the image features of multiple rotated images, and then fusing the stitched features with the image features of each rotated image, the final fused features can represent both the comprehensive characteristics of the text in the text image under multiple specified orientations and the characteristics of the text in the text image under each specified orientation. In this way, the multi-level and multi-dimensional characteristics represented by the fused features can be used to predict the text in the text image, which is conducive to more accurate text recognition.

[0071] Activation mapping can effectively amplify the parts of the spliced ​​features that are beneficial to text recognition. In this way, the activation features obtained by activation, as well as the target features and fusion features obtained by each image feature, will also amplify the parts of their features that are beneficial to text recognition, thereby further improving the accuracy of text recognition.

[0072] Alternatively, other methods can be used to fuse multiple image features, which are not limited here.

[0073] S304. Based on the position of characters in each rotated image, the fused features are classified to obtain the recognized text of the text image.

[0074] Since image feature extraction is a process of continuous sampling of the image, the feature values ​​extracted from the image have a mapping relationship with each position in the image. Therefore, in this embodiment, the classification process of fused features may specifically include: for each character in the text image, based on the position of the corresponding character in each rotated image, obtaining the mapping features of each character in the fused features; classifying the mapping features of each character to obtain the predicted character of each character; and combining the predicted characters of each character to obtain the recognized text of the text image.

[0075] Since the text contained in each rotated image is identical to that in the text image, the corresponding character for any given character in each rotated image refers to the character whose character arrangement is the same as that given character in each rotated image. Since there are multiple rotated images, this is equivalent to finding the feature portion in the fusion feature that matches the aforementioned multiple positions of any given character in the text image across multiple rotated images, and using this as the mapping feature for that given character.

[0076] In addition, one or more classification functions such as the normalized exponential function (softmax, a multi-classification function), the cross-entropy loss function, and the log-likelihood loss function can be used for feature classification, without any restrictions here.

[0077] In one embodiment, considering that the semantics of each character in a text can influence each other, the mapping features of other characters in each rotated image can, to some extent, help the mapping features of each character to perform more accurate semantic analysis during classification, obtain more accurate predicted characters, and thus facilitate more accurate text recognition.

[0078] The specific process of classifying the mapping features of each character to obtain the predicted character of each character can include: classifying and predicting the mapping features of each character based on the mapping features of other characters in each rotated image, in order to obtain the predicted character of each character.

[0079] In practice, an attention mechanism can be used to encode the mapping features of each character based on the mapping features of other characters in each rotated image, thereby obtaining the encoded features of each character. Finally, the encoded features of each character are classified to obtain the predicted character of each character.

[0080] In one possible implementation, the semantic influence of the characters preceding each character in the text on the semantics of each character is greater than the semantic influence of the characters following each character in the text on the semantics of each character.

[0081] Therefore, the specific process of classification processing may also include: obtaining the predicted characters of all characters in the text image, and obtaining the recognition result of the text image based on all the predicted characters;

[0082] The process of obtaining the predicted character of the i-th character in the text image includes: obtaining the mapping feature of the i-th character in the fused feature based on the position of the i-th character in each rotated image; and performing feature classification on the mapping feature corresponding to the i-th character based on the mapping features corresponding to the 1st to i-1th characters in the text image to obtain the predicted character corresponding to the i-th character in the text image. Here, the i-th character in the text image is any character in the text image, and i is a positive integer.

[0083] When the i-th character is not the first character in the text image, the specific process of classifying the mapping features corresponding to the i-th character based on the mapping features corresponding to the 1st to i-1th characters in the text image can include: concatenating the mapping features of the i-th character in the text image with the latent vector corresponding to the i-1th character in the text image to obtain the concatenated features; and classifying the concatenated features to obtain the predicted character corresponding to the i-th character in the text image.

[0084] The latent vector corresponding to the (i-1)th character is obtained based on the concatenated features of the (i-1)th character, the state vector of the (i-1)th character, and the state vector of the (i-2)th character in the text image.

[0085] When the i-th character is the first character in the text image, the specific process of performing feature classification on the mapping features corresponding to the first character in the text image may include: concatenating the mapping features of the first character in the text image with a preset latent vector to obtain the concatenated features; and performing feature classification on the concatenated features to obtain the predicted character corresponding to the i-th character in the text image.

[0086] Furthermore, based on the concatenated features corresponding to the first character and the preset state vector, the state vector corresponding to the first character can be obtained; based on the concatenated features corresponding to the first character and the state vector corresponding to the first character, the latent vector corresponding to the first character can be obtained.

[0087] Accordingly, based on the concatenated features corresponding to the i-th character and the state vector corresponding to the (i-1)-th character, the state vector corresponding to the i-th character can be obtained; based on the concatenated features corresponding to the i-th character and the state vector corresponding to the i-th character, the latent vector corresponding to the i-th character can be obtained. The preset latent vector and preset state vector can be manually set, or they can be set by the terminal device or server in the aforementioned image processing system; no limitation is made here.

[0088] Please see the appendix Figure 4 This illustrates a classification process involving feature fusion. The text image W is rotated in N specified orientations to obtain rotated images 1 to N, where N is a positive integer. Then, as... Figure 4 As shown, based on the position of the i-th character in each rotated image, the mapping feature corresponding to the i-th character in the text image W can be determined from the fused features.

[0089] By concatenating the mapping feature corresponding to the i-th character in the text image W with the latent vector corresponding to the (i-1)-th character in the text image W, we can obtain the concatenated feature corresponding to the i-th character in the text image W. By performing feature classification on the concatenated feature, we can obtain the predicted character of the i-th character in the text image W.

[0090] Simultaneously, based on the concatenated features corresponding to the i-th character in the text image W and the state vector corresponding to the (i-1)-th character in the text image W, the state vector corresponding to the i-th character in the text image W can be obtained; then, based on the concatenated features corresponding to the i-th character in the text image W and the state vector corresponding to the i-th character in the text image W, the latent vector corresponding to the i-th character in the text image W can be obtained.

[0091] The specific methods for obtaining the state vector and hidden vector corresponding to each character can be found in the calculation methods of the state vector and hidden vector in Long Short-Term Memory (LSTM) networks, and will not be elaborated here.

[0092] In this embodiment, by fusing the image features of multiple rotated images, the resulting fused features can characterize the text in the text image under multiple specified arrangement directions. This allows for comprehensive classification based on the fused features, taking into account the multidimensional characteristics of the text in the text image under multiple arrangement directions, which is beneficial for more accurate text recognition. Furthermore, this embodiment classifies the fused features based on the position of characters in each rotated image. This method makes the classification of fused features a character-level prediction, which is more accurate than predicting the entire text, further facilitating more accurate text recognition. Additionally, this embodiment uses image rotation to ensure that accurate text can be recognized from the text image regardless of the character arrangement direction.

[0093] In one embodiment of this application, another image processing method is provided, which can be applied to Figure 2The image processing system shown can be executed by a terminal device or a server, or by both. In this embodiment, the method is described using the server as an example. Figure 5 The diagram illustrates a flowchart of another image processing method. Figure 3 The method shown is an extension of the one described.

[0094] The details of S501 to S509 are as follows:

[0095] S501. Obtain the training images and the real text contained in the training images.

[0096] In this embodiment, the character arrangement direction of the real text contained in the training image can be the same as any one of a plurality of specified arrangement directions. Optionally, the character arrangement direction of the real text contained in the training image can also be different from all of the plurality of specified arrangement directions.

[0097] S502. Based on the direction difference between the character arrangement direction of the text in the training image and multiple specified arrangement directions, rotate the training image to obtain the reference image corresponding to each specified arrangement direction.

[0098] In this embodiment, the character arrangement direction of text in a training image can be identified using a direction classifier; then, the direction difference between the identified character arrangement direction and each specified arrangement direction is calculated, and the training image is rotated according to each direction difference to obtain a reference image corresponding to each specified arrangement direction. Specifically, the direction classifier can consist of a convolutional neural network and a classification function.

[0099] Optionally, the orientation classifier can also be integrated into the text recognition model. Furthermore, the specific implementation of step S502 can be found in the specific implementation of step S201, and will not be repeated here.

[0100] S503. Extract features from each reference image to obtain the image features of each reference image.

[0101] In this embodiment, the specific implementation method for feature extraction can be found in step S202, and will not be repeated here. Optionally, the multimodal encoder mentioned in step S202 can be integrated into the text recognition model.

[0102] S504. The image features of multiple reference images are fused to obtain fused training features.

[0103] In this embodiment, the specific implementation of feature fusion can be found in step S203, and will not be repeated here. Optionally, a feature fusion module can be integrated into the multimodal encoder to fuse the image features extracted by each feature extraction module in the multimodal encoder.

[0104] S505. A text recognition model is obtained by training based on real text and fused training features.

[0105] In this embodiment of the application, the training process of the text recognition model based on real text and fused training features may include: performing feature classification on the fused training features based on the position of reference characters in each reference image and the real characters in the real text that match the character arrangement order of each reference character to obtain the predicted characters corresponding to each reference character; calculating the character loss value between the real characters that match each reference character and the predicted characters corresponding to each reference character; and training in the direction of reducing the character loss value to obtain the text recognition model.

[0106] The character loss value between the true character and the predicted character can be calculated using loss functions such as sigmoid (S-shaped growth curve, a type of loss function) or cross-entropy loss function. Training in the direction of reducing character loss can specifically be done by reducing the loss value of all characters; alternatively, all character loss values ​​can be summed to obtain a total loss value, and then training can be done in the direction of reducing the total loss value.

[0107] In one embodiment, the prediction process for the predicted character corresponding to each reference character may include: obtaining the mapping feature of the j-th reference character in the training image in the fused training features based on the position of the j-th reference character in each reference image; concatenating the latent vector corresponding to the (j-1)-th reference character in the training image and the mapping feature corresponding to the j-th reference character in the training image to obtain the concatenated reference feature; and classifying the concatenated reference feature to obtain the predicted character corresponding to the j-th reference character in the training image.

[0108] Here, the j-th reference character in the training image is any character in the training image, where j is a positive integer; the latent vector corresponding to the (j-1)-th reference character is obtained based on the character features of the latent vector corresponding to the (j-2)-th reference character in the training image, the state vector corresponding to the (j-2)-th reference character, and the real character that matches the (j-1)-th reference character.

[0109] When j=1, the latent vector corresponding to the first reference character is obtained based on the preset latent vector, the preset state vector, and the mapping features of the first reference character. The method for obtaining the latent vector and state vector corresponding to any reference character can be found in the specific method for obtaining the latent vector and state vector in step S304, and will not be repeated here.

[0110] In a specific implementation, the text recognition model may include a decoder. During the training process of the text recognition model, the fused training features and the preset starting character can be input into the decoder in the text recognition model to obtain the predicted character of each reference character.

[0111] The preset starting character is used to indicate the first character of the text, so that the text recognition model can find the mapping feature of the first reference character in the training image based on the position of the first reference character in each rotated image, according to the prompt of the preset starting character. Then, the mapping feature of the first reference character is concatenated with the preset latent vector to obtain the concatenated feature corresponding to the first reference character. Finally, the concatenated feature corresponding to the first reference character is classified to obtain the predicted character of the first reference character.

[0112] After obtaining the predicted character of the first reference character, the mapping feature of the first reference character is equivalent to prompting the text recognition model to find the mapping feature of the reference characters after the first reference character. This allows the text recognition model to find the mapping feature of the second reference character in the training image based on the position of the second reference character in each reference image.

[0113] Optionally, if the predicted character of the j-th reference character is not a preset end character, the text recognition model can continue to find the mapping feature of the (j+1)-th reference character in the training image based on the position of the (j+1)-th reference character in each reference image, according to the hints from the mapping feature of the j-th reference character. If the predicted character of the j-th reference character is a preset end character, the text recognition model can combine the 1st to the (j-1)th reference characters to obtain the recognized text of the training image.

[0114] The preset end character can be set manually or by the terminal device or server in the image processing system mentioned above; no limitation is made here. For example, the preset end character can be "end" or "0", etc.

[0115] After the text recognition model predicts the actual last character in the training image (which is the (j-1)th reference character at this time), since the text recognition model does not know that the (j-1)th reference character is the last character in the training image, the text recognition model will continue to find the corresponding feature part in the fusion feature as the mapping feature of the j-th reference character based on the hint of the mapping feature of the (j-1)th reference character and based on the position after the (j-1)th reference character in each reference image.

[0116] However, since the image portion represented by the mapping feature of the j-th reference character actually has no character content, the predicted character based on the mapping feature of the j-th reference character will be the preset end character.

[0117] Specifically, in the prediction process of each reference character in this application embodiment, the predicted character of the previous reference character is not used as the prediction basis for the current reference character. Instead, the character features of the correct real character are forcibly used. This can effectively avoid the cumulative prediction error of the previous reference character affecting the prediction result of the subsequent reference character. As a result, each reference character no longer depends on the prediction result of the previous reference character, but on the correct sample result. This allows the text recognition model to be trained more quickly based on the correct sample result, which is beneficial to improving the training speed and training effect of the text recognition model.

[0118] S506. Based on the direction difference between the character arrangement direction of the text in the text image and multiple specified arrangement directions, rotate the text image to obtain the rotated image corresponding to each specified arrangement direction.

[0119] In this embodiment of the application, referring to step S502, the orientation classifier in the text recognition model trained in steps S501 to S506 can be used to rotate the text image, thereby obtaining the rotated image corresponding to each specified arrangement direction.

[0120] Furthermore, the specific implementation of step S506 can be found in the specific implementation of step S201, and will not be repeated here.

[0121] S507. Extract features from each rotated image to obtain the image features of each rotated image.

[0122] In this embodiment of the application, referring to step 503, the image features of each rotated image can be extracted through the feature extraction modules in the multimodal encoder of the text recognition model trained in steps S501 to S506.

[0123] Furthermore, the specific implementation of step S507 can be found in the specific implementation of step S202, and will not be repeated here.

[0124] S508. Fuse multiple image features to obtain fused features.

[0125] In this embodiment of the application, referring to step 504, the feature fusion module in the multimodal encoder of the text recognition model trained in steps S501 to S506 can be used to fuse multiple image features to obtain fused features.

[0126] Furthermore, the specific implementation of step S508 can be found in the specific implementation of step S203, and will not be repeated here.

[0127] S509. Based on the position of characters in each rotated image, the text recognition model classifies the fused features to obtain the recognized text of the text image.

[0128] In this embodiment, since the text recognition model mentioned above may include a decoder, the fused features and the feature vector of the preset starting character can be input into the decoder of the text recognition model trained through steps S501 to S506. The decoder outputs the predicted characters of each character in the text image to combine and obtain the recognized text of the text image. Furthermore, the specific implementation of step S509 can be found in the specific implementations of steps S204 and S505, and will not be repeated here.

[0129] In one possible implementation, since text with characters arranged in a horizontal direction and text with characters arranged in a horizontal direction are the two most frequently occurring types of text, multiple specified arrangement directions can include both horizontal and vertical arrangement directions.

[0130] Therefore, the text image can be rotated based on the directional difference between the character arrangement direction and the horizontal alignment direction to obtain a rotated image corresponding to the horizontal alignment direction; similarly, the text image can be rotated based on the directional difference between the character arrangement direction and the vertical alignment direction to obtain a rotated image corresponding to the vertical alignment direction. Features are then extracted from each of the two rotated images to obtain image features for each rotated image; these two image features are then fused to obtain fused features; finally, based on the position of the characters in each rotated image, the fused features are classified to obtain the recognized text of the text image.

[0131] In practice, multiple specified arrangement directions can be set as horizontal (or simply horizontal) and vertical (or simply vertical). Please see the appendix. Figure 6The diagram illustrates the recognition process of a text recognition model. Figure 6 As shown, the text recognition model trained for horizontal and vertical arrangement directions may include: a horizontal / vertical generator 601, a multimodal encoder 602, and a decoder 603. The multimodal encoder 602 may include a horizontal feature extraction module 604, a vertical feature extraction module 605, and a feature fusion module 606.

[0132] like Figure 6 As shown, after the text image 607 is input into the text recognition model, the horizontal and vertical image generator 601 in the text recognition model can be called to rotate the text image 607, resulting in a horizontal image 608 and a vertical image 609. Then, the multimodal encoder 602 in the text recognition model is called to process the horizontal image 608 and the vertical image 609.

[0133] Specifically, the horizontal feature extraction module 604 in the multimodal encoder 602 extracts features from the horizontal image 608 to obtain the image features of the horizontal image 608 (hereinafter referred to as horizontal features); and the vertical feature extraction module 605 in the multimodal encoder 602 extracts features from the vertical image 609 to obtain the image features of the vertical image 609 (hereinafter referred to as vertical features). Finally, the feature fusion module 606 in the multimodal encoder 602 fuses the horizontal and vertical features to obtain the fused features, which are then output.

[0134] After receiving the fused features output by the multimodal encoder 602, the decoder 603 can perform feature decoding (i.e., classification processing) on ​​the fused features, thereby outputting the recognized text 610 of the text image 607.

[0135] For details, please see the appendix. Figure 7 This diagram illustrates an image rotation process. The horizontal / vertical image generator 601 may include an orientation classifier 701 and a rotation module 702. (Example...) Figure 7 As shown, after the text image 607 is input into the horizontal / vertical layout generator 601, the direction classifier 701 identifies that the character arrangement direction of the text in the text image 607 is a horizontal direction. Then, the rotation module 702 determines that the direction difference between the character arrangement direction of the text in the text image 607 and the horizontal direction is 0 degrees, and rotates the text image 607 by 0 degrees to obtain the horizontal image 608. Furthermore, the rotation module 702 determines that the direction difference between the character arrangement direction of the text in the text image 607 and the vertical direction is 90 degrees, and rotates the text image 607 by 90 degrees to obtain the vertical image 609.

[0136] In practical applications, the orientation classifier can consist of a 5-layer convolutional neural network. Optionally, the input image can be set to be a text image. Furthermore, to more accurately identify the character arrangement orientation in the text image, the size of the text image can be set to h×w, where h and w are both positive numbers.

[0137] In addition, please see the appendix. Figure 8 The diagram illustrates a feature fusion process. The feature fusion module 606 in the multimodal encoder 602 can receive the horizontal features 801 output by the horizontal feature extraction module 604 and the vertical features 802 output by the vertical feature extraction module 605.

[0138] Then, as Figure 8 As shown, the feature fusion module 606 can concatenate the horizontal feature 801 and the vertical feature 802 to obtain the concatenated feature 803. Then, the feature fusion module 606 can call an activation function to perform activation mapping on the concatenated feature 803 to obtain the activated feature 804.

[0139] After obtaining the activation features, the feature fusion module 606 can multiply the activation features 804 with the horizontal features 801 to obtain the target features 805 corresponding to the horizontal image; and multiply the activation features 804 with the vertical features 802 to obtain the target features 806 corresponding to the vertical image.

[0140] Finally, the feature fusion module 606 adds the target features 805 corresponding to the horizontal image and the target features 806 corresponding to the vertical image to obtain the fused feature 807.

[0141] When there is a need to recognize text with multiple character orientations, traditional solutions train corresponding text recognition models for each orientation. Then, when an image containing text with a specific orientation is detected, the corresponding text recognition model is called for text recognition. This approach suffers from two problems: firstly, the varying amounts of training images containing text with different orientations lead to varying accuracy rates among the different text recognition models, resulting in overall low accuracy; secondly, training multiple text recognition models requires adjustments for each update or other event, leading to high management complexity.

[0142] In this embodiment, by rotating the image, regardless of the character arrangement direction of the text in the image, it can be used as a training image for the text recognition model. In this way, the training images used by multiple text recognition models in traditional schemes can be shared as the training data of the text recognition model in this embodiment. This can effectively expand the amount of training data of the text recognition model, which is conducive to training a text recognition model with higher recognition accuracy, thereby achieving the goal of improving the recognition accuracy of text in images.

[0143] Furthermore, the text recognition model trained in this embodiment can accurately recognize text images containing text with various character arrangement directions. Compared to the traditional approach that requires training multiple text recognition models, the text recognition model in this embodiment is easier to manage and facilitates timely updates and iterations, thereby improving the recognition accuracy of the text recognition model.

[0144] This application describes an apparatus embodiment that can be used to perform the image processing method described above. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the image processing method described above.

[0145] This application provides an image processing apparatus, such as... Figure 9 As shown, the device includes an image rotation unit 901, a feature acquisition unit 902, and a processing unit 903, wherein:

[0146] The image rotation unit 901 is used to rotate the text image based on the direction difference between the character arrangement direction of the text in the text image and multiple specified arrangement directions, so as to obtain the rotated image corresponding to each specified arrangement direction.

[0147] The feature acquisition unit 902 is used to extract features from each rotated image to obtain the image features of each rotated image.

[0148] The feature acquisition unit 902 is also used to fuse multiple image features to obtain fused features;

[0149] The processing unit 903 is used to classify the fused features based on the position of the characters in each rotated image to obtain the recognized text of the text image.

[0150] In one embodiment of this application, based on the aforementioned scheme, when the feature acquisition unit 902 fuses multiple image features to obtain fused features, it is specifically used to stitch multiple image features together to obtain stitched features; perform activation mapping processing on each feature value in the stitched features to obtain activation features; obtain target features of each rotated image based on the similarity between the activation features and each image feature; and fuse multiple target features to obtain fused features.

[0151] In one embodiment of this application, based on the aforementioned scheme, when the processing unit 903 performs classification processing on the fusion features based on the position of characters in each rotated image to obtain the recognized text of the text image, it can specifically be used to: obtain the predicted characters of all characters in the text image, and obtain the recognition result of the text image based on all predicted characters; wherein, obtaining the predicted character of the i-th character in the text image includes: obtaining the mapping feature of the i-th character in the text image in the fusion features based on the position of the i-th character in each rotated image; wherein, the i-th character in the text image is any character in the text image, and i is a positive integer; based on the mapping features corresponding to the 1st to i-1th characters in the text image respectively, performing feature classification on the mapping features corresponding to the i-th character to obtain the predicted character corresponding to the i-th character in the text image.

[0152] In one embodiment of this application, the i-th character is not the first character in the text image. Based on the aforementioned scheme, when the processing unit 903 performs feature classification on the mapping features corresponding to the i-th character based on the mapping features corresponding to the 1st to i-1th characters in the text image to obtain the predicted character corresponding to the i-th character in the text image, it can specifically be used to: concatenate the mapping features of the i-th character in the text image with the latent vector corresponding to the i-1th character in the text image to obtain the concatenated features; wherein, the latent vector corresponding to the i-1th character is obtained based on the concatenated features corresponding to the i-1th character, the state vector corresponding to the i-1th character, and the state vector corresponding to the i-2th character in the text image; perform feature classification on the concatenated features to obtain the predicted character corresponding to the i-th character in the text image.

[0153] In one embodiment of this application, the multiple specified arrangement directions include a horizontal arrangement direction and a vertical arrangement direction. Based on the aforementioned scheme, when the image rotation unit 901 rotates the text image based on the direction difference between the character arrangement direction of the text in the text image and the multiple specified arrangement directions to obtain the rotated image corresponding to each specified arrangement direction, it can specifically be used to: rotate the text image based on the direction difference between the character arrangement direction of the text in the text image and the horizontal arrangement direction to obtain the rotated image corresponding to the horizontal arrangement direction; and rotate the text image based on the direction difference between the character arrangement direction of the text in the text image and the vertical arrangement direction to obtain the rotated image corresponding to the vertical arrangement direction.

[0154] In one embodiment of this application, based on the aforementioned scheme, the processing unit 903 can specifically classify the fused features based on the position of characters in each rotated image using a text recognition model to obtain the recognized text of the text image. The text recognition model is trained in the following manner: acquiring a training image and the real text contained in the training image; rotating the training image based on the direction difference between the character arrangement direction of the text in the training image and multiple specified arrangement directions to obtain reference images corresponding to each specified arrangement direction; extracting features from each reference image to obtain the image features of each reference image; fusing the image features of multiple reference images to obtain fused training features; and training based on the real text and the fused training features to obtain the text recognition model.

[0155] In one embodiment of this application, based on the aforementioned scheme, when the processing unit 903 trains a text recognition model based on real text and fused training features to obtain a text recognition model, it can specifically be used to: classify the fused training features based on the position of reference characters in each reference image and the real characters in the real text that match the character arrangement order of each reference character to obtain the predicted characters corresponding to each reference character; calculate the character loss value between the real characters that match each reference character and the predicted characters corresponding to each reference character; and train in the direction of reducing the character loss value to obtain a text recognition model.

[0156] In one embodiment of this application, based on the aforementioned scheme, when the processing unit 903 performs feature classification on the fused training features based on the positions of reference characters in each reference image and the real characters in the real text that match the character arrangement order of each reference character, and obtains the predicted characters corresponding to each reference character, it can specifically be used to: obtain the mapping features of the j-th reference character in the training image in the fused training features according to the position of the j-th reference character in each reference image; wherein, the j-th reference character in the training image is any character in the training image, and j is a positive integer; concatenate the latent vector corresponding to the (j-1)-th reference character in the training image and the mapping features corresponding to the j-th reference character in the training image to obtain the concatenated reference features; wherein, the latent vector corresponding to the (j-1)-th reference character is obtained based on the latent vector corresponding to the (j-2)-th reference character in the training image, the state vector corresponding to the (j-2)-th reference character, and the character features of the real characters that match the (j-1)-th reference character; classify the concatenated reference features to obtain the predicted characters corresponding to the j-th reference character in the training image.

[0157] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0158] The apparatus provided in the above embodiments can be located within a terminal device or a server. By fusing the image features of multiple rotated images using the apparatus provided in this application, a fused feature can be obtained that characterizes the features of text in a text image under multiple specified arrangement directions. This allows for comprehensive classification based on the fused feature, taking into account the characteristics of text in the text image under multiple arrangement directions, which is beneficial for identifying more accurate text. Furthermore, in this application embodiment, the fused feature is classified based on the position of characters in each rotated image. This method makes the classification of the fused feature a character-level classification prediction, which is more accurate than predicting the entire text, and is beneficial for further identifying more accurate text.

[0159] Embodiments of this application also provide an electronic device, including one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, which, when executed by one or more processors, cause the electronic device to implement the image processing method described above.

[0160] Figure 10 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0161] It should be noted that, Figure 10 The computer system 1000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0162] like Figure 10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on a program stored in read-only memory (ROM) 1002 or a program loaded from storage portion 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0163] In some embodiments, the following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0164] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as a computer program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor (CPU) 1001, it performs various functions defined in the system of this application.

[0165] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory, flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0167] The units or modules described in the embodiments of this application can be implemented in software or hardware, and can also be located in a processor. The names of these units or modules do not necessarily limit the specific unit or module itself.

[0168] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0169] Another aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the image processing method as described above in the various embodiments.

[0170] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0171] Other embodiments of this application will readily conceive of by considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0172] The above content is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. An image processing method, characterized in that, The method includes: Based on the direction difference between the character arrangement direction of the text in the text image and multiple specified arrangement directions, the text image is rotated to obtain rotated images corresponding to each specified arrangement direction; Feature extraction is performed on each rotated image to obtain the image features of each rotated image; Multiple image features are fused to obtain fused features; Based on the position of the characters in each rotated image, the fused features are classified to obtain the recognized text of the text image.

2. The method according to claim 1, characterized in that, The process of fusing multiple image features to obtain fused features includes: The multiple image features are concatenated to obtain the concatenated features; Activation mapping is performed on each feature value in the spliced ​​features to obtain the activation features; The target features of each rotated image are obtained based on the similarity between the activation features and each image feature. The fused feature is obtained by fusing multiple target features.

3. The method according to claim 1, characterized in that, The step of classifying the fused features based on the positions of characters in each rotated image to obtain the recognized text of the text image includes: Obtain the predicted characters of all characters in the text image, and obtain the recognition result of the text image based on all predicted characters; The process of obtaining the predicted character of the i-th character in the text image includes: Based on the position of the i-th character in each of the rotated images, the mapping feature of the i-th character in the text image in the fusion feature is obtained; wherein, the i-th character in the text image is any character in the text image, and i is a positive integer; Based on the mapping features corresponding to the 1st to i-1th characters in the text image, feature classification is performed on the mapping features corresponding to the i-th character to obtain the predicted character corresponding to the i-th character in the text image.

4. The method according to claim 3, characterized in that, The i-th character is not the first character in the text image; the step of classifying the mapping features corresponding to the i-th character based on the mapping features corresponding to the 1st to i-1th characters in the text image to obtain the predicted character corresponding to the i-th character in the text image includes: The mapping feature of the i-th character in the text image is concatenated with the latent vector corresponding to the (i-1)-th character in the text image to obtain the concatenated feature; wherein, the latent vector corresponding to the (i-1)-th character is obtained based on the concatenated feature corresponding to the (i-1)-th character, the state vector corresponding to the (i-1)-th character, and the state vector corresponding to the (i-2)-th character in the text image; The concatenated features are classified to obtain the predicted character corresponding to the i-th character in the text image.

5. The method according to claim 1, characterized in that, The multiple specified arrangement directions include horizontal arrangement directions and vertical arrangement directions; the step of rotating the text image based on the direction difference between the character arrangement direction of the text in the text image and the multiple specified arrangement directions to obtain rotated images corresponding to each specified arrangement direction includes: Based on the directional difference between the character arrangement direction and the horizontal arrangement direction of the text in the text image, the text image is rotated to obtain a rotated image corresponding to the horizontal arrangement direction. Based on the directional difference between the character arrangement direction and the vertical arrangement direction in the text image, the text image is rotated to obtain a rotated image corresponding to the vertical arrangement direction.

6. The method according to any one of claims 1 to 5, characterized in that, The text recognition model classifies the fused features based on the positions of characters in each rotated image to obtain the recognized text of the text image. The text recognition model was trained in the following manner: Obtain training images and the real text contained within those training images; Based on the direction difference between the character arrangement direction of the text in the training image and the multiple specified arrangement directions, the training image is rotated to obtain the reference image corresponding to each specified arrangement direction. Feature extraction is performed on each reference image to obtain the image features of each reference image; The image features of multiple reference images are fused to obtain fused training features; The text recognition model is obtained by training based on the real text and the fused training features.

7. The method according to claim 6, characterized in that, The text recognition model is obtained by training based on the real text and the fused training features, including: Based on the position of the reference characters in each reference image and the real characters in the real text that match the character arrangement order of the reference characters, feature classification is performed on the fused training features to obtain the predicted characters corresponding to each reference character; Calculate the character loss value between the real character that matches each of the reference characters and the predicted character corresponding to each of the reference characters; The text recognition model is obtained by training in the direction of reducing the character loss value.

8. The method according to claim 7, characterized in that, The method involves classifying the fused training features based on the positions of reference characters in each reference image and the real characters in the real text that match the character order of the reference characters, to obtain the predicted characters corresponding to each reference character, including: Based on the position of the j-th reference character in each reference image, obtain the mapping feature of the j-th reference character in the training image in the fused training feature; wherein, the j-th reference character in the training image is any character in the training image, and j is a positive integer; The latent vector corresponding to the (j-1)th reference character in the training image and the mapping feature corresponding to the j-th reference character in the training image are concatenated to obtain the concatenated reference feature; wherein, the latent vector corresponding to the (j-1)th reference character is obtained based on the latent vector corresponding to the (j-2)th reference character in the training image, the state vector corresponding to the (j-2)th reference character, and the character features of the real character that matches the (j-1)th reference character; The splicing reference features are classified to obtain the predicted character corresponding to the j-th reference character in the training image.

9. An image processing apparatus, characterized in that, The device includes an image rotation unit, a feature acquisition unit, and a processing unit, wherein: The image rotation unit is used to rotate the text image based on the direction difference between the character arrangement direction of the text in the text image and multiple specified arrangement directions, so as to obtain the rotated image corresponding to each specified arrangement direction. The feature acquisition unit is used to extract features from each rotated image to obtain the image features of each rotated image. The feature acquisition unit is used to fuse multiple image features to obtain fused features; The processing unit is used to classify the fused features based on the positions of the characters in each of the rotated images to obtain the recognized text of the text image.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 8.

11. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image processing method as described in any one of claims 1 to 8.

Images