Image recognition method and system

Abstract

The present application discloses an image recognition method and system. The method includes: obtaining a first image and a second image, wherein the arrangement direction of the first text data contained in the first image is a first direction, and the arrangement direction of the second text data contained in the second image is a second direction; processing the first image and the second image using a text recognition model to obtain a first recognition result for the first image and a second recognition result for the second image; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result. The present application solves the technical problem in the related art that text recognition methods recognize text data arranged in multiple directions, wasting computing and storage resources.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and more specifically, to an image recognition method and system. Background Art

[0002] Currently, text recognition algorithms can be used to identify text data contained in images. However, since traditional text line recognition algorithms can only process text data arranged in one direction, when recognizing text data arranged in multiple directions, a simple solution is to train multiple models, each for processing text data arranged in one direction. However, this solution requires storing multiple models and performing calculations using each model separately, resulting in a waste of computing and storage resources.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide an image recognition method and system to at least solve the technical problem in related arts that text recognition methods recognize text data arranged in multiple directions, wasting computing and storage resources.

[0005] According to one aspect of an embodiment of the present application, an image recognition method is provided, including: acquiring a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the first text data is arranged in a first direction, and the second text data is arranged in a second direction; processing the first image and the second image using a text recognition model to obtain a first recognition result of the first image and a second recognition result of the second image; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0006] According to another aspect of an embodiment of the present application, an image recognition device is also provided, including: a first acquisition module, used to acquire a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; a processing module, used to process the first image and the second image using a text recognition model to obtain a first recognition result of the first image and a second recognition result of the second image; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0007] According to another aspect of an embodiment of the present application, an image recognition method is also provided, including: acquiring a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; performing feature extraction on the first image and the second image to obtain a first feature sequence of the first image and a second feature sequence of the second image; based on the first feature sequence, obtaining a first recognition result of the first image; based on the second feature sequence, obtaining a second recognition result of the second image.

[0008] According to another aspect of an embodiment of the present application, a storage medium is further provided. The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned image recognition method.

[0009] According to another aspect of an embodiment of the present application, a computing device is further provided, including: a memory and a processor, the memory is used to store a program, and the processor is used to run the program, wherein the above-mentioned image recognition method is executed when the program is run.

[0010] According to another aspect of an embodiment of the present application, an image recognition system is also provided, including: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: acquiring a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; using a text recognition model to process the first image and the second image to obtain a first recognition result of the first image and a second recognition result of the second image; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0011] In an embodiment of the present application, for an image containing text data arranged in two different directions, a single text recognition model can be used to process the image, obtaining recognition results for the text data arranged in different directions, thereby achieving the purpose of image recognition. It is easy to note that the use of two recognition model branches allows the simultaneous recognition of horizontal and vertical text lines using a single model, thereby avoiding the need to store two recognition models online, achieving the technical effect of saving computing and storage resources, and further resolving the technical problem in related technologies of wasting computing and storage resources by recognizing text data arranged in multiple directions. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image recognition method according to an embodiment of the present application;

[0014] Figure 2 is a flowchart of an image recognition method according to an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of the architecture of an optional image recognition method according to an embodiment of the present application;

[0016] Figure 4 is a schematic diagram of an optional generation of a mask matrix according to an embodiment of the present application;

[0017] Figure 5 is a schematic diagram of an image recognition device according to an embodiment of the present application;

[0018] Figure 6 is a flowchart of another image recognition method according to an embodiment of the present application; and

[0019] Figure 7 This is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0023] OCR: Optical Character Recognition, optical character recognition, can refer to the recognition of optical characters through image processing and pattern recognition technology.

[0024] CTC: Connectionist Temporal Classification, connectionist temporal classification, can be used to solve the problem that input sequences and output sequences are difficult to correspond one to one.

[0025] CRNN: Convolution Recurrent Neural Network, a convolutional recurrent neural network, can be a combination of convolutional neural network CNN and recurrent neural network RNN. Its network architecture mainly consists of three parts, including convolutional layer, recurrent layer and transcription layer.

[0026] Attention Mechanism: This mechanism can be used to improve the performance of RNN-based encoding and decoding models. By assigning different weights to each word in a sentence, it makes the learning of the neural network model more flexible. It can also serve as an alignment relationship to explain the alignment between translation input and output sentences.

[0027] BLSTM: Bidirectional Long Short-term Memory, a bidirectional long short-term memory neural network that can simultaneously utilize information from past and future moments. Its network consists of two ordinary RNNs: a forward RNN that utilizes past information and a reverse RNN that utilizes future information.

[0028] RESNET: Residual Neural Network, by adding direct channels in the network, allowing the original input information to be transmitted directly to the subsequent layers. The neural network in this layer does not need to learn the entire output, but only needs to learn the residual of the previous network output.

[0029] Mask: A mask can be a string of binary codes that performs a bitwise AND operation on the target field to mask the current input bit.

[0030] Existing text recognition algorithms can be mainly divided into two categories: CTC-based text recognition algorithms and Attention-based text recognition algorithms.

[0031] CTC-based text recognition algorithms consist of three main components: a convolutional neural network (CNN) for extracting image feature sequences; a recurrent neural network (RNN) for learning contextual information about text feature sequences; and a CTC decoder. By introducing a blank class, the CTC decoder solves the length-to-short alignment problem in sequence recognition and decodes the probability distribution output by the RNN into the final recognition result. In this type of approach, the CNN in the first component typically sets the image height to 1, requiring the input image to have a fixed height. This characteristic makes the model unable to process images containing text data arranged in a second direction.

[0032] Attention-based text recognition algorithms also consist of three main components. The first two components are consistent with CTC-based text recognition algorithms, but the third component utilizes an attention mechanism for decoding, outputting recognition results sequentially at each time point. Unlike CTC methods, the CNN component of attention-based text recognition algorithms can output two-dimensional feature maps. Therefore, in theory, this type of algorithm can simultaneously process images containing text data arranged in both horizontal and vertical directions. However, because the number of categories in Chinese is far greater than that in English, the time-based classification Attention algorithm is less effective in Chinese text line recognition. In addition, the forward pass of the Attention algorithm consumes more time than the CTC algorithm. Therefore, the Attention algorithm is not suitable for current optical text recognition scenarios.

[0033] Because traditional text line recognition algorithms can only process images with text lines arranged in one direction, the model's applicability is significantly limited. A simple solution is to train two models separately, but this requires storing both models online, resulting in a waste of resources.

[0034] In order to solve the above problems, this application optimizes and improves the text recognition algorithm and provides a model that can simultaneously recognize text data arranged in both horizontal and vertical directions. This not only avoids storing two models, but also improves the recognition effect of the model.

[0035] The above-mentioned text recognition algorithm can be applied to a variety of fields requiring text recognition. For example, in the field of assisted design on online shopping platforms, to help merchants design product detail pages, merchants can upload design templates. The algorithm can then identify text data arranged in different directions within the design template and replace it as needed to create the merchant's customized product detail page. Furthermore, for user convenience, the algorithm can be deployed as a SaaS (Software as a Service) service on a cloud server, allowing users to use the method to recognize text data arranged in different directions over the internet as needed.

[0036] Example 1

[0037] According to an embodiment of the present application, an image recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image recognition method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image recognition method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned image recognition method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0043] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the aforementioned computer device (or mobile device).

[0044] Under the above operating environment, this application provides Figure 2 The image recognition method shown. Figure 2 This is a flow chart of an image recognition method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0045] Step S202, acquiring a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the first text data is arranged in a first direction, and the second text data is arranged in a second direction;

[0046] The first image and the second image in the above steps may refer to text images extracted from the image to be processed, and the texts contained in the two images are arranged in different directions, which are the first direction and the second direction respectively. The image to be processed may be a template image uploaded by a merchant in the auxiliary design field of an online shopping platform, or an image uploaded by a user received in a SaaS service, but is not limited to this. In actual applications, the text in the image to be processed is usually arranged in a horizontal direction, and may also be arranged in a vertical direction. In the embodiment of the present application, the first direction is the horizontal direction and the second direction is the vertical direction as an example for illustration, but is not limited to this.

[0047] In an optional embodiment, the image to be processed containing horizontal text and vertical text can be processed using an image text extraction method to extract a first image containing horizontal text and a second image containing vertical text. The two images are then processed simultaneously using a model to obtain the final recognition result.

[0048] Step S204: Process the first image and the second image using a text recognition model to obtain a first recognition result for the first image and a second recognition result for the second image;

[0049] Among them, the text recognition model is used to input the first image and the second image into the feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0050] Optionally, the feature extraction model includes: multiple convolutional layers, excitation layers and pooling layers connected in sequence, wherein the parameter of the last pooling layer is 2*1; the first recognition model or the second recognition model includes: a convolutional neural network, a bidirectional long short-term memory neural network and a connectionist time classification.

[0051] In order to ensure that text data arranged in multiple directions can be recognized simultaneously by one model, the present application can improve the text recognition algorithm based on CTC. Since the underlying features of text arranged in the horizontal direction and text arranged in the vertical direction are similar, most of the existing convolutional neural network model can be shared to obtain a feature extraction model, thereby reducing the number of parameters. In addition, text data arranged in different directions can be recognized by different recognition models based on the existing model, and the structures of all recognition models are exactly the same.

[0052] The feature extraction model can be based on a VGG16 (Visual Geometry Group Network) architecture, extracting feature information from text line images for subsequent recognition. The first and second recognition models can have identical architectures, consisting of CNN, BLSTM, and CTC.

[0053] It should be noted that in order to enhance the feature extraction capability of the feature extraction model, the feature extraction model may also adopt ResNet, but is not limited thereto, and may also adopt other structures.

[0054] Since the sequence recognition model in the traditional CTC-based text recognition algorithm needs to process the image height to 1 and retain sufficient width, the model will include a 2*1 pooling layer. The first recognition model and the second recognition model can start after the 2*1 pooling layer appears, and the feature extraction model is before it.

[0055] For example, take the text data arranged in horizontal and vertical directions as an example. Figure 3 As shown, the overall framework of the text extraction model of the present application can be divided into three parts. The first part is data processing; the second part is a feature extraction model of a shared CNN layer, which is used to extract feature information of the first text data and the second text data contained in the image; the third part is a branch recognition network (including the above-mentioned first recognition model and second recognition model), which is used to recognize text data arranged in the horizontal and vertical directions respectively.

[0056] Based on the solution provided by the above-mentioned embodiment of the present application, for an image containing text data arranged in two different directions, a text recognition model can be used to process it, and the recognition results of the text data in different directions can be obtained, thereby achieving the purpose of image recognition. It is easy to notice that the two recognition model branches are used to realize the simultaneous recognition of horizontal text lines and vertical text lines through a single model, thereby avoiding the need to store two recognition models online, achieving the technical effect of saving computing and storage resources, and further solving the technical problem in the related art that text recognition methods recognize text data arranged in multiple directions, which wastes computing and storage resources.

[0057] In the above embodiment of the present application, before using the text recognition model to process the first image and the second image, the method also includes: obtaining the position of each first character in the first image, and the position of each second character in the second image; based on the position of each first character, generating a first mask matrix corresponding to the first image, and based on the position of each second character, generating a second mask matrix corresponding to the second image, wherein the first mask matrix is ​​used to represent the arrangement direction of the first text data, and the second mask matrix is ​​used to represent the arrangement direction of the second text data; splicing the first image, the first mask matrix, the second image and the second mask matrix to obtain an input matrix; using the text recognition model to process the input matrix to obtain a first recognition result and a second recognition result.

[0058] Optionally, the first and second mask matrices in the above steps can be asymmetric matrices that can identify certain directional information. The values ​​of the mask matrices are only related to the position and arrangement direction of the characters in the text data. In the embodiment of the present application, for horizontally arranged text data, the values ​​of the first mask matrix increase from left to right and from top to bottom. Therefore, the arrangement direction of the text data can be determined based on the numerical values ​​of the mask matrices.

[0059] In an optional embodiment, since the recognition process does not require high accuracy of character positions, the position of each character in the text data can be obtained through a simple projection segmentation method, and a mask matrix is ​​generated based on the character position and arrangement direction. The information of the mask matrix is ​​added to the original first image and the second image, and further processed using a text recognition model, thereby enhancing the model's ability to distinguish text data arranged in horizontal and vertical directions, and enabling the shared CNN layer to specifically learn feature information applicable to texts in different directions, ultimately improving the recognition effect of the model.

[0060] For example, still take the text data arranged in horizontal and vertical directions as an example. Figure 3 As shown in the figure, the data processing process can be to generate a mask matrix through the character position, and then concatenate each image with the corresponding Mask in the channel dimension. This four-channel matrix is ​​used as the input of the model.

[0061] In the above embodiment of the present application, generating a second mask matrix corresponding to the second image based on the position of each second character includes: rotating the second image to obtain a rotated image, wherein the arrangement direction of the second text data in the rotated image is the first direction; generating a preset matrix based on the position of each second character; and rotating the preset matrix to obtain a second mask matrix.

[0062] The preset matrix in the above steps can be a mask matrix corresponding to the rotated image. In an embodiment of the present application, for text data arranged horizontally, similar to the first mask matrix, the value of the preset matrix can increase from left to right and from top to bottom.

[0063] In an optional embodiment, the second image can be rotated so that the arrangement direction of the text data is the same as that in the first image, and then a mask matrix corresponding to the rotated image is generated through the character position. Then, by rotating the generated mask matrix, the mask matrix corresponding to the second image (i.e., the second mask matrix mentioned above) can be obtained.

[0064] It should be noted that the rotation direction of the second image is opposite to the rotation direction of the preset matrix.

[0065] For example, still taking the text data arranged in the horizontal and vertical directions as an example for illustration, as Figure 3 shown, first, the image containing the text data arranged in the vertical direction can be rotated to make the text data arranged in the horizontal direction. Then, a mask matrix is generated based on the character positions, and the mask matrix is rotated counterclockwise by 90 degrees to obtain the mask matrix corresponding to the original text data arranged in the vertical direction. Finally, each image is concatenated with the corresponding Mask in the channel dimension, and this four-channel matrix is used as the input of the model.

[0066] As Figure 4 shown, for the text data "Standardization Technical Committee" arranged in the horizontal direction, which contains 8 characters, thus, a mask matrix corresponding to each character can be generated, and the values of the mask matrix increase sequentially from left to right, and then an image with the mask matrix added is obtained. It should be noted that the size of the mask matrix is the same as the size of the region corresponding to the character. For example, if the size of the region corresponding to the character "标" is 32*50, then the size of the mask matrix corresponding to this character is 32*50. If the size of the region corresponding to each character is 32*50, then the size of the mask matrix corresponding to the image is 32*400, that is, the size of the mask matrix corresponding to the image is the same as the size of the image.

[0067] For the text data "Economic Inspection Brigade" arranged in the vertical direction, which contains 6 characters, thus, it can be rotated into text data arranged in the horizontal direction, and a mask matrix corresponding to each character is generated, and the values of the mask matrix increase sequentially from left to right. Further, the mask matrix is rotated counterclockwise by 90 degrees to obtain the final mask matrix, and then an image with the mask matrix added is obtained.

[0068] In the above embodiments of the present application, the method further includes: obtaining a plurality of training images, where the arrangement direction of the text data included in each training image is the first direction or the second direction; training an initial model using the plurality of training images to obtain a text recognition model.

[0069] The training data in the above steps may include images of text data arranged in different directions.

[0070] In an optional embodiment, images of text data arranged in different directions can be mixed, and the mixed images are used as training data to train the model. During the joint training process, the training data arranged in different directions can complement each other, enhance the feature extraction ability of the encoder, improve the generalization and robustness of the text recognition model, and make the recognition effect of the text recognition model stronger than that of the model trained with training data arranged in a single direction.

[0071] For example, still taking the text data arranged in the horizontal and vertical directions as an example for illustration, for as Figure 3The text recognition model shown can use a mixed training model of text data arranged in horizontal and vertical directions to obtain a trained text recognition model.

[0072] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0074] Example 2

[0075] According to an embodiment of the present application, an image recognition device for implementing the above-mentioned image recognition method is also provided. Figure 5 As shown, the device 500 includes: a first acquisition module 502 and a processing module 504.

[0076] Among them, the first acquisition module 502 is used to acquire the first image and the second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is the first direction, and the arrangement direction of the second text data is the second direction; the processing module 504 is used to use the text recognition model to process the first image and the second image to obtain a first recognition result of the first image and a second recognition result of the second image; wherein the text recognition model is used to input the first image and the second image into the feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0077] It should be noted that the first acquisition module 502 and the processing module 504 described above correspond to steps S202 to S204 in Example 1. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0078] In the above embodiment of the present application, the device further includes: a second acquisition module, a generation module and a splicing module.

[0079] Among them, the second acquisition module is used to obtain the position of each first character in the first image and the position of each second character in the second image; the generation module is used to generate a first mask matrix corresponding to the first image based on the position of each first character, and to generate a second mask matrix corresponding to the second image based on the position of each second character, wherein the first mask matrix is ​​used to represent the arrangement direction of the first text data, and the second mask matrix is ​​used to represent the arrangement direction of the second text data; the splicing module is used to splice the first image, the first mask matrix, the second image and the second mask matrix to obtain an input matrix; the processing module is also used to process the input matrix using a text recognition model to obtain a first recognition result and a second recognition result.

[0080] In the above embodiment of the present application, the generation module includes: a first rotation unit, a generation unit and a second rotation unit.

[0081] Among them, the first rotation unit is used to rotate the second image to obtain a rotated image, wherein the arrangement direction of the second text data in the rotated image is the first direction; the generation unit is used to generate a preset matrix based on the position of each second character; the second rotation unit is used to rotate the preset matrix to obtain a second mask matrix.

[0082] In the above embodiment of the present application, the device further includes: a third acquisition module and a training module.

[0083] Among them, the third acquisition module is used to obtain multiple training images, wherein the arrangement direction of the text data contained in each training image is the first direction or the second direction; the training module is used to use the multiple training images to train the initial model to obtain a text recognition model.

[0084] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0085] Example 3

[0086] According to an embodiment of the present application, an image recognition method is also provided.

[0087] Figure 6 FIG. 1 is a flow chart of another image recognition method according to an embodiment of the present application. Figure 6 As shown, the method includes the following steps:

[0088] Step S602: Acquire a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the first text data is arranged in a first direction, and the second text data is arranged in a second direction;

[0089] The first image and the second image in the above steps may refer to text images extracted from the image to be processed, and the texts contained in the two images are arranged in different directions, which are the first direction and the second direction respectively. The image to be processed may be a template image uploaded by a merchant in the auxiliary design field of an online shopping platform, or an image uploaded by a user received in a SaaS service, but is not limited to this. In actual applications, the text in the image to be processed is usually arranged in a horizontal direction, and may also be arranged in a vertical direction. In the embodiment of the present application, the first direction is the horizontal direction and the second direction is the vertical direction as an example for illustration, but is not limited to this.

[0090] Step S604: extract features from the first image and the second image to obtain a first feature sequence of the first image and a second feature sequence of the second image;

[0091] Step S606: obtaining a first recognition result of the first image based on the first feature sequence;

[0092] Step S608: Obtain a second recognition result of the second image based on the second feature sequence.

[0093] In the above embodiment of the present application, feature extraction is performed on the first image and the second image to obtain a first feature sequence of the first image and a second feature sequence of the second image, including: inputting the first image and the second image into a shared feature extraction model to obtain a first feature sequence and a second feature sequence.

[0094] Optionally, the feature extraction model includes: multiple sequentially connected convolutional layers, excitation layers, and pooling layers, wherein the parameter of the last pooling layer is 2*1. The main structure of the feature extraction model can be pruned based on VGG16 (Visual Geometry Group Network) to extract feature information of text line images for subsequent recognition.

[0095] In the above embodiment of the present application, obtaining a first recognition result of the first image based on the first feature sequence includes: inputting the first feature sequence into a first recognition model to obtain the first recognition result.

[0096] Optionally, the first recognition model includes: a convolutional neural network, a bidirectional long short-term memory neural network, and a connectionist temporal classification. The first recognition model may have a structure composed of a CNN, a BLSTM, and a CTC.

[0097] In the above embodiment of the present application, obtaining a second recognition result of the second image based on the second feature sequence includes: inputting the second feature sequence into a second recognition model to obtain the second recognition result.

[0098] Optionally, the second recognition model includes: a convolutional neural network, a bidirectional long short-term memory neural network, and a connectionist temporal classification. The structure of the second recognition model can be the same as the first recognition model, consisting of a CNN, a BLSTM, and a CTC.

[0099] In the above embodiment of the present application, before performing feature extraction on the first image and the second image, the method also includes: obtaining the position of each first character in the first image, and the position of each second character in the second image; generating a first mask matrix corresponding to the first image based on the position of each first character, and generating a second mask matrix corresponding to the second image based on the position of each second character, wherein the first mask matrix is ​​used to represent the arrangement direction of the first text data, and the second mask matrix is ​​used to represent the arrangement direction of the second text data; splicing the first image, the first mask matrix, the second image and the second mask matrix to obtain an input matrix; performing feature extraction on the input matrix to obtain a first feature sequence and a second feature sequence.

[0100] Optionally, the first and second mask matrices in the above steps can be asymmetric matrices that can identify certain directional information. The values ​​of the mask matrices are only related to the position and arrangement direction of the characters in the text data. In the embodiment of the present application, for horizontally arranged text data, the values ​​of the first mask matrix increase from left to right and from top to bottom. Therefore, the arrangement direction of the text data can be determined based on the numerical values ​​of the mask matrices.

[0101] In the above embodiment of the present application, generating a second mask matrix corresponding to the second image based on the position of each second character includes: rotating the second image to obtain a rotated image, wherein the arrangement direction of the second text data in the rotated image is the first direction; generating a preset matrix based on the position of each second character; and rotating the preset matrix to obtain a second mask matrix.

[0102] In the above embodiment of the present application, the method also includes: obtaining multiple training images, wherein the arrangement direction of the text data contained in each training image is the first direction or the second direction; using the multiple training images to train the feature extraction model, the first recognition model and the second recognition model to obtain a text recognition model.

[0103] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0104] Example 4

[0105] According to an embodiment of the present application, an image recognition system is further provided, including:

[0106] processor; and

[0107] The memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: obtaining a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; using a text recognition model to process the first image and the second image to obtain a first recognition result of the first image and a second recognition result of the second image; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0108] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0109] Example 5

[0110] The embodiment of the present application can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0111] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0112] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image recognition method: obtaining a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; using a text recognition model to process the first image and the second image to obtain a first recognition result of the first image and a second recognition result of the second image; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0113] Optionally, Figure 7 This is a structural block diagram of a computer terminal according to an embodiment of the present application. Figure 7 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 702 and a memory 704.

[0114] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image recognition method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image recognition method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtaining a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; using a text recognition model to process the first image and the second image to obtain a first recognition result of the first image and a second recognition result of the second image; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0116] Optionally, the processor may also execute the program code of the following steps: obtaining the position of each first character in the first image and the position of each second character in the second image; generating a first mask matrix corresponding to the first image based on the position of each first character, and generating a second mask matrix corresponding to the second image based on the position of each second character, wherein the first mask matrix is ​​used to represent the arrangement direction of the first text data, and the second mask matrix is ​​used to represent the arrangement direction of the second text data; splicing the first image, the first mask matrix, the second image, and the second mask matrix to obtain an input matrix; processing the input matrix using a text recognition model to obtain a first recognition result and a second recognition result.

[0117] Optionally, the processor may also execute the program code of the following steps: rotating the second image to obtain a rotated image, wherein the arrangement direction of the second text data in the rotated image is the first direction; generating a preset matrix based on the position of each second character; and rotating the preset matrix to obtain a second mask matrix.

[0118] Optionally, the processor may also execute the program code of the following steps: obtaining a plurality of training images, wherein the arrangement direction of the text data contained in each training image is the first direction or the second direction; and training the initial model using the plurality of training images to obtain a text recognition model.

[0119] The present invention provides an image recognition solution that uses a single text recognition model to identify text data arranged in two different orientations, thereby avoiding the need to store two recognition models online. This approach saves computing and storage resources and addresses the technical issue in related art where text recognition methods waste computing and storage resources when identifying text data arranged in multiple orientations.

[0120] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; perform feature extraction on the first image and the second image to obtain a first feature sequence of the first image and a second feature sequence of the second image; obtain a first recognition result of the first image based on the first feature sequence; obtain a second recognition result of the second image based on the second feature sequence.

[0121] It can be understood by those skilled in the art that Figure 7The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 7 It does not limit the structure of the above electronic device. For example, the computer terminal A may also include Figure 7 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 7 Different configurations shown.

[0122] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0123] Example 6

[0124] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image recognition method provided by the above embodiment.

[0125] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0126] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: acquiring a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; processing the first image and the second image using a text recognition model to obtain a first recognition result of the first image and a second recognition result of the second image; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into the first recognition model to obtain a first recognition result, and input the second feature sequence into the second recognition model to obtain a second recognition result.

[0127] Optionally, the storage medium is also configured to store program code for executing the following steps: obtaining the position of each first character in the first image and the position of each second character in the second image; generating a first mask matrix corresponding to the first image based on the position of each first character, and generating a second mask matrix corresponding to the second image based on the position of each second character, wherein the first mask matrix is ​​used to represent the arrangement direction of the first text data, and the second mask matrix is ​​used to represent the arrangement direction of the second text data; splicing the first image, the first mask matrix, the second image and the second mask matrix to obtain an input matrix; processing the input matrix using a text recognition model to obtain a first recognition result and a second recognition result.

[0128] Optionally, the storage medium is also configured to store program code for executing the following steps: rotating the second image to obtain a rotated image, wherein the arrangement direction of the second text data in the rotated image is the first direction; generating a preset matrix based on the position of each second character; and rotating the preset matrix to obtain a second mask matrix.

[0129] Optionally, the above-mentioned storage medium is also configured to store program code for executing the following steps: obtaining multiple training images, wherein the arrangement direction of the text data contained in each training image is the first direction or the second direction; using the multiple training images to train the initial model to obtain a text recognition model.

[0130] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: acquiring a first image and a second image, wherein the first image contains first text data, the second image contains second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; performing feature extraction on the first image and the second image to obtain a first feature sequence of the first image and a second feature sequence of the second image; acquiring a first recognition result of the first image based on the first feature sequence; and acquiring a second recognition result of the second image based on the second feature sequence.

[0131] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0132] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0134] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0137] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image recognition method, comprising: Extracting and processing an image to be processed that includes first text data and second text data to obtain a first image and a second image, wherein the first image includes the first text data, the second image includes the second text data, the first text data is arranged in a first direction, and the second text data is arranged in a second direction; Generate a first mask matrix corresponding to the first image, and generate a second mask matrix corresponding to the second image, wherein the first mask matrix is ​​used to represent the arrangement direction of the first text data, and the second mask matrix is ​​used to represent the arrangement direction of the second text data, the first mask matrix and the second mask matrix are asymmetric matrices, and values ​​of the first mask matrix are related to character positions and arrangement directions in the first text data, and values ​​of the second mask matrix are related to character positions and arrangement directions in the second text data; splicing the first image, the first mask matrix, the second image, and the second mask matrix to obtain an input matrix; Processing the input matrix using a text recognition model to obtain the first recognition result and the second recognition result; In which, the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into a first recognition model to obtain the first recognition result, and input the second feature sequence into a second recognition model to obtain the second recognition result.

2. The method according to claim 1, wherein Generating a first mask matrix corresponding to the first image and generating a second mask matrix corresponding to the second image, including: Obtaining a position of each first character in the first image and a position of each second character in the second image; Based on the position of each first character, the first mask matrix corresponding to the first image is generated, and based on the position of each second character, the second mask matrix corresponding to the second image is generated.

3. The method according to claim 2, wherein: Generating a second mask matrix corresponding to the second image based on the position of each second character includes: Rotating the second image to obtain a rotated image, wherein the arrangement direction of the second text data in the rotated image is the first direction; generating a preset matrix based on the position of each second character; The preset matrix is ​​rotated to obtain the second mask matrix.

4. The method according to claim 3, wherein: The first mask matrix and the second mask matrix are asymmetric matrices; the values ​​of the preset matrix or the first mask matrix increase from left to right and from top to bottom.

5. The method according to claim 1, wherein The method further comprises: Acquire a plurality of training images, wherein the arrangement direction of the text data contained in each training image is the first direction or the second direction; The initial model is trained using the multiple training images to obtain the text recognition model.

6. The method according to claim 1, wherein The feature extraction model includes: multiple convolutional layers, excitation layers and pooling layers connected in sequence, wherein the parameter of the last pooling layer is 2*1; the first recognition model or the second recognition model includes: a convolutional neural network, a bidirectional long short-term memory neural network and a connectionist time classification.

7. An image recognition method, comprising: Extracting and processing an image to be processed that includes first text data and second text data to obtain a first image and a second image, wherein the first image includes the first text data, the second image includes the second text data, the first text data is arranged in a first direction, and the second text data is arranged in a second direction; Generate a first mask matrix corresponding to the first image, and generate a second mask matrix corresponding to the second image, wherein the first mask matrix is ​​used to represent the arrangement direction of the first text data, and the second mask matrix is ​​used to represent the arrangement direction of the second text data, the first mask matrix and the second mask matrix are asymmetric matrices, and values ​​of the first mask matrix are related to character positions and arrangement directions in the first text data, and values ​​of the second mask matrix are related to character positions and arrangement directions in the second text data; splicing the first image, the first mask matrix, the second image, and the second mask matrix to obtain an input matrix; Performing feature extraction on the input matrix to obtain the first feature sequence and the second feature sequence; Obtaining a first recognition result of the first image based on the first feature sequence; Based on the second feature sequence, a second recognition result of the second image is obtained.

8. The method according to claim 7, wherein: Performing feature extraction on the first image and the second image to obtain a first feature sequence of the first image and a second feature sequence of the second image includes: The first image and the second image are input into a shared feature extraction model to obtain the first feature sequence and the second feature sequence.

9. The method according to claim 7, wherein: Obtaining a first recognition result of the first image based on the first feature sequence includes: The first feature sequence is input into a first recognition model to obtain the first recognition result.

10. The method according to claim 7, wherein: Acquiring a second recognition result of the second image based on the second feature sequence includes: The second feature sequence is input into a second recognition model to obtain the second recognition result.

11. The method according to claim 7, wherein: Generating a first mask matrix corresponding to the first image and generating a second mask matrix corresponding to the second image, including: Obtaining a position of each first character in the first image and a position of each second character in the second image; Based on the position of each first character, a first mask matrix corresponding to the first image is generated, and based on the position of each second character, a second mask matrix corresponding to the second image is generated.

12. A storage medium comprising a stored program, wherein: When the program is running, the device where the storage medium is located is controlled to execute the image recognition method according to any one of claims 1 to 11.

13. A computing device comprising: A memory and a processor, wherein the memory is used to store a program, and the processor is used to run the program, wherein the image recognition method according to any one of claims 1 to 11 is executed when the program is run.

14. An image recognition system comprising: processor; as well as A memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: extracting and processing the image to be processed containing first text data and second text data to obtain a first image and a second image, wherein the first image contains the first text data, the second image contains the second text data, the arrangement direction of the first text data is a first direction, and the arrangement direction of the second text data is a second direction; generating a first mask matrix corresponding to the first image, and generating a second mask matrix corresponding to the second image, wherein the first mask matrix is ​​used to represent the arrangement direction of the first text data, the second mask matrix is ​​used to represent the arrangement direction of the second text data, the first mask matrix and the second mask matrix are asymmetric matrices, and the The values ​​of the first mask matrix are related to the character positions and arrangement directions in the first text data, and the values ​​of the second mask matrix are related to the character positions and arrangement directions in the second text data; the first image, the first mask matrix, the second image and the second mask matrix are spliced ​​to obtain an input matrix; the input matrix is ​​processed using a text recognition model to obtain the first recognition result and the second recognition result; wherein the text recognition model is used to input the first image and the second image into a feature extraction model to obtain a first feature sequence of the first image and a second feature sequence of the second image, input the first feature sequence into a first recognition model to obtain the first recognition result, and input the second feature sequence into a second recognition model to obtain the second recognition result.

Images