Image recognition method, device and electronic equipment

Abstract

The application discloses an image recognition method and device and electronic equipment. It relates to the field of artificial intelligence, and the method comprises the following steps: obtaining a target image to be recognized; performing hole convolution processing on a first image feature of the target image in a feature enhancement network in a target image recognition model to obtain a second image feature; determining a plurality of candidate regions according to the second image feature, and determining a confidence degree corresponding to each of the plurality of candidate regions through a prediction network in the target image recognition model; and determining a candidate region with a confidence degree greater than or equal to a confidence threshold as a target region. Through the application, the problem that the position region of text in an image cannot be accurately positioned in the related art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to an image recognition method and device and electronic equipment. BACKGROUND

[0002] With the continuous development of the Internet in China, the Internet financial industry is also gradually growing, and online payment has gradually become the main way of people's daily payment. Under normal circumstances, manually entering the bank card number not only takes time but also is easy to misinput the digits, especially for the elderly with low cultural level. To avoid complex operations, how to efficiently and correctly bind the bank card has become a technology that needs to be developed. At the same time, in some natural scenes, due to the folding, wear and tear of the bank card, and the problem of blurred font on the bank card, it still faces great challenges to quickly and accurately detect text in complex scenes. Therefore, how to accurately locate the position area of the text in the image has become a technical problem to be solved.

[0003] In view of the problem that the position area of the text in the image cannot be accurately located in the related art, no effective solution has been proposed so far. SUMMARY

[0004] The main purpose of the present application is to provide an image recognition method, device and electronic equipment to solve the problem that the position area of the text in the image cannot be accurately located in the related art.

[0005] In order to achieve the above purpose, according to one aspect of the present application, an image recognition method is provided. The method comprises: acquiring a target image to be recognized, wherein the area ratio between a target region in the target image and the image region of the target image is less than or equal to a target threshold, and the target region is an image region displaying a text object; performing dilated convolution processing on a first image feature of the target image in a feature enhancement network in a target image recognition model to obtain a second image feature, wherein the first image feature is an image feature extracted from the target image by a feature extraction network in the target image recognition model, and the receptive field corresponding to the second image feature is larger than the receptive field corresponding to the first image feature; determining a plurality of candidate regions according to the second image feature, and determining the confidence of each candidate region corresponding to the plurality of candidate regions through a prediction network in the target image recognition model; and determining the candidate region with a confidence greater than or equal to a confidence threshold as the target region.

[0006] To achieve the above object, according to another aspect of the present application, an image recognition device is provided. The device comprises: an acquisition unit configured to acquire a target image to be recognized, wherein an area ratio between a target region in the target image and an image region of the target image is less than or equal to a target threshold, and the target region is an image region in which a text object is displayed; a processing unit configured to perform a dilated convolution on a first image feature of the target image in a feature enhancement network in a target image recognition model to obtain a second image feature, wherein the first image feature is an image feature extracted from the target image by a feature extraction network in the target image recognition model, and a receptive field corresponding to the second image feature is larger than a receptive field corresponding to the first image feature; a first determination unit configured to determine a plurality of candidate regions according to the second image feature, and determine a confidence corresponding to each of the plurality of candidate regions by a prediction network in the target image recognition model; and a second determination unit configured to determine a candidate region with a confidence greater than or equal to a confidence threshold as the target region.

[0007] The processing unit comprises a sampling unit and an input unit. The sampling unit is configured to sample the first image feature according to the dilated convolution with the target sampling rate in the feature enhancement module to obtain the second image feature.

[0008] The sampling unit comprises an acquisition unit and a sampling subunit. The acquisition unit is configured to acquire a dilation coefficient sequence and a dilated coefficient sequence matched with the dilation coefficient sequence, wherein each dilation coefficient in the dilation coefficient sequence is used to indicate a number of intervals between convolution kernels, and the sampling subunit is configured to sequentially acquire one dilation coefficient in the dilation coefficient sequence and one dilated coefficient corresponding to the one dilation coefficient in the dilated coefficient sequence, and sample the first image feature according to the target sampling rate based on the dilation coefficient and the dilated coefficient to obtain N reference features, wherein the dilation coefficient sequence comprises N dilation coefficients, the dilated coefficient sequence comprises N dilated coefficients corresponding to the N dilation coefficients respectively, and N is an integer greater than or equal to 1.

[0009] The aforementioned first determining unit includes a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit. The first processing unit is used to perform feature fusion processing on the second image features according to the feature fusion network in the target image recognition model to obtain fused features. The second processing unit is used to process the fused features using a target activation function and a first convolutional kernel to obtain a first prediction equation matrix, wherein the first prediction equation matrix is ​​used to indicate the first probability value of a pixel in the target image being a pixel in the text image. The third processing unit is used to process the fused features using a target activation function and a second convolutional kernel to obtain a second prediction equation matrix, wherein the second prediction equation matrix is ​​used to indicate the second probability value of a candidate region in the target image being a target region. The fourth processing unit is used to process the fused features using a target activation function and a third convolutional kernel to obtain a third prediction equation matrix, wherein the third prediction equation matrix is ​​used to indicate the third probability value of the region tilt angle of the candidate region in the target image.

[0010] The aforementioned image recognition device further includes a preprocessing unit and a feature extraction unit. The preprocessing unit is used to perform image preprocessing on the target image to obtain a reference image before performing dilated convolution processing on the first image features of the target image in the feature enhancement network of the target image recognition model to obtain the second image features. The feature extraction unit is used to extract features from the reference image using a feature extraction network after obtaining the reference image to obtain the first image features. The feature extraction network includes a deep residual network.

[0011] The aforementioned preprocessing unit includes a first processing subunit, a second processing subunit, a third processing subunit, and a segmentation unit. The first processing subunit is used to perform grayscale and binarization processing on the target image to obtain a first image. The second processing subunit is used to perform image enhancement processing on the first image to obtain a second image, wherein the image enhancement processing includes at least one of the following: grayscale transformation processing, histogram correction processing, image smoothing processing, and image sharpening processing. The third processing subunit is used to perform noise reduction processing on the second image to obtain a third image. The segmentation unit is used to perform image region segmentation operation on the third image to obtain a reference image, wherein the reference image includes the target text object.

[0012] The image recognition device further includes a model training unit. The model training unit is configured to obtain a reference image recognition model in a training stage and a training sample image before obtaining a target image to be recognized. The reference image recognition model includes a reference feature extraction network, a reference feature enhancement network, a reference feature fusion network, and a reference region prediction network. The training sample image is input into the reference image recognition model to obtain a plurality of reference regions for recognizing the training sample image. The reference region is an image region in the training sample image with a confidence value greater than or equal to a confidence threshold. A current training loss of the reference image recognition model is determined according to the reference region. In a case where the current training loss is greater than or equal to a loss threshold, the model parameters in the image recognition model are adjusted, and the next training sample image is obtained to train the reference image recognition model. In a case where the current training loss is less than the loss threshold, the reference image recognition model is determined as a target image recognition model.

[0013] The training unit is configured to determine the current training loss of the reference image recognition model according to the reference region. The target loss weight and the annotation information of the training sample image are obtained. The annotation information includes the region position of the annotation region and the annotation tilt angle of the annotation region. A first loss value is determined according to the reference region position of the reference region and the annotation region position. A second loss value is determined according to the reference tilt angle of the reference region and the annotation tilt angle, and the target loss weight. The current training loss is determined according to the first loss value and the second loss value.

[0014] To achieve the above object, according to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is configured to execute the above image recognition method when running.

[0015] To achieve the above object, according to another aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium. The processor executes the computer instructions to make the computer device execute the above image recognition method.

[0016] To achieve the above object, according to another aspect of the present application, a processor is provided for running a program. The program is configured to execute the above image recognition method when running.

[0017] To achieve the above object, according to another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory for storing one or more programs, wherein when the one or more programs are run by the one or more processors, the one or more processors execute the image recognition method.

[0018] According to the present application, the following steps are adopted: obtaining a target image to be recognized; performing hole convolution processing on a first image feature of the target image in a feature enhancement network in a target image recognition model to obtain a second image feature; determining a plurality of candidate regions according to the second image feature, and determining a confidence corresponding to each of the plurality of candidate regions through a prediction network in the target image recognition model; and determining a candidate region with a confidence greater than or equal to a confidence threshold as a target region, thereby solving the problem that in the related art, the position of text in an image cannot be accurately positioned. In this way, the position of the text in the image can be quickly and accurately positioned. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and the illustrative embodiments of the present application and their description serve the purpose of explaining the present application. The accompanying drawings should not be construed as an inappropriate limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of a hardware environment of an image recognition method according to an embodiment of the present application;

[0021] Figure 2 is a flowchart of an image recognition method according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of an image recognition method according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of another image recognition method according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of another image recognition method according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of another image recognition method according to an embodiment of the present application;

[0026] Figure 7 is a flowchart of another image recognition method according to an embodiment of the present application;

[0027] Figure 8 is a schematic diagram of another image recognition method according to an embodiment of the present application;

[0028] Figure 9 is a flowchart of still another image recognition method according to an embodiment of the present application;

[0029] Figure 10 is a flowchart of still another image recognition method according to an embodiment of the present application;

[0030] Figure 11 is a schematic diagram of an image recognition device according to an embodiment of the present application;

[0031] Figure 12 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] The image recognition method, device and electronic device in the present application can be used in the field of artificial intelligence, and can also be used in any field other than the field of artificial intelligence which needs image recognition, and the application field of the image recognition method, device and electronic device in the present application is not limited.

[0034] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should belong to the protection scope of the present application.

[0035] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0036] In order to facilitate the description, the following describes some nouns or terms related to the embodiments of the present application:

[0037] EAST: A high-efficiency and accurate scene text detection algorithm;

[0038] FEM: Feature Enhancement Module.

[0039] It should be noted that all relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. The acquisition, storage, and processing of data involved in this application all comply with relevant regulations. For example, this system has an interface with relevant users or organizations. Before obtaining images and other related information, it is necessary to send an acquisition request to the aforementioned users or organizations through the interface. After receiving consent information from the aforementioned users or organizations, the images and other related information are obtained. At the same time, the text information on the images obtained in this application is all authorized by the users or organizations.

[0040] According to one aspect of the present invention, an image recognition method is provided. As an optional implementation, the image recognition method described above can be applied to, but is not limited to, [examples of other methods]. Figure 1 The image recognition system shown consists of terminal device 102, server 104, and network 110. For example... Figure 1 As shown, terminal device 102 communicates with server 104 via network 110. This network can include, but is not limited to, wired networks and wireless networks. The wired network includes local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). The wireless network includes Bluetooth, Wi-Fi, and other networks enabling wireless communication. The terminal device can include, but is not limited to, at least one of the following: mobile phones (such as Android phones, iOS phones, etc.), laptops, tablets, PDAs, MIDs (Mobile Internet Devices), PADs, desktop computers, smart TVs, in-vehicle devices, etc. Terminal device 102 can include, but is not limited to, a display, processor, and memory. Server 104 can be a single server, a server cluster consisting of multiple servers, or a cloud server. The server includes a database and a processing engine.

[0041] The specific process can be summarized in the following steps:

[0042] In step S102, the terminal device 102 sends the target image to the server 104;

[0043] In steps S104 to S110, server 104 acquires the target image to be identified, wherein the area ratio between the target region and the image region of the target image is less than or equal to a target threshold, and the target region is an image region displaying text objects; in the feature enhancement network of the target image recognition model, the first image feature of the target image is subjected to dilated convolution processing to obtain the second image feature, wherein the first image feature is the image feature extracted by the feature extraction network of the target image recognition model based on the target image, and the receptive field corresponding to the second image feature is larger than the receptive field corresponding to the first image feature; multiple candidate regions are determined based on the second image feature, and the confidence level corresponding to each of the multiple candidate regions is determined by the prediction network of the target image recognition model; the candidate regions with a confidence level greater than or equal to the confidence threshold are determined as the target regions.

[0044] remove Figure 1 Beyond the examples shown, the above steps can be completed independently by the client or server, or collaboratively by both, such as by the terminal device 102 executing steps S104 to S110, thereby reducing the processing load on the server 104. The terminal device 102 includes, but is not limited to, handheld devices (such as mobile phones), laptops, desktop computers, and in-vehicle devices. This application does not limit the specific implementation of the terminal device 102.

[0045] The present invention will now be described in conjunction with preferred implementation steps. Figure 2 This is a flowchart of a response method for an account operation request provided in an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0046] Step S202: Obtain the target image to be identified, wherein the area ratio between the target region and the image region of the target image is less than or equal to the target threshold, and the target region is the image region displaying text objects;

[0047] It should be noted that the technical solution in this application can be applied to image recognition scenarios where the receptive field of the text portion of the target image to be recognized is insufficient, such as the scenario of locating and recognizing text objects in a bank card image. The target threshold can be preset or derived based on information such as the target image size, the predicted target region size, and the positional relationship between the target image size and the predicted target region size. The specific method for determining the target threshold is not limited in this application.

[0048] Step S204, performing a hole convolution processing on the first image feature of the target image in a feature enhancement network in the target image recognition model to obtain a second image feature, wherein the first image feature is an image feature extracted from the target image by a feature extraction network in the target image recognition model, and a receptive field corresponding to the second image feature is larger than a receptive field corresponding to the first image feature;

[0049] It should be noted that the feature enhancement network can be a feature enhancement network capable of expanding the receptive field, such as a FEM feature enhancement network, an ASPP feature enhancement network, or a feature enhancement network obtained by combining one or more groups of the above feature enhancement networks; the receptive field (Receptive Field) is used to indicate the size of the region on the input image that is mapped back from the pixel points on the feature map output by each layer of the convolutional neural network, i.e., the size of the image region that can be captured by the operator of the layer; the hole convolution can be understood as injecting holes into the standard convolution layer to expand the receptive field. Compared with the conventional convolution method, the dilation coefficient is introduced in the hole convolution, and the default value of the dilation coefficient in the conventional convolution is 1. The dilation coefficient refers to the number of intervals between the convolution kernels, that is, the hole convolution controls the receptive field without changing the size of the feature map (i.e., the hole convolution can effectively expand the receptive field of the target convolution layer without introducing more parameters), thereby extracting multi-scale information.

[0050] Step S206, determining a plurality of candidate regions according to the second image feature, and determining a confidence degree corresponding to each of the plurality of candidate regions by a prediction network in the target image recognition model;

[0051] It should be noted that when the hole convolution processing is performed on the first image feature to obtain the second image feature, the dilation coefficient in the hole convolution can be set to a plurality of coefficients of different sizes, thereby obtaining a plurality of second image features of different sizes. The processing of the obtained plurality of second image features by the activation function and the convolution kernel of different sizes can determine the candidate region corresponding to each second image feature. The activation function can use a plurality of activation functions such as a sigmoid function, a Leaky ReLU function, a Maxout function, and a tanh function. The activation function can normalize and smooth the gradient of each neuron to avoid the output value jumping. The activation function is equivalent to a scoring function, and the output range is 0 to 1. The result of the prediction is normalized to 0 to 1, and the size of the score is used to represent the size of the possibility of the prediction. The closer the score is to 1, the greater the possibility is. The size of the convolution kernel can be various, for example Figure 3 (a) shows a 3x3 size convolution kernel, Figure 3 (b) shows a 7x size convolution kernel.

[0052] Step S208, determining the candidate region with the confidence greater than or equal to the confidence threshold as the target region.

[0053] The image recognition method provided by the embodiment of the present application comprises the following steps: obtaining a target image to be recognized, wherein an area ratio between a target region in the target image and an image region of the target image is less than or equal to a target threshold, and the target region is an image region in which a text object is displayed; performing a hole convolution processing on a first image feature of the target image in a feature enhancement network in a target image recognition model to obtain a second image feature, wherein the first image feature is an image feature extracted from the target image by a feature extraction network in the target image recognition model, and a receptive field corresponding to the second image feature is greater than a receptive field corresponding to the first image feature; determining a plurality of candidate regions according to the second image feature, and determining a confidence corresponding to each of the plurality of candidate regions by a prediction network in the target image recognition model; and determining a candidate region with the confidence greater than or equal to a confidence threshold as the target region. The problem that the position region of the text in the image cannot be accurately positioned in the related art is solved, the problem that the traditional recognition technology is greatly affected by the scene (for example, the bank card is folded, worn, and the recognition scene light is dark, and the lamp factor cannot achieve a high-accuracy recognition effect) is overcome, and the feature enhancement network is added, which can not only expand the receptive field but also improve the learning ability of the model. In turn, the technical effect that the position region of the text in the image is accurately positioned is achieved, and the text in the position region is quickly and accurately recognized based on the accurately positioned position region.

[0054] As an optional solution, the hole convolution processing on the first image feature of the target image in the feature enhancement network in the target image recognition model to obtain the second image feature comprises the following steps:

[0055] S1, in the feature enhancement module, the first image feature is sampled according to the target sampling rate of the hole convolution to obtain the second image feature.

[0056] It should be noted that the feature enhancement module can be either the FEM module or the ASPP module. The FEM module uses support set features and a prior mask to adaptively enhance query features to overcome spatial inconsistency. The FEM module considers using four different sizes of convolutions (1×1, 3×3, 5×5, and 7×7, with dilation coefficients of 1, 6, 12, and 18, respectively) to extract features at different scales. Each scale is an independent branch. The outputs of the four different scales are then concatenated using the Concat method, and the concatenated result is fed into a 1×1 filter convolution with 320 output channels. Finally, bilinear upsampling is used to obtain the required spatial dimension for concatenation. The ASPP module samples the input features at different sampling rates, i.e., extracts input features from different scales, and then fuses the obtained features to obtain the final feature extraction result.

[0057] In dilated convolution, the size of the receptive field is controlled by the dilation coefficient. The larger the dilation coefficient, the larger the receptive field. At the same time, dilated convolution can effectively resolve the contradiction between resolution and receptive field. The feature map generated by dilated convolution can obtain a larger receptive field with the same input size. The larger the receptive field value, the stronger the network's perception of the original image, the more global information of the mobile image it can obtain, and the deeper the meaning of the image can be extracted.

[0058] like Figure 3 As shown in (a), the convolution kernel = 3×3, the holes = 0, and the dilation coefficient = 1, as... Figure 3 As shown in (b), the dilation coefficient = 2, the hole = 1, and the kernel size = 3 × 3, that is... Figure 3 In (a), the 3×3 convolutional kernel is expanded to... Figure 3 (b) uses a 7×7 convolution kernel, but during computation, the weights are non-zero only at the black dots. Therefore, the convolution kernel at the black dots only convolves with the image. Figure 3 The position of the convolution in (b) is relative to Figure 3 The position of (a) convolution is more diffuse.

[0059] The FEM feature enhancement module was initially combined with dilated convolution, enabling it to sample the given input using dilated convolutions at different sampling rates, essentially capturing the image context at multiple scales. For example... Figure 10In the flowchart shown, the FEM module uses four different sizes of convolution (dilation coefficients are: 1, 6, 12, 18) to extract features of different scales, each scale is an independent branch, then the output results of the four different scales are combined together using Concat concatenation, and then the concatenated results are sent to a 1x1 filter convolution with an output channel of 320, finally, the required spatial dimension is obtained by bilinear upsampling for splicing. The network structure diagram of the FEM feature enhancement module is shown in Figure 4 .

[0060] By the above method of sampling the first image features according to the target sampling rate to obtain the second image features, an optimized model obtained by improving the EAST model is obtained, that is, the feature enhancement module is combined in the EAST model, and the given input is sampled by the dilated convolution with different sampling rates, which is equivalent to capturing the context of the image at multiple scales. The second image features obtained by sampling at different scales not only improve the accuracy of positioning the image regions in the image, but also further improve the accuracy and efficiency of subsequent text object recognition.

[0061] As an optional solution, the above sampling the first image features according to the target sampling rate to obtain the second image features in the above feature enhancement module comprises:

[0062] S1, obtaining a dilation coefficient sequence and a hole coefficient sequence matched with the dilation coefficient sequence, wherein each dilation coefficient in the dilation coefficient sequence is used to indicate the number of intervals between the convolution kernels;

[0063] It should be noted that the dilation coefficient sequence and the hole coefficient sequence can be obtained, or one of them can be obtained, and the dilation coefficient and the hole coefficient can be used to indicate the number of intervals between the convolution kernels, for example, Figure 3 As shown in (a), the dilation coefficient = 1, which can be understood as the distance between the center positions of adjacent convolution kernels being one unit cell, and the hole coefficient = 0, which can be understood as the number of complete unit cells between adjacent convolution kernels being 0; as Figure 3 (b) shows that the dilation coefficient = 2 and the hole coefficient = 1, and the dilation coefficient sequence includes one or more dilation coefficients of different sizes.

[0064] S2, sequentially obtaining one dilation coefficient in the dilation coefficient sequence and one hole coefficient corresponding to the dilation coefficient, and sampling the first image features according to the target sampling rate based on the dilation coefficient and the hole coefficient to obtain N second image features, wherein the dilation coefficient sequence includes N dilation coefficients, the hole coefficient sequence includes N hole coefficients corresponding to the N dilation coefficients respectively, and N is an integer greater than or equal to 1.

[0065] By obtaining multiple dilation coefficients of different sizes, multiple different receptive field second image features corresponding to different dilation coefficients can be obtained, so as to locate the position region of the text object according to the image features of different receptive fields. The contradiction between resolution and receptive field can be effectively solved by the empty hole convolution. The feature mapping generated by the empty hole convolution can have the same size as the input, and a larger receptive field can be obtained, thereby improving the accuracy of locating the position region and further improving the accuracy and efficiency of text recognition.

[0066] As an optional solution, the above-mentioned multiple candidate regions are determined according to the above-mentioned second image features, and the confidence of each candidate region corresponding to the above-mentioned target image recognition model is determined by a prediction network.

[0067] S1, the feature fusion network in the above-mentioned target recognition model is used to perform feature fusion processing on the above-mentioned second image features, and a fusion feature is obtained.

[0068] It should be noted that in the feature fusion stage, the size of the feature map does not change at each stage, and finally the feature fusion layer outputs a feature of, for example, 128x128x32 dimensions to the output layer.

[0069] S2, the fusion feature is processed by using a target activation function and a first convolution kernel to obtain a first prediction equation matrix, wherein the first prediction equation matrix is used to indicate a first probability value of a pixel point in the target image being a pixel point in the text image.

[0070] It should be noted that the feature output from the feature fusion layer is processed by a fixed-size convolution kernel (for example, 1x1) with a specified output channel value (for example, 1) and an activation function (for example, sigmoid), and a regression prediction equation matrix of a target dimension (i.e., 128x128) can be obtained. This regression prediction equation matrix can be used to determine whether a pixel point is text. An activation function is added after each convolution layer to ensure that each convolution can complete its own convolution task. The main points are as follows: the output range of the activation function is 0 to 1, which is equivalent to normalizing each neuron by the activation function and smoothing the gradient, thereby avoiding jumping output values. The activation function is equivalent to a scoring function that normalizes the prediction result to between 0 and 1. The size of the score obtained is used to represent the size of the prediction possibility. The closer the score is to 1, the greater the possibility.

[0071] S3, the fusion feature is processed by using the above-mentioned target activation function and a second convolution kernel to obtain a second prediction equation matrix, wherein the second prediction equation matrix is used to indicate a second probability value of the candidate region in the target image being the target region.

[0072] It should be noted that the output of the feature fusion layer is subjected to a convolution kernel (i.e., 1x1) with the same size as the above-mentioned size and an output channel value of another set value (e.g., 4), and a regression prediction equation matrix of 128x128x4 dimensions obtained by an activation function (e.g., sigmoid), which can be used to indicate the regression prediction of the text box position coordinates.

[0073] S4, processing the fusion feature by using the target activation function and the third convolution kernel to obtain a third prediction equation matrix, wherein the third prediction equation matrix is used to indicate a third probability value of a region tilt angle of the candidate region in the target image.

[0074] It should be noted that the third convolution kernel and the first convolution kernel can be convolution kernels with the same channel and dilation coefficient. For example, the output of the feature fusion layer is subjected to a 1x1 convolution kernel with an output channel of 1 and a sigmoid activation function to obtain a prediction value of the text box angle of 128x128 dimensions.

[0075] By the feature fusion network in the target recognition model, the second image feature is subjected to feature fusion processing to obtain a fusion feature; the fusion feature is processed by using a target activation function and a first convolution kernel to obtain a first prediction equation matrix; the fusion feature is processed by using the target activation function and a second convolution kernel to obtain a second prediction equation matrix; and the fusion feature is processed by using the target activation function and a third convolution kernel to obtain a third prediction equation matrix. Different processing results corresponding to the convolution kernels are obtained by using the convolution kernels and the target activation function to process the fusion feature, which can achieve the effect of accurately positioning the target region according to different convolution kernels.

[0076] As an optional solution, before the first image feature of the target image is subjected to the atrous convolution processing in the feature enhancement network in the target image recognition model to obtain the second image feature, the method further includes:

[0077] S1, performing image preprocessing on the target image to obtain a reference image;

[0078] S2, extracting a feature of the reference image by using a feature extraction network to obtain the first image feature, wherein the feature extraction network includes a deep residual network.

[0079] It should be noted that the feature extraction network described above can be a structure ResNet-50 in ResNet, or a ResNetXt improved on ResNet. The structure in ResNet calls a pre-trained model on ImageNet as Backbone, which can improve the performance of the model. ResNetXt combines the idea of Inception, which obtains features through a series of convolution operations with the same topology, and finally concatenates them together.

[0080] By the above pre-processing of the target image, the reference image is obtained, which can not only reduce the calculation cost, but also greatly improve the performance of the analysis result, and is helpful to suppress the information irrelevant to feature extraction. The feature extraction network is used to extract features from the reference image to obtain first image features. The feature extraction network includes a deep residual network. The introduction of the deep residual network can extract deeper features, thereby improving the image recognition efficiency and accuracy.

[0081] As an optional solution, the above image pre-processing of the target image to obtain the reference image includes:

[0082] S1, the target image is subjected to grayscale processing and binarization processing to obtain a first image.

[0083] It should be noted that in the RGB model, if R=G=B, then the color represents a gray color, wherein the value of R=G=B is called a gray value. Therefore, each pixel of a gray image only needs one byte to store the gray value (also known as intensity value or brightness value). In simple terms, grayscale is to convert a color image into a black and white (brightness) image. Compared with a color image, a gray image occupies less memory and operates faster. A gray image can increase contrast in vision and highlight the target area. The above binarization is to set the pixel gray value of the image to 0 or 255, i.e. only black and white visual effects. When the gray value is 255, it represents the brightest (pure white). When the gray value is 0, it represents the darkest (pure black). The threshold T can be set to divide the image data into a pixel group greater than T and a pixel group less than T. Binarization makes the image clearer and facilitates the extraction of target objects.

[0084] S2, the first image is subjected to image enhancement processing to obtain a second image, wherein the image enhancement processing includes at least one of the following: gray scale change processing, histogram correction processing, image smoothing processing, and image sharpening processing.

[0085] It should be noted that the above gray scale change processing is mainly for independent pixel points, and the gray scale change processing mode includes at least one of the following: image inversion, logarithmic transformation, power law transformation, piecewise linear transformation, etc., and the piecewise linear transformation includes at least one of the following: contrast stretching, gray level layering, bit plane layering; the contrast stretching can transform the gray scale dense area to a relatively wide range, and compress the gray scale range of the area of interest; the gray level layering can directly enhance the gray scale of the target area (target area), and does not change the gray scale of other areas (areas in the image except the target area); the bit plane layering: the picture is composed of each bit 0 / 1 of each pixel, which helps to save the space occupied by data.

[0086] The above histogram correction can include histogram equalization and histogram specification. The histogram equalization is an enhancement method of making correction to the histogram of the original image by performing a certain transformation on the original image to make the gray scale histogram of the original image a uniform histogram. The histogram specification is an enhancement method of making correction to the histogram by making the gray scale histogram of the original image a histogram of a specified shape. When performing the histogram specification, the original image is first processed by histogram equalization.

[0087] The above image smoothing processing is used to adjust the pixel points in the image that have a large difference in pixel value from the surrounding pixel points to a value close to the pixel value of the surrounding pixel points. The image is smoothed by image filtering, and according to different convolution kernels and convolution methods, the smoothing methods mainly include: mean filtering, box filtering, Gaussian filtering, median filtering, and bilateral filtering.

[0088] The above image sharpening processing can increase the contrast of the edges and linear targets on the image, that is, edge enhancement. The result of sharpening highlights the edges and contours, and linear target information. The image sharpening processing method includes at least one of the following: gradient method, Roberts gradient, Prewitt and Sobel gradient, Laplace algorithm, and directional detection.

[0089] S3, performing noise reduction processing on the second image to obtain a third image;

[0090] It should be noted that the above noise reduction processing includes at least one of the following: mean filtering algorithm, median filtering, Wiener filtering, etc.

[0091] S4, performing image region segmentation operation on the third image to obtain a reference image, wherein the reference image includes a target text object.

[0092] It should be noted that the above-mentioned manner of performing image region segmentation operation on the third image includes methods based on threshold, edge, region, graph theory, energy functional, wavelet analysis and genetic algorithm, and the number of image regions determined according to the image region segmentation operation can be equal to 1 or greater than 1. In the case where the number of determined image regions is equal to 1, the image region is an approximate location region of the target text object predicted by the model based on the original image third image. In the case where the number of determined image regions is greater than 1, the plurality of image regions determined are a plurality of location regions of a plurality of text objects in the third image.

[0093] By the above-mentioned grayscale processing and binarization processing on the target image, the first image is obtained; the image enhancement processing is performed on the first image to obtain the second image; the noise reduction processing is performed on the second image to obtain the third image; the image region segmentation operation is performed on the third image to obtain the reference image, which not only can eliminate irrelevant information in the image, enhance the detectability of relevant information, reduce the memory occupied by the image, and maximize the simplification of data, but also can improve the reliability of subsequent feature extraction, feature recognition and the like.

[0094] As an optional solution, before the target image to be recognized is obtained, the method further includes:

[0095] S1, obtaining a reference image recognition model in a training stage and a training sample image, wherein the reference image recognition model comprises a reference feature extraction network, a reference feature enhancement network, a reference feature fusion network and a reference region prediction network;

[0096] S2, inputting the training sample image into the reference image recognition model to obtain a plurality of reference regions for recognizing the training sample image, wherein the reference region is an image region in the training sample image with a confidence greater than or equal to a confidence threshold;

[0097] It should be noted that the training sample image can be a clear image, a blurred image after the target object is damaged or worn (for example, an image after a card is damaged), or an unclear image due to shooting angle or shooting delay.

[0098] S3, determining a current training loss of the reference image recognition model according to the reference region;

[0099] S4, in the case where the current training loss is greater than or equal to a loss threshold, adjusting the model parameters in the image recognition model, and obtaining a next training sample image to train the reference image recognition model;

[0100] S5, in the case where the current training loss is less than the loss threshold, determining the reference image recognition model as the target image recognition model.

[0101] The reference image recognition model in the training stage and the training sample image are obtained, the training sample image is input into the reference image recognition model, a plurality of reference regions for recognizing the training sample image are obtained, the current training loss of the reference image recognition model is determined according to the reference regions, in the case that the current training loss is greater than or equal to a loss threshold, the model parameters in the image recognition model are adjusted, the next training sample image is obtained to train the reference image recognition model, and in the case that the current training loss is less than the loss threshold, the reference image recognition model is determined as the target image recognition model. Not only the classification problem of difficult samples and the tedious training operation of a conventional neural network model in the early stage and the difficulty in obtaining a large number of effective bank card pictures in a short time are solved, but also the problems of large manpower, material resources and financial resources for obtaining samples are solved, the accuracy and performance of the target data model are improved, the parameters are as close as possible to the real model, and therefore the output result of the target image recognition model obtained by training can be more accurate in subsequent use.

[0102] As an optional solution, the current training loss of the reference image recognition model is determined according to the reference regions, including:

[0103] S1, obtaining a target loss weight and annotation information of a training sample image, wherein the annotation information includes a region position of an annotation region and an annotation tilt angle of the annotation region;

[0104] S2, determining a first loss value according to a reference region position of the reference region and the annotation region position;

[0105] S3, determining a second loss value according to a reference tilt angle of the reference region and the annotation tilt angle and the target loss weight;

[0106] S4, determining the current training loss according to the first loss value and the second loss value.

[0107] The target loss weight and the annotation information of the training sample image are obtained, the first loss value is determined according to the reference region position of the reference region and the annotation region position, the second loss value is determined according to the reference tilt angle of the reference region and the annotation tilt angle and the target loss weight, and the current training loss is determined according to the first loss value and the second loss value. The current training loss is determined by the first loss value and the second loss value, so that whether the obtained image recognition model meets the model use requirement can be determined according to the size of the determined current training loss, and the training loss determined in this way is more referential, so that the image recognition model obtained by training is more reliable.

[0108] The network structure of EAST can be roughly divided into three parts: the feature extraction backbone for feature extraction, the feature fusion branch that merges multiple feature maps extracted by the feature extraction backbone into a single feature map, and the output part that outputs the confidence score map and the geometric information feature map.

[0109] When training the reference recognition model as described above, the training labels for the EAST algorithm are divided into two parts: the confidence score of the text region in one channel and the information of the text box set in five channels. To reduce background interference and focus more on the features of the text region, the labeled region needs to be shrunk. To obtain the confidence labels, the image is first converted into a matrix of the same size, with all initial values ​​set to 0. Then, the labeled text region is shrunk in the following manner. The process of obtaining the confidence labels is illustrated in the diagram below. Figure 5 As shown, the specific process is as follows:

[0110] S1, for quadrilateral Q={p i |i∈{1,2,3,4}},p i ={x i y i} is the coordinates of the vertices of the quadrilateral, starting from the top left vertex and recording them one by one in a counter-clockwise direction;

[0111] S2, calculate the reference length r for each vertex. i r i＝min (D(p i ,p (imod4)+1 ),D(p i ,p ((i+3)mod4+1) ), where D(p i ,p j ) is p i and p j The Euclidean distance between them;

[0112] S3, the longer and shorter opposite sides are determined by comparing the average lengths of the two sets of opposite sides;

[0113] S4, first shrink the longer pair of opposite edges, then shrink the shorter pair of opposite edges, that is, for edge (p i ,p (imod4)+1 The two vertices of the symbol are moved inward by 0.3r. i and 0.3r (imod4)+1 .

[0114] For the shrunk region, set the value in its corresponding matrix to 1 to represent a positive sample, i.e., the text region; the rest of the 0s represent negative samples, i.e., the background region.

[0115] A schematic diagram illustrating the process of obtaining geometric information from a text box is shown below. Figure 6As shown, in order to make the text box geometric information label, five channels of matrix with the same size as the picture are needed, and the positive sample region contraction method is the same as when obtaining the confidence label. The remaining steps are as follows:

[0116] S1, calculate the minimum circumscribed rectangle of the obtained original annotation region;

[0117] S2, calculate and save the distance from the pixel points of the positive sample region to the four edges of the minimum circumscribed rectangle in the corresponding positions of the four channels respectively;

[0118] S3, calculate the angle between the tilt direction of the minimum circumscribed rectangle and the horizontal direction, and save the angle information in the matrix of the fifth channel.

[0119] Optionally, the above image recognition method can be applied to any image recognition scene, and can also be used in a scene of recognizing target text in an image. The following will be described in combination with Figures 7-10 the scene of applying the above embodiment to bank card number image recognition. The overall flowchart of the algorithm is as shown in Figure 7

[0120] S701, the process starts;

[0121] S702, first, the image preprocessing operation is performed, and the specific steps are as follows:

[0122] S703, input the processed image into the EAST model based on Resnet-50, and extract the text information of the bank card image through convolution operation, and the specific processing process is as shown in S703 of Figure 7 , which includes S703-1, image feature extraction is performed on the image to be recognized (such as Figure 9 S901 in, feature extraction (ResNet-50)) to obtain feature maps of multiple scales. When performing image feature extraction, the application introduces a deep residual network ResNet-50 in the EAST structure, as shown in Figure 8 The structure parameters of the ResNet-50 pre-training network are shown in the figure. In Conv 1, 7x7 is the size of the convolution kernel, 64 is the number of convolution kernels, the number of convolution kernels determines the dimension of the final output matrix, stride is the step length, which represents the step size of the convolution kernel moving each time, 256x256 is the output size, max pool in Conv2_x represents that the maximum pooling algorithm is used, and the purpose is to refine the large features with small features. The 1x1 in the square brackets represents that 64 convolution kernels with a size of 1x1 are used for convolution, 3x3 identifies that three residual blocks are executed for three times, and the final output result is 128x128. Figure 8The padding value is set to 1 after calculation, aiming to extract more edge information, and the meanings of the remaining parameters are the same as above. S703-2, image feature fusion is performed on the obtained multi-scale feature map (such as Figure 9 S902 in the above, feature fusion) to obtain a feature map, and the position information of the text box is predicted on the obtained feature map; then non-maximum suppression and text box fusion are performed, and finally S703-3, the predicted text box is output (such as Figure 9 S903 in the above, output layer), the 32-channel feature map obtained by the feature fusion branch is mapped to a 1-channel confidence score map and a 5-channel text box geometric information map, and the geometric information map is divided into a 4-channel axial bounding box and a 1-channel rotation angle.

[0123] Figure 9 The confidence score 901 output by the output layer in the above is used to indicate the accuracy of the image recognition model, the text box 902 is used to output the position information of the target region determined by the reference image recognition model each time the training is performed, the text box rotation angle 903 is used to indicate the inclination angle of the target region (which can be based on 0 degrees, 90 degrees or 180 degrees, which is not limited in the present application), and the text box set information 904 is used to indicate the size of the text object in the target region.

[0124] The image processing flowchart in the improved EAST network structure is shown in the above Figure 9 The data processing flowchart in the FEM feature enhancement module is shown in the above Figure 10 The feature enhancement module considers using 1x1, 3x3, 5x5 and 7x7 convolution kernels of four different sizes to extract features of different scales, and then fusing the extracted features of multiple scales, which can effectively increase the width of the network and also increase the receptive field of the network, and enhance the feature extraction ability of the feature extraction backbone to different size text regions. However, 5x5 and 7x7 convolution kernels introduce more parameters, in order to reduce the parameters, the 3x3, 5x5 and 7x7 convolutions are split, for example, the 7x7 convolution can be split into a combination of 1x7 convolution and 7x1 convolution. Moreover, the shape of 1xn convolution is more similar to the shape of the text region, which is beneficial to extract the features of the text region and reduce the interference of the background.

[0125] S704, calculate the optimized loss function, and continue to train the algorithm model to make the detection result optimal.

[0126] The loss function is used for further optimization of the network model, and the smaller the function value obtained is, the higher the performance accuracy of the model prediction is. By using the improved loss function in the application, the text information in the to-be-identified image can be identified more quickly and accurately in different scenes. For the positioning of the digital region, since the bank card number image has uniqueness, the area occupied by the digital region is small, the background occupies a large proportion and is coincided with the digital region, therefore, the total loss function formula of the algorithm in the application is as follows: L = L s + λ g L g L = L s + λ g L g

[0127] Wherein, the total loss function L represents the classification loss of the segmented image background and the image text, the part of the target text object is marked as 1, and the part of the non-target text object is marked as 0, that is, the classification loss of the pixel point, L g is the loss of the geometric graph. λ g is the loss weight, and the value is 1.

[0128] In the training process, balanced cross-entropy is used to calculate the score channel loss to simplify the training process, and the formula is as follows:

[0129]

[0130] In the above formula, represents the predicted score value, Y * represents the label value of the sample, and the parameter β is the balance factor between positive and negative samples, and the formula is as follows:

[0131] L g The geometric graph loss is divided into two parts, the regression loss and the rotation angle loss, and the formulas are as follows:

[0132]

[0133] Wherein, L AABB is the regression loss, is the rotation angle loss, is the prediction of the rotation angle (predicted text inclination angle), and θ * represents the label value (real inclination angle of the text rectangle frame). AABB represents the distance from the pixel position to the upper, lower, left and right boundaries of the text rectangle, L g is the weighted sum of the regression loss and the rotation angle loss (collectively referred to as the geometric loss), and the calculation formula is as follows: L g = L AABB + λ θ L θ .

[0134] The algorithm in the application adopts a pre-trained network structure (as shown in the table below) Figure 6 The structure parameters of the RestNet-50 pre-training network are shown in the table below), because the parameters are relatively stable and have good similarity, and deeper image features can be extracted to achieve higher recognition accuracy, therefore, the ResNet-50 feature extraction network is adopted, and widening and deepening the network is the main way to improve the network effect, and the improved EAST algorithm network structure is as shown in the table below Figure 7 Figure 7 Unpool in the table below is a specific method of up-sampling, and the bilinear difference method is adopted in this paper.

[0135] Through the above embodiments of the application, the problems that the neural network model needs a large amount of pre-training, the bank card picture is a relatively private personal privacy, it is difficult to obtain a large number of effective bank card pictures in a short time, and a large amount of manpower, material resources and financial resources are needed to obtain samples can be overcome, the technical problems that the traditional recognition technology is greatly affected by the scene, for example, the bank card is folded, worn, the recognition scene light is dark, and the lamp factor, and high-accuracy recognition effect cannot be achieved can be solved, the problems that the existing EAST algorithm has insufficient positioning of the bank card mark in the natural scene can be overcome, the problems that the existing EAST algorithm has insufficient receptive field for sparse and long text dense text parts and uneven positive and negative samples can be solved, the deep residual network introduced can extract deeper features, the FEM combined with the hollow convolution can expand the receptive field, the problems of insufficient receptive field in the related art can be solved, the improved loss function solves the classification problem of difficult samples, and the entire technical solution can accurately locate the position area of the text object in the image, and the accuracy of locating the position area of the text object in the image is greatly improved.

[0136] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0137] The embodiment of the application further provides an image recognition device, and it should be noted that the image recognition device of the embodiment of the application can be used to execute the image recognition method provided by the embodiment of the application. The image recognition device provided by the embodiment of the application is introduced as follows.

[0138] Figure 11 is a schematic diagram of the image recognition device according to the embodiment of the application. As shown in Figure 11 the device comprises:

[0139] ​The acquisition unit 1102 is configured to acquire a target image to be recognized, wherein an area ratio between a target region in the target image and an image region of the target image is less than or equal to a target threshold, and the target region is an image region in which a text object is displayed.

[0140] The processing unit 1104 is configured to perform a hole convolution process on a first image feature of the target image in a feature enhancement network in a target image recognition model to obtain a second image feature, wherein the first image feature is an image feature extracted from the target image by a feature extraction network in the target image recognition model, and a receptive field corresponding to the second image feature is larger than a receptive field corresponding to the first image feature.

[0141] The first determination unit 1106 is configured to determine a plurality of candidate regions according to the second image feature, and determine a confidence corresponding to each of the plurality of candidate regions by a prediction network in the target image recognition model.

[0142] The second determination unit 1108 is configured to determine the target region as the candidate region whose confidence is greater than or equal to a confidence threshold.

[0143] Optionally, the processing unit includes a sampling unit and an input unit, the sampling unit is configured to sample the first image feature according to a target sampling rate in the feature enhancement module to obtain the second image feature.

[0144] Optionally, the sampling unit includes an acquisition unit and a sampling subunit, the acquisition unit is configured to acquire a dilation coefficient sequence and a hole coefficient sequence matched with the dilation coefficient sequence, wherein each dilation coefficient in the dilation coefficient sequence is used to indicate a number of intervals between convolution kernels, and the sampling subunit is configured to sequentially acquire one dilation coefficient and one hole coefficient corresponding to the dilation coefficient in the dilation coefficient sequence and the hole coefficient sequence, and sample the first image feature according to the target sampling rate based on the dilation coefficient and the hole coefficient to obtain N reference features, wherein the dilation coefficient sequence includes N dilation coefficients, the hole coefficient sequence includes N hole coefficients corresponding to the N dilation coefficients respectively, and N is an integer greater than or equal to 1.

[0145] Optionally, the first determining unit comprises a first processing unit, a second processing unit, a third processing unit and a fourth processing unit. The first processing unit is configured to perform feature fusion processing on the second image features according to a feature fusion network in the target image recognition model to obtain fused features. The second processing unit is configured to process the fused features by using a target activation function and a first convolution kernel to obtain a first prediction equation matrix, wherein the first prediction equation matrix is used to indicate a first probability value of a pixel point in the target image being a pixel point in the text image. The third processing unit is configured to process the fused features by using a target activation function and a second convolution kernel to obtain a second prediction equation matrix, wherein the second prediction equation matrix is used to indicate a second probability value of a candidate region in the target image being a target region. The fourth processing unit is configured to process the fused features by using a target activation function and a third convolution kernel to obtain a third prediction equation matrix, wherein the third prediction equation matrix is used to indicate a third probability value of a region tilt angle of the candidate region in the target image.

[0146] Optionally, the image recognition device further comprises a preprocessing unit and a feature extraction unit. The preprocessing unit is configured to perform image preprocessing on the target image to obtain a reference image before performing the hole convolution processing on the first image features of the target image in the feature enhancement network in the target image recognition model to obtain the second image features. The feature extraction unit is configured to perform feature extraction on the reference image by using a feature extraction network to obtain the first image features after obtaining the reference image, wherein the feature extraction network comprises a deep residual network.

[0147] Optionally, the preprocessing unit comprises a first processing subunit, a second processing subunit, a third processing subunit and a segmentation unit. The first processing subunit is configured to perform grayscale processing and binaryzation processing on the target image to obtain a first image. The second processing subunit is configured to perform image enhancement processing on the first image to obtain a second image, wherein the image enhancement processing comprises at least one of the following: grayscale change processing, histogram correction processing, image smoothing processing and image sharpening processing. The third processing subunit is configured to perform noise reduction processing on the second image to obtain a third image. The segmentation unit is configured to perform image region segmentation operation on the third image to obtain the reference image, wherein the reference image comprises a target text object.

[0148] Optionally, the image recognition device further comprises a model training unit, configured to obtain a reference image recognition model in a training stage and a training sample image before obtaining the target image to be recognized, wherein the reference image recognition model comprises a reference feature extraction network, a reference feature enhancement network, a reference feature fusion network and a reference region prediction network; the training sample image is input into the reference image recognition model to obtain a plurality of reference regions for recognizing the training sample image, wherein the reference region is an image region in the training sample image with a confidence greater than or equal to a confidence threshold; a current training loss of the reference image recognition model is determined according to the reference region; in a case where the current training loss is greater than or equal to a loss threshold, the model parameters in the image recognition model are adjusted, and the next training sample image is obtained to train the reference image recognition model; in a case where the current training loss is less than the loss threshold, the reference image recognition model is determined as the target image recognition model.

[0149] Optionally, the training unit is configured to determine the current training loss of the reference image recognition model according to the reference region, comprising: obtaining a target loss weight and annotation information of the training sample image, wherein the annotation information comprises a region position of an annotation region and an annotation tilt angle of the annotation region; determining a first loss value according to a reference region position of the reference region and the annotation region position; determining a second loss value according to a reference tilt angle of the reference region and the annotation tilt angle, and the target loss weight; and determining the current training loss according to the first loss value and the second loss value.

[0150] Optionally, in the embodiment, the embodiments to be implemented by each unit module can refer to the above-mentioned method embodiments, which will not be described here.

[0151] The image recognition device provided by the embodiment comprises an obtaining unit, configured to obtain a target image to be recognized, wherein an area ratio between a target region in the target image and an image region of the target image is less than or equal to a target threshold, and the target region is an image region displaying a text object; a processing unit, configured to perform a hole convolution processing on a first image feature of the target image in a feature enhancement network in a target image recognition model to obtain a second image feature, wherein the first image feature is an image feature extracted from the target image by a feature extraction network in the target image recognition model, and a receptive field corresponding to the second image feature is greater than a receptive field corresponding to the first image feature; a first determining unit, configured to determine a plurality of candidate regions according to the second image feature, and determine a confidence corresponding to each of the plurality of candidate regions by a prediction network in the target image recognition model; and a second determining unit, configured to determine a candidate region with a confidence greater than or equal to a confidence threshold as the target region, thereby solving the problem that the position region of the text in the image cannot be accurately positioned in the related art, and further achieving the effect of quickly and accurately positioning the position region of the text in the image.

[0152] The image recognition device includes a processor and a memory, and the acquisition unit, the processing unit, the first determination unit, the second determination unit, and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0153] The processor includes a core, and the core calls the corresponding program units in the memory. One or more cores can be provided, and the position area of the text in the image is quickly and accurately positioned by adjusting the core parameters.

[0154] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0155] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the image recognition method.

[0156] The embodiment of the present application provides a processor, and the processor is used to run a program, and the program is executed to realize the image recognition method.

[0157] As shown in Figure 12 , the embodiment of the present application provides an electronic device for implementing the image recognition method, and the electronic device can be a terminal device or a server as shown in Figure 12 . This embodiment takes the electronic device as an example for description. As shown in Figure 12 , the electronic device includes a memory 1202, a processor 1204, and a program stored in the memory 1202 and capable of running on the processor 1204, and the processor 1204 realizes the following steps when executing the program.

[0158] An image to be recognized is acquired, wherein an area ratio between a target region in the image and an image region of the image is less than or equal to a target threshold, the target region is an image region in which a text object is displayed; a first image feature of the image is subjected to a hole convolution processing in a feature enhancement network in a target image recognition model, to obtain a second image feature, wherein the first image feature is an image feature extracted from the image by a feature extraction network in the target image recognition model, and a receptive field corresponding to the second image feature is larger than a receptive field corresponding to the first image feature; a plurality of candidate regions are determined according to the second image feature, and a confidence degree corresponding to each of the plurality of candidate regions is determined by a prediction network in the target image recognition model; and a candidate region with a confidence degree greater than or equal to a confidence threshold is determined as the target region.

[0159] The first image feature of the target image is subjected to a hole convolution processing in a feature enhancement module in the target image recognition model to obtain a second image feature, comprising: in the feature enhancement module, the first image feature is sampled according to a target sampling rate of hole convolution to obtain the second image feature;

[0160] In the feature enhancement module, the first image feature is sampled according to a target sampling rate of hole convolution to obtain the second image feature, comprising: obtaining a dilation coefficient sequence and a hole coefficient sequence matched with the dilation coefficient sequence, wherein each dilation coefficient in the dilation coefficient sequence is used to indicate the interval number between convolution kernels; one dilation coefficient in the dilation coefficient sequence and one hole coefficient corresponding to the dilation coefficient are obtained in sequence, and the first image feature is sampled according to the target sampling rate based on the dilation coefficient and the hole coefficient to obtain N second image features, wherein the dilation coefficient sequence includes N dilation coefficients, the hole coefficient sequence includes N hole coefficients corresponding to the N dilation coefficients respectively, and N is an integer greater than or equal to 1;

[0161] According to the second image feature, a plurality of candidate regions are determined, and the confidence of each candidate region is determined through a prediction network in the target image recognition model, comprising: according to the feature fusion network in the target image recognition model, the second image feature is subjected to feature fusion processing to obtain a fusion feature; the fusion feature is processed by using a target activation function and a first convolution kernel to obtain a first prediction equation matrix, wherein the first prediction equation matrix is used to indicate a first probability value of a pixel point in the target image being a pixel point in a text image; the fusion feature is processed by using a target activation function and a second convolution kernel to obtain a second prediction equation matrix, wherein the second prediction equation matrix is used to indicate a second probability value of a candidate region in the target image being a target region; the fusion feature is processed by using a target activation function and a third convolution kernel to obtain a third prediction equation matrix, wherein the third prediction equation matrix is used to indicate a third probability value of a region tilt angle of the candidate region in the target image;

[0162] Before the first image feature of the target image is subjected to hole convolution processing in the feature enhancement network in the target image recognition model to obtain the second image feature, it further comprises: performing image preprocessing on the target image to obtain a reference image; the reference image is subjected to feature extraction by using a feature extraction network to obtain the first image feature, wherein the feature extraction network comprises a deep residual network;

[0163] The target image is preprocessed to obtain a reference image, including: performing grayscale processing and binarization processing on the target image to obtain a first image; performing image enhancement processing on the first image to obtain a second image, wherein the image enhancement processing includes at least one of the following: grayscale change processing, histogram correction processing, image smoothing processing, and image sharpening processing; performing noise reduction processing on the second image to obtain a third image; performing image region segmentation operation on the third image to obtain the reference image, wherein the reference image includes a target text object;

[0164] Before obtaining the target image to be recognized, the method further includes: obtaining a reference image recognition model in a training stage and a training sample image, wherein the reference image recognition model includes a reference feature extraction network, a reference feature enhancement network, a reference feature fusion network, and a reference region prediction network; inputting the training sample image into the reference image recognition model to obtain a plurality of reference regions for recognizing the training sample image, wherein the reference region is an image region in the training sample image with a confidence value greater than or equal to a confidence threshold; determining a current training loss of the reference image recognition model according to the reference region; in a case where the current training loss is greater than or equal to a loss threshold, adjusting model parameters in the image recognition model, and obtaining a next training sample image to train the reference image recognition model; in a case where the current training loss is less than the loss threshold, determining the reference image recognition model as the target image recognition model;

[0165] Determining the current training loss of the reference image recognition model according to the reference region includes: obtaining a target loss weight and annotation information of the training sample image, wherein the annotation information includes a region position of an annotation region and an annotation tilt angle of the annotation region; determining a first loss value according to a reference region position of the reference region and the annotation region position; determining a second loss value according to a reference tilt angle of the reference region and the annotation tilt angle, and the target loss weight; and determining the current training loss according to the first loss value and the second loss value. The device in this document can be a server, a PC, a PAD, a mobile phone, etc.

[0166] The present application also provides a computer program product adapted to execute a program of the following method steps when executed on a data processing device:

[0167] Obtaining a target image to be recognized, wherein an area ratio between a target region in the target image and an image region of the target image is less than or equal to a target threshold, and the target region is an image region in which a text object is displayed; performing a hole convolution process on a first image feature of the target image in a feature enhancement network in a target image recognition model to obtain a second image feature, wherein the first image feature is an image feature extracted from the target image by a feature extraction network in the target image recognition model, and a receptive field corresponding to the second image feature is larger than a receptive field corresponding to the first image feature; determining a plurality of candidate regions according to the second image feature, and determining a confidence corresponding to each of the plurality of candidate regions by a prediction network in the target image recognition model; and determining a candidate region with a confidence greater than or equal to a confidence threshold as the target region.

[0168] Performing a hole convolution process on a first image feature of the target image in a feature enhancement module in the target image recognition model to obtain a second image feature, comprising: sampling the first image feature according to a target sampling rate of hole convolution in the feature enhancement module to obtain the second image feature.

[0169] Sampling the first image feature according to a target sampling rate of hole convolution in the feature enhancement module to obtain the second image feature, comprising: obtaining a dilation coefficient sequence and a hole coefficient sequence matched with the dilation coefficient sequence, wherein each dilation coefficient in the dilation coefficient sequence is used to indicate a number of intervals between convolution kernels; sequentially obtaining a dilation coefficient and a hole coefficient corresponding to the dilation coefficient in the dilation coefficient sequence and the hole coefficient sequence, and sampling the first image feature according to the target sampling rate based on the dilation coefficient and the hole coefficient to obtain N second image features, wherein the dilation coefficient sequence includes N dilation coefficients, the hole coefficient sequence includes N hole coefficients corresponding to the N dilation coefficients respectively, and N is an integer greater than or equal to 1.

[0170] Determining a plurality of candidate regions according to the second image feature, and determining a confidence corresponding to each of the plurality of candidate regions by a prediction network in the target image recognition model, comprising: performing feature fusion processing on the second image feature according to a feature fusion network in the target image recognition model to obtain a fusion feature; processing the fusion feature by using a target activation function and a first convolution kernel to obtain a first prediction equation matrix, wherein the first prediction equation matrix is used to indicate a first probability value of a pixel point in the target image being a pixel point in a text image; processing the fusion feature by using a target activation function and a second convolution kernel to obtain a second prediction equation matrix, wherein the second prediction equation matrix is used to indicate a second probability value of a candidate region in the target image being the target region; and processing the fusion feature by using a target activation function and a third convolution kernel to obtain a third prediction equation matrix, wherein the third prediction equation matrix is used to indicate a third probability value of a region tilt angle of the candidate region in the target image.

[0171] Before the first image feature of the target image is processed by the hole convolution in the feature enhancement network in the target image recognition model to obtain the second image feature, the method further includes: performing image preprocessing on the target image to obtain a reference image; and extracting a feature of the reference image by using a feature extraction network to obtain the first image feature, wherein the feature extraction network includes a deep residual network.

[0172] The image preprocessing on the target image to obtain the reference image includes: performing grayscale processing and binaryzation processing on the target image to obtain a first image; performing image enhancement processing on the first image to obtain a second image, wherein the image enhancement processing includes at least one of the following: grayscale change processing, histogram correction processing, image smoothing processing, and image sharpening processing; performing noise reduction processing on the second image to obtain a third image; and performing image region segmentation operation on the third image to obtain the reference image, wherein the reference image includes a target text object.

[0173] Before the target image to be recognized is obtained, the method further includes: obtaining a reference image recognition model in a training stage and a training sample image, wherein the reference image recognition model includes a reference feature extraction network, a reference feature enhancement network, a reference feature fusion network, and a reference region prediction network; inputting the training sample image into the reference image recognition model to obtain a plurality of reference regions for recognizing the training sample image, wherein the reference region is an image region in the training sample image with a confidence value greater than or equal to a confidence threshold; determining a current training loss of the reference image recognition model according to the reference region; in a case where the current training loss is greater than or equal to a loss threshold, adjusting a model parameter in the image recognition model, and obtaining a next training sample image to train the reference image recognition model; and in a case where the current training loss is less than the loss threshold, determining the reference image recognition model as the target image recognition model.

[0174] The current training loss of the reference image recognition model is determined according to the reference region, which includes: obtaining a target loss weight and annotation information of the training sample image, wherein the annotation information includes a region position of an annotation region and an annotation tilt angle of the annotation region; determining a first loss value according to a reference region position of the reference region and the annotation region position; determining a second loss value according to a reference tilt angle of the reference region and the annotation tilt angle, and the target loss weight; and determining the current training loss according to the first loss value and the second loss value.

[0175] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0176] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0177] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0178] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0179] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0180] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or volatile / non-volatile random access memory (RAM), among others. The memory is an example of computer readable media.

[0181] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0182] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but also other elements not explicitly listed, or other elements inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0183] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0184] The above merely provides embodiments of the present application and is not intended to limit the present application. Various modifications and changes can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. An image recognition method, characterized in that, include: Obtain a target image to be identified, wherein the area ratio between the target region and the image region of the target image is less than or equal to a target threshold, and the target region is an image region displaying text objects; In the feature enhancement network of the target image recognition model, the first image feature of the target image is subjected to dilated convolution to obtain the second image feature. The first image feature is the image feature extracted by the feature extraction network of the target image recognition model based on the target image. The receptive field of the second image feature is larger than the receptive field of the first image feature. Multiple candidate regions are determined based on the second image features, and the confidence level of each of the multiple candidate regions is determined by the prediction network in the target image recognition model. The candidate regions with a confidence level greater than or equal to the confidence threshold are determined as the target regions; The step of determining multiple candidate regions based on the second image features and determining the confidence level of each candidate region through the prediction network in the target image recognition model includes: performing feature fusion processing on the second image features according to the feature fusion network in the target image recognition model to obtain fused features; processing the fused features using a target activation function and a first convolutional kernel to obtain a first prediction equation matrix, wherein the first prediction equation matrix is ​​used to indicate a first probability value that a pixel in the target image is a pixel in the text image; processing the fused features using the target activation function and a second convolutional kernel to obtain a second prediction equation matrix, wherein the second prediction equation matrix is ​​used to indicate a second probability value that the candidate region in the target image is the target region; and processing the fused features using the target activation function and a third convolutional kernel to obtain a third prediction equation matrix, wherein the third prediction equation matrix is ​​used to indicate a third probability value that the region tilt angle of the candidate region in the target image is the third probability value. Before acquiring the target image to be identified, the process further includes: acquiring a reference image recognition model and training sample images in the training phase, wherein the reference image recognition model includes a reference feature extraction network, a reference feature enhancement network, a reference feature fusion network, and a reference region prediction network; inputting the training sample images into the reference image recognition model to obtain multiple reference regions for recognizing the training sample images, wherein the reference regions are image regions in the training sample images with a confidence level greater than or equal to a confidence threshold; determining the current training loss of the reference image recognition model based on the reference regions; if the current training loss is greater than or equal to a loss threshold, adjusting the model parameters in the image recognition model and acquiring the next training sample image to train the reference image recognition model; if the current training loss is less than the loss threshold, determining the reference image recognition model as the target image recognition model.

2. The method according to claim 1, characterized in that, The second image feature is obtained by performing dilated convolution processing on the first image feature of the target image in the feature enhancement module of the target image recognition model, including: In the feature enhancement module, the first image features are sampled according to the dilated convolution at the target sampling rate to obtain the second image features.

3. The method according to claim 2, characterized in that, In the feature enhancement module, the first image features are sampled according to a dilated convolution at a target sampling rate to obtain the second image features, including: Obtain a sequence of dilation coefficients and a sequence of dilation coefficients that match the sequence of dilation coefficients, wherein each dilation coefficient in the sequence of dilation coefficients is used to indicate the number of intervals between convolutional kernels; The expansion coefficient sequence and the hole coefficient sequence are sequentially obtained, along with a corresponding hole coefficient. The first image feature is then sampled based on the expansion coefficient and the hole coefficient at the target sampling rate to obtain N second image features. The expansion coefficient sequence includes N expansion coefficients, and the hole coefficient sequence includes N hole coefficients corresponding to the N expansion coefficients, where N is an integer greater than or equal to 1.

4. The method according to claim 1, characterized in that, Before performing dilated convolution processing on the first image features of the target image in the feature enhancement network of the target image recognition model to obtain the second image features, the method further includes: The target image is preprocessed to obtain a reference image; The reference image is used to extract features to obtain the first image features, wherein the feature extraction network includes a deep residual network.

5. The method according to claim 4, characterized in that, The step of preprocessing the target image to obtain the reference image includes: The target image is subjected to grayscale and binarization processing to obtain a first image; The first image is subjected to image enhancement processing to obtain a second image, wherein the image enhancement processing includes at least one of the following: grayscale change processing, histogram correction processing, image smoothing processing, and image sharpening processing; The second image is subjected to noise reduction processing to obtain the third image; The third image is segmented to obtain the reference image, wherein the reference image includes the target text object.

6. The method according to claim 1, characterized in that, Determining the current training loss of the reference image recognition model based on the reference region includes: Obtain the target loss weights and the annotation information of the training sample images, wherein the annotation information includes the region location of the annotation region and the annotation tilt angle of the annotation region; The first loss value is determined based on the reference area position and the labeled area position of the reference area; The second loss value is determined based on the reference tilt angle of the reference area and the labeled tilt angle, as well as the target loss weight; The current training loss is determined based on the first loss value and the second loss value.

7. An image recognition device, characterized in that, include: An acquisition unit is used to acquire a target image to be identified, wherein the area ratio between the target region and the image region of the target image is less than or equal to a target threshold, and the target region is an image region displaying text objects; The processing unit is configured to perform dilated convolution processing on the first image feature of the target image in the feature enhancement network of the target image recognition model to obtain the second image feature, wherein the first image feature is the image feature extracted by the feature extraction network of the target image recognition model based on the target image, and the receptive field corresponding to the second image feature is larger than the receptive field corresponding to the first image feature. The first determining unit is used to determine multiple candidate regions based on the second image features, and to determine the confidence level of each of the multiple candidate regions through the prediction network in the target image recognition model. The second determining unit is used to determine the candidate regions with confidence levels greater than or equal to the confidence threshold as the target regions; The first determining unit is configured to: perform feature fusion processing on the second image features according to the feature fusion network in the target image recognition model to obtain fused features; process the fused features using a target activation function and a first convolutional kernel to obtain a first prediction equation matrix, wherein the first prediction equation matrix is ​​used to indicate a first probability value that a pixel in the target image is a pixel in the text image; process the fused features using the target activation function and a second convolutional kernel to obtain a second prediction equation matrix, wherein the second prediction equation matrix is ​​used to indicate a second probability value that the candidate region in the target image is the target region; and process the fused features using the target activation function and a third convolutional kernel to obtain a third prediction equation matrix, wherein the third prediction equation matrix is ​​used to indicate a third probability value that the region tilt angle of the candidate region in the target image is... The image recognition device is further configured to: acquire a reference image recognition model and training sample images in the training phase, wherein the reference image recognition model includes a reference feature extraction network, a reference feature enhancement network, a reference feature fusion network, and a reference region prediction network; input the training sample images into the reference image recognition model to obtain multiple reference regions for recognizing the training sample images, wherein the reference regions are image regions in the training sample images with a confidence level greater than or equal to a confidence threshold; determine the current training loss of the reference image recognition model based on the reference regions; if the current training loss is greater than or equal to a loss threshold, adjust the model parameters in the image recognition model and acquire the next training sample image to train the reference image recognition model; if the current training loss is less than the loss threshold, determine the reference image recognition model as the target image recognition model.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 6 when it runs.

9. An electronic device, characterized in that, It includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 6.

Images