Image processing method and device

Abstract

The invention discloses an image processing method and device. According to the specific scheme, at least two to-be-used nameplate images of a to-be-processed nameplate image under at least two resolutions are determined; based on the at least two to-be-used nameplate images, a to-be-applied nameplate image corresponding to the to-be-processed nameplate image is determined, and the to-be-applied nameplate image comprises at least one pixel point including the confidence coefficient of characters and a textbox area; the method comprises the steps of performing direction correction processing on characters in a textbox region in a to-be-applied nameplate image to obtain a target nameplate image; and determining target text information of the to-be-processed nameplate image based on the target nameplate image. According to the invention, the recognition accuracy of the text information in the nameplate image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method and device. Background Art

[0002] During the substation inspection process, it is usually necessary to read the text information of all nameplates in the substation to deal with various problems that may occur in the substation.

[0003] In the prior art, one way to identify the text information in the nameplate is to read the text information on the nameplate manually. However, it is difficult to ensure that the text information in the nameplate is read accurately and completely based on the manual method. Another way is to identify the text information in the nameplate image through a simple feature extraction network. However, when the feature extraction network processes the nameplate image in complex scenes, the recognition accuracy is often low, which is not conducive to the subsequent processing of the substation. Summary of the invention

[0004] The present invention provides an image processing method and device, which improves the recognition accuracy of text information in nameplate images.

[0005] According to one aspect of the present invention, there is provided an image processing method, the method comprising:

[0006] Determining at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed;

[0007] Based on at least two nameplate images to be used, determining a nameplate image to be applied corresponding to the nameplate image to be processed, wherein the nameplate image to be applied includes at least one pixel point including a confidence level of text and a text box area;

[0008] The target nameplate image is obtained by performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied;

[0009] Based on the target nameplate image, target text information of the nameplate image to be processed is determined.

[0010] According to another aspect of the present invention, there is provided an image processing device, the device comprising:

[0011] An image determination module, used to determine at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed;

[0012] A module for determining a nameplate image to be applied, used to determine a nameplate image to be applied corresponding to the nameplate image to be processed based on at least two nameplate images to be used, wherein the nameplate image to be applied includes at least one pixel point including a confidence level of text and a text box area;

[0013] A text correction module, used for performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied, to obtain a target nameplate image;

[0014] The text information determination module is used to determine the target text information of the nameplate image to be processed based on the target nameplate image.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the image processing method of any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method of any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, there is provided a computer program product, comprising a computer program, wherein the computer program implements the image processing method according to any embodiment of the present invention when executed by a processor.

[0021] The technical solution of the embodiment of the present invention facilitates the subsequent more accurate identification of image features in the nameplate image by determining at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed. Based on the at least two nameplate images to be used, the nameplate image to be applied corresponding to the nameplate image to be processed is determined. Based on this, it can be determined whether each pixel point in the nameplate image to be processed includes text, which facilitates the subsequent direction correction of the text. The target nameplate image is obtained by directionally correcting the text in the text box area in the nameplate image to be applied. According to the target nameplate image, the target text information of the nameplate image to be processed is determined. The present invention solves the problem of inaccurate and incomplete text information recognition caused by the recognition of text information in the nameplate based on manual methods or simple feature extraction networks in the prior art, and improves the recognition accuracy of text information in the nameplate image.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present invention;

[0025] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present invention;

[0026] Figure 3 is a structural schematic diagram of an image processing device provided by an embodiment of the present invention;

[0027] Figure 4 It is a schematic diagram of the structure of an electronic device for implementing the image processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1This is a flowchart of an image processing method provided by the first embodiment of the present invention. This embodiment is applicable to the case of identifying text information in a nameplate image to be processed. The method can be executed by an image processing device. The image processing device can be implemented in the form of hardware and / or software. The image processing device can be configured in an electronic device such as a mobile phone, a computer or a server. Figure 1 As shown, the method includes:

[0032] S110: Determine at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed.

[0033] The nameplate image to be processed may be an image containing a nameplate, and text information in the nameplate currently needs to be detected. The at least two nameplate images to be used at at least two resolutions may be obtained by scaling the nameplate image to be processed.

[0034] Specifically, the nameplate image to be processed is scaled to obtain at least two nameplate images to be used at at least two resolutions. The nameplate images to be used with different resolutions can facilitate subsequent extraction of image features corresponding to different scales, thereby improving the accuracy of image recognition.

[0035] Exemplarily, the nameplate image to be processed is scaled to obtain a nameplate image to be used whose image size is half of the image size of the nameplate image to be processed, a nameplate image to be used whose image size is one quarter of the image size of the nameplate image to be processed, and a nameplate image to be used whose image size is one eighth of the image size of the nameplate image to be processed. Based on this, three nameplate images to be used at three resolutions are obtained.

[0036] In an embodiment of the present invention, before processing the nameplate image to be processed, the nameplate image to be processed can be obtained based on the original nameplate image. The specific method can be: obtaining the original nameplate image captured by the camera device; and obtaining the nameplate image to be processed of the original nameplate image by performing lighting processing on the original nameplate image.

[0037] The original nameplate image may be an unprocessed image including the nameplate. The camera device may be a device for capturing the original nameplate image. Optionally, the camera device may be a camera of a monitoring device, an electronic device of a corresponding user, a drone camera capture device, or a robot for capturing images, etc. This embodiment does not limit the camera device.

[0038] When the original nameplate image is collected by the camera device, there may be occlusion or insufficient brightness. In this case, the collected original nameplate image is not conducive to the recognition and processing of text information in the subsequent image. Therefore, the original nameplate image can be illuminated to obtain the nameplate image to be processed. Illumination processing is to improve the brightness or contrast of the original nameplate image.

[0039] Specifically, an original nameplate image captured by a camera device is obtained. In order to improve the accuracy of subsequent image recognition, the original nameplate image is subjected to illumination processing to enhance the brightness and contrast of the original nameplate image, and a nameplate image to be processed corresponding to the original nameplate image is obtained.

[0040] Optionally, the method of performing illumination processing on the original nameplate image may be: for at least one pixel point in the original nameplate image, the pixel coordinates, pixel value and global illumination parameters of the pixel point are input into the illumination estimation model to obtain the ideal illumination value of the pixel point, so as to determine the nameplate image to be processed of the original nameplate image based on the ideal illumination value; or, the original nameplate image corresponding to different exposure parameters collected by the same device is obtained, and the original nameplate image is fused to obtain the nameplate image to be processed that fuses different exposure parameters.

[0041] The global illumination parameter may be a pre-set parameter for determining an ideal illumination value corresponding to a pixel. The illumination estimation model may be used to adjust the illumination conditions in the original nameplate image. Optionally, the illumination estimation model may be a model determined based on an adaptive illumination compensation algorithm. The ideal illumination value may be understood as a value capable of adjusting parameters such as brightness and contrast of the original nameplate image.

[0042] The same device can be a camera device used to capture the original nameplate image. For the same device, different aperture, shutter speed and sensitivity settings have different corresponding exposure parameters. Exposure parameters are key factors affecting the brightness of the original nameplate image.

[0043] Specifically, for at least one pixel in the original nameplate image, the pixel coordinates, pixel value and global illumination parameters of the current pixel are input into the illumination estimation model to obtain the ideal illumination value of the current pixel. Based on this, the ideal illumination value of each pixel in the original nameplate image can be obtained. According to the ideal illumination value of each pixel, the pixel value of each pixel of the original nameplate image is adjusted to improve the brightness and contrast of the original nameplate image, and the nameplate image to be processed is obtained.

[0044] Alternatively, the original nameplate image may be subjected to illumination processing by obtaining the original nameplate image under different exposure parameters of the same camera device. The original nameplate images under these different exposure parameters are subjected to image fusion processing using a pre-trained image fusion model to obtain a nameplate image to be processed that is fused with the original nameplate images under different exposure parameters. The image to be processed obtained based on the above method has higher contrast and brightness, which can improve the accuracy of subsequent image recognition.

[0045] For example, the illumination estimation model is used as an example to illustrate the model corresponding to the adaptive illumination compensation algorithm. The pixel value, pixel coordinates and global illumination parameters of each pixel in the original image are input into the illumination estimation model, and the ideal illumination value calculation function of the illumination estimation model is calculated, L(x , y) ＝ f((x , y), G), determine the ideal illumination value corresponding to each pixel. In the ideal illumination value calculation function, (x, y) represents the pixel coordinates of the current pixel, G represents the global illumination parameter, and L(x, y) is the ideal illumination value of the current pixel. Based on the ideal illumination value calculation function, the ideal illumination value of each pixel is obtained. The brightness and contrast of each pixel are adjusted according to the ideal illumination value to obtain the nameplate image to be processed corresponding to the original nameplate image.

[0046] For example, the image fusion model is described as a U-Net deep learning model, and the original nameplate images corresponding to different exposure parameters collected by the same device are obtained. The original nameplate images corresponding to different exposure parameters are input into the pre-trained U-Net deep learning model for image fusion processing to obtain the fused image, that is, the nameplate image to be processed.

[0047] In the embodiment of the present invention, the nameplate image to be used may be determined by performing resolution processing on the nameplate image to be processed according to at least two resolutions to obtain the nameplate image to be used at each resolution.

[0048] Specifically, at least two resolutions can be set according to actual needs. The nameplate image to be processed is scaled according to at least two resolutions to obtain the nameplate image to be used at the corresponding resolution, providing a richer image for subsequent image recognition.

[0049] S120: Determine a nameplate image to be applied corresponding to the nameplate image to be processed based on at least two nameplate images to be used, wherein the nameplate image to be applied includes at least one pixel point including a confidence level of text and a text box area.

[0050] The nameplate image to be used may be a feature image obtained by performing image feature extraction, image feature fusion and feature prediction on at least two nameplate images to be used. The nameplate image to be used includes the confidence that at least one pixel includes text and a text box area. The confidence that at least one pixel includes text can be understood as the credibility or probability that each pixel is a pixel corresponding to the text. The text box area may be a predicted text box area obtained by locating the display position of the text information in the nameplate image to be used.

[0051] Specifically, for at least two nameplate images to be used, image feature extraction processing is performed on each nameplate image to be used to obtain image features corresponding to each nameplate image to be used. Feature fusion processing is performed on at least two image features to obtain fusion features. Feature prediction processing is performed on the fusion features to obtain a nameplate image to be applied corresponding to the nameplate image to be processed. The nameplate image to be applied includes at least one pixel point including the confidence level of the text and the text box area. Based on this, the subsequent detection of the text direction in the nameplate image to be applied can be facilitated.

[0052] S130 , obtaining a target nameplate image by performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied.

[0053] The text in the text box area in the nameplate image to be applied may be presented in different directions. For example, the display direction of the text in the text box area may be horizontal, vertical, or inclined at a certain angle. The direction correction process can be used to correct the text in the text box area to a preset direction. Optionally, the preset direction can be a horizontal direction. The target nameplate image can be a feature image obtained after the direction of the text in the text box area in the nameplate image to be applied is corrected to a preset direction.

[0054] Specifically, direction detection is performed on the text in the text box area of ​​the target nameplate image. When it is determined that the display direction of the text in the text box is not the preset direction, direction correction processing is performed on the text in the text box area to obtain the target nameplate image.

[0055] In an embodiment of the present invention, the text direction correction processing method may be: based on the first target algorithm, the text in the text box area of ​​the target nameplate image is directionally corrected to obtain a target nameplate image in which the text is displayed in the first direction.

[0056] The first target algorithm may be an algorithm for correcting the direction of text in the text box area. Optionally, the first target algorithm may be a spatial transformer network (STN) algorithm. The first direction, i.e., the preset direction, may be a preset standard display direction of text in the text box area. Optionally, the first direction may be a horizontal direction.

[0057] Specifically, the direction of the text in the nameplate image to be applied is estimated by the first target algorithm, and it is determined that the display direction of the text in the nameplate image to be applied is not all in the first direction. In this case, the direction of the text in the nameplate image to be applied is corrected, and the display direction of the text in the text box area is adjusted to the first direction, so as to obtain a target nameplate image in which the display direction of the text is all in the first direction.

[0058] Exemplarily, the first target algorithm is the STN algorithm, and the first direction is the horizontal direction. The direction of the text in the text box area of ​​the nameplate image to be applied is estimated. When it is determined that the display direction of the text is multi-directional, the affine transformation matrix corresponding to the STN algorithm is used to perform direction correction processing on the multi-directional text to obtain a target nameplate image with the text display direction in the horizontal direction.

[0059] S140: Determine target text information of the nameplate image to be processed based on the target nameplate image.

[0060] The target text information may be understood as the text information in the nameplate image to be processed.

[0061] Specifically, feature extraction and feature prediction processing may be performed on the target nameplate image to determine the target text information in the nameplate image to be processed.

[0062] Exemplarily, feature extraction processing can be performed on the target nameplate image based on a preset network model to obtain a feature matrix corresponding to the target nameplate image. The feature matrix is ​​processed by another preset network model to extract the target text information corresponding to the feature matrix, that is, to obtain the target text information of the nameplate image to be processed.

[0063] The technical solution of this embodiment facilitates the subsequent more accurate identification of image features in the nameplate image by determining at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed. Based on the at least two nameplate images to be used, the nameplate image to be applied corresponding to the nameplate image to be processed is determined. Based on this, it can be determined whether each pixel in the nameplate image to be processed includes text, which facilitates the subsequent direction correction of the text. The target nameplate image is obtained by directionally correcting the text in the text box area in the nameplate image to be applied. According to the target nameplate image, the target text information of the nameplate image to be processed is determined. The present invention solves the problem of inaccurate and incomplete text information recognition caused by the recognition of text information in the nameplate based on manual methods or simple feature extraction networks in the prior art, and improves the recognition accuracy of text information in the nameplate image.

[0064] Embodiment 2

[0065] Figure 2 1 is a flowchart of an image processing method provided by Embodiment 2 of the present invention. This embodiment is a preferred embodiment of the above embodiment. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same as or corresponding to the above embodiment are not repeated here. Figure 2 As shown, the method includes:

[0066] S210: Determine at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed.

[0067] S220, inputting at least two nameplate images to be used into a pre-trained first neural network model, so as to extract image features of corresponding nameplate images to be used based on multiple feature extraction layers in the first neural network model.

[0068] Among them, the first neural network model can be pre-trained and used to determine the confidence that the pixel points in the nameplate image to be used include text, as well as the text box area. The first neural network model can include a feature extraction layer, a feature fusion layer and a prediction module. The feature extraction layer can be used to extract image features of the nameplate image to be used. The feature extraction layer can be composed of at least one residual module. The feature fusion layer can be used to perform feature fusion processing on feature images of different resolutions. The feature fusion layer includes at least one convolution module and an upsampling module. The prediction module is used to perform prediction processing on the fused features.

[0069] Specifically, at least two nameplate images to be used are input into a pre-trained first neural network model, so that feature extraction processing is performed on each nameplate image to be used through multiple feature extraction layers of the first neural network model to obtain image features corresponding to different resolutions.

[0070] Exemplarily, the first neural network model is a model determined based on the improved EAST (Efficient and AccurateScene Text Detector) algorithm as an example for explanation. The multiple feature extraction layers of the first neural network model can be composed of residual modules (Residual Network, ResNet). That is, each feature extraction layer includes at least one residual module. Based on at least one residual module of each feature extraction layer, feature extraction is performed on the corresponding nameplate image to be used to obtain image features corresponding to each nameplate image to be used.

[0071] S230, fusing the image features of at least two nameplate images to be used based on the feature fusion layer in the first neural network model to obtain fused features.

[0072] The feature fusion layer can be used to perform feature fusion processing on image features obtained based on multiple feature extraction layers. Optionally, the feature fusion layer can be determined by a multi-channel Fully Convolutional Networks (FCN).

[0073] Specifically, the feature fusion layer based on the first neural network model fuses the image features of at least two nameplate images to be used to obtain fused features, so as to determine the nameplate image to be used based on the fused features.

[0074] Exemplarily, a feature fusion layer includes at least two convolution modules, at least one feature fusion module and at least one upsampling module. The convolution module is used to extract deep-level image features corresponding to image features. The upsampling module is used to perform upsampling processing on image features. The feature fusion module is used to perform feature fusion processing on deep-level image features or upsampled image features.

[0075] The at least two convolution modules of the feature fusion layer respectively perform convolution feature extraction on the image features of at least two nameplate images to be used to obtain deep-level image features. The deep-level image features or image features are upsampled based on at least one upsampling module to obtain upsampled image features. The feature fusion module performs weighted fusion or splicing on image features of different scales or deep-level image features or upsampled image features to obtain fused features.

[0076] S240: Predict and process the fusion features based on the prediction module in the first neural network model to obtain a nameplate image to be applied.

[0077] The nameplate image to be applied includes at least one pixel including the confidence of the text and the text box area. The prediction module is used to perform prediction processing on the fusion feature. Optionally, the prediction module may include a non-maximum suppression algorithm. The non-maximum suppression algorithm is used to remove redundant text boxes and retain the best text box area.

[0078] Specifically, the prediction module of the first neural network model is used to perform prediction processing on the fusion features to obtain the nameplate image to be applied. The nameplate image to be applied includes at least one pixel point including the confidence of the text and the text box area.

[0079] S250 , obtaining a target nameplate image by performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied.

[0080] S260. Extract features of the target nameplate image based on the second neural network model to obtain a target feature matrix, wherein the target feature matrix includes a feature sequence corresponding to each column of pixels.

[0081] The second neural network model may be a model for extracting features from the target nameplate image. The second neural network model may be a residual network model. The target feature matrix may include a feature sequence corresponding to each column of pixel points in the target nameplate image.

[0082] Specifically, the target nameplate image is feature extracted according to the second neural network model to obtain a convolution feature map, and the feature sequence corresponding to each column of pixels in the convolution feature map is determined. According to the feature sequence corresponding to each column of pixels, the target feature matrix is ​​determined.

[0083] Exemplarily, the second neural network model is a residual network model as an example for explanation. The convolution features of the target nameplate image are extracted according to the residual network model to obtain a convolution feature map. And the feature sequence corresponding to each column of pixels in the convolution feature map is determined. According to the feature sequence corresponding to each column of pixels, a target feature matrix is ​​obtained, so as to determine the target text information in the nameplate image to be processed based on the target feature matrix.

[0084] S270. Process the target feature matrix based on a third neural network model that integrates at least two long short-term memory networks to extract target text information from the nameplate image to be processed.

[0085] Among them, the third neural network model integrated with at least two long short-term memory networks can be used to obtain features in at least two directions to determine the target text information. Optionally, the third neural network model integrated with at least two long short-term memory networks can be a bidirectional long short-term memory network model (Bidirectional Long Short-Term Memory, BiLSTM) composed of two long short-term memory networks (Long Short-Term Memory, LSTM) in different directions.

[0086] Specifically, the target feature matrix is ​​processed according to a third neural network model integrated with at least two long short-term memory networks to determine the context information of the characters in the nameplate image to be processed, and obtain the target text information corresponding to the nameplate image to be processed.

[0087] In the embodiment of the present invention, after the target text information is obtained, repeated texts and blank characters in the target text information may be removed based on the second target algorithm to update the target text information.

[0088] The second target algorithm may be used to remove repeated text and blank text in the target text information. Optionally, the second target algorithm may be a Connectionist Temporal Classification (CTC) algorithm.

[0089] Specifically, the target text information is processed using the second target algorithm to obtain a character probability distribution sequence corresponding to the target text information. The character probability distribution sequence is decoded, and repeated characters and blank characters in the character probability distribution sequence are deleted during the decoding process to obtain a text sequence, so as to update the target text information according to the text sequence.

[0090] It should be noted that the first neural network model, the second neural network model and the third neural network model can be lightweight network models determined based on knowledge distillation technology. Among them, the first neural network model, the second neural network model and the third neural network model can be lightweight network models determined after learning key information such as weight values ​​and loss functions in corresponding large and complex models using knowledge distillation technology.

[0091] In an embodiment of the present invention, in order to improve the processing speed of the first neural network model, the second neural network model and the third neural network model, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC) accelerator can be used to implement specific computing operations in hardware to avoid delays in image processing.

[0092] Optionally, the method also includes: pruning the first neural network model, the second neural network model and the third neural network model based on the pruning strategy and the memory usage strategy, respectively, to obtain lightweight first neural network model, second neural network model and third neural network model; wherein the pruning strategy is used to adjust the network nodes in the network model, and the memory usage strategy is used to adjust the data types used in the network nodes.

[0093] Among them, the pruning strategy can be used to gradually remove network layers or modules that have little impact on the output results of the network model while ensuring the recognition accuracy of the network model, so as to achieve the purpose of reducing the size of the network model. The memory usage strategy can be used to reduce the memory usage of the network model.

[0094] Specifically, the pruning methods of the first neural network model, the second neural network model and the third neural network model based on the pruning strategy are similar. To avoid redundancy, the pruning of the first neural network model based on the pruning strategy is explained. By analyzing the impact of changes in weights and activation values ​​in the feature extraction layer, feature fusion layer and prediction module of the first neural network model on the final output result, the importance evaluation results corresponding to each module and network node in the first neural network model are determined. Among them, the importance of each module in each layer can be determined based on the L1 norm, L2 norm and Taylor expansion formula. Modules or network nodes with smaller importance evaluation results are gradually removed to obtain a lightweight first neural network model. It should be noted that in the process of removing modules or network nodes, it is necessary to detect the performance changes of the first neural network model so as to prune the first neural network model while ensuring that the performance of the first neural network model meets the preset requirements.

[0095] The manner of adjusting the data types of the first neural network model, the second neural network model, and the third neural network model based on the memory footprint strategy is similar. To avoid repetition, the adjustment of the data types of the network nodes in the first neural network model based on the memory footprint strategy is used as an example for explanation. A preset data type is used instead of the data type used by each network node in the first neural network model. Optionally, the preset data type may be an 8-bit integer data type. The floating-point data type corresponding to the weight values ​​and activation values ​​of each network node of the first neural network model is converted to a preset data type to reduce storage requirements and computational overhead. The first neural network model is retrained so that the first neural network model can adapt to the preset data type. The model performance of the first neural network model is further optimized through quantization-aware training (QAT) so that the first neural network model can maintain a high recognition accuracy in a low-precision environment to obtain a lightweight first neural network model.

[0096] Exemplarily, the preset data type is an 8-bit integer data type as an example for explanation. Further optimizing the model performance of the first neural network model through quantization-aware training can be to insert a quantization operation in the forward propagation of the first neural network model to simulate the data processing of the first neural network model under a low-precision data type. By adjusting the learning rate, optimizer and other hyperparameters corresponding to the first neural network model, the first neural network model is trained and optimized to ensure that the first neural network model can still converge and achieve a higher model performance in a low-precision environment to obtain an optimized first neural network model. The model performance of the optimized first neural network model, such as accuracy, processing speed, memory occupancy and power consumption, is evaluated to obtain an evaluation result. According to the evaluation results, the model parameters of the optimized first neural network model are adjusted to obtain a lightweight first neural network model.

[0097] Optionally, the method also includes: when the next nameplate image to be processed is obtained, determining the changed area between the next nameplate image to be processed and the current nameplate image to be processed; processing the changed area based on the processing methods of S210 to S270 above to obtain the text information to be updated corresponding to the changed area; and updating the target text information based on the text information to be updated.

[0098] Specifically, for the nameplate image to be processed, it can be obtained by performing lighting processing on the original nameplate image directly obtained, or by taking a video of the corresponding nameplate and determining it through a frame in the video stream of the corresponding nameplate. If the nameplate image to be processed is a frame of the video stream of the corresponding nameplate, then after obtaining the target text information of the nameplate image to be processed of the current frame, for the nameplate image to be processed of the next frame, the changed area that is different from the nameplate image to be processed of the current frame can be determined. Through the above-mentioned determination method of S210 to S270, the changed area is detected and identified, and the text information to be updated is obtained, and the target text information is updated with the text information to be updated. Based on the above-mentioned incremental image processing method, unnecessary image recognition processing can be reduced and the recognition efficiency of the nameplate image can be improved.

[0099] The technical solution of this embodiment is to facilitate the subsequent more accurate identification of image features in the nameplate image by determining at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed. At least two nameplate images to be used are input into a pre-trained first neural network model to extract the image features of the corresponding nameplate images to be used based on multiple feature extraction layers in the first neural network model. The image features of at least two nameplate images to be used are fused based on the feature fusion layer in the first neural network model to obtain fusion features. The fusion features are predicted and processed based on the prediction module in the first neural network model to obtain the nameplate image to be applied. Based on this, it can be determined whether each pixel in the nameplate image to be processed includes text, which is convenient for subsequent direction correction of the text. The target nameplate image is obtained by direction correction of the text in the text box area in the nameplate image to be applied. Feature extraction is performed on the target nameplate image based on the second neural network model to obtain a target feature matrix. The target feature matrix is ​​processed based on a third neural network model integrated with at least two long short-term memory networks to extract the target text information in the nameplate image to be processed. The present invention solves the problem of inaccurate and incomplete text information recognition caused by recognizing text information in nameplates based on manual methods or simple feature extraction networks in the prior art, and improves the recognition accuracy of text information in nameplate images.

[0100] Embodiment 3

[0101] Figure 3 Schematic diagram of the structure of an image processing device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an image determination module 310 , a nameplate image determination module 320 , a text correction module 330 , and a text information determination module 340 .

[0102] An image determination module 310 is used to determine at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed; a nameplate image determination module 320 is used to determine a nameplate image to be applied corresponding to the nameplate image to be processed based on the at least two nameplate images to be used, wherein the nameplate image to be applied includes at least one pixel point including the confidence of the text and the text box area; a text correction module 330 is used to obtain a target nameplate image by performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied; a text information determination module 340 is used to determine the target text information of the nameplate image to be processed based on the target nameplate image.

[0103] The technical solution of this embodiment facilitates the subsequent more accurate identification of image features in the nameplate image by determining at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed. Based on the at least two nameplate images to be used, the nameplate image to be applied corresponding to the nameplate image to be processed is determined. Based on this, it can be determined whether each pixel in the nameplate image to be processed includes text, which facilitates the subsequent direction correction of the text. The target nameplate image is obtained by directionally correcting the text in the text box area in the nameplate image to be applied. According to the target nameplate image, the target text information of the nameplate image to be processed is determined. The present invention solves the problem of inaccurate and incomplete text information recognition caused by the recognition of text information in the nameplate based on manual methods or simple feature extraction networks in the prior art, and improves the recognition accuracy of text information in the nameplate image.

[0104] On the basis of the above embodiment, optionally, the image determination module is used to perform resolution processing on the nameplate image to be processed according to at least two resolutions to obtain the nameplate image to be used at each resolution.

[0105] Optionally, the module for determining the nameplate image to be used includes: an image feature extraction unit, used to input at least two nameplate images to be used into a pre-trained first neural network model, so as to extract image features of corresponding nameplate images to be used based on multiple feature extraction layers in the first neural network model; a fusion feature extraction unit, used to fuse the image features of at least two nameplate images to be used based on the feature fusion layer in the first neural network model to obtain a fusion feature; a feature prediction processing unit, used to predict the fusion feature based on the prediction module in the first neural network model to obtain the nameplate image to be used.

[0106] Optionally, a text correction module is used to perform direction correction on the text in the text box area of ​​the target nameplate image based on a first target algorithm to obtain a target nameplate image in which the display direction of the text is all in the first direction.

[0107] Optionally, the text information determination module includes: a feature matrix determination unit, used to extract features of the target nameplate image based on the second neural network model to obtain a target feature matrix, wherein the target feature matrix includes a feature sequence corresponding to each column of pixels; a target text information determination unit, used to process the target feature matrix based on a third neural network model integrated with at least two long short-term memory networks to extract target text information from the nameplate image to be processed.

[0108] Optionally, before the image determination module, the device also includes: an original image processing module, which includes: an original nameplate image acquisition unit, used to acquire the original nameplate image captured by the camera device; an image illumination processing unit, used to obtain a nameplate image to be processed of the original nameplate image by performing illumination processing on the original nameplate image.

[0109] Optionally, an image illumination processing unit is used to input the pixel coordinates, pixel value and global illumination parameters of at least one pixel point in the original nameplate image into an illumination estimation model to obtain an ideal illumination value of the pixel point, so as to determine the nameplate image to be processed of the original nameplate image based on the ideal illumination value; or, obtain the original nameplate image corresponding to different exposure parameters collected by the same device, and fuse the original nameplate image to obtain the nameplate image to be processed that fuses different exposure parameters.

[0110] Optionally, the device also includes: a model optimization module, which is used to prune the first neural network model, the second neural network model and the third neural network model based on a pruning strategy and a memory usage strategy, respectively, to obtain a lightweight first neural network model, a second neural network model and a third neural network model; wherein the pruning strategy is used to adjust the network nodes in the network model, and the memory usage strategy is used to adjust the data types used in the network nodes.

[0111] Optionally, after the text information determination module, the device further includes: a text information updating module, which is used to remove repeated texts and blank characters in the target text information based on the second target algorithm to update the target text information.

[0112] The image processing device provided by the embodiment of the present invention can execute the image processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0113] Embodiment 4

[0114] Figure 4 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0115] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0116] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as an image processing method.

[0118] In some embodiments, the image processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The computer programs for implementing the image processing method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] Embodiment 5

[0122] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute an image processing method, the method comprising:

[0123] Determine at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed; determine a nameplate image to be applied corresponding to the nameplate image to be processed based on the at least two nameplate images to be used, wherein the nameplate image to be applied includes at least one pixel point including the confidence of the text and a text box area; obtain a target nameplate image by performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied; determine the target text information of the nameplate image to be processed based on the target nameplate image.

[0124] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0127] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: include: Determining at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed; Based on the at least two nameplate images to be used, determining a nameplate image to be applied corresponding to the nameplate image to be processed, wherein the nameplate image to be applied includes at least one pixel point including a confidence level of text and a text box area; Obtaining a target nameplate image by performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied; Based on the target nameplate image, target text information of the nameplate image to be processed is determined.

2. The method according to claim 1, characterized in that The step of determining at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed comprises: Resolution processing is performed on the nameplate image to be processed according to the at least two resolutions to obtain the nameplate image to be used at each resolution.

3. The method according to claim 1, characterized in that The determining, based on the at least two nameplate images to be used, a nameplate image to be applied corresponding to the nameplate image to be processed comprises: Inputting the at least two nameplate images to be used into a pre-trained first neural network model, so as to respectively extract image features of the corresponding nameplate images to be used based on a plurality of feature extraction layers in the first neural network model; Based on the feature fusion layer in the first neural network model, image features of the at least two nameplate images to be used are fused to obtain fused features; The fusion feature is predicted and processed based on the prediction module in the first neural network model to obtain the nameplate image to be applied.

4. The method according to claim 1, characterized in that The step of performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied to obtain the target nameplate image includes: Direction correction is performed on the text in the text box area in the nameplate image to be applied based on the first target algorithm to obtain a target nameplate image in which the display direction of the text is all in the first direction.

5. The method according to claim 1, characterized in that The step of determining target text information of the nameplate image to be processed based on the target nameplate image includes: Extracting features of the target nameplate image based on the second neural network model to obtain a target feature matrix, wherein the target feature matrix includes a feature sequence corresponding to each column of pixels; The target feature matrix is ​​processed based on a third neural network model integrated with at least two long short-term memory networks to extract target text information in the nameplate image to be processed.

6. The method according to claim 1, characterized in that Before determining at least two nameplate images to be used of the nameplate image to be processed at at least two resolutions, the method further includes: Acquire the original nameplate image captured by the camera device; The nameplate image to be processed of the original nameplate image is obtained by performing illumination processing on the original nameplate image.

7. The method according to claim 6, characterized in that The step of performing illumination processing on the original nameplate image to obtain a nameplate image to be processed of the original nameplate image comprises: For at least one pixel point in the original nameplate image, input the pixel coordinates, pixel value and global illumination parameters of the pixel point into the illumination estimation model to obtain the ideal illumination value of the pixel point, so as to determine the nameplate image to be processed of the original nameplate image based on the ideal illumination value; or, The original nameplate images corresponding to different exposure parameters collected by the same device are obtained, and the original nameplate images are fused to obtain the nameplate images to be processed which are fused with different exposure parameters.

8. The method according to claim 1, characterized in that: Also includes: Based on the pruning strategy and the memory usage strategy, the first neural network model, the second neural network model and the third neural network model are pruned respectively to obtain lightweight first neural network models, second neural network models and third neural network models; The pruning strategy is used to adjust the network nodes in the network model, and the memory usage strategy is used to adjust the data types used in the network nodes.

9. The method according to claim 1, characterized in that: After obtaining the target text information, the method further includes: Based on the second target algorithm, duplicate text and blank text in the target text information are removed to update the target text information.

10. An image processing device, characterized in that: include: An image determination module, used to determine at least two nameplate images to be used at at least two resolutions of the nameplate image to be processed; A nameplate image to be applied determining module, used to determine a nameplate image to be applied corresponding to the nameplate image to be processed based on the at least two nameplate images to be used, wherein the nameplate image to be applied includes at least one pixel point including a confidence level of text and a text box area; A text correction module, used for obtaining a target nameplate image by performing direction correction processing on the text in the text box area of ​​the nameplate image to be applied; The text information determination module is used to determine the target text information of the nameplate image to be processed based on the target nameplate image.

Images