Low-light food package image recognition method, computer program product and terminal

Abstract

The invention discloses a low-illumination food package image recognition method, a computer program product and a terminal, and belongs to the technical field of image recognition. The joint training loss is the weighted sum of the reconstruction loss of the low-illumination image enhancement network, the first feature perception loss of the optical character recognition model, the classification loss of the optical character recognition model and the label loss of the optical character recognition model. According to the method, a joint training strategy is adopted, under the guidance of advanced information fed back by the optical character recognition model, the low-illumination image enhancement network can better understand and predict the features of the image under the low-illumination condition, and then optimization is continuously carried out to generate the high-illumination image more suitable for processing of the optical character recognition model; the method not only improves the precision of a low-illumination image optical character recognition task, but also enhances the robustness and generalization ability of an optical character recognition model under different illumination conditions.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a low-light food packaging image recognition method, a computer program product and a terminal. Background Art

[0002] With the rapid development of the global food industry, food packaging plays a vital role in ensuring food safety, enhancing brand image, and optimizing supply chain management. The text information on food packaging, such as production date, shelf life, ingredient list, barcode, etc., not only provides necessary information for consumers, but also provides key data support for production companies and logistics links. Therefore, accurate and fast recognition and processing of optical characters (OCR) on these packages has become an important technical requirement in the modern food industry.

[0003] Traditional optical character recognition technology can achieve high recognition accuracy under sufficient lighting conditions. However, in practical applications, food packaging often needs to be scanned and recognized in a variety of complex environments, such as storage environments under low-light conditions, night production lines, and mobile device usage scenarios. Although some studies have attempted to improve OCR performance under low-light conditions through methods such as image enhancement, and such methods may perform well in specific low-light scenarios, the performance of these methods will significantly decrease when environmental conditions change. Summary of the invention

[0004] The purpose of the present invention is to overcome the problems of the prior art and to provide a low-light food packaging image recognition method, a computer program product and a terminal.

[0005] The object of the present invention is to achieve the following technical solution: a low-light food packaging image recognition method, the method comprising the following steps: Building low-light image enhancement networks and optical character recognition models; Jointly train a low-light image enhancement network and an optical character recognition model, including: Pre-training the optical character recognition model using an image dataset under normal lighting conditions, freezing all trainable parameters in the optical character recognition model after the pre-training is completed, and updating the parameters of the low-light image enhancement network so that the low-light image enhancement network outputs a high-light image that can be recognized by the optical character recognition model, and inputting the high-light image into the optical character recognition model to obtain an optical character recognition result; The joint training loss used in the joint training is the weighted sum of the reconstruction loss of the low-light image enhancement network, the first feature perception loss of the optical character recognition model, the classification loss of the optical character recognition model, and the label loss of the optical character recognition model.

[0006] In one example, the low-light image enhancement network includes a downsampling module, an upsampling module, and a reconstruction module; A downsampling module is used to perform convolution processing, local and global feature extraction processing, and multi-scale feature downsampling processing on the input low-light image to obtain a first multi-scale feature map; the downsampling processing of each scale feature includes downsampling convolution processing, or the downsampling processing of each scale feature includes downsampling convolution processing, local and global feature extraction processing; An upsampling module is used to perform transposed convolution processing on the input low-light image and the first multi-scale feature map to obtain a first feature map, and then a multi-scale feature upsampling processing is performed on the first feature map to obtain a second multi-scale feature map; the upsampling processing of each scale feature includes local and global feature extraction processing, feature enhancement processing, or the upsampling processing of each scale feature includes local feature and global feature extraction, feature enhancement processing, and transposed convolution processing, and the input features of the upsampling processing of each scale feature also include extracted local features and global features of different scales; The reconstruction module is used to perform local and global feature extraction and convolution processing on the second multi-scale feature map to obtain a high-light image.

[0007] In one example, the local and global feature extraction process includes: The input features are input into the convolutional neural network branch and the Transformer branch respectively. The convolutional neural network branch performs multiple convolutions on the input features. The first intermediate feature map obtained by the convolution is downsampled to obtain a second intermediate feature map with the same dimension as the Transformer branch. The second intermediate feature map is input into the Transformer branch. The Transformer branch performs feature extraction and outputs a third intermediate feature map. The third intermediate feature map is transposed and convolved to obtain a fourth intermediate feature map with the same dimension as the CNN branch. The convolutional neural network branch performs multiple convolutions on the fourth intermediate feature map to output local and global features.

[0008] In one example, the feature enhancement process includes: The input features are processed by multi-scale convolution and then spliced. The spliced ​​feature maps are input into the convolution block attention module for multi-scale feature fusion to obtain refined features.

[0009] In one example, the optical character recognition model includes a text detection network and a character recognition network. The text detection network performs text detection on an input high-light image and extracts a character image. The character recognition subnetwork recognizes the input character image to obtain a text recognition result.

[0010] In one example, the text detection network is a differentiable binarization network, and the character recognition network is a convolutional recurrent neural network; The differentiable binarization network extracts multi-scale features from the input high-light image to obtain a multi-scale feature map, upsamples the multi-scale feature map to obtain a feature map of the same size, performs feature enhancement on the feature maps of the same size and then cascades them to obtain a fused feature map; predicts the probability map and threshold map of the text area based on the fused feature map, calculates an approximate binary map based on the probability map and the threshold map, generates a bounding box of the text based on the approximate binary map, and obtains a character image; The feature enhancement process includes: The input features are processed by multi-scale convolution and then spliced. The spliced ​​feature maps are input into the convolution block attention module for multi-scale feature fusion to obtain refined features. The convolutional recurrent neural network performs convolution and pooling processing on the input character image, serializes the extracted local features to obtain a feature vector, captures the character dependency and context information in the feature vector, decodes the character dependency and context information, and obtains the text recognition result.

[0011] In one example, the reconstruction loss of the low-light image enhancement network is The calculation expression is: ; in, is the color space difference loss, is the hyperparameter of color space difference loss; is the image frequency domain difference loss, is the hyperparameter of the image frequency domain difference loss; Second feature perceptual loss for image enhancement in low-light image enhancement networks, is the hyperparameter of the second perceptual loss; is the image structure similarity loss, is the hyperparameter of the image structure similarity loss.

[0012] In one example, the first feature perceptual loss of the optical character recognition model The calculation expression is: ; ; in, represents the first feature-perceptual loss, which is the cumulative sum of multiple feature-perceptual loss items; Represents single feature perception loss; represents the mean absolute error loss function; It is the feature representation of the enhanced high-light image; It is the feature representation of the normal illumination image; The label representing the feature-aware loss term; Indicates the number of feature extraction stages.

[0013] It should be further explained that the technical features corresponding to the various examples of the above method can be combined or replaced with each other to form a new technical solution.

[0014] The present invention also includes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the low-light food packaging image recognition method formed by any one of the above examples or a combination of multiple examples.

[0015] The present invention also includes a terminal, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor executes the computer instructions, the steps of the low-light food packaging image recognition method formed by any one or more of the above examples are executed.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. In one example, the present invention jointly trains a low-light image enhancement network and an optical character recognition model. Under the guidance of high-level information such as the position and content of text in the image fed back by the optical character recognition model, the low-light image enhancement network can better understand and predict the characteristics of the image under low-light conditions, and then continuously optimize to generate high-light images that are more suitable for processing by the optical character recognition model. This method not only improves the accuracy of the optical character recognition task of low-light images, but also enhances the robustness and generalization ability of the optical character recognition model under different lighting conditions, so that it can adapt to different environmental conditions.

[0018] 2. In one example, the low-light image enhancement network extracts and fuses multi-scale features through the downsampling module and the upsampling module respectively, which can effectively capture the local details and global structural information in the image. Therefore, when enhancing the low-light image, it not only retains the detailed texture of the image, but also maintains the overall structural consistency, thereby improving the image quality; the useful features of the image are enhanced through feature enhancement processing, so that the brightness and contrast of the image are improved. At the same time, the local and global features are further extracted and convolved in the reconstruction module, and the image reconstruction process is optimized, so that the enhanced high-light image is clearer and brighter, with richer details, so that the optical character recognition model can effectively recognize it.

[0019] 3. In one example, the CNN branch captures the local features and spatial information of the image, and the Transformer branch can effectively extract the global features and long-distance dependencies. The downsampling and transposed convolution operations are combined to perform dimensionality conversion, so that the feature map dimensions of the CNN branch and the Transformer branch remain consistent, which facilitates information interaction and fusion between features, enhances the comprehensive processing capabilities of local and global features, and can improve the accuracy and robustness of feature extraction.

[0020] 4. In one example, multi-scale convolution processing can capture detailed information of different scales in the image, thereby achieving a more comprehensive understanding of the image content; the spliced ​​feature map is input into the convolution block attention module, and the features are weighted and screened using the attention mechanism to obtain refined features containing rich image information, providing high-quality input for subsequent optical character recognition tasks, which is conducive to obtaining more accurate image processing results.

[0021] 5. In one example, a differentiable binarization network introduces feature enhancement processing, and combines multi-scale feature extraction, upsampling processing and other operations to obtain a fused feature map. Based on the fused feature map, the image content can be understood more comprehensively, and the detection capability of texts of different sizes and shapes is improved. At the same time, the binarization operation is embedded in the network for joint training and optimization, which can learn more precise text area boundaries and provide more accurate input information for optical character recognition tasks, thereby improving the accuracy and robustness of optical character recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0023] Figure 1 A method flow chart provided for an example of the present invention; Figure 2 A schematic diagram of a joint training framework provided for an example of the present invention; Figure 3 A schematic diagram of a low-light image enhancement network structure provided for an example of the present invention; Figure 4 A schematic diagram of a local and global feature extraction block structure provided for an example of the present invention; Figure 5 A schematic diagram of a feature enhancement block structure provided for an example of the present invention; Figure 6 A schematic diagram of a DB network structure provided for an example of the present invention; Figure 7 A schematic diagram of a convolutional recurrent neural network structure provided for an example of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the use of ordinal numbers (e.g., "first and second", "first to fourth", etc.) is to distinguish objects, and is not limited to this order, and cannot be understood as indicating or implying relative importance.

[0026] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, they can be directly connected, or indirectly connected through an intermediate medium, or they can be internally connected between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] In one example, if Figure 1 As shown, a low-light food packaging image recognition method is provided, and an end-to-end joint learning framework for image optical character recognition in a low-light restricted environment is designed, which can be used to recognize different types of images, including low-light food packaging images, low-light commodity label images, and other images including text information. The specific embodiment of the present invention illustrates the inventive concept of the method by processing the food packaging character recognition task under low light. The method comprises the following steps: S1: Build a Low-light Image Enhancement Network (LLIENet) and an Optical Character Recognition (OCR) model.

[0029] Among them, the low-light image enhancement network can be a convolutional neural network model Retinex-Net based on Retinex theory, an unsupervised generative adversarial network EnlightenGAN, etc., and the optical character recognition model can be a convolutional recurrent neural network (CRNN), an ultra-lightweight optical character recognition model PP-OCR, etc.

[0030] S2: Joint training of the low-light image enhancement network and the optical character recognition model, including: The optical character recognition model is pre-trained using a food packaging image dataset under normal lighting conditions. After the pre-training is completed, all trainable parameters in the optical character recognition model, including weights, biases, etc., are frozen. That is, only the parameters of the low-light image enhancement network, such as weights, are updated so that the low-light image enhancement network outputs a high-light image that can be recognized by the optical character recognition model. The high-light image is input into the optical character recognition model to obtain the optical character recognition result.

[0031] Furthermore, if Figure 2 As shown, in the joint training process in step S2, the joint training loss used is the weighted sum of the reconstruction loss of the low-light image enhancement network, the first feature perception loss of the optical character recognition model, the classification loss of the optical character recognition model, and the label loss of the optical character recognition model. In order to improve the image quality from qualitative and quantitative perspectives, the reconstruction loss is calculated based on the structural similarity, feature perception, color space difference, and frequency difference between the enhanced high-light image and the normal-light image. The mean absolute error (MAE) loss function is used to calculate the difference between the feature representations of the enhanced high-light image and the normal-light image, and then the first feature perception loss is calculated. The cross entropy loss function is used to calculate the difference between the character category probability distribution predicted by the OCR model and the true label, and then the classification loss is calculated. The cross entropy loss function is used to calculate the label difference based on the enhanced high-light image and the normal-light image, and then the label loss is calculated.

[0032] In this example, the joint learning method aims to enable the OCR model to guide the restoration process (image reconstruction process) of LLIENet, so that LLIENet can reconstruct high-light images that are more suitable for the OCR model and improve the accuracy of the OCR model. In the joint training strategy, the pre-trained OCR model plays the role of a supervisor to guide LLIENet to train, while LLIENet, as a student, uses the feedback information of the OCR model (information such as the position and content of the text in the image) to continuously optimize and expect to achieve the goals required by the OCR model. Specifically, given a low-light food packaging image, LLIENet is first applied to enhance the input image, and then the enhanced result (high-light image) is sent to the OCR model to perform the optical character recognition task, and then the optical character recognition result is output. In summary, the goals of the method of the present invention are: 1) Under the guidance of the high-level information of the OCR model, a visually good high-light image is reconstructed as the output of the low-light image enhancement model. 2) Achieve good accuracy in the low-light food packaging optical character recognition task.

[0033] In one example, a low-light image enhancement network includes a downsampling module, an upsampling module, and a reconstruction module. The downsampling module includes a first convolutional layer, a first local and global feature extraction block, and a multi-scale feature downsampling group connected in sequence, the multi-scale feature downsampling group includes a multi-scale feature downsampling block, each scale feature downsampling block includes a downsampling convolutional layer, or each scale feature downsampling block includes a downsampling convolutional layer, a local and global feature extraction block, and the first convolutional layer receives an input low-light image. Figure 3 As shown, the downsampling module includes the first convolution layer, the first local and global feature extraction block (the first CNN-Trans block, CNN is a convolutional neural network branch, and Trans represents a Transformer branch), the first downsampling convolution layer, the second local and global feature extraction block (the second CNN-Trans block), the second downsampling convolution layer, the third local and global feature extraction block (the third CNN-Trans block), and the third downsampling convolution layer, which are connected in sequence. The first downsampling convolution layer and the second local and global feature extraction block constitute a first scale feature downsampling block, the second downsampling convolution layer and the third local and global feature extraction block constitute a second scale feature downsampling block, and the second downsampling convolution layer is a third scale feature downsampling block. The downsampling convolution in each scale feature downsampling block is a 2×2 convolution with a stride of 2. In this example, the downsampling module is used to perform a 1×1 convolution layer preprocessing on the input low-light image, and then the image is transferred to the first local and global feature extraction block to extract local and global features, and then the depth downsampling of three scales is performed to obtain the first multi-scale feature map. The downsampling process of each scale feature includes downsampling convolution process, or the downsampling process of each scale feature includes downsampling convolution process, local and global feature extraction process.

[0034] Furthermore, the upsampling module aims to recover brightness information and dimensional information from the downsampled features. The upsampling module also includes a CNN-Trans block and introduces a new feature enhancement block. Specifically, the upsampling module includes a first transposed convolution layer, a multi-scale feature upsampling group, the multi-scale feature upsampling group includes a multi-scale feature upsampling block, the multi-scale feature upsampling block includes a local and global feature extraction block group, a feature enhancement layer, or a multi-scale feature upsampling block includes a local and global feature extraction block group, a feature enhancement processing layer and a transposed convolution layer. The input end of each multi-scale feature upsampling block is connected to different local and global feature extraction blocks to realize feature fusion processing. The first transposed convolution layer receives the input low-light image and the first multi-scale feature map. Figure 3 As shown, the upsampling module includes a first transposed convolution layer, a first local and global feature extraction block group (first CNN-Trans block group), a first feature enhancement block, a second transposed convolution layer, a second local and global feature extraction block group (second CNN-Trans block group), a second feature enhancement block, a third transposed convolution layer, a fourth local and global feature extraction block (fourth CNN-Trans block), and a third feature enhancement block. The first local and global feature extraction block group and the second local and global feature extraction block group each include two sequentially connected local and global feature extraction blocks. The first local and global The feature extraction block group, the first feature enhancement block, and the second transposed convolution layer constitute a first scale feature upsampling block, the second local and global feature extraction block group, the second feature enhancement block, and the third transposed convolution layer constitute a second scale feature upsampling block, the fourth local and global feature extraction block and the third feature enhancement block constitute a third scale feature upsampling block, the first local and global feature extraction block is connected to the fourth local and global feature extraction block, the second local and global feature extraction block is connected to the second local and global feature extraction block group, and the third local and global feature extraction block is connected to the first local and global feature extraction block group. In the upsampling module, the input low-light image and the first multi-scale feature map are first transposed convolutionally processed through a 2×2 transposed convolution layer with a stride of 2 to obtain a first feature map, and then the first feature map is subjected to multi-scale feature upsampling to obtain a second multi-scale feature map. Among them, the upsampling processing of each scale feature includes local and global feature extraction processing, feature enhancement processing, or the upsampling processing of each scale feature includes local feature and global feature extraction, feature enhancement processing, and 2×2 transposed convolution processing with a stride of 2, and the upsampling processing of each scale feature includes feature fusion processing with the corresponding local features and global features, and then the original down-sampled features are deeply upsampled to obtain a second multi-scale feature map of a larger scale.

[0035] Further, the reconstruction module is used to perform high-quality image reconstruction on the second multi-scale feature map, including a fifth local and global feature extraction block (fifth CNN-Trans block) connected in sequence, a plurality of convolutional layers, such as Figure 3 As shown, the reconstruction module includes the fifth local and global feature extraction block, the second convolution layer, and the third convolution layer connected in sequence, and the third convolution layer outputs a highlight image that can be recognized by the optical character recognition network. In this example, the reconstruction module is used to perform local and global feature extraction processing on the second multi-scale feature map to obtain pure features, and then the features are input into two 1×1 convolution layers to realize the reconstruction of the three-channel highlight image, thereby obtaining a highlight image.

[0036] In one example, the local and global feature extraction processing is implemented by a local and global feature extraction block, the local and global feature extraction block includes a convolutional neural network (CNN) branch and a Transformer branch, the convolutional neural network branch includes a first convolution group and a second convolution group, the Transformer branch is a Transformer encoding block, the first convolution group is connected to the input end of the Transformer encoding block via a downsampling processing block, and the output end of the Transformer encoding block is connected to the second convolution group via a transposed convolution block. Specifically, Figure 4As shown, the local and global feature extraction block includes a transform layer, which is used to transform the input feature map and then input it into the CNN branch and the Transformer branch. The CNN branch includes a first convolution group and a second convolution group connected in sequence. The first convolution group includes a 1×1 first convolution-batch normalization layer, a 3×3 second convolution-batch normalization layer, and a 1×1 third convolution-batch normalization layer. The second convolution group includes a 1×1 fourth convolution-batch normalization layer, a 3×3 fifth convolution-batch normalization layer, and a 1×1 sixth convolution-batch normalization layer. The input end residual of the first convolution-batch normalization layer is connected to the input end of the fourth convolution-batch normalization layer, and the input end residual of the fourth convolution-batch normalization layer is connected to the output end of the sixth convolution-batch normalization layer. Each convolution group uses a residual connection to reduce information loss and gradient disappearance. Optionally, the convolution-batch normalization layer can be replaced by a convolution layer. The Transformer encoding block of the Transformer branch includes the second normalization layer, multi-head self-attention layer, third normalization layer, and multi-layer perceptron connected in sequence. The input residual of the second normalization layer is connected to the input of the third normalization layer, and the input residual of the third normalization layer is connected to the output of the multi-layer perceptron. The downsampling processing block includes the first normalization layer, the fourth downsampling convolution layer, and the fourth convolution layer connected in sequence. The output of the second convolution-batch normalization layer is connected to the first normalization layer, and the output of the fourth convolution layer is connected to the second normalization layer. The feature map extracted by the CNN branch is downsampled to the feature map of the same dimension as the Transformer branch through the downsampling processing block and then enters the Transformer branch. The transposed convolution block includes a first batch normalization layer, a fifth convolution layer, and a fourth transposed convolution layer connected in sequence, the output end of the multilayer perceptron is connected to the first batch normalization layer, the output end of the fourth transposed convolution layer is connected to the fifth convolution-batch normalization layer, and the feature map output by the Transformer branch is converted into a feature map of the same dimension as the CNN branch by the transposed convolution block and then sent to the second convolution group of the CNN branch. Specifically, the local and global feature extraction processing of the present invention is to input the input features into the convolutional neural network branch and the Transformer branch respectively after the conversion processing, the convolutional neural network branch performs multiple convolution processing on the input features, and the first intermediate feature map obtained by the convolution processing is downsampled to obtain a second intermediate feature map with the same dimension as the Transformer branch. The second intermediate feature map is input into the Transformer branch, and the Transformer branch outputs a third intermediate feature map after feature extraction processing, and the third intermediate feature map is transposed convolution processing is performed on the third intermediate feature map to obtain a fourth intermediate feature map with the same dimension as the CNN branch. The convolutional neural network branch performs multiple convolution processing on the fourth intermediate feature map and outputs local and global features.It should be noted that the CNN branch is mainly responsible for extracting local features, that is, capturing local information in the input features through multiple convolution operations; the Transformer branch is mainly responsible for extracting global features, capturing the global dependencies between features through the self-attention mechanism. During the entire feature extraction process, local features and global features are processed and interacted by different branches, and feature fusion is performed to provide rich feature representations for subsequent tasks.

[0037] In one example, feature enhancement processing is implemented by a feature enhancement (FE) block, such as Figure 5 As shown in the figure, the feature enhancement block includes the third convolution group, the first concatenation layer and the convolutional block attention module (CBAM). The third convolution group includes the seventh convolution layer of 3×3, the eighth convolution layer of 5×5 and the ninth convolution layer of 7×7 in parallel. The output ends of the seventh convolution layer, the eighth convolution layer and the ninth convolution layer are all connected to the first concatenation layer. The convolutional block attention module includes a channel attention module and a spatial attention module. The channel attention module obtains the global spatial information of the input feature map in the channel dimension through global average pooling and global maximum pooling operations, and then sends these two global spatial information to the shared multi-layer perceptron (MLP) for processing, and finally generates the channel attention weight; the spatial attention module obtains refined features by weighting different channel features to strengthen the feature channels that contribute more to the current task, so that the feature representation is more focused on the target information and reduces the interference of noise and redundant information. The feature enhancement processing in this example is to perform multi-scale convolution processing on the input features and then splice them, and input the spliced ​​feature map into the convolution block attention module to perform multi-scale feature fusion to obtain refined features.

[0038] In one example, an optical character recognition model includes a text detection network and a character recognition network. The text detection network performs text detection on an input high-light image and extracts a character image. The character recognition subnetwork recognizes the input character image to obtain a text recognition result.

[0039] Optionally, the text detection network is a Differentiable Binarization (DB) network, and the character recognition network is a convolutional recurrent neural network (CRNN), and the two parts jointly complete the OCR task. The present invention introduces a feature enhancement block in the DB network to perform feature enhancement processing on the features before splicing at each level. Figure 6As shown, the DB network includes a feature pyramid backbone network, multiple upsampling layers, a feature enhancement block group and a differentiable binarization (DB) module. The feature enhancement block group includes a fourth feature enhancement block, a fifth feature enhancement block, a sixth feature enhancement block and a seventh feature enhancement block. The specific structure of each feature enhancement block is as follows Figure 5 As shown, no further details are given here. Among them, the feature pyramid backbone network performs multiple convolution and pooling operations (multi-scale feature extraction) on the high-light image to generate a series of feature maps of different scales, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 in size. The feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 in size are upsampled through multiple upsampling layers to obtain four upsampled feature maps of the same size. The feature maps of the same size are then enhanced through the fourth feature enhancement block, the fifth feature enhancement block, the sixth feature enhancement block, and the seventh feature enhancement block. The four feature maps of the same size are cascaded through the second splicing layer to obtain a fused feature map; the probability map and threshold map of the text area are predicted based on the fused feature map, and the DB module calculates an approximate binary map based on the probability map and the threshold map, and then generates a bounding box of the text to obtain a character image.

[0040] Furthermore, after upsampling the feature maps of sizes 1 / 4, 1 / 8, 1 / 16, and 1 / 32 through multiple upsampling layers, four upsampled feature maps of the same size are obtained, specifically including: The feature maps of sizes 1 / 16 and 1 / 32 are upsampled once and then element-summed to obtain a first intermediate upsampled feature map, the first intermediate upsampled feature map is upsampled twice and the feature map of size 1 / 8 is upsampled once and then element-summed to obtain a second intermediate upsampled feature map, the second intermediate upsampled feature map is upsampled twice and the feature map of size 1 / 4 is upsampled once and then element-summed to obtain a third intermediate upsampled feature map, the third intermediate upsampled feature map is convolved to obtain the first upsampled feature map, the second intermediate upsampled feature map is convolved and upsampled twice to obtain the second upsampled feature map, the first intermediate upsampled feature map is convolved and upsampled four times to obtain the third upsampled feature map, the feature map of size 1 / 32 is convolved and upsampled eight times to obtain a fourth upsampled feature map, and the sizes of the first upsampled feature map, the second upsampled feature map, the third upsampled feature map and the fourth upsampled feature map are the same.

[0041] Furthermore, if Figure 7As shown, the convolutional recurrent neural network includes a sequentially connected convolutional pooling group, an eleventh convolutional layer, a twelfth convolutional layer, a second maximum pooling layer, a convolutional batch normalization group, a third maximum pooling layer, a fourteenth convolutional layer, a mapping to sequence layer, a first bidirectional long short-term memory network, a second bidirectional long short-term memory network and a transcription layer (Transcription), the convolutional pooling group includes two convolutional pooling blocks, the convolutional pooling block includes a sequentially connected tenth convolutional layer and a first maximum pooling layer, the convolutional batch normalization group includes two convolutional batch normalization blocks, and the convolutional batch normalization block includes a sequentially connected thirteenth convolutional layer and a second batch normalization layer. Through the convolutional recurrent neural network with the above structure, the input character image is subjected to convolution processing, maximum pooling processing and batch normalization processing to extract local features, and the extracted local feature input is mapped to the sequence layer for serialization processing to obtain a feature vector. The character dependency and context information in the feature vector are captured through the first bidirectional long short-term memory network and the second bidirectional long short-term memory network, and then the character dependency and context information are decoded through the transcription layer to obtain the text recognition result.

[0042] In an example, the reconstruction loss of a low-light image enhancement network The calculation expression is: ; in, is the color space difference loss, is the hyperparameter of color space difference loss; is the image frequency domain difference loss, is the hyperparameter of the image frequency domain difference loss; Second feature perceptual loss for image enhancement in low-light image enhancement networks, is the hyperparameter of the second perceptual loss; is the image structure similarity loss, is the hyperparameter of the image structure similarity loss.

[0043] Specifically, for color space difference loss In order to make the enhanced high-light image and normal-light image as consistent as possible in the image space domain, the prediction error of the image is measured in the HSV and HSI color spaces respectively, and the color space difference loss The calculation expression is: ; in represents LLIENet; LL is the input low-light image; and They are the representations of the enhanced high-light image and normal-light image in the HSV color space; and They represent the enhanced highlight image and normal illumination image in the HSI color space respectively.

[0044] For image frequency domain difference loss , is a high-frequency component loss function used to measure the error of high-frequency components between the enhanced high-light image and the normal-light image. The present invention uses the difference in high-frequency components between the enhanced high-light image and the normal-light image in the horizontal and vertical directions as the image frequency domain loss. The specific calculation formula is as follows: ; in, and They represent the high-frequency components of the enhanced high-light image and normal-light image in the horizontal and vertical directions, respectively.

[0045] In order to further evaluate the difference between images, the present invention also introduces the second feature perception loss when the low-light image enhancement network performs image enhancement processing, which is used to measure the feature similarity between the enhanced high-light image and the normal-light image. The calculation expression is: ; in, , and Represent the image width, height and number of channels respectively; Respectively represent the image width index, image height index and image channel number index; It is a pre-trained ultra-deep convolutional network (VGG neural network) for large-scale image recognition; represents the output of the low-light image enhancement network; Represents the feature representation of a normal lighting image extracted by a pre-trained VGG neural network.

[0046] For image structure similarity loss , the calculation expression is: ; ; in, represents the structural similarity index; LL is the low-light image; is a normal lighting image; Indicates different images that need to be measured; , Respectively represent images ,image The average pixel value of and Respectively represent images ,image The variance of Representing images ,image The covariance of and are two constants. =0.01 2 , =0.03 2 .

[0047] In one example, the first feature perceptual loss of the OCR model The calculation expression is: ; ; in, represents the first feature-perceptual loss, which is the cumulative sum of multiple feature-perceptual loss items; represents a single feature perception loss, including the difference between 1 / 4 feature maps of the enhanced high-light image and the normal-light image processed by the two OCR models respectively, the difference between 1 / 8 feature maps of the enhanced high-light image and the normal-light image processed by the two OCR models respectively, the difference between 1 / 16 feature maps of the enhanced high-light image and the normal-light image processed by the two OCR models respectively, the difference between 1 / 32 feature maps of the enhanced high-light image and the normal-light image processed by the two OCR models respectively, the difference between the feature representations of the enhanced high-light image and the normal-light image output by the eleventh convolutional layers of the two OCR models respectively, the difference between the feature representations of the enhanced high-light image and the normal-light image output by the thirteenth convolutional layers of the two OCR models respectively, the difference between the feature representations of the enhanced high-light image and the normal-light image output by the mapping-to-sequence layers of the two OCR models respectively, and the difference between the feature representations of the enhanced high-light image and the normal-light image output by the transcription layers of the two OCR models respectively; represents the mean absolute error loss function; It is the feature representation of the enhanced high-light image; is the feature representation of the normal illumination image; S represents the number of feature extraction stages (the number of stages in the multi-stage network architecture).

[0048] The reconstruction loss , the first feature perception loss Calculate the examples to combine and get the joint training loss The calculation expression is: ; The cross entropy loss function is used to calculate the difference between the character category probability distribution predicted by the OCR model and the true label, and then the classification loss is calculated. ; The cross entropy loss function is used to calculate the label difference between the enhanced high-light image and the normal-light image, and then the label loss is calculated .

[0049] Combining the above examples, we get a preferred example of the present invention. At this time, we first establish the following Figure 3-Figure 5 The low-light image enhancement network LLIENet shown in the figure and Figure 6-Figure 7 The OCR model shown in the figure is then jointly trained for LLIENet and the OCR model, that is, the optical character recognition model is pre-trained using a food packaging image dataset under normal lighting conditions, all trainable parameters in the optical character recognition model are frozen after the pre-training is completed, the parameters of the low-light image enhancement network are updated, so that the low-light image enhancement network outputs a high-light image that the optical character recognition model can recognize, and the high-light image is input into the optical character recognition model to obtain the optical character recognition result. The joint training loss used in the joint training process is the weighted sum of the reconstruction loss of the low-light image enhancement network, the first feature perception loss of the optical character recognition model, the classification loss of the optical character recognition model, and the label loss of the optical character recognition model. Finally, the LIENet and OCR models that have completed the joint training are used to perform the low-light enhancement task and the optical character recognition task respectively to obtain the final optical character recognition result, that is, to recognize the text information in the low-light image.

[0050] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the low-light food packaging image recognition method formed by any one of the above examples or a combination of multiple examples. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0051] The present invention also provides a terminal, which has the same inventive concept as any example or a combination of multiple examples corresponding to the above-mentioned low-light food packaging image recognition method, including a memory and a processor, the memory stores computer instructions that can be run on the processor, and the processor executes the steps of the above-mentioned low-light food packaging image recognition method when running the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0052] In one example, the terminal, i.e., the electronic device, is presented in the form of a general-purpose computing device, and the components of the electronic device may include but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0053] The storage unit stores a program code, and the program code can be executed by the processing unit, so that the processing unit performs the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit can perform the above-mentioned low-light food packaging image recognition method.

[0054] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0055] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0056] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0057] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0058] Through the above description, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method of the exemplary embodiment of the present application.

[0059] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A low-light food packaging image recognition method, characterized in that: The method comprises the following steps: Building low-light image enhancement networks and optical character recognition models; Jointly train a low-light image enhancement network and an optical character recognition model, including: Pre-training the optical character recognition model using an image dataset under normal lighting conditions, freezing all trainable parameters in the optical character recognition model after the pre-training is completed, and updating the parameters of the low-light image enhancement network so that the low-light image enhancement network outputs a high-light image that can be recognized by the optical character recognition model, and inputting the high-light image into the optical character recognition model to obtain an optical character recognition result; The joint training loss used in the joint training is the weighted sum of the reconstruction loss of the low-light image enhancement network, the first feature perception loss of the optical character recognition model, the classification loss of the optical character recognition model, and the label loss of the optical character recognition model.

2. The low-light food packaging image recognition method according to claim 1, characterized in that: The low-light image enhancement network includes a downsampling module, an upsampling module and a reconstruction module; A downsampling module is used to perform convolution processing, local and global feature extraction processing, and multi-scale feature downsampling processing on the input low-light image to obtain a first multi-scale feature map; the downsampling processing of each scale feature includes downsampling convolution processing, or the downsampling processing of each scale feature includes downsampling convolution processing, local and global feature extraction processing; An upsampling module is used to perform transposed convolution processing on the input low-light image and the first multi-scale feature map to obtain a first feature map, and then a multi-scale feature upsampling processing is performed on the first feature map to obtain a second multi-scale feature map; the upsampling processing of each scale feature includes local and global feature extraction processing, feature enhancement processing, or the upsampling processing of each scale feature includes local feature and global feature extraction, feature enhancement processing, and transposed convolution processing, and the input features of the upsampling processing of each scale feature also include extracted local features and global features of different scales; The reconstruction module is used to perform local and global feature extraction and convolution processing on the second multi-scale feature map to obtain a high-light image.

3. The low-light food packaging image recognition method according to claim 2, characterized in that: The local and global feature extraction process includes: The input features are input into the convolutional neural network branch and the Transformer branch respectively. The convolutional neural network branch performs multiple convolutions on the input features. The first intermediate feature map obtained by the convolution is downsampled to obtain a second intermediate feature map with the same dimension as the Transformer branch. The second intermediate feature map is input into the Transformer branch. The Transformer branch performs feature extraction and outputs a third intermediate feature map. The third intermediate feature map is transposed and convolved to obtain a fourth intermediate feature map with the same dimension as the CNN branch. The convolutional neural network branch performs multiple convolutions on the fourth intermediate feature map to output local and global features.

4. The low-light food packaging image recognition method according to claim 2, characterized in that: The feature enhancement process includes: The input features are processed by multi-scale convolution and then spliced. The spliced ​​feature maps are input into the convolution block attention module for multi-scale feature fusion to obtain refined features.

5. The low-light food packaging image recognition method according to claim 1, characterized in that: The optical character recognition model includes a text detection network and a character recognition network. The text detection network performs text detection on an input high-light image and extracts a character image. The character recognition subnetwork recognizes the input character image to obtain a text recognition result.

6. The low-light food packaging image recognition method according to claim 5, characterized in that: The text detection network is a differentiable binarization network, and the character recognition network is a convolutional recurrent neural network; The differentiable binarization network extracts multi-scale features from the input high-light image to obtain a multi-scale feature map, upsamples the multi-scale feature map to obtain a feature map of the same size, performs feature enhancement on the feature maps of the same size and then cascades them to obtain a fused feature map; predicts the probability map and threshold map of the text area based on the fused feature map, calculates an approximate binary map based on the probability map and the threshold map, generates a bounding box of the text based on the approximate binary map, and obtains a character image; The feature enhancement process includes: The input features are processed by multi-scale convolution and then spliced. The spliced ​​feature maps are input into the convolution block attention module for multi-scale feature fusion to obtain refined features. The convolutional recurrent neural network performs convolution and pooling processing on the input character image, serializes the extracted local features to obtain a feature vector, captures the character dependency and context information in the feature vector, decodes the character dependency and context information, and obtains the text recognition result.

7. The low-light food packaging image recognition method according to claim 1, characterized in that: Reconstruction loss of the low-light image enhancement network The calculation expression is: ； in, is the color space difference loss, is the hyperparameter of color space difference loss; is the image frequency domain difference loss, is the hyperparameter of the image frequency domain difference loss; Second feature perceptual loss for image enhancement in low-light image enhancement networks, is the hyperparameter of the second perceptual loss; is the image structure similarity loss, is the hyperparameter of the image structure similarity loss.

8. The low-light food packaging image recognition method according to claim 1, characterized in that: The first feature perceptual loss of the optical character recognition model The calculation expression is: ； ； in, represents the first feature-perceptual loss, which is the cumulative sum of multiple feature-perceptual loss items; Represents single feature perception loss; represents the mean absolute error loss function; It is the feature representation of the enhanced high-light image; It is the feature representation of the normal illumination image; Indicates the number of feature extraction stages.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the low-light food packaging image recognition method according to any one of claims 1 to 8 are implemented.

10. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, characterized in that: When the processor runs the computer instructions, the processor performs the steps of the low-light food packaging image recognition method according to any one of claims 1 to 8.

Images