Image coloring method, electronic device, storage medium and computer program product

Abstract

The embodiments of the present application disclose an image colorization method, an electronic device, a storage medium and a computer program product, the method comprising: receiving an image to be colored; expressing the image to be colored at different scales to obtain images to be colored at different scales corresponding to at least two Luts, wherein each Lut records a mapping relationship between multiple traversed image blocks and numerical values ​​corresponding to each channel of a single pixel, and the traversed image blocks in each Lut have the same size; obtaining at least two color images of different scales according to the images to be colored at different scales and the at least two Luts; and fusing the at least two color images of different scales to obtain a colored target image.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to an image coloring method, electronic equipment, storage medium and computer program product. Background Art

[0002] Image colorization technology is a technology that restores color images from grayscale images. For example, when there is only the Y channel value, the R channel value, the G channel value, and the B channel value are predicted. Existing technologies mainly rely on deep learning, such as using instance segmentation with convolutional neural networks to generate image colors, or using generative adversarial neural networks to generate image colors. However, because existing technologies often pursue effects too much, they require a large number of GPUs to implement, resulting in image colorization technology being unable to be implemented on low-power devices. Summary of the invention

[0003] The embodiments of the present application provide an image colorization method, an electronic device, a storage medium, and a computer program product to solve the technical problem that the image colorization technology in the prior art cannot be implemented on low-power devices.

[0004] According to a first aspect of the present application, a method for coloring an image is disclosed, the method comprising:

[0005] Receiving an image to be colored;

[0006] The image to be colored is expressed at different scales to obtain images to be colored at different scales corresponding to at least two Luts, wherein each Lut records a mapping relationship between multiple traversed image blocks and values ​​corresponding to each channel of a single pixel, and the traversed image blocks in each Lut have the same size;

[0007] Obtaining at least two color images of different scales according to the to-be-colored images of different scales and the at least two Luts;

[0008] The at least two color images of different scales are fused to obtain a colored target image.

[0009] According to a second aspect of the present application, an image coloring device is disclosed, the device comprising:

[0010] A receiving module, used for receiving an image to be colored;

[0011] A transformation module, used for expressing the image to be colored at different scales to obtain images to be colored at different scales corresponding to at least two Luts, wherein each Lut records a mapping relationship between multiple traversed image blocks and values ​​corresponding to each channel of a single pixel, and the traversed image blocks in each Lut have the same size;

[0012] A table lookup module, used for obtaining at least two color images of different scales according to the to-be-colored images of different scales and the at least two Luts;

[0013] The fusion module is used to fuse the at least two color images of different scales to obtain a colored target image.

[0014] According to a third aspect of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the image coloring method as in the first aspect.

[0015] According to a fourth aspect of the present application, a computer-readable storage medium is disclosed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the image colorization method as in the first aspect is implemented.

[0016] According to a fifth aspect of the present application, a computer program product is disclosed, comprising a computer program / instruction, which implements the image coloring method in the first aspect when executed by a processor.

[0017] In an embodiment of the present application, after receiving the image to be colored, the image to be colored can be expressed at different scales to obtain images to be colored of different scales corresponding to at least two Luts, wherein each Lut records the mapping relationship between multiple traversed image blocks and the numerical values ​​corresponding to each channel of a single pixel, and the size of the traversed image blocks in each Lut is the same; at least two color images of different scales are obtained according to the images to be colored of different scales and at least two Luts; the at least two color images of different scales are fused to obtain the colored target image. Compared with the prior art, in an embodiment of the present application, since the storage space occupied by the at least two Luts used in coloring the image is limited, the table lookup process also occupies a limited amount of GPU, and the information recorded in the at least two Luts can accurately and comprehensively characterize the multiple colors between the local blocks of the image, the image coloring technology is implemented on low-power devices while ensuring the correctness of the image coloring result. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of a small receptive field network training process of an embodiment of the present application;

[0019] Figure 2 This is an example diagram of an embodiment of the present application showing the same receptive field on images of different scales;

[0020] Figure 3 This is an example diagram of a network structure of a small receptive field network according to an embodiment of the present application;

[0021] Figure 4 This is an example diagram of a small receptive field network training process according to an embodiment of the present application;

[0022] Figure 5 is a flowchart of a Lut generation process of an embodiment of the present application;

[0023] Figure 6 This is an example diagram of the Lut generation process of one embodiment of the present application;

[0024] Figure 7 is a flow chart of an image coloring method according to an embodiment of the present application;

[0025] Figure 8 is an example diagram of a table lookup process of an embodiment of the present application;

[0026] Fig. 9 is an example diagram of an image coloring method according to an embodiment of the present application;

[0027] Fig.10 is a schematic structural diagram of an image coloring device according to an embodiment of the present application;

[0028] Fig.11 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0031] In recent years, research on computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence has made important progress. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving many types of technologies such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks, etc. Computer vision, as an important branch of artificial intelligence, specifically allows machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, automatic driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, and smart temperature measurement.

[0032] Taking the field of image processing as an example, at present, image colorization technology is a common technology in the field of image processing. It is a technology that restores color images from grayscale images. For example, when there is only a Y channel value, the R channel value, G channel value and B channel value are predicted.

[0033] Existing technologies mainly rely on deep learning, such as using instance segmentation with convolutional neural networks to generate image colors, or using generative adversarial neural networks to generate image colors. However, because existing technologies often pursue effects too much, they require a large number of GPUs to achieve, resulting in image colorization technology being unable to be implemented on low-power devices.

[0034] In order to solve the above technical problems, the embodiments of the present application provide an image colorization method, an electronic device, a storage medium and a computer program product.

[0035] To facilitate understanding, the overall processing flow of the technical solution provided in the embodiment of the present application is first introduced.

[0036] In order to solve the problem that image colorization technology cannot be implemented on low-power devices, the technical solution provided in the embodiment of the present application mainly adopts the following technical means:

[0037] 1. Pre-train at least two small receptive field networks;

[0038] 2. Based on at least two small receptive field networks obtained through training, at least two Luts (Look up tables) are generated, wherein each Lut records the mapping relationship between multiple traversed image blocks and the values ​​corresponding to each channel of a single pixel point, and the traversed image blocks in each Lut have the same size;

[0039] 3. Store at least two Luts on a low-power device. When coloring is required, for each Lut, traverse each image block in the image to be colored, find the values ​​corresponding to each channel of the single pixel point corresponding to each image block in the Lut, obtain the table lookup result of each Lut, merge multiple table lookup results, and obtain the final colored image.

[0040] In the embodiment of the present application, the traversed image block is a grayscale image block, that is, the pixels in the traversed image block correspond to only one channel, and the value of the channel is a grayscale value. For ease of description, "image block" may be used instead of "traversed image block" for description.

[0041] In the embodiment of the present application, compared with a grayscale image in which one pixel corresponds to one channel, a color image in which one pixel corresponds to multiple channels, and each channel has a specific value, that is, the value corresponding to each channel.

[0042] For example, for an image in RGB format, the values ​​corresponding to each channel of a pixel point refer to: the value of the R channel, the value of the G channel, and the value of the B channel of the pixel point.

[0043] For example, for an image in YUV format, the values ​​corresponding to each channel of a pixel point refer to: the value of the Y channel, the value of the U channel, and the value of the V channel of the pixel point.

[0044] The above model training process and the process of generating Lut can be executed on a device with strong computing power, and the at least two Luts generated are stored on a low-power device. Since the model training process and the process of generating Lut do not need to be executed on a low-power device, and the storage space occupied by Lut is limited, and the GPU occupied by the table lookup process is also limited, the image colorization technology can be implemented on a low-power device.

[0045] In the embodiment of the present application, the reason for using a small receptive field network is that it is directly related to the size of the final generated Lut. If the Lut is large, the storage space of the device it occupies is also large, and the GPU occupied by the table lookup process is also large, which makes it difficult to land on low-power devices. The Lut generated based on the small receptive field network is relatively small, and the storage space it occupies is also small, so it can be landed on low-power devices.

[0046] In the embodiment of the present application, the reason for using at least two small receptive field networks is that: it is not possible to use only one small receptive field network, because the receptive field is small, and a point in the feature map obtained by the network corresponds to a relatively small area of ​​the original image, resulting in no difference between images of different grayscale blocks for the network, and failure to learn multiple colors. In order to overcome this defect, it is necessary to train at least two small receptive field networks. As for how many to train, you can refer to the storage and computing capabilities of low-power electronic devices. If the capabilities are relatively strong and can store multiple Luts, you can train more small receptive field networks; or, if the size of the sample image in the training data is relatively small, the resolution of the image is relatively low after multiple downsampling, the details of the image cannot be seen clearly, and accurate features cannot be extracted. It becomes meaningless to train many networks. At this time, you can train fewer small receptive field networks.

[0047] In the embodiment of the present application, the size and shape of the traversed image block need to be the same as the size and shape of the receptive field of the small receptive field network.

[0048] In the embodiment of the present application, one table lookup result corresponds to one colored color image, and at least two colored color images are fused to obtain a final colored image.

[0049] Next, the training process of the small receptive field network mentioned above, the process of generating Lut based on the small receptive field network, and the process of image coloring based on Lut are introduced in turn.

[0050] Figure 1 is a flowchart of a small receptive field network training process of an embodiment of the present application, such as Figure 1 As shown, the following steps may be included: step 101, step 102, step 103 and step 104, wherein,

[0051] In step 101, a training set is obtained, and sample images in the training set are expressed at different scales to obtain sample images of at least two scales, wherein the sample images include grayscale sample images and corresponding color sample images.

[0052] In the embodiment of the present application, the training set includes multiple sample images, each of which includes a grayscale sample image and a corresponding color sample image. In practical applications, in order to ensure the accuracy of the training results, the training set may include a large number of sample images; in order to ensure the accuracy of the subsequent image coloring effect, the sample images may be high-resolution images, such as 4K images; or, in order to reduce the training cost, low-resolution images may also be used, which is not limited in the embodiment of the present application.

[0053] The “multi-scale” mentioned in the embodiments of the present application refers to: an original scale and a scale smaller than the original scale.

[0054] Considering the disadvantage of a small receptive field network, the network cannot learn multiple colors because the receptive field is too small. Images with different grayscale blocks have no difference to the network, so it cannot learn multiple colors. A common example is that the handle of an eggplant is green, but the network has no difference between the purple and green parts of the eggplant, which results in the color of the eggplant handle being purple. Considering that reducing the scale of the image is equivalent to increasing the receptive field, it is more conducive to learning the correct color, such as Figure 2 shown.

[0055] In view of this, in an embodiment of the present application, in order to learn the correct colors in the image, the "receptive field" can be increased from the perspective of training data. Specifically, the sample images in the same training set can be expressed at different scales to obtain sample images of at least two scales. At least two small receptive field networks are trained based on the sample images of at least two scales, thereby ensuring that the network can correctly learn all colors in the sample image.

[0056] In the embodiment of the present application, the sample images in the training set may be downsampled by different multiples to obtain expressions of the sample images at different scales.

[0057] For example, taking a sample image Y_0 in the training set as an example, Y_0 is downsampled by 4 times to obtain Y_1, and Y_0 is downsampled by 16 times to obtain Y_2. At this time, the expression of Y_0 at three scales can be obtained: Y_0 at the original scale, Y_0 at 1 / 4 scale (i.e. Y_1), and Y_0 at 1 / 16 scale (i.e. Y_2). The same processing is performed on the remaining sample images in the training set to obtain the expression of each sample image in the training set at different scales.

[0058] In step 102, an initial network and an initial loss function corresponding to the sample image of each scale are constructed, wherein the receptive fields of different initial networks are the same and the value of the receptive field is less than a preset value.

[0059] In the embodiment of the present application, different initial networks are constructed for sample images of different scales to train different small receptive field networks.

[0060] In the embodiment of the present application, the relationship between different initial networks can be summarized as follows:

[0061] 1. The network structures of different initial networks are generally the same. Specifically, the size and shape of the receptive field of each initial network are the same, and the value of the receptive field is less than the preset value, to ensure that the size and shape of the receptive field of each small receptive field network obtained by training are the same, so that the format of each Lut table generated based on each small receptive field network is the same, which is convenient for subsequent table lookup for image coloring. In practical applications, the preset value can be 4 or 5. Due to the different scales of the sample images, the network structures of different initial networks have some differences in details. For example, as the scale of the sample image decreases, the depth of the constructed initial network gradually becomes shallower.

[0062] In some embodiments, the general network structure of the initial network can be Figure 3 The network structure shown in the figure has a receptive field value of 4. The network structure includes: multiple convolutional layers, wherein the convolution kernel of the first convolutional layer has a structure size (Kernel Size) of 2×2, a stride (Stride) of 1, and a padding (Padding) of 0. The convolution kernels of the remaining convolutional layers have a structure size of 1, a stride of 1, and a padding of 0.

[0063] Before inputting a grayscale image of size (H, W, 1) into the network of this network structure, it is necessary to fill a circle of 0s on the upper and left edges of the image so that the size of the input image becomes (H+1, W+1, 1). (H+1, W+1, 1) is processed by the first convolution layer to obtain an image of size (H, W, 16). The image of size (H, W, 16) is processed by the second convolution layer to obtain an image of size (H, W, 32), and so on. Finally, the network outputs a result image of size (H, W, 3), that is, a color image. Among them, H in (H, W, N) represents the height of the image, W represents the width of the image, and N represents the number of channels.

[0064] It is understandable that due to Figure 3 In the network structure shown, the size of the convolution kernel of the first convolution layer is 2×2, and the sizes of the convolution kernels of the remaining convolution layers are all 1, so the receptive field of the network structure is 4. If you want to construct a network with a receptive field of 5, you can modify the size of the convolution kernel of the first convolution layer in the network structure to 1×5 or 5×1, and the sizes of the convolution kernels of the remaining convolution layers remain unchanged and remain 1.

[0065] 2. Initial weight coefficients of different initial networks: Before training, the weight coefficients need to be initialized. The initialization strategy is the same, so the initial weight coefficients can be consistent.

[0066] 3. When constructing different initial networks, corresponding loss functions will be constructed for each network. However, during training, all constructed loss functions need to be weighted as the loss function used in the actual training process. The reason is: if each initial network is trained separately, each initial network uses its corresponding loss function for training, that is, each network is trained separately, which can only ensure that the small receptive field network obtained by each training is in the best state, and cannot ensure that all networks reach the best state at the same time. Therefore, it is necessary to train all the initial networks together to ensure that all networks are in the best state when used at the same time, that is, the overall effect is the best. The means used for training together is to merge the initial loss functions of each initial network into a new loss function, and all initial networks are trained using the new loss function.

[0067] In step 103, a target loss function is generated based on the constructed initial loss function.

[0068] In the embodiment of the present application, considering that the actual receptive field of the small-scale sample image is larger for the network, the learned area in the image is wider and the range is larger, so when generating the target loss function, the weight of the initial loss function corresponding to the small-scale sample image can be set to be larger, and the weight of the initial loss function corresponding to the large-scale sample image can be set to be smaller. In this case, the above step 103 can specifically include the following steps:

[0069] A weighted sum operation is performed on all constructed initial loss functions to obtain the target loss function, where the smaller the scale of the sample image, the larger the weight value of the corresponding initial loss function.

[0070] In step 104, the grayscale sample images in the sample images of each scale are input into the corresponding initial network for processing, and the corresponding predicted color images are output. Based on the predicted color images, the color sample images corresponding to the grayscale sample images, and the target loss function, the initial network is trained until the model converges, and a small receptive field network corresponding to the sample images of each scale is obtained.

[0071] In the embodiment of the present application, when the model is actually trained, the loss functions used by all initial networks are target loss functions rather than their respective initial loss functions.

[0072] In some embodiments, when the network structure of the initial network is Figure 3 When the network structure is shown, the network training process can be as follows:

[0073] 1. Input a grayscale image of size (H+1,W+1,1), pass it through 16 convolution kernels of size (2,2,1), perform convolution with a step size of 1 to obtain a feature map of size (H,W,16), the feature map passes through the relu activation function, and then passes through the BN layer to obtain a feature map of size (H,W,16);

[0074] 2. The feature map of the previous step with a size of (H, W, 16) is convolved with 32 convolution kernels of size (1, 1, 16) with a step size of 1 to obtain a feature map of size (H, W, 32). The feature map passes through the relu activation function and then passes through the BN layer to obtain a feature map of size (H, W, 32);

[0075] 3. The feature map of the previous step with a size of (H, W, 32) is convolved with 64 convolution kernels of size (1, 1, 32) with a step size of 1 to obtain a feature map of size (H, W, 64). The feature map passes through the relu activation function and then passes through the BN layer to obtain a feature map of size (H, W, 64);

[0076] 4. The feature map of the previous step with a size of (H, W, 64) is convolved with 128 convolution kernels of size (1, 1, 64) with a step size of 1 to obtain a feature map of size (H, W, 128). The feature map passes through the relu activation function and then passes through the BN layer to obtain a feature map of size (H, W, 128);

[0077] 5. The feature map of the previous step with a size of (H, W, 128) is convolved with 64 convolution kernels of size (1, 1, 128) with a step size of 1 to obtain a feature map of size (H, W, 64). The feature map passes through the relu activation function and then passes through the BN layer to obtain a feature map of size (H, W, 64);

[0078] 6. The feature map of the previous step with a size of (H, W, 64) is convolved with 32 convolution kernels of size (1, 1, 64) with a step size of 1 to obtain a feature map of size (H, W, 32). The feature map passes through the relu activation function and then passes through the BN layer to obtain a feature map of size (H, W, 32);

[0079] 7. The feature map of the previous step with a size of (H, W, 32) is convolved with 16 convolution kernels of size (1, 1, 32) with a step size of 1 to obtain a feature map of size (H, W, 16). The feature map passes through the relu activation function and then passes through the BN layer to obtain a feature map of size (H, W, 16);

[0080] 8. The feature map of the previous step with a size of (H, W, 16) is convolved with 3 convolution kernels of size (1, 1, 16) with a step size of 1 to obtain a result map of size (H, W, 3).

[0081] In an example, a color sample image in RGB format is taken as an example, Figure 4 As shown, three small receptive field networks can be trained. For the sample grayscale image Y_0 of the original scale, it can be downsampled by 4 times to obtain Y_1 and 16 times to obtain Y_2, where Y_0, Y_1 and Y_2 are the expressions of Y_0 at three scales.

[0082] Afterwards, a small receptive field network Net_0 corresponding to Y_0, a small receptive field network Net_1 corresponding to Y_1, and a small receptive field network Net_2 corresponding to Y_2 are constructed, where the network structures of Net_0, Net_1 and Net_2 are roughly the same, and the values ​​of the initial weight coefficients are roughly the same.

[0083] Finally, train Net_0, Net_1 and Net_2 at the same time: input Y_0 into Net_0 for processing, output the predicted color image RGB_0, calculate the loss of the color sample images corresponding to RGB_0 and Y_0 according to the target loss function, update the weight coefficient in Net_0, and repeat the above process until the model converges to obtain a trained small receptive field network.

[0084] Input Y_1 into Net_1 for processing, output the predicted color image RGB_1, calculate the loss of the color sample images corresponding to RGB_1 and Y_1 (that is, 4 times downsampling of the color sample image corresponding to Y_0) according to the target loss function, update the weight coefficient in Net_1, and repeat the above process until the model converges to obtain a trained small receptive field network.

[0085] Input Y_2 to Net_2 for processing, output the predicted color image RGB_2, calculate the loss of the color sample images corresponding to RGB_2 and Y_2 (i.e., 16 times downsampling of the color sample image corresponding to Y_0) according to the target loss function, update the weight coefficient in Net_2, and repeat the above process until the model converges to obtain a trained small receptive field network. So far, three trained small receptive field networks have been obtained.

[0086] It can be seen from the above embodiment that in this embodiment, at least two small receptive field networks are obtained by training based on the expression of sample images of the same training set at different scales, which can overcome the problem that a single small receptive field network cannot distinguish the differences between local blocks of the image, and can learn multiple colors between local blocks of the image. Therefore, it can ensure the correctness of the image coloring result when the at least two Luts generated based on the at least two small receptive field networks are used to color the image.

[0087] Figure 5is a flowchart of the Lut generation process of an embodiment of the present application, such as Figure 5 As shown, the following steps may be included: step 501, step 502 and step 503, wherein:

[0088] In step 501, a traversal image block having the same receptive field size and shape as the small receptive field network is constructed.

[0089] In the embodiment of the present application, each traversed image block contains one or more pixel points.

[0090] In the embodiment of the present application, the size and shape of the traversed image blocks are the same as the receptive field size and shape of the small receptive field network. For example, when the receptive field is 2×2, the arrangement of the image blocks is also 2×2.

[0091] In step 502, the grayscale value of each pixel in the traversed image block is assigned a value of 0 to 255, and the traversed image blocks after the assignment are input into each trained small receptive field network for processing, and the numerical value corresponding to each channel of a single pixel is output.

[0092] In the embodiment of the present application, when assigning grayscale values ​​to the traversed image blocks, it should be ensured that the grayscale values ​​of each pixel in the traversed image blocks cover all situations.

[0093] For example, when the image blocks are arranged in 2×2, the grayscale value of each pixel in the image block should cover 4 256 The specific situations are:

[0094]

[0095] In the embodiment of the present application, the input is a grayscale image block, and the output is the numerical value corresponding to each channel of a single pixel. The reason is: taking a network with a 2×2 receptive field as an example, corresponding to the network learning process, the network is learned through a 2×2 receptive field. When the network is converted into a table, the conversion process must be consistent with the network, taking the original image as input and the feature map as an output, outputting a pixel point, corresponding to the 2×2 pixel points in the original image.

[0096] In step 503, for each small receptive field network obtained through training, a mapping relationship is established between each traversed image block after assignment and the numerical values ​​corresponding to each channel of a single pixel point output by the small receptive field network, and the mapping relationship is recorded in Lut to obtain the Lut corresponding to each small receptive field network.

[0097] In one example, there are three networks with small receptive fields, namely Net_0, Net_1 and Net_2, and the receptive fields are all 2×2. Correspondingly, the arrangement of image blocks is also 2×2, and the grayscale value of each pixel in the image block should cover 4 256The specific situations are:

[0098]

[0099] like Figure 6 As shown, the above 4 256 The image blocks of each case are input into Net_0 for processing, and 4 256 The mapping relationship between the image block in the above case and the values ​​corresponding to each channel of a single pixel is stored in the lookup table Lut_0. 256 The image blocks of each case are input into Net_1 for processing, and 4 256 The mapping relationship between the image block in the above case and the values ​​corresponding to each channel of a single pixel is stored in the lookup table Lut_1. 256 The image blocks of each case are input into Net_2 for processing, and 4 256 The mapping relationship between the image block in this case and the values ​​corresponding to each channel of a single pixel point is stored in the lookup table Lut_2.

[0100] Taking a small receptive field network with a receptive field of 2×2 as an example, the size of Lut generated by each small receptive field network is (256, 256, 256, 256, 3), where the first four items represent the grayscale values ​​of the image block, and the last item represents the values ​​of the three channels of the corresponding single pixel.

[0101] according to Figure 6 The example diagram shown can obtain three tables Lut_0, Lut_1 and Lut_2. Considering that if the original table is used, the size may be too large and occupy too much storage space, when it is actually stored on a low-power electronic device, the Lut corresponding to each small receptive field network can be reduced to obtain the corresponding reduced Lut. For example, a value is stored every 16 intervals, so that the size of the table is reduced to (17,17,17,17,3). When it is necessary to look up the table for coloring, the reduced table can be interpolated and the table can be looked up based on the interpolated table.

[0102] It is understandable that the larger the receptive field, the higher the dimension of the generated lookup table Lut. For example, when the receptive field is 2×2, the size of the table is (256,256,256,256,3); when the receptive field is 5, the size of the table is (256,256,256,256,256,3), and the size of the table increases geometrically. It also further illustrates that a small receptive field network is needed to generate Lut.

[0103] As can be seen from the above embodiment, in this embodiment, at least two small receptive field networks can be pre-trained based on sample images of at least two scales, and at least two small receptive field networks obtained through training traverse image blocks of specified grayscale values ​​to generate at least two Luts, each Lut records the mapping relationship between multiple traversed image blocks and the numerical values ​​corresponding to each channel of a single pixel point, and then the generated at least two Luts are stored in a low-power device. When coloring is required, for each Lut, traverse each image block in the image to be colored, search the Lut table for the numerical values ​​corresponding to each channel corresponding to each image block, obtain the table lookup results of each Lut table, and merge multiple table lookup results to obtain the final colored image.

[0104] Compared with the prior art, in the embodiment of the present application, since at least two Luts used in coloring the image are generated based on at least two small receptive field networks, and at least two small receptive field networks are obtained based on the expression training of sample images of the same training set at different scales, the problem that a single small receptive field network cannot distinguish the differences between local blocks of the same grayscale image can be overcome, that is, multiple colors between local blocks of the grayscale image can be learned, so the correctness of the image coloring result can be ensured. In addition, the model training process and the process of generating Lut can be executed on a device with strong computing power, and the generated Lut can be stored on a low-power device. Since the model training process and the process of generating a table do not need to be executed on a low-power device, and the storage space occupied by the generated Lut is limited, and the table lookup process also occupies a limited amount of GPU, while ensuring the correctness of the image coloring result, the image coloring technology is implemented on a low-power device.

[0105] Figure 7 is a flow chart of an image coloring method according to an embodiment of the present application. Figure 7 As shown, the method may include the following steps: step 701, step 702 and step 703, wherein:

[0106] In step 701, an image to be colored is received.

[0107] In step 702, the image to be colored is expressed in different sizes to obtain images to be colored in different scales corresponding to at least two Luts.

[0108] In the embodiment of the present application, each Lut records the mapping relationship between multiple traversed image blocks and the numerical values ​​corresponding to each channel of a single pixel point, and the sizes of the traversed image blocks in each Lut are the same.

[0109] In the embodiment of the present application, Lut can be Figure 5The lookup table generated in the method of the illustrated embodiment, specifically, different Luts are obtained by traversing image blocks of specified grayscale values ​​through different small receptive field networks, wherein the receptive fields of different small receptive field networks are the same and the values ​​of the receptive fields are less than preset values, and different small receptive field networks are obtained based on expression training of sample images in the same training set at different scales, and the size and shape of each traversed image block are the same as the size and shape of the receptive field of the small receptive field network.

[0110] In the embodiment of the present application, since at least two Luts are generated based on different small receptive field networks, and different small receptive field networks are trained based on sample images of different scales, when performing a table lookup, it is first necessary to adjust the scale of the image to be colored to the scale of the sample image corresponding to each Lut, and then perform a table lookup.

[0111] In the embodiment of the present application, downsampling can be used to express the colored image in different sizes.

[0112] In step 703, at least two color images of different scales are obtained according to the to-be-colored images of different scales and at least two Luts.

[0113] In an embodiment of the present application, for each scale of the image to be colored, the values ​​corresponding to the channels corresponding to each pixel can be searched from the corresponding Lut according to the image blocks corresponding to each pixel in the image to be colored at each scale, and at least two color images of different scales can be obtained based on the values ​​corresponding to the channels corresponding to the found pixels.

[0114] In an embodiment of the present application, when at least two Luts stored locally are reduced Luts, interpolation operations can be performed on the reduced Luts to obtain corresponding interpolated Luts; at least two color images of different scales are obtained based on the images to be colored of different scales and the interpolated Luts.

[0115] In an example, Fig. 9 As shown, the image to be colored is P1, and the local storage is Figure 6The three tables Lut_0, Lut_1 and Lut_2 shown in the figure, when looking up the tables, according to the image blocks corresponding to each pixel in P1, the values ​​corresponding to each channel of each pixel (such as RGB values) are searched from Lut_0, and the color image Res_0 is generated according to the values ​​corresponding to each channel of each pixel found; P1 is downsampled by 4 times to obtain P2, according to the image blocks corresponding to each pixel in P2, the values ​​corresponding to each channel of each pixel are searched from Lut_1, and the color image Res_1 is generated according to the values ​​corresponding to each channel of each pixel found; P1 is downsampled by 16 times to obtain P3, according to the image blocks corresponding to each pixel in P3, the values ​​corresponding to each channel of each pixel are searched from Lut_2, and the color image Res_2 is generated according to the values ​​corresponding to each channel of each pixel found; so far, the color images Res_0, Res_1 and Res_2 of three scales are obtained.

[0116] For ease of understanding, take the colored image as an RGB format image as an example, combined with Figure 8 The grayscale image shown describes the process of table lookup based on image blocks.

[0117] like Figure 8 As shown in the figure, the scale of the grayscale image is 5×5 (5×5 is used as an example for ease of description, and the image size in actual applications is much larger than 5×5). The RGB value of the pixel at position (0,0) in the image is obtained by the image block and Lut composed of the four pixels at positions (0,0), (0,1), (1,0) and (1,1) in the image; the RGB value of the pixel at position (0,1) in the image is obtained by the image block and Lut composed of the four pixels at positions (0,1), (0,2), (1,1) and (0,2); similarly, the RGB values ​​of the pixels at positions (0,2) and (0,3) in the image can be obtained; the RGB value of the pixel at position (0,4) in the image can be obtained by first adding a column of 0s on the right and then looking up the table; or the column (0,4) can be cut off.

[0118] In step 704, at least two color images of different scales are fused to obtain a colored target image.

[0119] In an embodiment of the present application, considering that the connected area in the image is usually a larger area, a complete area, for example, the area where the eggplant is located in the image is a connected area, in order to improve the accuracy of the coloring effect, the connected area information of the images to be colored of different scales can be obtained, and according to the connected area information, at least two color images of different scales are fused to obtain the colored target image.

[0120] In some embodiments, when color images of different scales are fused according to the connected region information of the image to be colored, considering that the result obtained by learning on the small-scale image is more accurate than that of the large-scale image, when performing image fusion, for the connected region, the small-scale color image is preferentially used for filling. Accordingly, the above step 703 may specifically include the following steps (not shown in the figure): replacing the values ​​corresponding to each channel of the pixel points in the connected region of the image to be colored of the next smallest scale with the values ​​corresponding to each channel of the pixel points at the corresponding position of the color image of the smallest scale, and replacing the values ​​corresponding to each channel of the pixel points outside the connected region of the image to be colored of the next smallest scale with the values ​​corresponding to each channel of the pixel points at the corresponding position of the color image of the next smallest scale, to obtain a first fused image;

[0121] Replace the values ​​corresponding to the channels of the pixels in the connected area of ​​the previous scale image to be colored at the next smallest scale with the values ​​corresponding to the channels of the pixels at the corresponding positions of the first fused image, and replace the values ​​corresponding to the channels of the pixels outside the connected area of ​​the previous scale image to be colored at the next smallest scale with the values ​​corresponding to the channels of the pixels at the corresponding positions of the previous scale color image at the next smallest scale, to obtain a second fused image;

[0122] The same fusion process is applied to other large-scale color images in sequence until the image fusion is completed on the largest-scale color image to obtain the colored target image.

[0123] More specifically, a segmentation mask may be used to achieve fusion. In this case, the above step 703 may include the following steps (not shown in the figure): step 7031, step 7032, step 7033, step 7034 and step 7035, wherein:

[0124] In step 7031, at least two images P1, ..., P N The connected region detection is performed on the original scale image P1 to be colored, and the segmentation mask map Mask1 corresponding to the connected region in P1 is obtained, where P j The scale is larger than P j+1 The scale of , 1≤j≤N, N is the number of different scales.

[0125] In the embodiment of the present application, the pixel value of the area where the connected area in Mask1 is located is 1, and the pixel value of other areas is 0.

[0126] In step 7032, Mask1 is downsampled multiple times to obtain images P2, ..., P N-1 The corresponding segmentation mask images Mask2,…,Mask N-1 , where Mask i Pi The corresponding segmentation mask image, 2≤i≤N-1.

[0127] In step 7033, P in at least two color images of different scales is N The corresponding color image is upsampled to obtain the color image Q N , Q N The scale of P is the same as that of at least two color images of different scales. N-1 The corresponding color images have the same scale; use Mask N-1 Q N and P N-1 The corresponding color images are fused to obtain the first fused image, where Q N The fusion weight is Mask N-1 , P N-1 The corresponding color image fusion weight is 1-Mask N-1 .

[0128] In step 7034, the first fused image is upsampled to obtain a color image Q N-1 , Q N-1 The scale of P is the same as that of at least two color images of different scales. N-2 The corresponding color images have the same scale and use Mask N-2 Q N-1 and P N-2 The corresponding color images are fused to obtain the second fused image, where Q N-1 The fusion weight is Mask N-2 , P N-2 The corresponding color image fusion weight is 1-Mask N-2 .

[0129] In step 7035, the same fusion process is applied to at least two color images of different scales P in sequence. N-3 The color image corresponding to P1, …, the color image corresponding to P1, until the image fusion is completed on the color image corresponding to P1 to obtain the N-1th fused image, where the N-1th fused image is the colored target image.

[0130] In one example, Fig. 9 For example, Fig. 9 As shown, at least two scales of images to be colored are P1, P2 and P3. The original scale image P1 is passed through the lookup table Lut_0 to obtain the original scale color image RES_0, the 1 / 4 scale image P2 is passed through the lookup table Lut_1 to obtain the 1 / 4 scale color image RES_1, and the 1 / 16 scale image P3 is passed through the lookup table Lut_2 to obtain the 1 / 16 scale color image RES_2.

[0131] When performing fusion, firstly, the connected area of ​​P1 is detected to obtain the segmentation mask map Mask1 corresponding to the connected area in P1;

[0132] After that, Mask1 is downsampled 4 times to obtain the segmentation mask image Mask2 corresponding to the connected area in P2 (the reason for not directly performing connected area detection on the colored image P2 here is that the coloring process will eventually return to the original scale, and detecting connected area detection at the original scale is the most accurate);

[0133] After that, the color image RES_2 is upsampled 4 times and fused with RES_1. The fusion weight map is Mask2 (the color of the upsampled result of RES_2 is used inside the connected area), and the result RES_12 is obtained.

[0134] Finally, RES_12 is upsampled 4 times and fused with RES_0. The fusion weight map is Mask1 (the color of the result of RES_12 upsampling inside the connected area), and the result RES is obtained. RES is the final colored target image.

[0135] In an embodiment of the present application, in order to increase the fusion speed, a weighted sum operation may be performed on pixel values ​​of at least two color images of different scales to obtain a colored target image.

[0136] In one example, Fig. 9 For example, when RES_2 is upsampled 4 times and then fused with RES_1, a weight of 4 times is given to RES_2 and a weight of 1 times is given to RES_1, and a weighted sum operation is performed pixel by pixel to obtain RES_12; then, when RES_12 is upsampled 4 times and then fused with RES_0, a weight of 4 times is given to RES_12 and a weight of 1 times is given to RES_0, and a weighted sum operation is performed pixel by pixel to obtain the final coloring result RES.

[0137] It can be seen from the above embodiment that in this embodiment, since the storage space occupied by the at least two LUTs used in coloring the image is limited, the table lookup process occupies a limited amount of GPU space, and the information recorded in the at least two LUTs can accurately and comprehensively characterize the multiple colors between local blocks of the image, the image coloring technology is implemented on low-power devices while ensuring the correctness of the image coloring results.

[0138] In summary, in the embodiment of the present application, since at least two Luts used in coloring the image are generated based on at least two small receptive field networks, and at least two small receptive field networks are obtained based on the expression training of sample images of the same training set at different scales, the problem that a single small receptive field network cannot distinguish the differences between local blocks of the same grayscale image can be overcome, that is, multiple colors between local blocks of the grayscale image can be learned, so the correctness of the image coloring result can be ensured. In addition, the model training process and the process of generating Lut can be executed on a device with strong computing power, and the generated Lut can be stored on a low-power device. Since the model training process and the process of generating a table do not need to be executed on a low-power device, and the storage space occupied by the generated Lut is limited, and the table lookup process also occupies a limited amount of GPU, while ensuring the correctness of the image coloring result, the image coloring technology is implemented on a low-power device.

[0139] Fig.10 FIG. 1 is a schematic diagram of the structure of an image coloring device according to an embodiment of the present application. Fig.10 As shown, the image coloring device 1000 may include: a receiving module 1001, a transforming module 1002, a table lookup module 1003 and a fusion module 1004, wherein:

[0140] A receiving module 1001 is used to receive an image to be colored;

[0141] The transformation module 1002 is used to express the image to be colored in different scales to obtain images to be colored in different scales corresponding to at least two Luts, wherein each Lut records the mapping relationship between multiple traversed image blocks and the values ​​corresponding to each channel of a single pixel, and the traversed image blocks in each Lut have the same size;

[0142] A table lookup module 1003, configured to obtain at least two color images of different scales according to the to-be-colored images of different scales and the at least two Luts;

[0143] The fusion module 1004 is used to fuse the at least two color images of different scales to obtain a colored target image.

[0144] It can be seen from the above embodiment that in this embodiment, since the storage space occupied by the at least two LUTs used in coloring the image is limited, the table lookup process occupies a limited amount of GPU space, and the information recorded in the at least two LUTs can accurately and comprehensively characterize the multiple colors between local blocks of the image, the image coloring technology is implemented on low-power devices while ensuring the correctness of the image coloring results.

[0145] Optionally, as an embodiment, the fusion module 1004 may include:

[0146] The fusion submodule is used to obtain the connected area information of the to-be-colored images of different scales, and fuse the at least two color images of different scales according to the connected area information to obtain the colored target image.

[0147] Optionally, as an embodiment, the fusion submodule is specifically used for:

[0148] The values ​​corresponding to the channels of the pixels in the connected area of ​​the image to be colored at the next smallest scale are replaced with the values ​​corresponding to the channels of the pixels at the corresponding positions of the color image at the smallest scale, and the values ​​corresponding to the channels of the pixels outside the connected area of ​​the image to be colored at the next smallest scale are replaced with the values ​​corresponding to the channels of the pixels at the corresponding positions of the color image at the next smallest scale, to obtain a first fused image;

[0149] Replace the values ​​corresponding to the channels of the pixels in the connected area of ​​the previous scale image to be colored at the next smallest scale with the values ​​corresponding to the channels of the pixels at the corresponding positions of the first fused image, and replace the values ​​corresponding to the channels of the pixels outside the connected area of ​​the previous scale image to be colored at the next smallest scale with the values ​​corresponding to the channels of the pixels at the corresponding positions of the previous scale color image at the next smallest scale, to obtain a second fused image;

[0150] The same fusion process is applied to other large-scale color images in sequence until the image fusion is completed on the largest-scale color image to obtain the colored target image.

[0151] Optionally, as an embodiment, different Luts are obtained by traversing image blocks of specified grayscale values ​​through different small receptive field networks; wherein, the receptive fields of different small receptive field networks are the same and the values ​​of the receptive fields are less than a preset value, and the different small receptive field networks are obtained based on the expression training of sample images in the same training set at different scales, and the size and shape of each traversed image block are the same as the size and shape of the receptive field of the small receptive field network.

[0152] Optionally, as an embodiment, the image coloring device 1000 may further include:

[0153] The training module is used to: obtain a training set, express the sample images in the training set at different scales, and obtain sample images at at least two scales, wherein the sample images include grayscale sample images and corresponding color sample images;

[0154] Construct the initial network and initial loss function corresponding to the sample image of each scale, where the receptive fields of different initial networks are the same and the value of the receptive field is less than the preset value;

[0155] Generate the target loss function based on the constructed initial loss function;

[0156] The grayscale sample image in each scale of the sample image is input into the corresponding initial network for processing, and the corresponding predicted color image is output. Based on the predicted color image, the color sample image corresponding to the grayscale sample image and the target loss function, the initial network is trained until the model converges, so as to obtain a small receptive field network corresponding to the sample image of each scale.

[0157] Optionally, as an embodiment, the training module is specifically used to:

[0158] A weighted sum operation is performed on all constructed initial loss functions to obtain the target loss function, where the smaller the scale of the sample image, the larger the weight value of the corresponding initial loss function.

[0159] Optionally, as an embodiment, the image coloring device 1000 may further include:

[0160] A lookup table generation module, used to: construct a traversal image block with the same receptive field size and shape as the small receptive field network;

[0161] Assigning a grayscale value of 0 to 255 to each pixel in the traversed image block, inputting the traversed image blocks after the assignment into each trained small receptive field network for processing, and outputting the value corresponding to each channel of a single pixel;

[0162] For each small receptive field network obtained through training, a mapping relationship is established between each traversed image block after assignment and the numerical values ​​corresponding to each channel of a single pixel point output by the small receptive field network, and the mapping relationship is recorded in Lut to obtain the Lut corresponding to each small receptive field network.

[0163] Optionally, as an embodiment, the image coloring device 1000 may further include:

[0164] The reduction module is used to reduce the Lut corresponding to each small receptive field network to obtain the reduced Lut;

[0165] The table lookup module 1003 may include:

[0166] An interpolation submodule, used for performing interpolation operation on the reduced Lut to obtain the corresponding interpolated Lut;

[0167] The table lookup submodule is used to obtain at least two color images of different scales according to the to-be-colored images of different scales and the interpolated Lut.

[0168] Any step in the embodiment of the image coloring method provided in the present application and the specific operation in any step can be completed by the corresponding module in the image coloring device. The process of the corresponding operation completed by each module in the image coloring device refers to the process of the corresponding operation described in the embodiment of the image coloring method.

[0169] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0170] Fig.11 1 is a block diagram of an electronic device according to an embodiment of the present application. The electronic device includes a processing component 1122, which further includes one or more processors, and a memory resource represented by a memory 1132 for storing instructions executable by the processing component 1122, such as an application. The application stored in the memory 1132 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1122 is configured to execute instructions to perform the above method.

[0171] The electronic device may also include a power supply component 1126 configured to perform power management of the electronic device, a wired or wireless network interface 1150 configured to connect the electronic device to a network, and an input / output (I / O) interface 1158. The electronic device may operate based on an operating system stored in the memory 1132, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.

[0172] According to another embodiment of the present application, the present application further provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps in the image colorization method described in any of the above embodiments.

[0173] According to another embodiment of the present application, the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the image colorization method described in any of the above embodiments.

[0174] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0175] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0177] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0178] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0179] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0180] The above is a detailed introduction to an image coloring method, electronic device, storage medium and computer program product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image coloring method, characterized in that: The method comprises: Receiving an image to be colored; The image to be colored is expressed at different scales to obtain images to be colored at different scales corresponding to at least two Luts, wherein the image to be colored is expressed at different scales by sampling the image to be colored at different magnifications to obtain images at different magnifications, wherein each Lut records a mapping relationship between multiple traversed image blocks and values ​​corresponding to each channel of a single pixel, and the sizes of the traversed image blocks in each Lut are the same; Different Luts are obtained by traversing image blocks with specified grayscale values ​​through different small receptive field networks; Among them, the receptive fields of different small receptive field networks are the same and the values ​​of the receptive fields are less than a preset value, the different small receptive field networks are obtained based on the expression training of sample images in the same training set at different scales, and the size and shape of each traversed image block are the same as the size and shape of the receptive field of the small receptive field network; Obtaining at least two color images of different scales according to the to-be-colored images of different scales and the at least two Luts; The at least two color images of different scales are fused to obtain a colored target image.

2. The method according to claim 1, characterized in that The step of fusing the at least two color images of different scales to obtain a colored target image includes: The connected region information of the to-be-colored images of different scales is obtained, and the at least two color images of different scales are fused according to the connected region information to obtain a colored target image.

3. The method according to claim 2, characterized in that The step of fusing the at least two color images of different scales according to the connected region information to obtain a colored target image includes: The values ​​corresponding to the channels of the pixels in the connected area of ​​the image to be colored at the next smallest scale are replaced with the values ​​corresponding to the channels of the pixels at the corresponding positions of the color image at the smallest scale, and the values ​​corresponding to the channels of the pixels outside the connected area of ​​the image to be colored at the next smallest scale are replaced with the values ​​corresponding to the channels of the pixels at the corresponding positions of the color image at the next smallest scale, to obtain a first fused image; Replace the values ​​corresponding to the channels of the pixels in the connected area of ​​the previous scale image to be colored at the next smallest scale with the values ​​corresponding to the channels of the pixels at the corresponding positions of the first fused image, and replace the values ​​corresponding to the channels of the pixels outside the connected area of ​​the previous scale image to be colored at the next smallest scale with the values ​​corresponding to the channels of the pixels at the corresponding positions of the previous scale color image at the next smallest scale, to obtain a second fused image; The same fusion process is applied to other large-scale color images in sequence until the image fusion is completed on the largest-scale color image to obtain the colored target image.

4. The method according to claim 1, characterized in that: The training process of the different small receptive field networks includes: Acquire a training set, express sample images in the training set at different scales, and obtain sample images at at least two scales, wherein the sample images include grayscale sample images and corresponding color sample images; Construct the initial network and initial loss function corresponding to the sample image of each scale, where the receptive fields of different initial networks are the same and the value of the receptive field is less than the preset value; Generate the target loss function based on the constructed initial loss function; The grayscale sample image in each scale of the sample image is input into the corresponding initial network for processing, and the corresponding predicted color image is output. Based on the predicted color image, the color sample image corresponding to the grayscale sample image and the target loss function, the initial network is trained until the model converges, so as to obtain a small receptive field network corresponding to the sample image of each scale.

5. The method according to claim 4, characterized in that The target loss function is generated based on the constructed initial loss function, including: A weighted sum operation is performed on all constructed initial loss functions to obtain the target loss function, where the smaller the scale of the sample image, the larger the weight value of the corresponding initial loss function.

6. The method according to any one of claims 4-5, characterized in that: The generation process of the different Luts includes: Constructing a traversal image block having the same receptive field size and shape as the small receptive field network; Assigning a grayscale value of 0 to 255 to each pixel in the traversed image block, inputting the traversed image blocks after the assignment into each trained small receptive field network for processing, and outputting the value corresponding to each channel of a single pixel; For each small receptive field network obtained through training, a mapping relationship is established between each traversed image block after assignment and the numerical values ​​corresponding to each channel of a single pixel point output by the small receptive field network, and the mapping relationship is recorded in Lut to obtain the Lut corresponding to each small receptive field network.

7. The method according to claim 6, characterized in that After recording the mapping relationship into Lut and obtaining the Lut corresponding to each small receptive field network, the method further includes: Reduce the Lut corresponding to each small receptive field network to obtain the reduced Lut; The step of obtaining at least two color images of different scales according to the to-be-colored images of different scales and the at least two Luts includes: Performing interpolation operation on the reduced Lut to obtain the corresponding interpolated Lut; At least two color images of different scales are obtained according to the to-be-colored images of different scales and the interpolated Lut.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images