Image processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide an image processing method and device, electronic equipment and storage medium, and relate to the technical field of image processing. The method comprises: obtaining a target object's to-be-processed image (for example, a grayscale digital image); performing image feature extraction on the to-be-processed image to obtain a first feature map; determining coloring reference information of the to-be-processed image based on a color system to which the to-be-processed image belongs, and fusing the coloring reference information to the first feature map to obtain a second feature map having coloring reference information; coloring the second feature map based on a reference palette obtained by training a first training set to obtain an image coloring result of the to-be-processed image, the first training set comprising at least one color image of the target object, and the reference palette being a set of colors contained in the color image of the target object in the first training set. The embodiments of the present application can effectively improve the coloring accuracy of grayscale digital images, and further improve the coloring effect of the to-be-processed image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] As people's demands for image visual quality continue to increase, grayscale images are gradually being replaced by color images in many application scenarios. For example, in daily life, users expect to obtain clearer photos with more realistic colors. Therefore, black and white photos have gradually been replaced by color photos. Alternatively, for scientific imaging observation of marine plankton, replacing grayscale images with color images helps in the identification and classification of target species, thus improving the accuracy of users' observations of marine plankton.

[0003] It is understandable that some application scenarios where acquiring color images is difficult still exist. In these scenarios, due to the imaging principles, it may be impossible or inconvenient to directly perform true-color imaging of the target in the scene, yet the application ultimately requires a true-color image. Therefore, indirect methods can be used to obtain true-color images of the target of interest in these scenarios. For example, colorization algorithms can be used to colorize grayscale images. However, the colorization effect of the color image obtained by existing colorization algorithms often differs significantly from the true color effect of the target of interest, and cannot yet meet the needs of many subsequent applications, such as scientific imaging.

[0004] As can be seen from the above, how to improve the accuracy of coloring and enhance the coloring effect is a problem that still needs to be solved. Summary of the Invention

[0005] This application provides an image processing method, apparatus, electronic device, and storage medium, which can solve the problem of poor coloring effect in related technologies. The technical solution is as follows:

[0006] According to one aspect of the embodiments of this application, an image processing method includes: acquiring an image of a target object to be processed; extracting image features from the image to be processed to obtain a first feature map; determining coloring reference information of the image to be processed based on the color system to which the image to be processed belongs, and fusing the coloring reference information into the first feature map to obtain a second feature map having the coloring reference information; coloring the second feature map based on a reference color palette trained from a first training set to obtain an image coloring result of the image to be processed, wherein the first training set includes at least one color image of the target object, and the reference color palette is a set of colors contained in the color image of the target object in the first training set.

[0007] According to one aspect of the embodiments of this application, an image processing apparatus includes: an image acquisition module for acquiring an image to be processed of a target object; a feature extraction module for extracting image features from the image to be processed to obtain a first feature map; an information fusion module for determining coloring reference information of the image to be processed based on the color system to which the image to be processed belongs, and fusing the coloring reference information into the first feature map to obtain a second feature map having the coloring reference information; and an image coloring module for coloring the second feature map based on a reference color palette trained from a first training set to obtain an image coloring result of the image to be processed, wherein the first training set includes at least one color image of the target object, and the reference color palette is a set of colors contained in the color image of the target object in the first training set.

[0008] In an exemplary embodiment, the information fusion module includes: a color classification unit, configured to classify the color system of the image to be processed and determine the color system category of the image to be processed; and a color encoding unit, configured to encode the color system category of the image to be processed to obtain a color system encoding vector of the image to be processed, which serves as coloring reference information for the image to be processed.

[0009] In an exemplary embodiment, the image coloring module includes: a color classification unit, configured to classify the pixels in the second feature map by color and determine the color category of the pixels in the second feature map; and a coloring unit, configured to color the second feature map according to the colors contained in the color image of the target object indicated by the reference color palette and the color categories of the pixels in the second feature map, to obtain the image coloring result of the image to be processed.

[0010] In an exemplary embodiment, the coloring unit includes: a color encoding subunit, configured to encode the color category of each pixel in the second feature map to obtain a color encoding vector for the pixel; a vector extraction subunit, configured to extract a palette vector corresponding to the color encoding vector of the pixel from the reference palette based on the color encoding vector of the pixel, the palette vector being used to represent the colors contained in the color image of the target object; a decoding subunit, configured to decode the color encoding vector of the pixel in the second feature map according to the extracted palette vector to obtain a color channel to be fused; and a fusion subunit, configured to use the grayscale of the pixels in the image to be processed as the luminance channel to be fused, and to fuse the color channel and the luminance channel to obtain the image coloring result of the image to be processed.

[0011] In an exemplary embodiment, the apparatus further includes: a network construction module for constructing a visual network for coloring the image to be processed, the visual network including a color palette module, a coloring reference module, and a coloring module, wherein the color palette module is used to train the reference color palette from the first training set; the coloring reference module includes a classifier and an encoder for determining the coloring reference information of the image to be processed; the coloring module includes an extractor, a combiner, and a decoder, the extractor being used to obtain the first feature map; the combiner being used to fuse the coloring reference information into the first feature map to obtain a second feature map having the coloring reference information; and the decoder being used to obtain the image coloring result of the image to be processed.

[0012] In an exemplary embodiment, the color palette module includes: an image acquisition unit for acquiring the first training set; an image traversal unit for traversing the color images of the target object in the first training set, the traversal including: determining regions of interest in the traversed color images; extracting color information from pixels in the determined regions of interest; a clustering unit for clustering the colors contained in the color images of the target object in the first training set based on the extracted color information; and training the reference color palette until a set condition is met.

[0013] In an exemplary embodiment, the apparatus further includes: an evaluation module, configured to perform coloring effect evaluation processing on the image coloring result obtained based on the coloring of the visual network, to obtain a coloring evaluation result of the visual network, the coloring evaluation result being used to represent the coloring effect of the image coloring result; the evaluation module includes: an image pair acquisition unit, configured to acquire a second training set, the second training set including at least one image pair of the target object, the image pair including a color image and a corresponding grayscale image; a color feature extraction unit, configured to extract color features from the image coloring results of the color image and the corresponding grayscale image in the image pair, respectively, to obtain color features of the color image and the image coloring result; a difference determination unit, configured to determine the color difference between the color image and the image coloring result based on the color features of the color image and the image coloring result; and an evaluation unit, configured to determine the coloring evaluation result of the visual network based on the color difference between the color image and the image coloring result.

[0014] According to one aspect of the present application, an electronic device includes: at least one processor, at least one memory, and at least one communication bus, wherein the memory stores a computer program, and the processor reads the computer program from the memory via the communication bus; when the computer program is executed by the processor, it implements the image processing method described above.

[0015] According to one aspect of the embodiments of this application, a storage medium stores a computer program thereon, which, when executed by a processor, implements the image processing method as described above.

[0016] According to one aspect of the embodiments of this application, a computer program product includes a computer program stored in a storage medium. The processor of an electronic device reads the computer program from the storage medium and executes the computer program, causing the electronic device to implement the image processing method as described above when executed.

[0017] The beneficial effects of the technical solution provided in this application are:

[0018] In the above technical solution, based on the image to be processed of the target object, image features are extracted to obtain a first feature map. The coloring reference information of the image to be processed is then fused into the first feature map to obtain a second feature map with coloring reference information. Based on the reference color palette trained by the first training set, the second feature map is colored to obtain the image coloring result of the image to be processed. Thus, by using the reference color palette and coloring reference information, the image to be processed is guided to be automatically colored to obtain a color image of the target object. This not only provides additional color information but also improves the accuracy of coloring, thereby effectively solving the problem of poor coloring effect in related technologies. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0020] Figure 1 This is a schematic diagram based on the implementation environment involved in this application;

[0021] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0022] Figure 3 yes Figure 2 A flowchart of step 370 in one embodiment corresponds to the following example;

[0023] Figure 4 This is a schematic diagram of the architecture of a visual network according to an exemplary embodiment;

[0024] Figure 5 This is a flowchart illustrating a method for training a reference palette according to an exemplary embodiment;

[0025] Figure 6 yes Figure 3 A flowchart of step 373 in one embodiment corresponds to the following example;

[0026] Figure 7 This is a flowchart illustrating a method for evaluating a visual network according to an exemplary embodiment;

[0027] Figure 8 This is a structural block diagram of an image processing apparatus according to an exemplary embodiment;

[0028] Figure 9 This is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0029] Figure 10 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0030] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0031] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0032] As mentioned earlier, in some application scenarios where there are defects in acquiring color images, the coloring effect of the color image obtained through coloring is poor.

[0033] For example, some underwater cameras are already capable of color imaging for marine plankton, which is more conducive to improving the observation accuracy of marine plankton compared to grayscale images. However, the aforementioned color imaging process requires white light illumination, which can easily induce phototaxis in marine plankton, leading to deviations in the measurement of marine plankton abundance and greatly affecting the observation accuracy of marine plankton.

[0034] To address this issue, an underwater imaging method using red or near-infrared light illumination is proposed. Since most marine plankton are insensitive to longer wavelengths of light, using red or near-infrared light instead of white light effectively avoids the aggregation of marine plankton caused by phototaxis, thus preventing deviations in the measurement of marine plankton abundance. However, this underwater imaging method still has a drawback: using red or near-infrared light illumination results in the underwater camera only acquiring grayscale images. To facilitate subsequent identification of marine plankton, these grayscale images need to be converted into color images. However, the colorization effect obtained by current colorization algorithms differs significantly from the true color effect, which is not conducive to the subsequent application of the color image.

[0035] As can be seen from the above, the related technologies still have limitations, such as poor coloring effects.

[0036] Therefore, the image processing method provided in this application can effectively improve the coloring accuracy and thus improve the coloring effect. Accordingly, the image processing method is applicable to an image processing device that can be deployed on an electronic device configured with a von Neumann architecture, such as a desktop computer, a laptop computer, a server, etc.

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0038] Figure 1 This is a schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is merely an example adapted to the present invention and should not be considered as providing any limitation on the scope of the invention.

[0039] The implementation environment includes a data acquisition terminal 110 and a server terminal 130.

[0040] Specifically, the acquisition terminal 110 can also be considered an image acquisition device, including but not limited to electronic devices with shooting functions such as cameras, camcorders, and video recorders. For example, the acquisition terminal 110 is an underwater camera.

[0041] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server, or it can be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide background services, such as, but not limited to, automatic coloring services.

[0042] The server 130 and the acquisition terminal 110 establish a network communication connection in advance via wired or wireless means, and data transmission between the server 130 and the acquisition terminal 110 is realized through this network communication connection. The transmitted data includes, but is not limited to, the image of the target object to be processed, etc.

[0043] In one application scenario, through the interaction between the acquisition terminal 110 and the server 130, the acquisition terminal 110 takes a picture of the target object and acquires the image to be processed, and uploads the image to be processed to the server 130 to request the server 130 to provide automatic coloring service.

[0044] For server 130, after receiving the image of the target object to be processed uploaded by acquisition terminal 110, it calls the automatic coloring service to color the image of the target object to be processed, so that the image of the target object to be processed is restored to a color image through color restoration, thereby solving the problem of poor coloring effect in related technologies.

[0045] Please see Figure 2 This application provides an underwater image enhancement method, which is applicable to electronic devices, specifically, [the electronic devices may be...]. Figure 1 The server 130 in the implementation environment is shown.

[0046] The method is illustrated using an electronic device as the execution subject for each step, but this does not constitute a specific limitation.

[0047] like Figure 2 As shown, the method may include the following steps:

[0048] Step 310: Obtain the image of the target object to be processed.

[0049] The image to be processed is composed of Figure 1 The image acquisition device shown is used to capture and generate images of the environment containing the target object. The target object refers to an object in the captured environment, which can be various organisms exhibiting phototaxis. For example, the target object can be an underwater organism, specifically a marine plankton.

[0050] It is understood that shooting can be a single shot or continuous shooting. For the same target object, continuous shooting yields a video clip, and the image to be processed can be any frame from that video. Conversely, multiple shots yield multiple photographs, and the image to be processed can be any one of those photographs. In other words, image processing in this embodiment can be based on any frame from a video clip or any one of multiple photographs; simply put, image processing is performed frame by frame.

[0051] Regarding the acquisition of the image to be processed, the image can originate from images captured and acquired in real time by an image acquisition device, or it can be images captured and acquired by an image acquisition device within a historical time period that are pre-stored in the electronic device. Therefore, for the electronic device, after the image acquisition device captures and acquires the image to be processed, it can process the image in real time, or it can pre-store it for later processing. For example, it can process the image when the CPU of the electronic device is low, or it can process the image according to the instructions of the staff. Thus, the image processing in this embodiment can be for real-time images or for historical images. In one possible implementation, the image to be processed of the target object refers to a grayscale image of the target object; correspondingly, the image acquisition device can be a grayscale camera, etc. In this way, since grayscale images have higher spatial resolution and smaller data volume, it is beneficial to reduce the pressure on image processing, storage, and transmission of the electronic device, thereby improving the image processing efficiency of the electronic device.

[0052] Step 330: Extract image features from the image to be processed to obtain the first feature map.

[0053] The first feature map refers to the image carrying the image features of the image to be processed. It can also be considered as an accurate description of the image features of the image to be processed. It should be understood that different images to be processed will have different extracted image features, and correspondingly, the first feature maps will also be different.

[0054] In one possible implementation, image feature extraction is achieved through convolutional kernels. It should be noted that using different numbers and sizes of convolutional kernels will yield first feature maps of different sizes, reflecting the image to be processed at different scales, thereby improving the accuracy of subsequent target object observation.

[0055] Step 350: Based on the color system of the image to be processed, determine the color reference information of the image to be processed, and fuse the color reference information into the first feature map to obtain a second feature map with color reference information.

[0056] First, it should be noted that the color system of the image to be processed refers to the family of colors contained in the image. In one possible implementation, the color system of the image to be processed includes, but is not limited to: white, red, yellow, green, and blue.

[0057] The inventors recognized that most target objects in the image to be processed will only exhibit one or two color schemes, while other color schemes are rarely seen. In other words, the colors of most target objects are relatively monotonous, and correspondingly, the color scheme of the image to be processed is also relatively monotonous. Based on this, in this embodiment, coloring reference information of the image to be processed is introduced, so as to serve as a reference for subsequent coloring of the image to be processed. Specifically, the coloring reference information is fused into the first feature map to obtain a second feature map with coloring reference information.

[0058] The coloring reference information is used to indicate the color system to which the image to be processed belongs. It can also be considered as an accurate description of the color system to which the image to be processed belongs. It should be understood that if the color system to which the image to be processed belongs is different, the coloring reference information of the image to be processed will also be different.

[0059] Therefore, compared to the first feature map which carries the image features of the image to be processed, the second feature map also carries the color system to which the image to be processed belongs. It can also be understood that the second feature map not only describes the image features of the image to be processed, but also describes the color system to which the image to be processed belongs. This makes it possible to use the second feature map as the coloring input for the subsequent coloring of the image to be processed, thereby improving the accuracy of coloring.

[0060] Step 370: Based on the reference color palette trained from the first training set, the second feature map is colored to obtain the image coloring result of the image to be processed.

[0061] The reference color palette is the set of colors contained in the color images of the target object in the first training set. It can also be understood as indicating several colors contained in the color images of the target object in the first training set. This reference color palette is obtained by training a base model on the first training set, which can be various machine learning models. The first training set includes at least one color image of the target object. It is worth noting that, unlike the image to be processed, the color image is generated by a color camera capturing and collecting images of the environment containing the target object.

[0062] In one possible implementation, color is represented by the RGB color space; in another, color is represented by ab values ​​in the LAB color space. For example, based on the RGB color space, white can be represented as (255, 255, 255), or based on the ab values ​​in the LAB color space, white can be represented as (0, 0).

[0063] Regarding the coloring process of the image to be processed, in one possible implementation, such as Figure 3 As shown, step 370 may include the following steps:

[0064] Step 371: Perform color classification on the pixels in the second feature map to determine the color category of the pixels in the second feature map.

[0065] Color classification essentially involves using a classifier to calculate the probability that a pixel belongs to different color categories. It should be understood that a higher probability indicates a greater likelihood that the pixel belongs to the corresponding color category. In one possible implementation, the classifier is implemented using the softmax activation function.

[0066] Step 373: Based on the colors contained in the color image of the target object indicated by the reference color palette and the color categories of the pixels in the second feature map, perform coloring of the second feature map to obtain the image coloring result of the image to be processed.

[0067] For example, based on the colors contained in the color image of the target object indicated by the reference color palette, each pixel in the second feature map is traversed. If the color category of the traversed pixel is white, then the pixel will be colored according to the white indicated by the reference color palette. Similarly, if the color category of the traversed pixel is red, then the pixel will be colored according to the red indicated by the reference color palette, and so on. In other words, coloring is essentially assigning a value to the pixel of the color category indicated by the reference color palette. For example, based on the RGB color space, the white indicated by the reference color palette is represented as (255, 255, 255), then the pixel belonging to the white category is assigned the value (255, 255, 255), thus completing the white coloring of the pixel.

[0068] Once all pixels in the second feature map have been traversed, the image coloring result of the image to be processed can be obtained.

[0069] Through the above process, the color image of the target object is obtained by automatically coloring the image to be processed by guiding the reference color palette. Compared with grayscale images, it can not only provide additional color information, but also improve the accuracy of coloring, thereby effectively solving the problem of poor coloring effect in related technologies.

[0070] Figure 4 A schematic diagram of the architecture of a visual network in one embodiment is shown, which is used to colorize an image to be processed. Figure 4As shown, the visual network 600 includes a palette module 610, a coloring reference module 630, and a coloring module 650. During the coloring process of the visual network on the image to be processed, the coloring reference module 630 and the coloring module 650 can be executed in parallel; this is not a specific limitation.

[0071] Now combined Figures 4 to 6 The architecture of visual networks will be explained in detail below:

[0072] Please see Figure 4 The palette module 610 is used to obtain a reference palette by training the first training set.

[0073] In one possible implementation, such as Figure 5 As shown, the training process for the reference palette may include the following steps:

[0074] Step 410: Obtain the first training set.

[0075] The first training set includes at least one color image of the target object. For example, if the target object is marine plankton, the color image of the target object is generated by an underwater camera.

[0076] Step 430: Traverse the color images of the target objects in the first training set.

[0077] The traversal process includes: determining the region of interest (ROI) in the traversed color image; and extracting color information from the pixels in the determined ROI.

[0078] First, it should be noted that the inventors recognized that color images captured by underwater cameras can be considered as dark-field images with an almost entirely black background. Therefore, in this embodiment, the training process of the reference color palette uses the region of interest (ROI) in the color image as the data basis to improve the accuracy of the training process. The ROI refers to the area of ​​the target object in the color image. Correspondingly, the background area refers to the area in the color image excluding the ROI. In other words, the color image consists of the ROI and the background area.

[0079] In one possible implementation, the region of interest is determined using an image segmentation method. This image segmentation method includes, but is not limited to, edge-based segmentation, threshold-based segmentation, region-based segmentation, and other similar image segmentation methods.

[0080] Secondly, pixel color information refers to the color of the pixel. In one possible implementation, the color information is represented by the RGB color space; in another possible implementation, the color information is represented by the ab values ​​in the LAB color space.

[0081] Once all color images of the target object in the first training set have been traversed, multiple color information can be extracted, which represent the original color of the target object.

[0082] Step 450: Based on the extracted color information, cluster the colors contained in the color images of the target objects in the first training set until the set conditions are met, and train to obtain a reference color palette.

[0083] The inventors realized that compared to the extremely wide color gamut of natural scenes in the real world under the sunlight spectrum, the color gamut of marine plankton images is much smaller. Therefore, in this embodiment, the training process of the reference color palette refers to integrating the original colors of the target object through a clustering algorithm, thereby simplifying the number of colors and thus simplifying the coloring process of the image to be processed. As can be seen from the above, the training process of the reference color palette is essentially a classification process that divides all the colors contained in the color images of the target object in the first training set into several color categories, so as to simplify the coloring process of the image to be processed.

[0084] In one possible implementation, the clustering algorithm includes, but is not limited to: clustering based on the mean of cluster centers, clustering based on sliding windows, density-based clustering, expectation-maximization clustering based on Gaussian mixture models, clustering based on agglomerative hierarchy, graph-based clustering, and so on. In one possible implementation, the clustering algorithm uses a clustering algorithm based on the mean of cluster centers, specifically the K-means clustering algorithm.

[0085] Taking the K-means clustering algorithm as an example, let's illustrate the clustering process of colors contained in the color images of target objects in the first training set:

[0086] The extracted color information is treated as an object. To facilitate clustering, this object is represented by ab values ​​based on the LAB color space. For example, the color information of pixel A can be represented as A = {a, b}.

[0087] For multiple objects, first randomly select K objects as initial cluster centers, then calculate the distance between each object and these initial cluster centers, and assign each object to the nearest cluster center based on the distance. Thus, the cluster centers and the objects assigned to those cluster centers form a cluster.

[0088] As objects are assigned, the cluster centers of each cluster are recalculated based on the existing objects in that cluster until the set conditions are met, ultimately resulting in K cluster centers. K can be flexibly adjusted according to the actual needs of the application scenario; no limitation is imposed here.

[0089] It should be noted that the set conditions can refer to the number of iterations exceeding the iteration threshold, or it can refer to the completion of all object allocation or the K cluster centers no longer changing during the recalculation process, or it can refer to the minimum of the sum of squared deviations within the cluster, the local minimum of the sum of squared errors, etc., without any limitation here.

[0090] Thus, the K cluster centers represent the K colors contained in the color images of the target objects in the first training set, and these are used as K palette vectors to form a reference palette.

[0091] Under the above embodiments, the training of the reference color palette is realized. It can also be considered that, based on the color images of the target objects in the first training set, a reference color palette suitable for the grayscale image coloring of the target objects is customized. This not only simplifies the coloring process of the image to be processed, that is, reduces all the colors contained in the color images of the target objects in the first training set to K colors, simplifying the color search space in the coloring process, but also achieves better coloring effect, which is conducive to improving the coloring accuracy of images in specific fields.

[0092] Please refer back to Figure 4 The coloring reference module 630 includes a classifier 631 and an encoder 633 for determining coloring reference information for the image to be processed.

[0093] The classifier 631 is used to classify the color system of the image to be processed and determine the color system category of the image to be processed.

[0094] Specifically, color classification essentially involves using classifier 631 to calculate the probability that the image to be processed belongs to different color categories. It should be understood that a higher probability indicates a greater likelihood that the image to be processed belongs to the corresponding color category. In one possible implementation, classifier 631 is implemented using a softmax activation function.

[0095] Encoder 633 is used to encode the color category of the image to be processed, and obtain the color coding vector of the image to be processed, which serves as the coloring reference information of the image to be processed.

[0096] Specifically, assume that the image to be processed can be divided into 5 color categories: white, red, yellow, green, and blue, and represented by integers L = 1 to 5 respectively.

[0097] After the classifier 631 determines the color category L of the image to be processed, the encoder 633 encodes the color category L to generate color reference information for the image to be processed.

[0098] In one possible implementation, the encoding employs a discrete encoding method, which means using embedding to encode the color category L to obtain color reference information. For example, embedding could refer to the word2vec algorithm.

[0099] In one possible implementation, the encoding adopts a continuous encoding method, which means that the coloring reference information R is obtained by using the following calculation formula (1):

[0100] R = L⁻¹ + P (1);

[0101] Where P represents the probability that the image to be processed belongs to color category L.

[0102] As described in the above embodiments, additional coloring reference information is generated to guide the coloring module 650 in subsequent coloring, thereby improving the coloring accuracy of the image to be processed.

[0103] Continue reading Figure 4 The coloring module 650 includes an extractor 651, a combiner 653, and a decoder 655, and is used to color the image to be processed.

[0104] The extractor 651 is used to extract image features from the image to be processed to obtain a first feature map.

[0105] Specifically, such as Figure 4 As shown, the extractor 651 consists of 4 modules, each containing 3 convolutional layers, of which 2 convolutional layers have 3×3 kernels and the other convolutional layer has 1×1 kernel.

[0106] Combiner 653 is used to fuse the color reference information obtained by encoder 633 into the first feature map to obtain a second feature map with color reference information.

[0107] In one possible implementation, the coloring reference information is decomposed into first reference information and second reference information, which are then fused with the first feature map to obtain a second feature map with coloring reference information.

[0108] Specifically, such as Figure 4 As shown, the combiner 653 consists of two modules, each containing a convolutional layer with a 3×3 kernel and a custom layer for fusing color reference information.

[0109] Regarding the specific process of integration, such as Figure 4 As shown, the coloring reference information R is decomposed into first reference information α and second reference information β through a convolutional layer. The first reference information α and the second reference information β are then fused with the first feature map through a custom layer to obtain a second feature map with coloring reference information. This second feature map is calculated by the following formula (2):

[0110] fm ref =αfm ex +β (2);

[0111] Among them, fm ref The second feature map, fm ex Let α represent the first feature map, and let β represent the first reference information and the second reference information, that is, the decomposed coloring reference information.

[0112] The decoder 655 is used to color the second feature map based on the reference color palette trained by the color palette module 610, so as to obtain the image coloring result of the image to be processed.

[0113] Specifically, in one possible implementation, such as Figure 6 As shown, step 373 may include the following steps:

[0114] Step 3731: For each pixel in the second feature map, encode the color category of the pixel to obtain the color encoding vector of the pixel.

[0115] The color encoding vector of a pixel is used to represent the color category of the pixel.

[0116] In one possible implementation, the encoding uses a one-hot encoding method, whereby the color encoding vector of a pixel is essentially a one-hot vector.

[0117] Step 3733: Based on the pixel's color encoding vector, extract the palette vector corresponding to the pixel's color encoding vector from the reference palette.

[0118] The palette vector is used to represent the color categories contained in the color image of the target object.

[0119] Step 3735: Based on the extracted palette vector, decode the color encoding vector of the pixel in the second feature map to obtain the color channel to be fused.

[0120] Specifically, decoding refers to multiplying the extracted palette vector with the color encoding vector of the corresponding pixel in the second feature map.

[0121] Step 3737: Use the grayscale of the pixels in the image to be processed as the luminance channel to be fused, and fuse the color channel and the luminance channel to obtain the image coloring result of the image to be processed.

[0122] In other words, the image coloring result of the image to be processed is formed by splicing the color channel and the brightness channel.

[0123] The above embodiments enable the coloring of the image to be processed.

[0124] Now combined Figure 4 and Figure 7 Taking the convolutional neural network model as the basic model of visual networks as an example, the training process of the coloring reference module and the coloring module in the visual network, that is, the training process of the convolutional neural network model, is explained as follows:

[0125] First, it should be noted that the inventors realized that due to the huge differences in the number of different types of marine plankton, this obvious imbalance in quantity will lead to an imbalance in the number of colors of different types of marine plankton. For example, the color images in the second training set may have a lot of yellow, while red is relatively rare. If the convolutional neural network model is trained directly using this training set, it will cause the visual network to have a dominant color effect, that is, the visual network only learns a few main colors.

[0126] Based on this, for image pairs in the second training set, pixels containing common colors are considered easily distinguishable samples, while pixels containing rare colors are considered difficult to distinguish samples. A loss function is set to improve the color imbalance problem; that is, by assigning more weight to rare colors to train the convolutional neural network model, the convolutional neural network model can better focus on rare colors, thereby avoiding the dominant color effect. The formula for calculating this loss function is as follows:

[0127] loss pixle (x, p) t )=-ω t (1-p t ) γ log(p t (3):

[0128]

[0129] In formula (3), loss pixle (x) represents the loss incurred when classifying the color of pixel x, p t ω represents the probability that pixel x belongs to the correct color category. t This represents the weight of pixel x belonging to the correct color category when calculating this loss. It should be noted that ω... t It is determined during the training process of the reference color palette. Color quantization is performed using color images of the target object in the first training set to obtain the quantity information of each color class. The more colors there are, the greater the weight ω is assigned to that color class. t The smaller ω is, the better. t It can be used to balance the differences between different color categories. Here, γ is a modulation factor used to balance the weights between easy and difficult samples, making the weight of difficult samples larger than that of easy samples.

[0130] In the calculation formula (4), S represents the set of difficult-to-classify samples, and N is the number of difficult-to-classify samples. It should be noted that the generation process of S is as follows: Based on the image pairs in the second training set, the loss generated when color classification of all pixels is sorted first, and then the N pixels with the largest loss are selected to form S. This method can make the convolutional network model focus on the pixels with large loss, which is beneficial to improving the training efficiency of the model.

[0131] The loss of the coloring reference module and the loss of the coloring module are both calculated using formula (4). Then, the total loss is calculated from the losses of these two modules using formula (5). This total loss is used for training the convolutional neural network model.

[0132] loss final =μloss ref +θloss color (5):

[0133] Where, loss final The total loss is... ref The loss is the loss of the coloring reference module. color Let μ be the loss of the shading module, and θ be the scaling factors of the two modules, respectively.

[0134] The training process of a convolutional neural network model includes:

[0135] The parameters of the convolutional neural network model are randomly initialized, and the total loss is calculated based on the images in the second training set. final If the total loss final If the minimum value is not reached, the parameters of the convolutional neural network model are updated, and the total loss is calculated again based on the image pairs in the second training set. final until the total loss final When the minimum value is reached, the convolutional neural network model converges to obtain the visual network.

[0136] Of course, the iteration threshold can be set according to the actual needs of the application scenario to accelerate the training efficiency of the model. At the same time, setting different iteration thresholds can also meet the different coloring accuracy requirements of the visual network. For example, a larger iteration threshold is beneficial to improving the accuracy of the visual network in coloring the image to be processed.

[0137] As a result, visual networks acquire the ability to colorize the images to be processed.

[0138] The evaluation of visual networks specifically includes: evaluating the coloring effect of image coloring results obtained based on visual networks to obtain the coloring evaluation result of the visual network. The coloring evaluation result is used to represent the coloring effect of the image coloring result.

[0139] Please see Figure 7 In one possible implementation, the evaluation process may include the following steps:

[0140] Step 510: Obtain the second training set.

[0141] The second training set includes at least one image pair of the target object, which includes a color image and a corresponding grayscale image.

[0142] For example, for the target object being marine plankton, the image pair can be obtained using a dual-channel dark-field imaging system. This dual-channel dark-field imaging system includes two channels, one of which is equipped with a color camera and the other with a grayscale camera. By simultaneously capturing images of the same marine plankton in the underwater environment using the color and grayscale cameras configured in these two channels, a color image and a corresponding grayscale image of the marine plankton can be obtained, thereby forming an image pair of the marine plankton, which can then be added to the second training set required for the visual network training process.

[0143] Step 530: For the color image and the corresponding grayscale image in the image pair, extract color features from the image coloring results of the color image and the grayscale image respectively to obtain the color features of the color image and the image coloring results.

[0144] Color features are digital representations of the colors contained in an image. They can also be understood as an accurate description of the colors contained in an image. It should be understood that different images will have different color features.

[0145] In one possible implementation, the color features include at least one of the following: a color histogram, a color aggregation vector, a color autocorrelation graph, color gradient information, and descriptive statistics of a single-channel histogram.

[0146] Step 550: Determine the color difference between the color image and the image coloring result based on the color characteristics of the color image and the image coloring result.

[0147] Color difference reflects the difference in the number and spatial distribution of colors between the image coloring result and the color image, and is used to evaluate the coloring effect of the image coloring result.

[0148] In one possible implementation, color difference refers to the Euclidean distance between the color features of a color image and the colorized image. This color difference is calculated using the following formula (6):

[0149]

[0150] Where X and Y represent the color features of the color image and the image coloring result, and l represents the vector length.

[0151] Step 570: Based on the color difference between the color image and the image coloring result, determine the coloring evaluation result of the visual network.

[0152] As can be seen from the above, the smaller the CDSIM value, the better the coloring effect of the image coloring result and the better the coloring evaluation result of the visual network.

[0153] It is worth mentioning that if the color evaluation effect of the visual network is not good, on the one hand, the color reference module and color module in the visual network can be trained again based on the second training set. On the other hand, the reference color palette can be trained again based on the first training set until the visual network has a good color evaluation result, which is more conducive to improving the accuracy of coloring and thus improving the coloring effect.

[0154] The following are embodiments of the apparatus described in this application, which can be used to execute the image processing method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the image processing method involved in this application.

[0155] Please see Figure 8 This application provides an image processing device 900, including but not limited to: an image acquisition module 910, a feature extraction module 930, an information fusion module 950, and an image coloring module 970.

[0156] The image acquisition module 910 is used to acquire the image of the target object to be processed.

[0157] The feature extraction module 930 is used to extract image features from the image to be processed to obtain the first feature map.

[0158] The information fusion module 950 is used to determine the color reference information of the image to be processed based on the color system to which the image to be processed belongs, and to fuse the color reference information into the first feature map to obtain a second feature map with color reference information.

[0159] The image coloring module 970 is used to color the second feature map based on a reference color palette trained from the first training set, so as to obtain the image coloring result of the image to be processed. The first training set includes at least one color image of the target object, and the reference color palette is the set of colors contained in the color image of the target object in the first training set.

[0160] It should be noted that the image processing device provided in the above embodiments is only illustrated by the division of the above functional modules when performing image processing. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the image processing device will be divided into different functional modules to complete all or part of the functions described above.

[0161] Furthermore, the image processing apparatus and image processing method embodiments provided in the above embodiments belong to the same concept, and the specific way in which each module performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0162] Figure 9 A schematic diagram of the structure of a server is shown according to an exemplary embodiment. This server is suitable for... Figure 1 The server 130 in the implementation environment is shown.

[0163] It should be noted that this server is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Nor should this server be interpreted as requiring or depending on any specific feature. Figure 9 One or more components of the exemplary server 2000 shown.

[0164] The hardware architecture of Server 2000 can vary significantly due to differences in configuration or performance, such as... Figure 9 As shown, the server 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0165] Specifically, power supply 210 is used to provide operating voltage for the various hardware devices on server 2000.

[0166] Interface 230 includes at least one wired or wireless network interface for interacting with external devices. For example, to perform... Figure 1 The diagram shows the interaction between the acquisition terminal 110 and the server terminal 130 in the implementation environment.

[0167] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 9 As shown, this does not constitute a specific limitation.

[0168] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0169] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0170] Application 253 is a computer program that performs at least one specific task based on operating system 251, and may include at least one module ( Figure 9 (Not shown), each module can contain a computer program for server 2000. For example, the data monitoring device can be considered as application 253 deployed on server 2000.

[0171] Data 255 can be photos, pictures, etc. stored on a disk, or images to be processed, color images, etc., stored in memory 250.

[0172] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer programs stored in the memory 250, thereby performing operations and processing on massive amounts of data 255 stored in the memory 250. For example, an image processing method may be performed by the central processing unit 270 reading a series of computer programs stored in the memory 250.

[0173] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0174] Please see Figure 10 This application provides an electronic device 4000, which may include: desktop computer, laptop computer, server, etc.

[0175] exist Figure 10 The electronic device 4000 includes at least one processor 4001, at least one communication bus 4002, and at least one memory 4003.

[0176] The processor 4001 and memory 4003 are connected, for example, via a communication bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0177] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0178] The communication bus 4002 may include a path for transmitting information between the aforementioned components. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0179] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0180] The memory 4003 stores a computer program, and the processor 4001 reads the computer program stored in the memory 4003 through the communication bus 4002.

[0181] When the computer program is executed by the processor 4001, it implements the image processing methods in the above embodiments.

[0182] Furthermore, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the image processing methods described in the above embodiments.

[0183] This application provides a computer program product including a computer program stored in a storage medium. An electronic device's processor reads the computer program from the storage medium and executes the computer program, causing the electronic device to perform the image processing methods described in the above embodiments.

[0184] Compared with related technologies, the image processing methods in the embodiments of this application provide a new approach to automatic coloring in specific fields. They can improve the coloring accuracy of images in specific fields by simplifying and customizing color palettes. A customized self-guided automatic coloring visual network is designed, which achieves better results in the coloring of marine plankton. A new evaluation index, color difference in similarity (CDSIM), is provided for evaluating the color similarity of images. A new design idea is provided for the development of in-situ color imaging instruments for marine plankton. Instruments designed based on this idea are expected to avoid the impact of phototaxis on measurement accuracy and alleviate the pressure of storing and transmitting marine observation data, which will help realize long-term continuous in-situ marine observation.

[0185] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0186] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image of the target object to be processed; Image features are extracted from the image to be processed to obtain a first feature map; Based on the color system to which the image to be processed belongs, the coloring reference information of the image to be processed is determined, and the coloring reference information is fused into the first feature map to obtain a second feature map with the coloring reference information; Based on the reference color palette obtained by training the first training set, the second feature map is colored to obtain the image coloring result of the image to be processed. The first training set includes at least one color image of the target object, and the reference color palette is the set of colors contained in the color image of the target object in the first training set. The step of determining the color reference information of the image to be processed based on the color system to which the image to be processed belongs includes: classifying the image to be processed by color system to determine the color system category of the image to be processed; encoding the color system category of the image to be processed to obtain the color system encoding vector of the image to be processed, which serves as the color reference information of the image to be processed. The process of training a reference color palette using a first training set includes: acquiring the first training set; traversing the color images of the target object in the first training set, wherein the traversal includes: determining regions of interest in the traversed color images; extracting color information from pixels in the determined regions of interest; clustering the colors contained in the color images of the target object in the first training set based on the extracted color information; and training the reference color palette until a set condition is met.

2. The method as described in claim 1, characterized in that, The step of coloring the second feature map based on a reference color palette trained from the first training set to obtain the image coloring result of the image to be processed includes: The pixels in the second feature map are classified by color to determine the color category of the pixels in the second feature map; Based on the colors contained in the color image of the target object indicated by the reference color palette and the color categories of the pixels in the second feature map, the second feature map is colored to obtain the image coloring result of the image to be processed.

3. The method as described in claim 2, characterized in that, The step of coloring the second feature map based on the colors contained in the color image of the target object indicated by the reference color palette and the color categories of pixels in the second feature map to obtain the image coloring result of the image to be processed includes: For each pixel in the second feature map, the color category of the pixel is encoded to obtain the color encoding vector of the pixel; Based on the color encoding vector of the pixel, a palette vector corresponding to the color encoding vector of the pixel is extracted from the reference palette. The palette vector is used to represent the colors contained in the color image of the target object. Based on the extracted palette vector, the color encoding vector of the pixel in the second feature map is decoded to obtain the color channel to be fused; The grayscale of the pixels in the image to be processed is used as the luminance channel to be fused, and the color channel and the luminance channel are fused to obtain the image coloring result of the image to be processed.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: constructing a visual network for coloring the image to be processed, the visual network including a color palette module, a coloring reference module, and a coloring module, wherein... The color palette module is used to train the reference color palette from the first training set; The coloring reference module includes a classifier and an encoder, used to determine the coloring reference information of the image to be processed; The coloring module includes an extractor, a combiner, and a decoder. The extractor is used to obtain the first feature map. The combiner is used to fuse the coloring reference information into the first feature map to obtain the second feature map with the coloring reference information. The decoder is used to obtain the image coloring result of the image to be processed.

5. The method as described in claim 4, characterized in that, The method further includes: performing a coloring effect evaluation process on the image coloring result obtained based on the visual network coloring to obtain a coloring evaluation result of the visual network, wherein the coloring evaluation result is used to represent the coloring effect of the image coloring result; The step of evaluating the coloring effect of the image coloring result obtained based on the visual network coloring, to obtain the coloring evaluation result of the visual network, includes: Obtain a second training set, the second training set including at least one image pair of the target object, the image pair including a color image and a corresponding grayscale image; For the color image and the corresponding grayscale image in the image pair, color features are extracted from the image coloring results of the color image and the grayscale image respectively to obtain the color features of the color image and the image coloring results; Based on the color characteristics of the color image and the image coloring result, determine the color difference between the color image and the image coloring result; The color evaluation result of the visual network is determined based on the color difference between the color image and the colorization result of the image.

6. An image processing apparatus, characterized in that, The device includes: The image acquisition module is used to acquire the image of the target object to be processed. The feature extraction module is used to extract image features from the image to be processed to obtain a first feature map; An information fusion module is used to determine the color reference information of the image to be processed based on the color system to which the image to be processed belongs, and to fuse the color reference information into the first feature map to obtain a second feature map with the color reference information. The step of determining the color reference information of the image to be processed based on the color system to which the image to be processed belongs includes: classifying the image to be processed according to its color system to determine the color system category of the image to be processed; encoding the color system category of the image to be processed to obtain a color system encoding vector of the image to be processed, which serves as the color reference information of the image to be processed. An image coloring module is used to color the second feature map based on a reference color palette trained from a first training set, thereby obtaining the image coloring result of the image to be processed. The first training set includes at least one color image of the target object, and the reference color palette is a set of colors contained in the color images of the target object in the first training set. The step of training the reference color palette from the first training set includes: acquiring the first training set; traversing the color images of the target object in the first training set, wherein the traversal includes: determining regions of interest in the traversed color images; extracting color information from the pixels in the determined regions of interest; clustering the colors contained in the color images of the target object in the first training set based on the extracted color information; and training the reference color palette until a set condition is met.

7. An electronic device, characterized in that, include: At least one processor, at least one memory, and at least one communication bus, wherein, The memory stores a computer program, and the processor reads the computer program from the memory via the communication bus; When the computer program is executed by the processor, it implements the image processing method according to any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 5.

Images