Image processing method, model training method, related device and equipment

Abstract

Embodiments of the present application disclose an image processing method, a model training method, related devices and equipment, which are used to improve the stability of the model, thereby improving the stability of the coloring effect. The method of the embodiments of the present application comprises: obtaining a plurality of first feature elements from a black-and-white first image through a feature extraction network, and obtaining a plurality of second feature elements from a colored second image. According to the similarity between the plurality of first feature elements and the plurality of second feature elements, and the similarity between the plurality of first feature elements, N first target elements are determined from the plurality of second feature elements, N being a positive integer. The plurality of first feature elements and the N first target elements are fused to obtain fused features, and the fused features are processed through an image reconstruction network to obtain a colored enhanced image corresponding to the first image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to image processing methods, model training methods, related devices and equipment. Background Technology

[0002] With the development of artificial intelligence, fully convolutional neural network (CNN) methods in deep learning have made significant progress in many fields of AI. They have achieved remarkable results in image restoration and image quality enhancement tasks related to computer vision.

[0003] In one image processing method, a target detection network is used to crop K instances from a black and white input image. Then, the input image and the K instances are colored separately, and the K+1 colored results are fused to obtain a color image corresponding to the input image.

[0004] This method only focuses on the coloring effect of a single image, without considering the correlation between image frames, and the model used has poor stability, resulting in unstable coloring effects of image frames and phenomena such as color jumps in the video sequence. Summary of the Invention

[0005] This application provides image processing methods, model training methods, related devices and equipment to improve the stability of models, thereby improving the stability of shading effects.

[0006] The first aspect of this application provides an image processing method, including:

[0007] A black-and-white first image and a color second image are input into a feature extraction network. The network extracts multiple first feature elements from the first image and multiple second feature elements from the second image. The goal of image processing is to colorize the black-and-white first image. This process requires referencing the features of the second image using the model. Therefore, it's necessary to select target elements that meet certain criteria from the multiple second feature elements to facilitate feature fusion.

[0008] Specifically, firstly, based on the similarity between multiple first feature elements and multiple second feature elements, the multiple second feature elements are filtered to determine M second feature elements, where M is a positive integer. Then, based on the similarity between these M second feature elements and the multiple first feature elements, as well as the similarity among the multiple first feature elements, these M second feature elements are filtered to determine N first target elements, where N is a positive integer. After obtaining the N first target elements, the multiple first feature elements and the N first target elements are fused to obtain fused features. The fused features are input into an image reconstruction network, and through the processing of the image reconstruction network, the enhanced image corresponding to the first image is obtained.

[0009] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0010] In this embodiment, when determining the target elements corresponding to each first feature element in the first image, not only is the similarity between the first feature element and the second feature element considered, selecting M first target elements from the second feature elements; furthermore, based on the similarity between the M second feature elements and multiple first feature elements, as well as the similarity among multiple first feature elements, these M second feature elements are further filtered to obtain N first target elements that are ultimately fused with the first feature elements. This multiple filtering makes the referenced second feature elements for each first feature element more accurate, and also ensures that identical or similar first feature elements can be fused with identical or similar second feature elements. This guarantees that identical or similar first feature elements can achieve the same or similar fusion effect even in different images, thereby ensuring the stability between different image frames. In other words, the stability of the model is improved, thus improving the stability of the coloring effect.

[0011] In conjunction with the first aspect, in the first implementation of the first aspect of this application, the similarity between one of the multiple first feature elements and one of the M second feature elements is greater than or equal to a first threshold, and the similarity between this first feature element and one of the N first target elements is greater than or equal to a second threshold. The values ​​of the first threshold and the second threshold can be determined according to the needs of the actual application, and are not specifically limited here.

[0012] In this embodiment of the application, by setting the similarity between the first feature element and the second feature element, and the similarity between the first feature element and the first feature element corresponding to the second feature element, the similarity between the first target element and the first feature element obtained after multiple screenings is ensured, thereby improving the fusion effect and making the coloring effect closer to the ideal color.

[0013] In conjunction with the first aspect or the first implementation of the first aspect, in the second implementation of the first aspect of this application, before fusing the feature elements to obtain the fused feature, second feature elements can be repeatedly searched for multiple first feature elements to further improve the fusion effect. Specifically, firstly, based on the similarity between multiple first feature elements and multiple second feature elements, multiple second feature elements are filtered to determine X second feature elements, where X is a positive integer. Then, based on the similarity between these X second feature elements and multiple first feature elements, and the similarity between multiple first feature elements, these X second feature elements are filtered to determine Y second target elements, where Y is a positive integer. After obtaining Y second target elements, multiple third target elements are determined from N first target elements and Y second target elements. Then, the multiple first feature elements and the multiple third target elements are fused to obtain the fused feature.

[0014] In this embodiment, the second feature element is searched repeatedly for multiple first feature elements to further improve the fusion effect.

[0015] In conjunction with the second implementation of the first aspect, in the third implementation of the first aspect of this application, the similarity between one of the multiple first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between this first feature element and one of the Y second target elements is greater than or equal to a fourth threshold. The values ​​of the third and fourth thresholds can be determined according to the needs of the actual application, and are not specifically limited here.

[0016] In this embodiment of the application, by setting the similarity between the first feature element and the second feature element, and the similarity between the first feature element and the first feature element corresponding to the second feature element, the similarity between the second target element obtained after multiple screenings and the first feature element is ensured, thereby improving the fusion effect and making the coloring effect closer to the ideal color.

[0017] In conjunction with the first aspect and any of the first to third implementations of the first aspect, in the fourth implementation of the first aspect of this application, the fusion features can be processed by an image reconstruction network to obtain multiple color images corresponding to the first image. Specifically, the image reconstruction network includes multiple branches, each branch including at least one image reconstruction module. The parameters of each image reconstruction module are different, and each image reconstruction module can process the fusion features or intermediate features, including convolution processing, regularization, and upsampling, or convolution processing and regularization processing, etc., which are not limited here. Through multiple branches, the fusion features are processed respectively to obtain multiple enhanced images corresponding to the first image, each enhanced image is a color image, and each color image has a different color. The number of branches included in the image reconstruction network is determined according to the needs of the actual application, which are not limited here.

[0018] In this embodiment, multiple enhanced images with different colors corresponding to the first image are obtained through an image reconstruction network, thereby improving the diversity of the network output.

[0019] In conjunction with the fourth implementation of the first aspect, in the fifth implementation of the first aspect of this application, any two branches in the image reconstruction network have at least one or more identical image reconstruction modules. That is, multiple branches can share the output of the same image reconstruction module, which reduces the parameters configured in the image reconstruction network and simplifies its configuration.

[0020] In conjunction with the first aspect and any of the first to fifth implementations of the first aspect, the sixth implementation of the first aspect of this application can further obtain enhancement parameters, which are used to determine the fusion features. The first image is input into a pre-trained model, and multiple intermediate features are extracted from the first image. These intermediate features are the outputs of different network layers of the pre-trained model. They can include outputs from shallow layers or deep layers, ensuring that the multiple intermediate features comprehensively reflect the first image. The multiple intermediate features are concatenated to obtain concatenated features, which are then processed by a neural network to obtain the unit vector corresponding to the first image. Based on the unit vector, an enhancement feature lookup table is consulted to determine the enhancement parameters corresponding to the first image. The enhancement feature lookup table includes multiple key-value pairs. Based on the similarity between the unit vector and the key, the value corresponding to the key most similar to the unit vector is used as the enhancement parameter.

[0021] In this embodiment of the application, the enhancement parameters corresponding to the first image can also be determined by querying the enhancement feature lookup table, and the enhancement parameters can be used in the process of determining the fusion features, thus enriching the determination method of enhancement features.

[0022] In conjunction with the first aspect and any of the first to sixth implementations of the first aspect, the seventh implementation of the first aspect of this application specifically involves obtaining feature elements from an image by using a feature extraction network to obtain a first feature map of the first image and a second feature map of the second image, then obtaining multiple first feature elements from the first feature map and multiple second feature maps from the second feature map. There are various ways to obtain feature elements from the feature map, such as through sliding window sampling or other methods, for example, downsampling based on mean pooling or max pooling; the specific method is not limited here.

[0023] In conjunction with the first aspect and any of the first to seventh implementations of the first aspect, in the eighth implementation of the first aspect of this application, the plurality of second feature elements further include a fourth target element, and the similarity between the fourth target element and any one of the plurality of first feature elements is less than a first threshold.

[0024] In conjunction with the first aspect and any of the first to eighth implementations of the first aspect, in the ninth implementation of the first aspect of this application, the plurality of second feature elements further include a fifth target element, and the similarity between the first feature element corresponding to the fifth target element and any one of the plurality of first feature elements is less than a second threshold.

[0025] The second aspect of this application provides a model training method, including:

[0026] The model learns the style of a colored second image and transforms a black-and-white first image into a colored enhanced image. Specifically, the black-and-white first image and the colored second image are input into a feature extraction network. This network extracts multiple first feature elements from the first image and multiple second feature elements from the second image. Based on the similarity between the first and second feature elements, the second feature elements are filtered to determine M second feature elements (M being a positive integer). Then, based on the similarity between these M second feature elements and the first feature elements, as well as the similarity among the first feature elements, these M second feature elements are filtered to determine N first target elements (N being a positive integer). After obtaining the N first target elements, the first feature elements and the N first target elements are fused to obtain a fused feature. This fused feature is input into an image reconstruction network, which processes it to obtain the enhanced image corresponding to the first image. The model is then iteratively trained, updating the parameters of the feature extraction network and the image reconstruction network, until the model converges or reaches a preset first iteration number.

[0027] In this embodiment, when determining the target elements corresponding to each first feature element in the first image, not only is the similarity between the first feature element and the second feature element considered, selecting M first target elements from the second feature elements; furthermore, based on the similarity between the M second feature elements and multiple first feature elements, as well as the similarity among multiple first feature elements, these M second feature elements are further filtered to obtain N first target elements that are ultimately fused with the first feature elements. This multiple filtering makes the referenced second feature elements for each first feature element more accurate, and also ensures that identical or similar first feature elements can be fused with identical or similar second feature elements. This guarantees that identical or similar first feature elements can achieve the same or similar fusion effect even in different images, thereby ensuring the stability between different image frames. In other words, the stability of the model is improved, thus improving the stability of the coloring effect.

[0028] In conjunction with the second aspect, in the first implementation of the second aspect of this application, the similarity between one of the multiple first feature elements and one of the M second feature elements is greater than or equal to a first threshold, and the similarity between this first feature element and one of the N first target elements is greater than or equal to a second threshold. The values ​​of the first threshold and the second threshold can be determined according to the needs of the actual application, and are not specifically limited here.

[0029] In this embodiment of the application, by setting the similarity between the first feature element and the second feature element, and the similarity between the first feature element and the first feature element corresponding to the second feature element, the similarity between the first target element and the first feature element obtained after multiple screenings is ensured, thereby improving the fusion effect and making the coloring effect closer to the ideal color.

[0030] In conjunction with the second aspect or the first implementation of the second aspect, in the second implementation of the embodiments of this application, before fusing the feature elements to obtain the fused feature, second feature elements can be repeatedly searched for multiple first feature elements to further improve the fusion effect. Specifically, firstly, based on the similarity between multiple first feature elements and multiple second feature elements, multiple second feature elements are filtered to determine X second feature elements, where X is a positive integer. Then, based on the similarity between these X second feature elements and multiple first feature elements, as well as the similarity between multiple first feature elements, these X second feature elements are filtered to determine Y second target elements, where Y is a positive integer. After obtaining Y second target elements, multiple third target elements are determined from N first target elements and Y second target elements. Then, the multiple first feature elements and the multiple third target elements are fused to obtain the fused feature.

[0031] In this embodiment, the second feature element is searched repeatedly for multiple first feature elements to further improve the fusion effect.

[0032] In conjunction with the second implementation of the second aspect, in the third implementation of the second aspect of this application, the similarity between one of the multiple first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between this first feature element and one of the Y second target elements is greater than or equal to a fourth threshold. The values ​​of the third and fourth thresholds can be determined according to the needs of the actual application, and are not specifically limited here.

[0033] In this embodiment of the application, by setting the similarity between the first feature element and the second feature element, and the similarity between the first feature element and the first feature element corresponding to the second feature element, the similarity between the second target element obtained after multiple screenings and the first feature element is ensured, thereby improving the fusion effect and making the coloring effect closer to the ideal color.

[0034] In conjunction with the second aspect and any of the first to third implementations of the second aspect, in the fourth implementation of the second aspect of this application, the fusion features can be processed by an image reconstruction network to obtain multiple color images corresponding to the first image. Specifically, the image reconstruction network includes branches, and each branch includes at least one image reconstruction module. The parameters of each image reconstruction module are different, and each image reconstruction module can process the fusion features or intermediate features, such as convolution processing, regularization, and upsampling, or include convolution processing and regularization processing, etc., which are not limited here. Through multiple branches, the fusion features are processed separately to obtain multiple enhanced images corresponding to the first image, each enhanced image is a color image, and each color image has a different color. The number of branches included in the image reconstruction network is determined according to the needs of the actual application, which are not limited here.

[0035] In this embodiment, multiple enhanced images with different colors corresponding to the first image are obtained through an image reconstruction network, thereby improving the diversity of the network output.

[0036] In conjunction with the fourth implementation of the second aspect, in the fifth implementation of the second aspect of this application, any two branches in the image reconstruction network have at least one or more identical image reconstruction modules. That is, multiple branches can share the output of the same image reconstruction module, which reduces the parameters configured in the image reconstruction network and simplifies its configuration.

[0037] In conjunction with the second aspect and any of the first to fifth implementations of the second aspect, the sixth implementation of the second aspect of this application further includes enhancement parameters used to determine the fusion features. The first image is input into a pre-trained model, and multiple intermediate features are extracted from the first image. These intermediate features are the outputs of different network layers of the pre-trained model. They can include outputs from shallow layers or deep layers, ensuring that the multiple intermediate features comprehensively reflect the first image. The multiple intermediate features are concatenated to obtain concatenated features, which are then processed by a neural network to obtain the unit vector corresponding to the first image. Based on the unit vector, an enhancement feature lookup table is consulted to determine the enhancement parameters corresponding to the first image. The enhancement feature lookup table includes multiple key-value pairs. Based on the similarity between the unit vector and the key, the value corresponding to the key most similar to the unit vector is used as the enhancement parameter. Then, the neural network and the enhancement feature lookup table are iteratively trained to update the parameters of the neural network and the enhancement feature lookup table until the neural network and the enhancement feature lookup table converge, or until a preset second iteration number is reached, making the enhancement parameters infinitely close to the true values.

[0038] In this embodiment, the enhancement parameters corresponding to the first image can also be determined by querying the enhancement feature lookup table. These enhancement parameters can be used during image processing using the model, especially in determining fusion features, thus enriching the methods for determining enhancement features.

[0039] In conjunction with the second aspect and any of the first to sixth implementations of the second aspect, the seventh implementation of the second aspect of this application specifically involves obtaining feature elements from an image by using a feature extraction network to obtain a first feature map of the first image and a second feature map of the second image, then obtaining multiple first feature elements from the first feature map and multiple second feature maps from the second feature map. There are various ways to obtain feature elements from the feature map, such as through sliding window sampling or other methods, for example, downsampling based on mean pooling or max pooling; the specific method is not limited here.

[0040] In conjunction with the second aspect and any of the first to seventh implementations of the second aspect, in the eighth implementation of the second aspect of this application, the plurality of second feature elements further include a fourth target element, and the similarity between the fourth target element and any one of the plurality of first feature elements is less than a first threshold.

[0041] In conjunction with the second aspect and any of the first to eighth implementations of the second aspect, in the ninth implementation of the second aspect of this application, the plurality of second feature elements further include a fifth target element, and the first feature element corresponding to the fifth target element has a similarity to any one of the plurality of first feature elements that is less than a second threshold.

[0042] A third aspect of this application provides an execution apparatus, comprising:

[0043] The acquisition unit is used to acquire multiple first feature elements from a first image and multiple second feature elements from a second image through a feature extraction network, wherein the first image is a black and white image and the second image is a color image.

[0044] The determining unit is used to determine M second feature elements from multiple second feature elements based on the similarity between multiple first feature elements and multiple second feature elements, where M is a positive integer; and to determine N first target elements from the M second feature elements based on the similarity between the M second feature elements and multiple first feature elements, as well as the similarity between multiple first feature elements, where N is a positive integer.

[0045] The processing unit is used to fuse multiple first feature elements and N first target elements to obtain fused features, which are used to reconstruct the enhanced image corresponding to the first image; the fused features are processed by an image reconstruction network to obtain the enhanced image corresponding to the first image, and the enhanced image is a color image.

[0046] The actuator is used to perform the method of the first aspect described above. The beneficial effects shown in this aspect are similar to those of the first aspect, and will not be repeated here.

[0047] A fourth aspect of this application provides a training apparatus, comprising:

[0048] The acquisition unit is used to acquire multiple first feature elements from a first image and multiple second feature elements from a second image through a feature extraction network. The first image is a black and white image, and the second image is a color image.

[0049] The determining unit is used to determine M second feature elements from multiple second feature elements based on the similarity between multiple first feature elements and multiple second feature elements, where M is a positive integer; and to determine N first target elements from the M second feature elements based on the similarity between the M second feature elements and multiple first feature elements, as well as the similarity between multiple first feature elements, where N is a positive integer.

[0050] The processing unit is used to fuse multiple first feature elements and N first target elements to obtain fused features, which are used to reconstruct the enhanced image corresponding to the first image; the fused features are processed by an image reconstruction network to obtain the enhanced image corresponding to the first image, which is a color image; the model is iteratively trained, and the parameters of the feature extraction network and the image reconstruction network are updated until the model converges or reaches the preset first iteration number.

[0051] The training device is used to perform the method described in the second aspect above. The beneficial effects shown in this aspect are similar to those in the second aspect, and will not be repeated here.

[0052] A fifth aspect of this application provides an execution device including a processor, a memory, and a communication interface. The processor, memory, and communication interface are connected together, and the processor is used to execute the method described in the first aspect. The beneficial effects shown in this aspect are similar to those in the first aspect, and will not be repeated here.

[0053] A sixth aspect of this application provides a training device, including a processor, a memory, and a communication interface. The processor, memory, and communication interface are connected together, and the processor is used to execute the method described in the second aspect. The beneficial effects shown in this aspect are similar to those in the second aspect, and will not be repeated here.

[0054] A seventh aspect of this application provides a computer-readable storage medium storing a program that, when executed by a computer, performs the method of the first or second aspect.

[0055] The eighth aspect of this application provides a computer program product, characterized in that when the computer program product is executed on a computer, the computer executes the method of the first aspect or the second aspect.

[0056] A ninth aspect of this application provides a chip including a processor and a data interface. The data interface and the processor are interconnected via a circuit. The processor reads instructions stored in a memory through the data interface to execute the method of the first aspect or the second aspect. The data interface in the chip can be an input / output interface, a pin, or a circuit, etc.

[0057] The beneficial effects shown in aspects seven through nine are similar to those in aspect one or two, and will not be repeated here. Attached Figure Description

[0058] Figure 1 A schematic diagram of the main framework of artificial intelligence;

[0059] Figure 2 A schematic diagram of the system architecture of the image processing system provided in the embodiments of this application;

[0060] Figure 3 A schematic diagram of the structure of the model provided in the embodiments of this application;

[0061] Figure 4 Another structural schematic diagram of the model provided in the embodiments of this application;

[0062] Figure 5 A schematic flowchart of the model training method provided in the embodiments of this application;

[0063] Figure 6 A schematic diagram of a feature extraction network provided in an embodiment of this application;

[0064] Figure 7 A schematic diagram of the bidirectional lookup process provided in the embodiments of this application;

[0065] Figure 8 A schematic diagram of an image reconstruction network provided in an embodiment of this application;

[0066] Figure 9 A schematic diagram of a feature enhancement module provided in an embodiment of this application;

[0067] Figure 10 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0068] Figure 11 A schematic diagram of the execution device provided in the embodiments of this application;

[0069] Figure 12 A schematic diagram of the training device provided in the embodiments of this application;

[0070] Figure 13 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0071] Figure 14 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0072] Figure 15 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0073] This application provides image processing methods, model training methods, related devices and equipment to improve the stability of models, thereby improving the stability of shading effects.

[0074] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0075] The above-mentioned artificial intelligence framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0076] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0077] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0078] (1) Infrastructure.

[0079] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, including central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs) and other hardware acceleration chips. The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0080] (2) Data.

[0081] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0082] (3) Data processing.

[0083] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0084] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0085] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0086] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0087] (4) General ability.

[0088] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0089] (5) Smart products and industry applications.

[0090] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, smart cities, and intelligent terminals.

[0091] Please refer to the following. Figure 2 , Figure 2 This is a system architecture diagram of an image processing system provided in an embodiment of this application. Figure 2 In the illustrated embodiment, the image processing system includes an execution device 210, a training device 220, a database 230, a terminal device 240, a data storage system 250, and a data acquisition device 260. The execution device 210 includes a computing module 211 and an input / output (I / O) interface 212.

[0092] Data acquisition device 260 is used to acquire a large-scale training dataset and store the training dataset in database 230. Training device 220 trains the model 201 constructed in this application based on the training dataset maintained in database 230. The trained model 201 runs on execution device 210. Execution device 210 can access data, code, etc. in data storage system 250, and can also store data, code, etc. in data storage system 250.

[0093] Regarding the image processing method in this embodiment, the data acquired by the data acquisition device 260 can be a black-and-white first image and a color second image. The execution device 210 can acquire the first and second images stored in the database 230 or the data storage system 250, or input by the terminal device 240 through the I / O interface 212, and process the first and second images to obtain feature maps corresponding to each image. Then, the execution device 210 can send the feature maps to the database 230 through the I / O interface 212, or send the feature maps to the terminal device 240 through the I / O interface 212, and the terminal device 240 will then send the feature maps to the database 230. The training device 220 can acquire the feature maps in the database 230 as training data for the training model 201 until the model 201 converges or reaches a preset number of training iterations, thus completing the training. The enhanced image predicted by the execution device 210 can be output to the terminal device 240 through the I / O interface 212, or directly displayed on the display interface of the terminal device 240.

[0094] It is important to note that Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application, and does not constitute any limitation on the positional relationships between various devices, components, modules, etc. For example, in Figure 2 In the illustrated embodiment, the data storage system 250 is an external memory relative to the execution device 210. In some alternative embodiments, the data storage system 250 may also be integrated into the execution device 210. Figure 2 In the illustrated embodiment, the terminal device 240 is an external device relative to the execution device 210. In some optional embodiments, the terminal device 240 may also be integrated into the execution device 210. This is not limited here.

[0095] This application provides a model for image processing that can convert black and white images into color images. The model may include a feature extraction network, a bidirectional search fusion module, and an image reconstruction network. It may also include a feature extraction network, a feature enhancement module, a bidirectional search fusion module, and an image reconstruction network; the specific components are not limited here.

[0096] The model provided in the embodiments of this application will be described below. Please refer to [link / reference]. Figure 3 and Figure 4 , Figure 3 and Figure 4 All of these are schematic diagrams of the models provided in the embodiments of this application.

[0097] like Figure 3As shown, model 300 includes a feature extraction network 301, a bidirectional search fusion module 302, and an image reconstruction network 303. In some optional embodiments, the model provided in this application also includes a feature enhancement module, such as... Figure 4 As shown, model 400, based on model 300, also includes feature enhancement module 404.

[0098] Based on model 300, the principle of the image processing method provided in this application embodiment is briefly introduced: Feature extraction network 301 extracts features from a black-and-white first image and a color second image to obtain a first feature map corresponding to the first image and a second feature map corresponding to the second image. Bidirectional lookup fusion module 302 extracts multiple first feature elements from the first feature map and multiple second feature elements from the second feature map. Based on the similarity between the multiple first feature elements and the multiple second feature elements, and the similarity between the multiple first feature elements, it determines the second feature element corresponding to each first feature element. Then, it fuses each first feature element and its corresponding second feature element to obtain a fused feature. Image reconstruction network 303 processes the fused feature to obtain an enhanced image corresponding to the first image, which is a color image. The second image can also be called a reference image, providing a reference for the conversion of the first image.

[0099] In model 400, the feature extraction module 401, the bidirectional search fusion module 402, and the image reconstruction network module 403 are respectively referred to in the previously described feature extraction module 301, bidirectional search fusion module 302, and image reconstruction network module 303, and will not be repeated here. Model 400 also includes a feature enhancement processing module 404, which processes the first image to obtain enhancement parameters. These enhancement parameters participate in the feature fusion process, making the fused features more accurate.

[0100] It is understood that the application of a model generally includes two stages: the training stage and the inference stage. The model training method and image processing method provided in the embodiments of this application will be described below from the perspectives of these two stages.

[0101] I. Training Phase.

[0102] Please see Figure 5 , Figure 5 A flowchart illustrating the model training method provided in this application embodiment includes the following steps:

[0103] 501. Using a feature extraction network, multiple first feature elements are obtained from a first image and multiple second feature elements are obtained from a second image, wherein the first image is a black and white image and the second image is a color image.

[0104] The training device acquires a black-and-white first image and a color second image. Through a feature extraction network, it obtains a first feature map corresponding to the first image and a second feature map corresponding to the second image. A bidirectional lookup fusion module then obtains multiple first feature elements from the first feature map and multiple second feature elements from the second feature map.

[0105] Optionally, the training device can obtain feature elements from the feature map using sliding window sampling. Assuming the first feature map has a size of 5×5, using sliding window sampling with a stride of 2 and a feature element size of 1×1, four first feature elements can be obtained from the first feature map. In practical applications, feature elements can also be obtained using other methods, such as downsampling based on mean pooling or max pooling; specific methods are not limited here.

[0106] Optionally, the display objects included in the second image can be of the same type as those included in the first image, such as trees, clouds, and blue sky; or they can include display objects of different types, which is not limited here. The second image can also be called a reference image. The model can learn the style, color, etc. of the second image and transform the first image into a color image with the same or similar style as the second image. The number of second images can be one or more, which is not limited here.

[0107] The feature extraction network comprises multiple units, each of which may include convolutional modules, regularization modules, and downsampling modules; it may also include only convolutional modules and regularization modules, but the specifics are not limited here. Figure 6 Taking an example, the feature extraction network provided in the embodiments of this application will be described. Please refer to... Figure 6 , Figure 6 This is a schematic diagram of a feature extraction network provided in an embodiment of this application.

[0108] like Figure 6 As shown, the feature extraction network comprises three units. The first two units each include a convolution (Conv) module, a regularization (Norm) module, and a downsampling (Bilinear) module. The third unit includes a convolution module and a regularization module. The convolution module, as the foundation of the neural network, performs convolution processing on the first and second images respectively. Since the model in this embodiment receives not only a black-and-white first image but also a color second image, a regularization module is used for regularization processing to accommodate multiple inputs. The downsampling module reduces the resolution of the first and second images, thereby reducing the amount of data during computation and saving computational resources.

[0109] It is important to note that Figure 6This is merely an example of a feature extraction network and does not constitute a limitation on the feature extraction network provided in this application. In practical applications, the feature extraction network may include more or fewer units, which is not limited here. The number of downsampling modules may be different from or the same as the number of units in the feature extraction network, which is not limited here.

[0110] 502. Based on the similarity between multiple first feature elements and multiple second feature elements, determine M second feature elements from the multiple second feature elements, where M is a positive integer.

[0111] After the training device acquires multiple first feature elements and multiple second feature elements, it uses a bidirectional search fusion module to find the corresponding second feature element for each first feature element. First, based on the similarity between the multiple first feature elements and the multiple second feature elements, the multiple second feature elements are filtered to determine M second feature elements. The similarity between one of these M second feature elements and one of the multiple first feature elements is greater than or equal to a first threshold. This initial filtering process of multiple second feature elements can be called the forward search process.

[0112] It should be noted that the value of the first threshold is determined based on the needs of the actual application, and no specific limit is imposed here. It is understandable that a smaller first threshold allows for more comprehensive filtering of multiple second feature elements; a larger first threshold reduces the computational load.

[0113] 503. Based on the similarity between M second feature elements and multiple first feature elements, and the similarity between multiple first feature elements, determine N first target elements from the M second feature elements, where N is a positive integer.

[0114] After obtaining M second feature elements, the training device performs a reverse lookup on these M second feature elements. Based on the similarity between these M second feature elements and multiple first feature elements, the first feature elements with the highest similarity are determined in descending order of similarity. Then, based on the similarity between the multiple first feature elements and the first feature elements corresponding to these M second feature elements, the first feature elements corresponding to these M second feature elements are filtered to select N first target elements. These N first target elements are contained within the M second feature elements. The similarity between one of the first feature elements corresponding to these N first target elements and one of the multiple first feature elements is greater than or equal to a second threshold. In short, this process involves a reverse lookup on the second feature elements, followed by filtering the first feature elements found through the reverse lookup, ultimately determining N first target elements from the M second feature elements.

[0115] To clearly illustrate the process shown in steps 502 and 503, i.e., the bidirectional lookup process, please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of a bidirectional lookup process provided in an embodiment of this application.

[0116] like Figure 7 As shown, taking the search for the first target element corresponding to the first feature element A1 as an example, the following explanation is provided. Based on the similarity between the first feature element A1 and multiple second feature elements, two second feature elements with the highest similarity to the first feature element A1 are determined from among the multiple second feature elements; these are the second feature element B1 and the second feature element B2. Specifically, the similarity between the first feature element A1 and either the second feature element B1 or the second feature element B2 is greater than a first threshold.

[0117] Perform a reverse lookup on these two second feature elements, searching for the first feature element for each second feature element in descending order of similarity. Figure 7 In the illustrated embodiment, it is assumed that for each second feature element, two first feature elements are searched. Second feature element B1 is used to find first feature elements A2 and A3, and second feature element B2 is used to find first feature elements A4 and A5. Then, based on the similarity between first feature element A1 and each of the first feature elements A2 to A5, these four first feature elements are filtered to determine the first feature elements whose similarity to first feature element A1 is greater than or equal to a second threshold.

[0118] If only the similarity between the first feature element A3 and the first feature element A1 is greater than or equal to the second threshold, then the second feature element B1 can be determined as the first target element corresponding to the first feature element A1. If the similarity between the first feature elements A3 and A4 and the first feature element A1 is greater than or equal to the second threshold, and the similarity between the first feature elements A3 and A1 is higher than the similarity between the first feature elements A4 and A1, then the training device will select the second feature element B1 corresponding to the first feature element A3 as the first target element corresponding to the first feature element A1. If the similarity between the first feature elements A3 and A4 and the first feature element A1 is greater than or equal to the second threshold, and the similarity between the first feature elements A3 and A1 is equal to the similarity between the first feature elements A4 and A1, then the training device can select either the second feature element B1 or B2, or select the second feature element B1 or B2 with the higher similarity to the first feature element A1, as the first target element corresponding to the first feature element A1. If the similarity between the first feature elements A2 to A5 and the first feature element A1 is less than the second threshold, then the training device determines that the first feature element A1 does not have a corresponding first target element.

[0119] In summary, the bidirectional search process can be described as follows: For any first feature element A, forward search finds the *a* second feature elements with the highest similarity to it. For each of the *a* second feature elements, backward search finds the *b* first feature elements with the highest similarity to it. In other words, for each first feature element, *a*×*b* first feature elements are found. Here, *a* and *b* are both positive integers, and *a* and *b* can be the same or different; no specific restrictions are imposed here.

[0120] Simultaneously, a first threshold is set for the similarity between the first feature element and the second feature element, and a second threshold is set for the similarity between the first feature element and these a×b first feature elements. All found first and second feature elements are then filtered based on these two thresholds. If multiple second feature elements satisfy the criteria, the first feature element with the highest similarity to first feature element A is selected as the first target element. If no second feature element satisfies the criteria, the first target element corresponding to first feature element A is considered to be 0.

[0121] Based on the above principle, the first target element corresponding to each first feature element is determined from multiple second feature elements, thereby determining N first target elements corresponding to multiple first feature elements.

[0122] In this embodiment of the application, by setting the similarity between the first feature element and the second feature element, and the similarity between the first feature element and the first feature element corresponding to the second feature element, the similarity between the first target element and the first feature element obtained after multiple screenings is ensured, thereby improving the fusion effect and making the coloring effect closer to the ideal color.

[0123] 504. Fuse multiple first feature elements and N first target elements to obtain fused features, which are used to reconstruct the enhanced image corresponding to the first image.

[0124] After determining N primary target elements, the training device fuses multiple primary feature elements with the N primary target elements to obtain fused features. There are various fusion methods, including element-wise addition, channel-wise concatenation, and other methods such as normalizing the elements and then incorporating ground truth values. The specific methods for incorporating ground truth values ​​are also varied, including adding or subtracting the mean, or multiplying by the variance or standard deviation of the ground truth values.

[0125] 505. The image reconstruction network processes the fusion features to obtain the enhanced image corresponding to the first image. The enhanced image is a color image.

[0126] After obtaining the fusion features, the training device inputs these features into the image reconstruction network. Through processing by the image reconstruction network, a color-enhanced image corresponding to the first image is obtained. The number of enhanced images can be one or more; there is no specific limitation here.

[0127] In some optional embodiments, the image reconstruction network includes multiple branches, each branch including at least one image reconstruction module. The training device processes the fusion features through the multiple branches respectively to obtain multiple enhanced images corresponding to the first image, and each enhanced image has a different color. Any two branches include at least one or at least two identical image reconstruction modules. Each image reconstruction module may include a processing submodule and a downsampling submodule, and may also include a processing submodule; the specific details are not limited here. Figure 8 For example, the image reconstruction network provided in the embodiments of this application will be described. Please refer to... Figure 8 , Figure 8 This is a schematic diagram of an image reconstruction network provided in an embodiment of this application. Figure 8 The image reconstruction network shown can also be called a multi-tailed reconstruction network.

[0128] like Figure 8As shown, both branch one and branch two include image reconstruction module 810 and image reconstruction module 821. Branch one also includes image reconstruction module 831, and branch two includes image reconstruction module 832. Branch one and branch two share the output of image reconstruction module 810 (i.e., the first intermediate feature obtained after processing the fused features) and the output of image reconstruction module 821 (i.e., the second intermediate feature obtained after processing the first intermediate feature). Both image reconstruction modules 831 and 832 have processing submodules, but the parameters of the processing submodules are different, resulting in different colors in the enhanced images obtained after processing.

[0129] like Figure 8 As shown, image reconstruction modules 810, 821, 822, and 831 have the same structure, all including a processing submodule and an upsampling submodule. The remaining image reconstruction modules also have the same structure, including a processing submodule. However, the parameters of the processing submodules differ across the image reconstruction modules. The processing submodule is used to perform operations such as convolution on the fused features or intermediate features obtained from the image reconstruction modules. The upsampling submodule is used to increase the resolution of the feature map to determine the resolution of the output enhanced image.

[0130] Optionally, the number of upsampling submodules in a branch of the model can be different from or the same as the number of downsampling submodules in the feature extraction network; this is not limited here. If the number of upsampling submodules in a branch of the model is the same as the number of downsampling submodules in the feature extraction network, then the resolution of the final output enhanced image is the same as the resolution of the first image. If the number of upsampling submodules is greater than the number of downsampling submodules in the feature extraction network, then the resolution of the enhanced image is greater than the resolution of the first image. If the number of upsampling submodules is less than the number of downsampling submodules in the feature extraction network, then the resolution of the enhanced image is less than the resolution of the first image.

[0131] For example, suppose the feature extraction network in the model is Figure 6 The feature extraction network shown has the first image processed through two downsampling modules. Therefore, after... Figure 8 In the enhanced images output by the image reconstruction network shown, the resolution of the enhanced image at output one is greater than that of the first image, while the resolutions of the enhanced images at outputs two through five are the same as those of the first image. In this embodiment, the resolution of the enhanced images is enriched by flexibly configuring the number of upsampling submodules in the image reconstruction network.

[0132] In an image reconstruction network, any two branches among multiple branches include at least one or at least two identical image reconstruction modules. For example... Figure 8 As shown, branch six and all other branches include image reconstruction module 810; branches one and two both include image reconstruction module 810 and image reconstruction module 821; branches three to five all include image reconstruction module 810 and image reconstruction module 822. In practical applications, each branch may include a greater number of the same image reconstruction modules, which is not limited here.

[0133] By using this progressive setup, different branches in the image reconstruction network can share the output of the same image reconstruction module, thereby reducing the number of parameters in the image reconstruction network. In other words, without significantly increasing the number of parameters, multiple different results can be output, reducing the complexity of the network layout.

[0134] In this embodiment of the application, the image reconstruction network may further include at least one target image reconstruction module, the output of which serves as the input to at least two image reconstruction modules. Figure 8 In the illustrated embodiment, taking image reconstruction module 822 as the target image reconstruction module as an example, the output of image reconstruction module 822 serves as the input to image reconstruction modules 833, 834, and 835. This setup further reduces the number of parameters in the image reconstruction network and lowers the complexity of the network layout. Figure 8 In the embodiment shown, the image reconstruction network is divided into three levels. In practical applications, the image reconstruction network may include more or fewer levels, which is not limited here.

[0135] The training device can also set different loss functions for the image reconstruction network to obtain multiple enhanced images with different colors. Assume... Figure 8 The network outputs shown are O1, O2, O3, O4, and O5, respectively. The cosine similarity between Oi and Oj is defined as sim. ij The following loss function can be used to increase the difference in the effects of the five outputs:

[0136]

[0137] It is important to note that Figure 8 This is merely an example of an image reconstruction network and does not constitute a limitation on the image reconstruction network provided in this application. In practical applications, image reconstruction networks may include more or fewer branches, which is not limited here.

[0138] In this embodiment, multiple enhanced images with different colors corresponding to the first image are obtained through an image reconstruction network, thereby improving the diversity of the network output.

[0139] 506. Iterate through the training of the model, updating the parameters of the feature extraction network and the image reconstruction network until the model converges or reaches the preset first iteration number.

[0140] The training equipment iteratively trains the model, continuously updating the parameters of the feature extraction network and the image reconstruction network until the model converges and achieves good results, or until the preset first iteration number is reached.

[0141] In this embodiment, when determining the target elements corresponding to each first feature element in the first image, not only is the similarity between the first feature element and the second feature element considered, selecting M first target elements from the second feature elements; furthermore, based on the similarity between the M second feature elements and multiple first feature elements, as well as the similarity among multiple first feature elements, these M second feature elements are further filtered to obtain N first target elements that are ultimately fused with the first feature elements. This multiple filtering makes the referenced second feature elements for each first feature element more accurate, and also ensures that identical or similar first feature elements can be fused with identical or similar second feature elements. This guarantees that identical or similar first feature elements can achieve the same or similar fusion effect even in different images, thereby ensuring the stability between different image frames. In other words, the stability of the model is improved, thus improving the stability of the coloring effect.

[0142] In some optional embodiments, prior to step 504, the training device may further search for the second feature elements corresponding to the multiple first feature elements multiple times to further improve the accuracy of the model. Prior to step 504, the training device may also determine X second feature elements from the multiple second feature elements based on the similarity between the multiple first feature elements and the multiple second feature elements, where X is a positive integer. Then, based on the similarity between the X second feature elements and the multiple first feature elements, and the similarity between the multiple first feature elements, Y second target elements are determined from the X second feature elements, where Y is a positive integer. Wherein, the similarity between one of the multiple first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between one first feature element and a first feature element corresponding to the Y second target elements is greater than or equal to a fourth threshold.

[0143] It should be noted that the process of determining Y second target elements from multiple second feature elements can be performed before or after step 503, as long as it is performed before step 504. No specific limitation is made here.

[0144] Furthermore, there are multiple possibilities for determining Y second target elements from multiple second feature elements. The following explanation, taking the process after step 503 as an example, illustrates the possible scenarios:

[0145] 1. Use multiple primary feature elements as the starting point for a bidirectional search.

[0146] To make this clear, let's combine... Figure 7 The following explanation is provided. Assume that after the training device performs a bidirectional search on the first feature element A1, the first target element corresponding to the first feature element A1 is the second feature element B1. Then, for the first feature element A1, a bidirectional search is performed again following a process similar to steps 502 and 503 to obtain the second target element corresponding to the first feature element A1. The specific process is similar to steps 502 and 503, except that the third threshold used for the forward search is different from the first threshold in step 502, and / or the fourth threshold used for the reverse search is different from the second threshold in step 503. Specific details are not limited here.

[0147] Understandably, if the third threshold is greater than the first threshold, and / or the fourth threshold is greater than the second threshold, the scope of the search will be more accurate, further improving the similarity between the second target element and the first feature element; if the third threshold is less than the first threshold, and / or the fourth threshold is less than the second threshold, the scope of the search will be more comprehensive.

[0148] Based on the above principle, the second target element corresponding to each first feature element is determined from multiple second feature elements, thereby determining Y second target elements corresponding to multiple first feature elements.

[0149] 2. Using the first feature element found by reverse searching through M second feature elements as the starting point, perform a bidirectional search.

[0150] To make this clear, let's combine... Figure 7 To illustrate, let's take one of the multiple first feature elements, A1, as an example. Assume that after the bidirectional search processes in steps 502 and 503, among the four first feature elements retrieved by the reverse search of second feature elements B1 and B2, first feature element A2 has the highest similarity to first feature element A1. Then, in the subsequent search process, a bidirectional search is performed on first feature element A2 following a similar process to steps 502 and 503 to obtain the second feature element corresponding to first feature element A2. This second feature element corresponding to first feature element A2 is then used as the second target element corresponding to first feature element A1.

[0151] It should be noted that in this method, the values ​​of the third and fourth thresholds are determined according to the actual application requirements. The value of the third threshold can be the same as or different from the value of the first threshold; the value of the fourth threshold can be the same as or different from the value of the second threshold. No specific restrictions are imposed here.

[0152] Understandably, in this approach, the previous round of bidirectional search ensures the similarity between the first feature element A2 found in the previous round and the initial first feature element A1, meaning that the first feature element A1 and the first feature element A2 are the same or similar. Then, using the first feature element A2 found in the previous round as the starting point for this round of bidirectional search, another bidirectional search is performed, and the finally selected second feature element is used as the second target element corresponding to the initial first feature element A1. This ensures that the same or similar feature elements can correspond to the same target element. Therefore, if processing consecutive image frames in a video sequence, the same or similar first feature elements in different image frames can refer to the same second feature element, ensuring that the same or similar first feature elements have the same or similar representation in different enhanced images, guaranteeing the stability between image frames and reducing the possibility of video sequence jumps. For example, in a black and white movie, the same or similar leaves will all appear as the same or similar green in the color video obtained after model processing, and the playback of the color video will not have obvious jumps.

[0153] Based on the above principle, the second target element corresponding to each first feature element is determined from multiple second feature elements, thereby determining Y second target elements corresponding to multiple first feature elements.

[0154] It should be noted that in practical applications, the training device can perform more bidirectional search processes, but this is not limited here.

[0155] In this embodiment, by setting the similarity between the first feature element and the second feature element, as well as the similarity between the first feature element and the first feature element corresponding to the second feature element, the similarity between the second target element obtained after multiple screenings and the first feature element is ensured, thereby improving the fusion effect and making the colored effect closer to the ideal color. Repeatedly searching for second feature elements for multiple first feature elements further enhances the fusion effect.

[0156] After obtaining Y second target elements through training, there are multiple ways to obtain the fused features. The following explanation uses the process of two bidirectional searches as an example.

[0157] The training device determines multiple third target elements from N first target elements and Y second target elements. Then, it fuses these multiple first feature elements and multiple third target elements to obtain a fused feature. Specifically, assuming one first feature element corresponds to one first target element and one second target element, during fusion, these three elements can be fused together. Alternatively, the first feature element can be fused with the filtered first and second target elements. Other methods are also possible, such as processing the first and second target elements (e.g., weighted averaging) and then fusing the first feature element with the processed element. The specific method is not limited here. There are several possibilities for filtering the first and second target elements: selecting target elements with higher similarity to the first feature element, or selecting target elements with higher similarity from the first feature elements found by reverse searching between the first and second target elements. If a first feature element corresponds to only one first target element or one second target element, then the first feature element and its corresponding target element can be fused.

[0158] In addition, there are various ways to fuse feature elements, similar to step 504, as shown in step 504, which will not be repeated here.

[0159] In some optional embodiments, the model also includes a feature enhancement module, which can be trained by a training device. This part is described below; please refer to [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram of the feature enhancement module.

[0160] like Figure 9 As shown, the feature enhancement module includes a pre-trained model 901, a neural network 902, and an enhanced feature lookup table 903. The training device extracts multiple intermediate features from the first image using the pre-trained model 901; these intermediate features are the outputs of different network layers of the pre-trained model. Figure 9 In the illustrated embodiment, three intermediate features are extracted using a pre-trained model. These three intermediate features may include the outputs of shallow network layers and deep network layers in the pre-trained model. This extraction method enables the extracted intermediate features to fully reflect the first image.

[0161] After obtaining the intermediate features, the training device concatenates multiple intermediate features to obtain concatenated features. These concatenated features are then processed by a neural network 902 to obtain the unit vector corresponding to the first image. The neural network 902 can include multiple fully connected layers or multiple convolutional layers. These layers process the concatenated features and normalize the processing results across channels, transforming the multidimensional concatenated features into a one-dimensional unit vector.

[0162] Based on the unit vector, the enhancement parameters corresponding to the first image are determined by querying the enhancement feature lookup table 903. The enhancement feature lookup table 903 includes multiple key-value pairs, where the dimension of the key is the same as the dimension of the unit vector, and the value has any size. The lookup operation can be performed by calculating the similarity between the unit vector and each key, and then taking the value corresponding to the key with the highest similarity as the enhancement parameter.

[0163] The training device iteratively trains the neural network and the augmented feature lookup table, updating their parameters until they converge or reach a preset second iteration. The closer the augmented parameters output by the feature augmentation module after training are to the true values, the better.

[0164] It is important to note that Figure 9 This is just an example and does not constitute a limitation on intermediate features. In practical applications, more or fewer intermediate features can be extracted, but this is not limited here.

[0165] Alternatively, the pre-trained model can be of various types, such as the VGG19 network, the RESNET50 network, or other pre-trained models capable of processing images; no specific limitations are made here.

[0166] In this embodiment, the enhancement parameters corresponding to the first image can also be determined by querying the enhancement feature lookup table. These enhancement parameters can be used during image processing using the model, especially in determining fusion features, thus enriching the methods for determining enhancement features.

[0167] In some optional embodiments, the plurality of second feature elements further include a fourth target element, wherein the similarity between the fourth target element and any one of the plurality of first feature elements is less than the first threshold.

[0168] In some optional embodiments, the plurality of second feature elements further includes a fifth target element, wherein the similarity between the first feature element corresponding to the fifth target element and any one of the plurality of first feature elements is less than a second threshold. That is, the similarity between the first feature element found by reverse lookup from the fifth target element and any one of the first feature elements is less than the second threshold.

[0169] II. Reasoning Stage.

[0170] In this embodiment, the inference phase refers to the process by which the execution device processes the first image using the iteratively trained model. Please refer to... Figure 10 , Figure 10 A flowchart illustrating the image processing method provided in this application embodiment includes the following steps:

[0171] 1001. Using a feature extraction network, multiple first feature elements are obtained from a first image and multiple second feature elements are obtained from a second image, wherein the first image is a black and white image and the second image is a color image.

[0172] The device acquires a black-and-white first image and a color second image. Through a feature extraction network, it obtains a first feature map corresponding to the first image and a second feature map corresponding to the second image. A bidirectional lookup fusion module then obtains multiple first feature elements from the first feature map and multiple second feature elements from the second feature map.

[0173] In this embodiment, the specific implementation of step 1001 is similar to that of step 501 described above, except that the execution subject of step 1001 is an execution device, and the first image processed is not a training image but an input image (i.e., the real image to be processed). The remaining parts can be found in [reference needed]. Figure 5 The description of step 501 in the corresponding embodiment will not be repeated here.

[0174] 1002. Based on the similarity between multiple first feature elements and multiple second feature elements, determine M second feature elements from the multiple second feature elements, where M is a positive integer.

[0175] After the execution device acquires multiple first feature elements and multiple second feature elements, it uses a bidirectional search fusion module to find the corresponding second feature element for each first feature element. First, based on the similarity between the multiple first feature elements and the multiple second feature elements, the multiple second feature elements are filtered to determine M second feature elements. The similarity between one of these M second feature elements and one of the multiple first feature elements is greater than or equal to a first threshold. This initial filtering process of multiple second feature elements can be called the forward search process.

[0176] It should be noted that the value of the first threshold is determined based on the needs of the actual application, and no specific limit is imposed here. It is understandable that a smaller first threshold allows for more comprehensive filtering of multiple second feature elements; a larger first threshold reduces the computational load.

[0177] In this embodiment, the specific implementation of step 1002 is similar to that of step 502 described above, except that the execution subject of step 1002 is the execution device. The remaining details can be found in [reference needed]. Figure 5 The description of step 502 in the corresponding embodiment will not be repeated here.

[0178] 1003. Based on the similarity between M second feature elements and multiple first feature elements, and the similarity between multiple first feature elements, determine N first target elements from the M second feature elements, where N is a positive integer.

[0179] After obtaining M second feature elements, the execution device performs a reverse lookup on these M second feature elements. Based on the similarity between these M second feature elements and multiple first feature elements, the first feature elements with the highest similarity are determined in descending order of similarity. Then, based on the similarity between the multiple first feature elements and the first feature elements corresponding to these M second feature elements, the first feature elements corresponding to these M second feature elements are filtered to select N first target elements. These N first target elements are contained within the M second feature elements. The similarity between one of the first feature elements corresponding to these N first target elements and one of the multiple first feature elements is greater than or equal to a second threshold. In short, this process involves a reverse lookup on the second feature elements, followed by filtering the first feature elements found through the reverse lookup, ultimately determining N first target elements from the M second feature elements.

[0180] In this embodiment of the application, by setting the similarity between the first feature element and the second feature element, and the similarity between the first feature element and the first feature element corresponding to the second feature element, the similarity between the first target element and the first feature element obtained after multiple screenings is ensured, thereby improving the fusion effect and making the coloring effect closer to the ideal color.

[0181] In this embodiment, the specific implementation of step 1003 is similar to that of step 503 described above, except that the execution subject of step 1003 is the execution device. The remaining details can be found in [the relevant documentation]. Figure 5 The description of step 503 in the corresponding embodiment will not be repeated here.

[0182] 1004. Fuse multiple first feature elements and N first target elements to obtain fused features, which are used to reconstruct the enhanced image corresponding to the first image;

[0183] After determining N first target elements, the execution device fuses multiple first feature elements and N first target elements to obtain fused features. There are various fusion methods, including element-wise addition, channel-wise concatenation, and other methods such as normalizing elements and introducing relevant values ​​of enhancement parameters; specific methods are not limited here.

[0184] It is important to note that if the execution device uses a method of normalizing elements and then introducing relevant values ​​of enhancement parameters for feature element fusion, then the model structure utilized by the execution device is as follows: Figure 4 The model shown is 400. Specifically, the training device extracts multiple intermediate features from the first image using a pre-trained model. These intermediate features are the outputs of different network layers of the pre-trained model. Then, these intermediate features are concatenated to obtain concatenated features. These concatenated features are then processed by a neural network to obtain the unit vector corresponding to the first image. Finally, based on the unit vector, an enhancement feature lookup table is consulted to determine the enhancement parameters corresponding to the first image. A more detailed process is described in... Figure 9 As illustrated in the examples, this will not be repeated here. There are various ways to introduce relevant values ​​of the enhancement parameter, including adding or subtracting the mean, or multiplying by the variance or standard deviation of the enhancement parameter, etc., which are not specifically limited here.

[0185] In this embodiment of the application, the enhancement parameters corresponding to the first image can also be determined by querying the enhancement feature lookup table, and the enhancement parameters can be used in the process of determining the fusion features, thus enriching the determination method of enhancement features.

[0186] In this embodiment, the specific implementation of step 1004 is similar to that of step 504 described above, except that the execution subject of step 1004 is the execution device, and if the feature elements are fused by introducing the correlation value of the enhancement parameter after normalizing the elements, the correlation value of the enhancement parameter is introduced instead of the true value in step 504. The remaining details can be found in [reference needed]. Figure 5 The description of step 504 in the corresponding embodiment will not be repeated here.

[0187] 1005. The image reconstruction network processes the fusion features to obtain the enhanced image corresponding to the first image. The enhanced image is a color image.

[0188] After obtaining the fusion features, the training device inputs these features into the image reconstruction network. Through processing by the image reconstruction network, a color-enhanced image corresponding to the first image is obtained. The number of enhanced images can be one or more; there is no specific limitation here.

[0189] In some alternative embodiments, the image reconstruction network can be as follows: Figure 8 As shown, the system includes multiple branches, each of which includes at least one image reconstruction module. The execution device processes the fusion features through each of the multiple branches to obtain multiple enhanced images corresponding to the first image, and each enhanced image has a different color. Any two branches include at least one or at least two identical image reconstruction modules.

[0190] In this embodiment, the specific implementation of step 1005 is similar to that of step 505 described above, except that the execution subject of step 1005 is the execution device. The remaining details can be found in [the relevant documentation]. Figure 5 The description of step 504 in the corresponding embodiment will not be repeated here.

[0191] In this embodiment, multiple enhanced images with different colors corresponding to the first image are obtained through an image reconstruction network, thereby improving the diversity of the network output.

[0192] In this embodiment, when determining the target elements corresponding to each first feature element in the first image, not only is the similarity between the first feature element and the second feature element considered, selecting M first target elements from the second feature elements; furthermore, based on the similarity between the M second feature elements and multiple first feature elements, as well as the similarity among multiple first feature elements, these M second feature elements are further filtered to obtain N first target elements that are ultimately fused with the first feature elements. This multiple filtering makes the referenced second feature elements for each first feature element more accurate, and also ensures that identical or similar first feature elements can be fused with identical or similar second feature elements. This guarantees that identical or similar first feature elements can achieve the same or similar fusion effect even in different images, thereby ensuring the stability between different image frames. In other words, the stability of the model is improved, thus improving the stability of the coloring effect.

[0193] Furthermore, based on experimental results, the beneficial effects of the image processing method provided in the embodiments of this application are explained, as shown in Table 1 below:

[0194] Table 1

[0195] PSNR 27.68 31.23 34.64 35.89 SSIM 0.8362 0.8898 0.9469 0.9494

[0196] Among them, the InstaColor method, FAV method, and DeepRemaster method are all existing methods capable of converting black and white images into color images. PSNR, or Peak Signal-to-Noise Ratio, is used to evaluate image quality; a higher value indicates higher quality of the enhanced image. SSIM, or Structural Similarity, reflects the similarity between the first image and the second image (i.e., the image to be processed and the reference image); a higher value indicates that the color style of the output enhanced image is closer to that of the reference image, meaning the enhanced image's colors are closer to the ideal colors. As shown in Table 1, the technical solution of this application is superior to these solutions and possesses excellent performance.

[0197] In some optional embodiments, before step 1004, the execution device may search for the second feature elements corresponding to the multiple first feature elements multiple times. Taking two searches as an example, before step 504, the execution device may also determine X second feature elements from the multiple second feature elements based on the similarity between the multiple first feature elements and the multiple second feature elements, where X is a positive integer. Then, based on the similarity between the X second feature elements and the multiple first feature elements, and the similarity between the multiple first feature elements, Y second target elements are determined from the X second feature elements, where Y is a positive integer. Wherein, the similarity between one of the multiple first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between one first feature element and one of the Y second target elements is greater than or equal to a fourth threshold. The values ​​of the third and fourth thresholds are determined according to the needs of the actual application. The value of the third threshold can be the same as or different from the value of the first threshold; the value of the fourth threshold can be the same as or different from the value of the second threshold, and no specific limitation is made here.

[0198] The process by which the execution device determines Y second target elements from multiple second feature elements is similar to the process by which the training device determines Y second target elements from multiple second feature elements, as explained above, and will not be repeated here.

[0199] It should be noted that in practical applications, the execution device can perform more bidirectional lookup processes, but this is not limited here.

[0200] In this embodiment, the second feature element is repeatedly searched for multiple first feature elements to further improve the fusion effect. Simultaneously, by setting the similarity between the first and second feature elements, and the similarity between the first and second feature elements corresponding to the first feature elements, the similarity between the second target element obtained after multiple screenings and the first feature elements is ensured, thereby improving the fusion effect and making the colored effect closer to the ideal color.

[0201] In some optional embodiments, the plurality of second feature elements further include a fourth target element, wherein the similarity between the fourth target element and any one of the plurality of first feature elements is less than a first threshold.

[0202] In some optional embodiments, the plurality of second feature elements further include a fifth target element, wherein the first feature element corresponding to the fifth target element has a similarity to any one of the plurality of first feature elements that is less than a second threshold.

[0203] Next, the relevant devices and equipment provided in the embodiments of this application will be described.

[0204] Please see Figure 11 , Figure 11 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1100 includes:

[0205] The acquisition unit 1101 is used to acquire multiple first feature elements from a first image and multiple second feature elements from a second image through a feature extraction network, wherein the first image is a black and white image and the second image is a color image.

[0206] The determining unit 1102 is used to determine M second feature elements from multiple second feature elements based on the similarity between multiple first feature elements and multiple second feature elements, where M is a positive integer; and to determine N first target elements from the M second feature elements based on the similarity between the M second feature elements and multiple first feature elements, as well as the similarity between multiple first feature elements, where N is a positive integer.

[0207] The processing unit 1103 is used to fuse multiple first feature elements and N first target elements to obtain fused features, which are used to reconstruct the enhanced image corresponding to the first image; the fused features are processed by an image reconstruction network to obtain the enhanced image corresponding to the first image, and the enhanced image is a color image.

[0208] In some optional embodiments, the similarity between one of the multiple first feature elements and one of the M second feature elements is greater than or equal to a first threshold, and the similarity between one first feature element and one of the N first target elements is greater than or equal to a second threshold.

[0209] In some optional embodiments, the determining unit 1102 is further configured to determine X second feature elements from the plurality of second feature elements based on the similarity between the plurality of first feature elements and the plurality of second feature elements, where X is a positive integer; and to determine Y second target elements from the X second feature elements based on the similarity between the X second feature elements and the plurality of first feature elements, and the similarity between the plurality of first feature elements, where Y is a positive integer.

[0210] The processing unit 1103 is specifically used to determine multiple third target elements from N first target elements and Y second target elements; and to fuse multiple first feature elements and multiple third target elements to obtain fused features.

[0211] In some optional embodiments, the similarity between one of the multiple first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between one first feature element and one of the Y second target elements is greater than or equal to a fourth threshold.

[0212] In some alternative embodiments, the image reconstruction network includes multiple branches, each of which includes at least one image reconstruction module.

[0213] The processing unit 1103 is specifically used to process the fusion features through multiple branches to obtain the enhanced image corresponding to the first image. There are multiple enhanced images, and each enhanced image has a different color.

[0214] In some alternative embodiments, any two of the multiple branches include at least one or at least two identical image reconstruction modules.

[0215] In some optional embodiments, the processing unit 1103 is further configured to extract multiple intermediate features from the first image using a pre-trained model, wherein the multiple intermediate features are the outputs of different network layers of the pre-trained model; concatenate the multiple intermediate features to obtain concatenated features; process the concatenated features through a neural network to obtain a unit vector corresponding to the first image; and query an enhancement feature lookup table based on the unit vector to determine the enhancement parameters corresponding to the first image, wherein the enhancement parameters are used to determine the fusion features.

[0216] In some optional embodiments, the acquisition unit 1101 is specifically used to acquire a first feature map corresponding to the first image and a second feature map corresponding to the second image through a feature extraction network; acquire multiple first feature elements from the first feature map and multiple second feature elements from the second feature map.

[0217] The actuator 1100 can perform the aforementioned Figure 1 , Figure 2 and Figure 10 The operations performed by the execution device in the illustrated embodiment will not be described in detail here.

[0218] Please see Figure 12 , Figure 12 A schematic diagram of a training device provided in an embodiment of this application is shown. The training device 1200 includes:

[0219] The acquisition unit 1201 is used to acquire multiple first feature elements from a first image and multiple second feature elements from a second image through a feature extraction network. The first image is a black and white image and the second image is a color image.

[0220] The determining unit 1202 is used to determine M second feature elements from multiple second feature elements based on the similarity between multiple first feature elements and multiple second feature elements, where M is a positive integer; and to determine N first target elements from the M second feature elements based on the similarity between the M second feature elements and multiple first feature elements, as well as the similarity between multiple first feature elements, where N is a positive integer.

[0221] The processing unit 1203 is used to fuse multiple first feature elements and N first target elements to obtain fused features, which are used to reconstruct the enhanced image corresponding to the first image; the fused features are processed by an image reconstruction network to obtain the enhanced image corresponding to the first image, which is a color image; the model is iteratively trained, and the parameters of the feature extraction network and the image reconstruction network are updated until the model converges or reaches the preset first iteration number.

[0222] In some optional embodiments, the similarity between one of the multiple first feature elements and one of the M second feature elements is greater than or equal to a first threshold, and the similarity between one first feature element and one of the N first target elements is greater than or equal to a second threshold.

[0223] In some optional embodiments, the determining unit 1202 is further configured to determine X second feature elements from the plurality of second feature elements based on the similarity between the plurality of first feature elements and the plurality of second feature elements, where X is a positive integer; and to determine Y second target elements from the X second feature elements based on the similarity between the X second feature elements and the plurality of first feature elements, and the similarity between the plurality of first feature elements, where Y is a positive integer.

[0224] The processing unit 1203 is specifically used to determine multiple third target elements from N first target elements and Y second target elements; and to fuse multiple first feature elements and multiple third target elements to obtain fused features.

[0225] In some optional embodiments, the similarity between one of the multiple first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between one first feature element and one of the Y second target elements is greater than or equal to a fourth threshold.

[0226] In some alternative embodiments, the image reconstruction network includes multiple branches, each of which includes at least one image reconstruction module;

[0227] The processing unit 1203 is specifically used to process the fusion features through multiple branches to obtain the enhanced image corresponding to the first image. There are multiple enhanced images, and each enhanced image has a different color.

[0228] In some alternative embodiments, any two of the multiple branches include at least one or at least two identical image reconstruction modules.

[0229] In some optional embodiments, the processing unit 1203 is further configured to extract multiple intermediate features from the first image using a pre-trained model, wherein the multiple intermediate features are the outputs of different network layers of the pre-trained model; concatenate the multiple intermediate features to obtain concatenated features; process the concatenated features through a neural network to obtain a unit vector corresponding to the first image; query an enhancement feature lookup table based on the unit vector to determine the enhancement parameters corresponding to the first image, wherein the enhancement parameters are used to determine the fusion features; iteratively train the neural network and the enhancement feature lookup table, updating the parameters of the neural network and the parameters of the enhancement feature lookup table until the neural network and the enhancement feature lookup table converge, or reach a preset second iteration number.

[0230] In some optional embodiments, the acquisition unit 1201 is specifically used to: acquire a first feature map corresponding to the first image and a second feature map corresponding to the second image through a feature extraction network; acquire a plurality of first feature elements from the first feature map and acquire a plurality of second feature elements from the second feature map.

[0231] The training device 1200 can perform the aforementioned... Figures 1 to 9 The operations performed by the training device in the illustrated embodiment will not be described in detail here.

[0232] The execution device provided in the embodiments of this application will be described below. Please refer to [link / reference]. Figure 13 , Figure 13 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1300 includes a processor 1301 and a memory 1302, wherein the memory 1302 stores one or more application programs or data.

[0233] The memory 1302 can be volatile or persistent storage. The program stored in the memory 1302 can include one or more modules, each of which can be used to implement a series of operations performed by the execution device 1300. Furthermore, the processor 1301 can communicate with the memory 1302 and execute a series of instructions stored in the memory 1302 on the execution device 1300. The processor 1301 can be a central processing unit (CPU), a single-core processor, or other types of processors, such as a dual-core processor; specific limitations are not specified here.

[0234] The execution device 1300 may also include one or more communication interfaces 1303, and one or more operating systems, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM wait.

[0235] The execution device 1300 can perform the aforementioned Figure 1 , Figure 2 and Figure 10 The operations performed by the execution device in the illustrated embodiment will not be described in detail here.

[0236] The training equipment provided in the embodiments of this application will be described below. Please refer to [link / reference]. Figure 14 , Figure 14 This is a schematic diagram of a training device provided in an embodiment of this application. The training device 1400 includes a processor 1401 and a memory 1402, wherein the memory 1402 stores one or more applications or data.

[0237] The memory 1402 can be volatile or persistent storage. The program stored in the memory 1402 can include one or more modules, each of which can be used to execute a series of operations performed by the training device 1400. Furthermore, the processor 1401 can communicate with the memory 1402 to execute a series of instructions stored in the memory 1402 on the training device 1400. The processor 1401 can be a central processing unit (CPU), a single-core processor, or other types of processors, such as a dual-core processor; specific limitations are not specified here.

[0238] Training device 1400 may also include one or more communication interfaces 1403, and one or more operating systems, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM wait.

[0239] The training device 1400 can perform the aforementioned tasks. Figures 1 to 9 The operations performed by the training device in the illustrated embodiment will not be described in detail here.

[0240] Please see Figure 15 , Figure 15 This is a schematic diagram of the chip provided in this application. The chip can be represented as a neural network processor 1500. The neural network processor 1500 is mounted on the main CPU as a coprocessor, and the host CPU allocates tasks to it. The core of the neural network processor 1500 is the arithmetic circuit 1503. The controller 1504 can control the arithmetic circuit 1503 to extract data from the weight memory 1502 or the input memory 1501 and perform calculations.

[0241] In some implementations, the arithmetic circuit 1503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 1503 can be a two-dimensional pulsating array. The arithmetic circuit 1503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1503 is a general-purpose matrix processor.

[0242] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1508.

[0243] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 via direct memory access controller (DMAC) 1505. Input data is also transferred to unified memory 1506 via DMAC.

[0244] The bus interface unit 1510 (BIU) can be used to enable interaction between the main CPU, DMAC, and instruction fetch buffer 1509 (IFB) via a bus. The instruction fetch buffer 1509 is used to store instructions used by the controller 1504.

[0245] The vector computation unit 1507 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computations in non-convolutional / fully connected layers of neural networks, such as pixel-level summation and upsampling of feature planes.

[0246] in, Figures 1 to 10 The operations of each layer in the neural network shown in the corresponding embodiment can be performed by the operation circuit 1503 or the vector calculation unit 1507.

[0247] The processor mentioned above can be a central processing unit, a microprocessor, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0248] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0249] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0250] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0251] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0252] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image processing method, characterized in that, include: Multiple first feature elements are obtained from a first image and multiple second feature elements are obtained from a second image through a feature extraction network, wherein the first image is a black and white image and the second image is a color image; Based on the similarity between the plurality of first feature elements and the plurality of second feature elements, M second feature elements are determined from the plurality of second feature elements, where M is a positive integer; Based on the similarity between the M second feature elements and the plurality of first feature elements, and the similarity between the plurality of first feature elements, the first feature elements corresponding to the M second feature elements are filtered to determine N first target elements, where N is a positive integer; By fusing the plurality of first feature elements and the N first target elements, a fused feature is obtained, which is used to reconstruct the enhanced image corresponding to the first image; The fusion features are processed by an image reconstruction network to obtain the enhanced image corresponding to the first image, wherein the enhanced image is a color image.

2. The method according to claim 1, characterized in that, The similarity between one of the plurality of first feature elements and one of the M second feature elements is greater than or equal to a first threshold, and the similarity between one of the first feature elements and one of the first feature elements corresponding to the N first target elements is greater than or equal to a second threshold.

3. The method according to claim 1 or 2, characterized in that, Prior to obtaining the fusion features, the method further includes: Based on the similarity between the plurality of first feature elements and the plurality of second feature elements, X second feature elements are determined from the plurality of second feature elements, where X is a positive integer; Based on the similarity between the X second feature elements and the plurality of first feature elements, and the similarity among the plurality of first feature elements, Y second target elements are determined from the X second feature elements, where Y is a positive integer; The fusion of the plurality of first feature elements and the N first target elements to obtain the fused feature includes: From the N first target elements and the Y second target elements, determine a plurality of third target elements; The fused feature is obtained by fusing the plurality of first feature elements and the plurality of third target elements.

4. The method according to claim 3, characterized in that, The similarity between one of the plurality of first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between one of the first feature elements and one of the first feature elements corresponding to the Y second target elements is greater than or equal to a fourth threshold.

5. The method according to claim 1, characterized in that, The image reconstruction network includes multiple branches, and each of the multiple branches includes at least one image reconstruction module; The step of processing the fusion features through an image reconstruction network to obtain the enhanced image corresponding to the first image includes: The fusion features are processed by the multiple branches respectively to obtain the enhanced image corresponding to the first image. There are multiple enhanced images, and each enhanced image has a different color.

6. The method according to claim 5, characterized in that, Any two of the plurality of branches include at least one or at least two identical image reconstruction modules.

7. The method according to claim 1, characterized in that, The method further includes: Multiple intermediate features are extracted from the first image using a pre-trained model, and these multiple intermediate features are the outputs of different network layers of the pre-trained model. By concatenating the multiple intermediate features, a concatenated feature is obtained; The splicing features are processed by a neural network to obtain the unit vector corresponding to the first image; Based on the unit vector, an enhancement feature lookup table is queried to determine the enhancement parameters corresponding to the first image. The enhancement parameters are used to determine the fusion features.

8. The method according to claim 1, characterized in that, The step of obtaining multiple first feature elements from a first image and multiple second feature elements from a second image through a feature extraction network includes: The feature extraction network is used to obtain a first feature map corresponding to the first image and a second feature map corresponding to the second image. The plurality of first feature elements are obtained from the first feature map, and the plurality of second feature elements are obtained from the second feature map.

9. The method according to claim 1, characterized in that, The plurality of second feature elements also include a fourth target element, wherein the similarity between the fourth target element and any one of the plurality of first feature elements is less than a first threshold.

10. The method according to claim 1, characterized in that, The plurality of second feature elements also include a fifth target element, wherein the first feature element corresponding to the fifth target element has a similarity to any one of the plurality of first feature elements that is less than a second threshold.

11. A model training method, characterized in that, include: Multiple first feature elements are obtained from a first image and multiple second feature elements are obtained from a second image using a feature extraction network. The first image is a black and white image and the second image is a color image. Based on the similarity between the plurality of first feature elements and the plurality of second feature elements, M second feature elements are determined from the plurality of second feature elements, where M is a positive integer; Based on the similarity between the M second feature elements and the plurality of first feature elements, and the similarity between the plurality of first feature elements, the first feature elements corresponding to the M second feature elements are filtered to determine N first target elements, where N is a positive integer; By fusing the plurality of first feature elements and the N first target elements, a fused feature is obtained, which is used to reconstruct the enhanced image corresponding to the first image; The fusion features are processed by an image reconstruction network to obtain the enhanced image corresponding to the first image, wherein the enhanced image is a color image. The model is trained iteratively, and the parameters of the feature extraction network and the image reconstruction network are updated until the model converges or reaches the preset first iteration number.

12. The method according to claim 11, characterized in that, The similarity between one of the plurality of first feature elements and one of the M second feature elements is greater than or equal to a first threshold, and the similarity between one of the first feature elements and one of the first feature elements corresponding to the N first target elements is greater than or equal to a second threshold.

13. The method according to claim 11 or 12, characterized in that, Prior to obtaining the fusion features, the method further includes: Based on the similarity between the plurality of first feature elements and the plurality of second feature elements, X second feature elements are determined from the plurality of second feature elements, where X is a positive integer; Based on the similarity between the X second feature elements and the plurality of first feature elements, and the similarity among the plurality of first feature elements, Y second target elements are determined from the X second feature elements, where Y is a positive integer; The fusion of the plurality of first feature elements and the N first target elements to obtain the fused feature includes: From the N first target elements and the Y second target elements, determine a plurality of third target elements; The fused feature is obtained by fusing the plurality of first feature elements and the plurality of third target elements.

14. The method according to claim 13, characterized in that, The similarity between one of the plurality of first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between one of the first feature elements and one of the first feature elements corresponding to the Y second target elements is greater than or equal to a fourth threshold.

15. The method according to claim 11, characterized in that, The image reconstruction network includes multiple branches, and each of the multiple branches includes at least one image reconstruction module; The step of processing the fusion features through an image reconstruction network to obtain the enhanced image corresponding to the first image includes: The fusion features are processed by the multiple branches respectively to obtain the enhanced image corresponding to the first image. There are multiple enhanced images, and each enhanced image has a different color.

16. The method according to claim 15, characterized in that, Any two of the plurality of branches include at least one or at least two identical image reconstruction modules.

17. The method according to claim 11, characterized in that, The method further includes: Multiple intermediate features are extracted from the first image using a pre-trained model, and these multiple intermediate features are the outputs of different network layers of the pre-trained model. By concatenating the multiple intermediate features, a concatenated feature is obtained; The splicing features are processed by a neural network to obtain the unit vector corresponding to the first image; Based on the unit vector, the enhancement feature lookup table is queried to determine the enhancement parameters corresponding to the first image, and the enhancement parameters are used to determine the fusion features; The neural network and the enhanced feature lookup table are iteratively trained, and the parameters of the neural network and the enhanced feature lookup table are updated until the neural network and the enhanced feature lookup table converge, or until a preset second iteration number is reached.

18. The method according to claim 11, characterized in that, The step of obtaining multiple first feature elements from a first image and multiple second feature elements from a second image through a feature extraction network includes: The feature extraction network is used to obtain a first feature map corresponding to the first image and a second feature map corresponding to the second image. The plurality of first feature elements are obtained from the first feature map, and the plurality of second feature elements are obtained from the second feature map.

19. The method according to claim 11, characterized in that, The plurality of second feature elements also include a fourth target element, wherein the similarity between the fourth target element and any one of the plurality of first feature elements is less than a first threshold.

20. The method according to claim 11, characterized in that, The plurality of second feature elements also include a fifth target element, wherein the first feature element corresponding to the fifth target element has a similarity to any one of the plurality of first feature elements that is less than a second threshold.

21. An actuator, characterized in that, include: The acquisition unit is used to acquire multiple first feature elements from a first image and multiple second feature elements from a second image through a feature extraction network, wherein the first image is a black and white image and the second image is a color image; Determine the unit, used for: Based on the similarity between the plurality of first feature elements and the plurality of second feature elements, M second feature elements are determined from the plurality of second feature elements, where M is a positive integer; Based on the similarity between the M second feature elements and the plurality of first feature elements, and the similarity between the plurality of first feature elements, the first feature elements corresponding to the M second feature elements are filtered to determine N first target elements, where N is a positive integer; Processing unit, used for: By fusing the plurality of first feature elements and the N first target elements, a fused feature is obtained, which is used to reconstruct the enhanced image corresponding to the first image; The fusion features are processed by an image reconstruction network to obtain the enhanced image corresponding to the first image, wherein the enhanced image is a color image.

22. The apparatus according to claim 21, characterized in that, The similarity between one of the plurality of first feature elements and one of the M second feature elements is greater than or equal to a first threshold, and the similarity between one of the first feature elements and one of the first feature elements corresponding to the N first target elements is greater than or equal to a second threshold.

23. The apparatus according to claim 21 or 22, characterized in that, The determining unit is further configured to: Based on the similarity between the plurality of first feature elements and the plurality of second feature elements, X second feature elements are determined from the plurality of second feature elements, where X is a positive integer; Based on the similarity between the X second feature elements and the plurality of first feature elements, and the similarity among the plurality of first feature elements, Y second target elements are determined from the X second feature elements, where Y is a positive integer; The processing unit is specifically used for: From the N first target elements and the Y second target elements, determine a plurality of third target elements; The fused feature is obtained by fusing the plurality of first feature elements and the plurality of third target elements.

24. The apparatus according to claim 23, characterized in that, The similarity between one of the plurality of first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between one of the first feature elements and one of the first feature elements corresponding to the Y second target elements is greater than or equal to a fourth threshold.

25. The apparatus according to claim 21, characterized in that, The image reconstruction network includes multiple branches, and each of the multiple branches includes at least one image reconstruction module; The processing unit is specifically used to process the fusion features through the multiple branches to obtain the enhanced image corresponding to the first image. The number of enhanced images is multiple, and each enhanced image has a different color.

26. The apparatus according to claim 21, characterized in that, The processing unit is further configured to: Multiple intermediate features are extracted from the first image using a pre-trained model, and these multiple intermediate features are the outputs of different network layers of the pre-trained model. By concatenating the multiple intermediate features, a concatenated feature is obtained; The splicing features are processed by a neural network to obtain the unit vector corresponding to the first image; Based on the unit vector, an enhancement feature lookup table is queried to determine the enhancement parameters corresponding to the first image. The enhancement parameters are used to determine the fusion features.

27. The apparatus according to claim 21, characterized in that, The acquisition unit is specifically used for: The feature extraction network is used to obtain a first feature map corresponding to the first image and a second feature map corresponding to the second image. The plurality of first feature elements are obtained from the first feature map, and the plurality of second feature elements are obtained from the second feature map.

28. A training device, characterized in that, include: The acquisition unit is used to acquire multiple first feature elements from a first image and multiple second feature elements from a second image through a feature extraction network, wherein the first image is a black and white image and the second image is a color image; Determine the unit, used for: Based on the similarity between the plurality of first feature elements and the plurality of second feature elements, M second feature elements are determined from the plurality of second feature elements, where M is a positive integer; Based on the similarity between the M second feature elements and the plurality of first feature elements, and the similarity between the plurality of first feature elements, the first feature elements corresponding to the M second feature elements are filtered to determine N first target elements, where N is a positive integer; Processing unit, used for: By fusing the plurality of first feature elements and the N first target elements, a fused feature is obtained, which is used to reconstruct the enhanced image corresponding to the first image; The fusion features are processed by an image reconstruction network to obtain the enhanced image corresponding to the first image, wherein the enhanced image is a color image. The model is trained iteratively, and the parameters of the feature extraction network and the image reconstruction network are updated until the model converges or reaches the preset first iteration number.

29. The apparatus according to claim 28, characterized in that, The similarity between one of the plurality of first feature elements and one of the M second feature elements is greater than or equal to a first threshold, and the similarity between one of the first feature elements and one of the first feature elements corresponding to the N first target elements is greater than or equal to a second threshold.

30. The apparatus according to claim 28 or 29, characterized in that, The determining unit is further configured to: Based on the similarity between the plurality of first feature elements and the plurality of second feature elements, X second feature elements are determined from the plurality of second feature elements, where X is a positive integer; Based on the similarity between the X second feature elements and the plurality of first feature elements, and the similarity among the plurality of first feature elements, Y second target elements are determined from the X second feature elements, where Y is a positive integer; The processing unit is specifically used for: From the N first target elements and the Y second target elements, determine a plurality of third target elements; The fused feature is obtained by fusing the plurality of first feature elements and the plurality of third target elements.

31. The apparatus according to claim 30, characterized in that, The similarity between one of the plurality of first feature elements and one of the X second feature elements is greater than or equal to a third threshold, and the similarity between one of the first feature elements and one of the first feature elements corresponding to the Y second target elements is greater than or equal to a fourth threshold.

32. The apparatus according to claim 28, characterized in that, The image reconstruction network includes multiple branches, and each of the multiple branches includes at least one image reconstruction module; The processing unit is specifically used to process the fusion features through the multiple branches to obtain the enhanced image corresponding to the first image. The number of enhanced images is multiple, and each enhanced image has a different color.

33. The apparatus according to claim 28, characterized in that, The processing unit is further configured to: Multiple intermediate features are extracted from the first image using a pre-trained model, and these multiple intermediate features are the outputs of different network layers of the pre-trained model. By concatenating the multiple intermediate features, a concatenated feature is obtained; The splicing features are processed by a neural network to obtain the unit vector corresponding to the first image; Based on the unit vector, the enhancement feature lookup table is queried to determine the enhancement parameters corresponding to the first image, and the enhancement parameters are used to determine the fusion features; The neural network and the enhanced feature lookup table are iteratively trained, and the parameters of the neural network and the enhanced feature lookup table are updated until the neural network and the enhanced feature lookup table converge, or until a preset second iteration number is reached.

34. The apparatus according to claim 28, characterized in that, The acquisition unit is specifically used for: The feature extraction network is used to obtain a first feature map corresponding to the first image and a second feature map corresponding to the second image. The plurality of first feature elements are obtained from the first feature map, and the plurality of second feature elements are obtained from the second feature map.

35. An execution device, characterized in that, Includes processor, memory, and communication interface; The processor, the memory, and the communication interface are connected; The processor is used to perform the method according to any one of claims 1 to 10.

36. A training device, characterized in that, Includes processor, memory, and communication interface; The processor, the memory, and the communication interface are connected; The processor is used to perform the method according to any one of claims 11 to 20.

37. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed by the computer, performs the method according to any one of claims 1 to 20.

38. A computer program product, characterized in that, When the computer program product is executed on a computer, the computer performs the method according to any one of claims 1 to 20.

39. A chip comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method of any one of claims 1 to 20.

Images