Image processing method and related device

Abstract

The embodiment of the present application discloses an image processing method for application in the field of artificial intelligence, comprising: obtaining an image to be processed; processing the image to be processed through a first network to obtain a first feature, wherein the first network is configured to extract at least a feature for image enhancement; processing the image to be processed through a second network to obtain a second feature, wherein the second network is configured to extract at least a semantic segmentation feature; generating a third feature based on the first feature and the second feature; obtaining a semantic segmentation result of the image to be processed; generating a fourth feature based on the third feature and the semantic segmentation result of the image to be processed; and reconstructing the fourth feature to obtain a target image. By introducing the semantic features and semantic segmentation results of the image during the image enhancement process, different image enhancement intensities can be used for different semantic regions, accurately maintaining texture details and improving the authenticity of texture details after image enhancement.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method and related devices. Background Art

[0002] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0003] Deep learning methods have been a key driver of recent developments in artificial intelligence, achieving remarkable results in a variety of computer vision tasks. In the field of image enhancement (also known as image quality enhancement), deep learning-based methods have surpassed traditional methods.

[0004] However, the current image enhancement network based on deep learning has an unnatural image enhancement effect, and the image texture details obtained after processing by the image enhancement network are not realistic. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method and related devices for improving image enhancement effects.

[0006] In a first aspect, the present application provides an image processing method, comprising: obtaining an image to be processed, which may be, for example, an image that requires image enhancement; processing the image to be processed through a first network to obtain a first feature, wherein the first network is configured to at least extract features for image enhancement, which may be, for example, low-level features of an image; processing the image to be processed through a second network to obtain a second feature, wherein the second network is configured to at least extract semantic segmentation features, which may be, for example, high-level features of an image; generating a third feature based on the first and second features; obtaining a semantic segmentation result of the image to be processed; generating a fourth feature based on the third feature and the semantic segmentation result of the image to be processed; and reconstructing the fourth feature to obtain a target image.

[0007] In this scheme, by introducing the semantic features and semantic segmentation results of the image during the image enhancement process, and fusing the semantic features and semantic segmentation results with the features used for image enhancement, different image enhancement intensities can be used for different semantic areas, accurately maintaining texture details and improving the authenticity of texture details after image enhancement.

[0008] Optionally, in a possible implementation, obtaining the semantic segmentation result of the image to be processed includes: processing the third feature through a third network to obtain the semantic segmentation result of the image to be processed.

[0009] Because the third feature is a fusion of low-level image features related to image enhancement and high-level image features related to semantic segmentation, processing the third feature allows for the introduction of low-level image features alongside high-level semantic segmentation features. This allows for the semantic segmentation results of the image to be processed to be obtained based on features at different levels, thereby improving the accuracy of the obtained semantic segmentation results.

[0010] Optionally, in a possible implementation, generating a third feature based on the first feature and the second feature includes: performing feature fusion processing on the first feature and the second feature to obtain the third feature; generating a fourth feature based on the third feature and the semantic segmentation result of the image to be processed includes: performing feature fusion processing on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature.

[0011] Optionally, in a possible implementation, the feature fusion processing includes summation processing, multiplication processing, cascade processing, or cascade processing and convolution processing.

[0012] Optionally, in a possible implementation, before generating the fourth feature based on the third feature and the semantic segmentation result of the image to be processed, the method also includes: processing the third feature to obtain a fifth feature; generating the fourth feature based on the third feature and the semantic segmentation result of the image to be processed includes: generating the fourth feature based on the fifth feature and the semantic segmentation result of the image to be processed.

[0013] That is, after extracting the third feature, the image processing device performs feature fusion based on the fifth feature obtained through further feature extraction and the semantic segmentation result of the image to be processed. By further extracting the third feature obtained after feature fusion, a finer-grained feature can be extracted based on the third feature, thereby improving the accuracy of the fourth feature obtained through subsequent feature fusion.

[0014] Optionally, in one possible implementation, processing the image to be processed using the second network to obtain the second feature includes: preprocessing the image to be processed to obtain a preprocessed feature; downsampling the preprocessed feature to obtain a downsampled feature; processing the downsampled feature using the second network to obtain a sixth feature; and upsampling the sixth feature to obtain the second feature of the image to be processed. By reducing the resolution of the preprocessed feature through the downsampling operation, the computational complexity of extracting semantic segmentation features can be reduced, thereby lowering the computing power requirements of the image processing device.

[0015] Optionally, in one possible implementation, the method is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image defogging, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

[0016] The second aspect of the present application provides a model training method, comprising: obtaining a training sample pair, the training sample pair comprising a first image and a second image, the quality of the first image being lower than that of the second image; processing the first image through an image processing model to be trained to obtain a predicted image, wherein the image processing model to be trained is used to obtain the image to be processed; processing the first image through a first network to obtain a first feature, the first network being configured to at least extract features for image enhancement; processing the first image through a second network to obtain a second feature, the second network being configured to at least extract a semantic segmentation feature; generating a third feature based on the first feature and the second feature; obtaining a semantic segmentation result of the first image; generating a fourth feature based on the third feature and the semantic segmentation result of the first image; reconstructing the fourth feature to obtain a predicted image; obtaining a first loss based on the second image in the training sample pair and the predicted image, the first loss being used to describe the difference between the second image and the predicted image; updating the model parameters of the image processing model to be trained at least based on the first loss until the model training conditions are met to obtain an image processing model.

[0017] Optionally, in a possible implementation, the image processing model to be trained is further used to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

[0018] Optionally, in a possible implementation, the image processing model to be trained is also used to: obtain the true semantic segmentation result of the first image; obtain a second loss based on the semantic segmentation prediction result and the true semantic segmentation result, and the second loss is used to describe the difference between the semantic segmentation prediction result and the true semantic segmentation result; update the model parameters of the image processing model to be trained at least based on the first loss and the second loss until the model training conditions are met to obtain the image processing model.

[0019] Optionally, in a possible implementation, the image processing model to be trained is also used to: perform feature fusion processing on the first feature and the second feature to obtain the third feature; and generate the fourth feature based on the third feature and the semantic segmentation result of the first image, including: performing feature fusion processing on the third feature and the semantic segmentation result of the first image to obtain the fourth feature.

[0020] Optionally, in a possible implementation, the feature fusion processing includes summation processing, cascade processing, or cascade processing and convolution processing; the feature fusion processing includes summation processing, cascade processing, or cascade processing and convolution processing.

[0021] Optionally, in a possible implementation, the image processing model to be trained is also used to process the third feature to obtain a fifth feature; and generate a fourth feature based on the third feature and the semantic segmentation result of the first image.

[0022] Optionally, in a possible implementation, the image processing model to be trained is also used to preprocess the first image to obtain preprocessing features; downsample the preprocessing features to obtain downsampled features; process the downsampled features through the second network to obtain sixth features; and upsample the sixth features to obtain second features of the first image.

[0023] Optionally, in one possible implementation, the image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

[0024] The third aspect of the present application provides an image processing device, including: an acquisition unit and a processing unit; the acquisition unit is used to acquire an image to be processed; the processing unit is used to process the image to be processed through a first network to obtain a first feature, and the first network is configured to at least extract features for image enhancement; the image to be processed is processed through a second network to obtain a second feature, and the second network is configured to at least extract semantic segmentation features; a third feature is generated based on the first feature and the second feature; the acquisition unit is also used to obtain a semantic segmentation result of the image to be processed; the processing unit is also used to generate a fourth feature based on the third feature and the semantic segmentation result of the image to be processed; the fourth feature is reconstructed to obtain a target image.

[0025] Optionally, in a possible implementation, the processing unit is further used to process the third feature through a third network to obtain a semantic segmentation result of the image to be processed.

[0026] Optionally, in a possible implementation, the processing unit is further used to perform feature fusion processing on the first feature and the second feature to obtain the third feature; and perform feature fusion processing on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature.

[0027] Optionally, in a possible implementation, the feature fusion processing includes at least one of summation processing, multiplication processing, cascade processing and cascade convolution processing.

[0028] Optionally, in a possible implementation, the processing unit is further used to process the third feature to obtain a fifth feature; and generate a fourth feature based on the fifth feature and the semantic segmentation result of the image to be processed.

[0029] Optionally, in a possible implementation, the processing unit is further used to preprocess the image to be processed to obtain preprocessing features; downsample the preprocessing features to obtain downsampled features; process the downsampled features through the second network to obtain a sixth feature; and upsample the sixth feature to obtain a second feature of the image to be processed.

[0030] Optionally, in one possible implementation, the image processing device is used to perform at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image defogging, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

[0031] The fourth aspect of the present application provides a model training device, comprising: an acquisition unit and a training unit; the acquisition unit is used to acquire a training sample pair, the training sample pair including a first image and a second image, the quality of the first image being lower than that of the second image; the training unit is used to process the first image through an image processing model to be trained to obtain a predicted image, wherein the image processing model to be trained is used to acquire the image to be processed; the first image is processed by a first network to obtain a first feature, and the first network is configured to at least extract features for image enhancement; the first image is processed by a second network to obtain a second feature, and the second network is configured to at least extract semantic segmentation features; a third feature is generated based on the first feature and the second feature; a semantic segmentation result of the first image is obtained; a fourth feature is generated based on the third feature and the semantic segmentation result of the first image; an image is reconstructed on the fourth feature to obtain a predicted image; a first loss is obtained based on the second image in the training sample pair and the predicted image, and the first loss is used to describe the difference between the second image and the predicted image; the model parameters of the image processing model to be trained are updated at least based on the first loss until the model training conditions are met to obtain an image processing model.

[0032] Optionally, in a possible implementation, the training unit is further used to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

[0033] Optionally, in a possible implementation, the training unit is also used to obtain the true semantic segmentation result of the first image; obtain a second loss based on the semantic segmentation prediction result and the true semantic segmentation result, and the second loss is used to describe the difference between the semantic segmentation prediction result and the true semantic segmentation result; update the model parameters of the image processing model to be trained at least based on the first loss and the second loss until the model training conditions are met, thereby obtaining an image processing model.

[0034] Optionally, in a possible implementation, the training unit is further used to perform feature fusion processing on the first feature and the second feature to obtain the third feature; and perform feature fusion processing on the third feature and the semantic segmentation result of the first image to obtain the fourth feature.

[0035] Optionally, in a possible implementation, the feature fusion processing includes at least one of summation processing, multiplication processing, cascade processing and cascade convolution processing.

[0036] Optionally, in a possible implementation, the training unit is further used to process the third feature to obtain a fifth feature; and generate a fourth feature based on the third feature and the semantic segmentation result of the first image.

[0037] Optionally, in a possible implementation, the training unit is also used to preprocess the first image to obtain preprocessing features; downsample the preprocessing features to obtain downsampled features; process the downsampled features through the second network to obtain sixth features; and upsample the sixth features to obtain second features of the first image.

[0038] Optionally, in one possible implementation, the image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

[0039] In a fifth aspect, the present application provides an image processing device, which may include a processor coupled to a memory, wherein the memory stores program instructions. When the program instructions stored in the memory are executed by the processor, the method described in the first aspect is implemented. For details of the steps in each possible implementation of the first aspect performed by the processor, please refer to the first aspect and will not be repeated here.

[0040] In a sixth aspect, the present application provides a model training device, which may include a processor coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the second aspect is implemented. For the steps in each possible implementation of the second aspect executed by the processor, please refer to the second aspect for details, and no further description is given here.

[0041] In a seventh aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the first aspect.

[0042] In an eighth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method described in the second aspect.

[0043] In a ninth aspect, the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect.

[0044] In a tenth aspect, the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the second aspect.

[0045] In an eleventh aspect, the present application provides a computer program, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0046] A twelfth aspect of the present application provides a computer program which, when executed on a computer, enables the computer to execute the method described in the second aspect above.

[0047] In a thirteenth aspect of the present application, a chip system is provided, which includes a processor for supporting a server or a threshold value acquisition device to implement the functions involved in the above aspects, for example, sending or processing the data and / or information involved in the above methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the structure of the artificial intelligence main framework provided in the embodiment of the present application;

[0049] Figure 2a An image processing system provided in an embodiment of the present application;

[0050] Figure 2b Another image processing system provided in an embodiment of the present application;

[0051] Figure 2c A schematic diagram of an image processing device according to an embodiment of the present application;

[0052] Figure 3a A schematic diagram of a system 100 architecture provided in an embodiment of the present application;

[0053] Figure 3b A schematic diagram of image semantic segmentation provided in an embodiment of the present application;

[0054] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the structure of a densely connected dilated convolutional network provided in an embodiment of the present application;

[0056] Figure 6a A schematic diagram of an architecture for image processing provided in an embodiment of the present application;

[0057] Figure 6b A schematic diagram of a network structure for image processing provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of an objective indicator comparison provided in an embodiment of the present application;

[0059] Figure 8 Another objective indicator comparison diagram provided in an embodiment of the present application;

[0060] Figure 9 A schematic diagram of image comparison provided in the embodiments of the present application;

[0061] Figure 10 A flow chart of a model training method provided in an embodiment of the present application

[0062] Figure 11 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0063] Figure 12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0064] Figure 13 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0065] Figure 14 A schematic diagram of the structure of the training device provided in an embodiment of the present application;

[0066] Figure 15 A schematic diagram of the structure of the chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0068] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0069] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0070] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The following diagram illustrates a structural diagram of the AI ​​framework. This framework is explained below from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it encompasses the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed progression from "data-information-knowledge-wisdom." The "IT value chain," encompassing the entire process from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0071] (1) Infrastructure

[0072] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0073] (2) Data

[0074] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0075] (3) Data processing

[0076] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0077] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0078] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0079] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0080] (4) General ability

[0081] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0082] (5) Smart products and industry applications

[0083] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0084] Next, several application scenarios of this application are introduced.

[0085] Figure 2a An image processing system provided in an embodiment of the present application includes a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, personal computer, or information processing center. The user device is the initiator of image processing and, as the initiator of an image enhancement request, is typically initiated by a user through the user device.

[0086] The aforementioned data processing device can be a device or server with data processing capabilities, such as a cloud server, network server, application server, or management server. The data processing device receives image enhancement requests from smart terminals via an interactive interface and then performs image processing using methods such as machine learning, deep learning, search, reasoning, and decision-making through its memory and data processing processor. The memory in the data processing device is a general term that includes local storage and a database that stores historical data. The database can be located on the data processing device or on other network servers.

[0087] exist Figure 2a In the image processing system shown, a user device can receive user instructions. For example, the user device can obtain an image input / selected by the user and then initiate a request to a data processing device, causing the data processing device to perform image enhancement processing applications (such as image super-resolution reconstruction, image denoising, image dehazing, image deblurring, and image contrast enhancement) on the image obtained by the user device, thereby obtaining a corresponding processing result for the image. For example, the user device can obtain an image input by the user and then initiate an image denoising request to the data processing device, causing the data processing device to perform image denoising on the image, thereby obtaining a denoised image.

[0088] exist Figure 2a In the embodiment of the present application, the data processing device can execute the image processing method.

[0089] Figure 2b Another image processing system provided in the embodiment of the present application is Figure 2b In the process, the user device directly acts as a data processing device. The user device can directly obtain input from the user and process it directly by the hardware of the user device itself. The specific process is the same as Figure 2a Similarly, please refer to the above description and will not be repeated here.

[0090] exist Figure 2b In the image processing system shown, the user device can receive user instructions. For example, the user device can obtain an image selected by the user in the user device, and then the user device itself performs image processing applications (such as image super-resolution reconstruction, image denoising, image defogging, image deblurring, and image contrast enhancement, etc.) on the image, thereby obtaining the corresponding processing results for the image.

[0091] exist Figure 2b In the embodiment of the present application, the user equipment itself can execute the image processing method.

[0092] Figure 2c It is a schematic diagram of the image processing related equipment provided in the embodiment of the present application.

[0093] above Figure 2a and Figure 2b The user equipment in the example may be Figure 2c The local device 301 or the local device 302 in Figure 2a The data processing device in the embodiment can be Figure 2cThe execution device 210 in the embodiment, wherein the data storage system 250 can store the data to be processed by the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or can be set on the cloud or other network servers.

[0094] Figure 2a and Figure 2b The processor in the image processing apparatus can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to perform image processing applications on the image, thereby obtaining corresponding processing results.

[0095] Figure 3a This is a schematic diagram of a system 100 architecture provided in an embodiment of the present application. Figure 3a In the embodiment of the present application, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data may include: various tasks to be scheduled, callable resources and other parameters.

[0096] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing (such as implementing the functions of the neural network in this application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0097] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0098] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from training samples collected by the data collection device 160.

[0099] exist Figure 3aIn the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in the form of display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, the collection can also be performed without client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0100] It is worth noting that Figure 3a This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 3a In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110. Figure 3a As shown, a neural network can be obtained by training using the training device 120 .

[0101] The present application also provides a chip comprising a neural network processor NPU. The chip can be set as follows Figure 3a The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 3a The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rules.

[0102] The neural network processor (NPU) is mounted as a coprocessor on the host central processing unit (CPU), and the host CPU assigns tasks to it. The core of the NPU is the arithmetic circuit. The controller controls the arithmetic circuit to extract data from the memory (weight memory or input memory) and perform calculations.

[0103] In some implementations, the arithmetic circuit includes multiple processing engines (PEs). In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.

[0104] For example, consider an input matrix A, a weight matrix B, and an output matrix C. The computational circuit retrieves the corresponding data for matrix B from the weight memory and caches it on each PE within the computational circuit. The computational circuit then performs a matrix operation on the matrix A data from the input memory and matrix B, storing the partial or final matrix results in the accumulator.

[0105] The vector computation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit can be used for network calculations of non-convolutional / non-FC layers in neural networks, such as pooling, batch normalization, local response normalization, etc.

[0106] In some implementations, the vector computation unit can store the processed output vectors in a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the computation circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vectors can be used as activation inputs to the computation circuit, such as for use in subsequent layers in a neural network.

[0107] The unified memory is used to store input data and output data.

[0108] The weight data is directly transferred from the external memory to the input memory and / or unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.

[0109] The bus interface unit (BIU) is used to implement interaction between the main CPU, DMAC and instruction fetch memory through the bus.

[0110] An instruction fetch buffer connected to the controller, used to store instructions used by the controller;

[0111] The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.

[0112] Generally, unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and external memory is memory outside the NPU, which can be double data rate synchronous dynamic random access memory (DDRSDRAM), high bandwidth memory (HBM) or other readable and writable memory.

[0113] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0114] (1) Neural Network

[0115] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0116]

[0117] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0118] The work of each layer in the neural network can be expressed mathematically as To describe: From a physical perspective, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., from the row space to the column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimensionality increase / decrease; 2. Zoom in / out; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the object being classified is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. The vector W determines the spatial transformation from the input space to the output space mentioned above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0119] Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value, and this adjustment is continued until the neural network can predict the desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0120] (2) Backpropagation algorithm

[0121] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0122] (3) Image enhancement

[0123] Image enhancement refers to the processing of an image's brightness, color, contrast, saturation, dynamic range, and other characteristics to meet certain specific indicators. Simply put, by purposefully emphasizing the overall or local characteristics of an image during image processing, an originally unclear image can be made clearer or certain features of interest can be emphasized, the differences between features of different objects in the image can be magnified, and features of no interest can be suppressed, thereby improving image quality and enriching the amount of image information. This can enhance image interpretation and recognition, and meet the needs of certain special analyses. For example, image enhancement can include, but is not limited to, image super-resolution reconstruction, image denoising, image dehazing, image deblurring, and image contrast enhancement.

[0124] (4) Image semantic segmentation

[0125] Image semantic segmentation refers to the process of dividing pixels of an image into different categories according to certain rules (such as lighting and categories). In simple terms, the goal of image semantic segmentation is to label each pixel in the image with a label, that is, to label the object category to which each pixel in the image belongs. These labels can include people, animals, cars, flowers, furniture, etc. Figure 3b , Figure 3b This is a schematic diagram of image semantic segmentation provided in an embodiment of the present application. Figure 3b As shown in Figure 3, image semantic segmentation can be used to divide an image into different sub-regions according to categories at the pixel level, such as buildings, sky, plants, and other sub-regions.

[0126] The method provided in this application is described below from the training side of the neural network and the application side of the neural network.

[0127] The training method of the neural network provided in the embodiment of the present application involves image processing, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, and performs symbolic and formalized intelligent information modeling, extraction, preprocessing, and training on the training data (such as the image in the present application), and finally obtains a trained image processing model; and the image processing method provided in the embodiment of the present application can use the above-mentioned trained image processing model to input input data (such as the image to be processed in the present application) into the trained image processing model to obtain output data (such as the target image in the present application). It should be noted that the training method of the image processing model and the image processing method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0128] See Figure 4 , Figure 4 This is a flow chart of an image processing method provided in an embodiment of the present application. Figure 4 As shown, an image processing method provided in an embodiment of the present application includes the following steps:

[0129] Step 401: Obtain an image to be processed.

[0130] In this embodiment, the image processing device may obtain an image to be processed, and the image to be processed may be, for example, an image requiring image enhancement.

[0131] It is understandable that when the image processing device is deployed in an unmanned vehicle, the image processing device can obtain the street view image collected by the unmanned vehicle during driving through the camera. When the image processing device is deployed in a robot, the image processing device can obtain the real-life image of the robot's environment in real time. When the image processing device is deployed in a security device (such as a surveillance camera), the image processing device can obtain the real-life image collected by the surveillance camera in real time. When the image processing device is deployed on a handheld device such as a mobile phone or tablet computer, the image processing device can obtain photos taken by the user or pictures downloaded from the website, and these images can all be used as images to be processed.

[0132] Step 402: Process the image to be processed by a first network to obtain a first feature, where the first network is configured to extract at least a feature for image enhancement.

[0133] In this embodiment, the first network may be a backbone network related to image enhancement, such as a convolutional neural network, and the first network is configured to extract at least features for image enhancement, such as low-level image features. For example, low-level image features may refer to small details in an image, such as high-frequency details such as edges, corners, colors, pixels, gradients, and textures.

[0134] It is understood that different networks can be used for different image enhancement tasks to meet the requirements of the image enhancement tasks. For example, when the image enhancement task is image super-resolution reconstruction, the first network can use a residual network (ResNet); when the image enhancement task is image contrast enhancement, the first network can use a Unet network.

[0135] Specifically, the image processing method provided in this embodiment can be applied to different image enhancement tasks, such as image super-resolution reconstruction, image denoising, image defogging, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

[0136] In one possible embodiment, before the image to be processed is processed by the first network, the image to be processed may be preprocessed by a preprocessing network to obtain preprocessing features. The obtained preprocessing features are then processed by the first network to obtain the first features. The preprocessing network may be, for example, a convolutional neural network. By preprocessing the image to be processed, irrelevant information in the image to be processed can be eliminated, useful real information can be restored, the detectability of relevant information can be enhanced, and the data can be simplified to the greatest extent, thereby improving the reliability of feature extraction.

[0137] It can be understood that the preprocessing network can also be included in the first network, that is, the first network includes the preprocessing network, and the first feature can be obtained by processing the image to be processed through the first network.

[0138] Step 403: Process the image to be processed through a second network to obtain a second feature, where the second network is configured to extract at least a semantic segmentation feature.

[0139] In this embodiment, the second network can be a backbone network related to image semantic segmentation, such as a convolutional neural network, which is configured to extract at least features for image semantic segmentation, such as high-level features of the image. For example, high-level features of the image can refer to features that can reflect the semantic information of the image based on low-level features of the image. Generally, high-level features of the image can be used to recognize and detect the shape of targets or objects in the image, and have richer semantic information.

[0140] In one possible embodiment, the second network can be, for example, a densely connected atrous convolutional network. The atrous convolutional network can increase the receptive field, while the densely connected atrous convolutional network can capture multi-scale information. Together, these two networks can generate accurate high-level image feature information relevant to semantic segmentation.

[0141] The dilated convolutional network actually introduces a dilation rate (also known as the dilation number) into the standard convolutional network. This parameter defines the spacing between the values ​​when the convolution kernel processes data, thereby increasing the receptive field. Generally speaking, the receptive field is used to represent the size of the receptive range of different neurons within the network to the original image, or in other words, the size of the area mapped by the pixels on the feature map output by each layer of the convolutional network on the original image. By increasing the receptive field, the pixels on the feature map can respond to a large enough area in the image to capture information about large objects, thereby obtaining accurate semantic information.

[0142] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a densely connected dilated convolutional network provided in an embodiment of the present application. Figure 5 As shown in the figure, a densely connected dilated convolutional network consists of multiple layers, each of which includes a dilated convolutional network (dilated conv) and a leaky relu (leaky relu). The output of each layer in the densely connected dilated convolutional network serves as the input to each subsequent layer, enabling feature reuse. By directly outputting features from lower-level networks to each subsequent higher-level network for aggregation, this reduces feature loss caused by passing through intermediate layers, thereby better utilizing features from lower-level networks.

[0143] It is understandable that the embodiments of the present application illustrate the second network as a densely connected dilated convolutional network. In actual situations, the second network can also be other neural networks, which is not specifically limited here.

[0144] In a possible embodiment, the processing of the image to be processed through the second network to obtain the second feature can specifically include: the image processing device preprocesses the image to be processed to obtain a preprocessed feature, for example, preprocessing the image to be processed through the preprocessing network described in step 402 above; downsampling the preprocessed feature to obtain a downsampled feature; processing the downsampled feature through the second network to obtain a sixth feature; and upsampling the sixth feature to obtain the second feature of the image to be processed.

[0145] In this embodiment, by downsampling the preprocessed features, downsampled features with reduced resolution can be output; after the downsampled features are processed by the second network to obtain the sixth features, the sixth features are upsampled to generate a second feature with the same resolution as the preprocessed features, that is, the resolution of the restored features.

[0146] In practical applications, the downsampling multiple can be determined based on the desired processing accuracy and the computing power of the target hardware platform, and is not specifically limited here. Generally speaking, the larger the downsampling multiple, the lower the processing accuracy and the smaller the amount of calculation, that is, the lower the computing power requirement; the smaller the downsampling multiple, the higher the processing accuracy, but the amount of calculation is also higher, that is, the higher the computing power requirement. The upsampling multiple needs to be consistent with the downsampling multiple to ensure the resolution of the restored features. Among them, the methods that can be used to perform upsampling include but are not limited to deconvolution, bilinear interpolation upsampling, and nearest neighbor interpolation upsampling. The upsampling method is not specifically limited here.

[0147] Step 404: Generate a third feature based on the first feature and the second feature.

[0148] In one possible embodiment, the image processing device may perform feature fusion processing on the first feature and the second feature to obtain the third feature. Exemplarily, the feature fusion processing may include at least one of a summation process, a multiplication process, a cascade process, and a cascade convolution process, where cascade convolution processing refers to performing a cascade process and a convolution process. In practical applications, a corresponding feature fusion processing method may be adopted based on actual needs, and is not specifically limited here.

[0149] In this embodiment, by fusing the first feature and the second feature, low-level image features related to image enhancement and high-level image features related to semantic segmentation can be effectively fused to achieve complementarity of features at different levels, thereby improving the robustness of the network.

[0150] Step 405: Obtain the semantic segmentation result of the image to be processed.

[0151] In one possible embodiment, the image processing device may process the third feature to obtain a semantic segmentation result of the image to be processed. Since the third feature is a feature that is a fusion of low-level image features related to image enhancement and high-level image features related to semantic segmentation. Therefore, by processing the third feature, low-level image features can be introduced on the basis of high-level image features related to semantic segmentation, that is, the semantic segmentation result of the image to be processed can be obtained based on features at different levels, thereby improving the accuracy of the obtained semantic segmentation result. Exemplarily, the image processing device may perform a convolution operation on the third feature through a convolutional network to obtain a semantic segmentation result of the image to be processed.

[0152] In another possible embodiment, the image processing device may process the second feature output by the second network to obtain a semantic segmentation result for the image to be processed, i.e., obtain the semantic segmentation result for the image directly based on features related to semantic segmentation. For example, the image processing device may perform a convolution operation on the second feature using a convolutional network to obtain the semantic segmentation result for the image to be processed.

[0153] Step 406: Generate a fourth feature based on the third feature and the semantic segmentation result of the image to be processed.

[0154] In one possible embodiment, the image processing device may perform feature fusion processing on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature. The specific method of feature fusion processing can be referred to as described in step 404 and will not be repeated here.

[0155] In one possible embodiment, after obtaining the third feature, the image processing device may process the third feature, for example, further extract the third feature to obtain a fifth feature. The image processing device generates a fourth feature based on the fifth feature and the semantic segmentation result of the image to be processed. That is, after extracting the third feature, the image processing device performs feature fusion based on the fifth feature obtained by further feature extraction and the semantic segmentation result of the image to be processed. By further extracting the third feature obtained after feature fusion, a finer-grained feature can be extracted based on the third feature to improve the accuracy of the fourth feature obtained by subsequent feature fusion.

[0156] Step 407: reconstruct the fourth feature to obtain a target image.

[0157] In this embodiment, after obtaining the fourth feature after two feature fusion processes, the image processing device can reconstruct the fourth feature, for example, perform a convolution post-processing operation on the fourth feature, to obtain a target image, which is the image obtained after image enhancement.

[0158] In this embodiment, during the image enhancement process, the semantic segmentation features and the semantic segmentation results are fused with the relevant image enhancement features through two feature fusion processes, thereby achieving feature information complementarity, so that different image enhancement intensities can be used for different semantic areas, accurately maintaining texture details and improving the authenticity of texture details after image enhancement.

[0159] It should be understood that the execution entity of steps 401 to 407 (ie, the image processing device) can be a terminal device or a server on the cloud side, and steps 401 to 407 can also be obtained by data processing and interaction between the terminal device and the server.

[0160] For ease of understanding, the following will describe in detail how the image processing method provided in this embodiment achieves image defogging with reference to specific examples.

[0161] See Figure 6a and Figure 6b , Figure 6a A schematic diagram of an architecture for image processing provided in an embodiment of the present application; Figure 6b This is a schematic diagram of a network structure for image processing provided in an embodiment of the present application. Figure 6a and 6b As shown, the architecture may include:

[0162] The preprocessing unit 100 is configured to receive a foggy, low-contrast image and preprocess the image to generate a preprocessing feature F. The preprocessing unit 100 may be, for example, a convolutional network that performs a convolution operation on the received image (e.g., a 12-megapixel, 3000*4000 resolution image) to generate the preprocessing feature F. The resolution of the preprocessing feature F is the same as that of the image, i.e., 3000*4000 resolution.

[0163] The first feature extraction unit 101 is used to extract features from the preprocessing feature F, for example, extract low-level features of the image from the preprocessing feature F to obtain the first feature F L. The first feature extraction unit 101 can adopt a backbone network related to the defogging task, such as a multi-stage cascaded convolutional network + instance normalization (IN) network. Based on the IN network, a highly nonlinear contrast normalization effect can be learned, so that the deviation in the image appearance such as brightness, color, style, etc. will not affect the final prediction result. At the same time, it can also improve the compatibility of the extracted low-level image features and the subsequently extracted high-level image features. The number of cascade layers N can be determined according to the desired processing accuracy and the computing power of the target hardware platform. Generally speaking, the larger the number of cascade layers N, the higher the accuracy of feature extraction, and the greater the amount of calculation.

[0164] The downsampling unit 200 is used to perform a preprocessing operation on the preprocessing feature F, perform downsampling processing, and obtain a downsampled feature F with reduced resolution. down For example, the downsampling unit 200 may be a pooling layer network, which performs a k*k average pooling downsampling operation on the preprocessed feature F to obtain the downsampled feature F. down The downsampling factor k can be determined based on the desired processing accuracy and the computing power of the target hardware platform. Generally speaking, the smaller the number of cascade layers N, the higher the feature extraction accuracy, but also the greater the computational effort. For example, the value of k can be 4, which means that both the width and height of the feature are downsampled by a factor of 4.

[0165] The second feature extraction unit 201 is used to extract the downsampled features F down Perform feature extraction processing, such as downsampling feature F down Extract high-level features of the image and obtain the sixth feature F down-seg Exemplarily, the second feature extraction unit 201 may be a densely connected dilated convolutional network.

[0166] Upsampling unit 202, used for upsampling the second feature F down-seg Perform upsampling to obtain the second feature F with the same resolution as the original input image H-seg The sampling method adopted by the upsampling unit 202 may be, for example, deconvolution, bilinear interpolation upsampling, or nearest neighbor interpolation upsampling, and the upsampling multiple is the same as the downsampling multiple.

[0167] The first feature fusion unit 102 is used to combine the first feature FL and the second feature F down-seg Perform feature fusion processing, such as the first feature FL and the second feature F down-seg Perform cascade operation to obtain the fused third feature F fusion1 .

[0168] The third feature extraction unit 103 is used to extract the third feature F fusion1 Do further feature extraction, such as using a convolutional network to extract the third feature F fusion1 Perform convolution processing to extract fine-grained features, namely the fifth feature F fine .

[0169] Semantic result prediction unit 203, used to predict the third feature F fusion1 Make semantic segmentation result predictions, for example, perform post-processing operations on the third feature Ffusion1 through a convolutional network to obtain the semantic segmentation result corresponding to the input image.

[0170] The second feature fusion unit 104 is used to combine the fifth feature F fine And the semantic segmentation result corresponding to the input image, perform feature fusion processing, for example, the fifth feature F fine And the semantic segmentation results corresponding to the input image are cascaded to obtain the fused fourth feature F fusion2 .

[0171] The image reconstruction unit 105 is used to reconstruct the fused fourth feature F fusion2 Perform image reconstruction, for example, by using a convolutional network to reconstruct the fourth feature F fusion2 Perform post-processing operations to obtain the defogging image.

[0172] Taking the method of this embodiment for image dehazing as an example, this embodiment is tested on an open source simulation dataset to compare the method with existing dehazing algorithms.

[0173] See Figure 7 , Figure 7 This is a schematic diagram of an objective indicator comparison provided by the embodiment of this application. Figure 7 It can be seen that compared with various existing dehazing algorithms, the image processing method provided by the embodiment of the present application has a higher Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM).

[0174] PSNR is an engineering term that represents the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy. PSNR is commonly used in fields such as image processing to measure signal reconstruction quality, and is typically defined as mean squared error. Generally speaking, the higher the PSNR, the smaller the deviation from the true value.

[0175] SSIM is an indicator for measuring the similarity between two images, mainly based on luminance, contrast and structure to evaluate the similarity of images.

[0176] In addition, when different levels of noise are added to the open source simulation dataset, the method of this embodiment has higher PSNR and SSIM than the existing dehazing methods, that is, the method of this embodiment has stronger robustness and stability. Figure 8 , Figure 8 Another objective indicator comparison diagram provided in an embodiment of the present application.

[0177] This example was tested on a real foggy dataset, and it was found that the method of this example can produce clearer and more transparent results without artifacts. Existing defogging algorithms have problems such as insufficient defogging, resulting in low contrast in defogging images, or excessive defogging, resulting in loss of texture details in some local areas. For more details, please refer to Figure 9 , Figure 9 This is a schematic diagram of image comparison provided in the embodiment of this application. Figure 9 It can be seen that the method of this embodiment in the lower right corner, on the basis of defogging, well preserves the details of areas such as green plants, floors, and ground. The picture is transparent and natural, and the visual effect is optimal.

[0178] See Figure 10 , Figure 10 A flow chart of a model training method provided in an embodiment of the present application. Figure 10 As shown, a model training method provided in an embodiment of the present application includes the following steps:

[0179] Step 1001: Acquire a training sample pair, where the training sample pair includes a first image and a second image, and the quality of the first image is lower than that of the second image.

[0180] In this embodiment, before the image training device performs model training, a pair of training samples can be obtained. The first image and the second image are two images of the same scene, and the image quality of the first image is lower than that of the second image. Image quality refers to one or more of color, brightness, saturation, contrast, dynamic range, resolution, texture detail, and clarity. For example, if the first image is foggy and the second image is non-foggy, the brightness, contrast, and clarity of the first image are all lower than those of the second image.

[0181] Step 1002: Process the first image through the image processing model to be trained to obtain a predicted image, wherein the image processing model to be trained is used to obtain the image to be processed; process the first image through a first network to obtain a first feature, and the first network is configured to at least extract features for image enhancement; process the first image through a second network to obtain a second feature, and the second network is configured to at least extract a semantic segmentation feature; generate a third feature based on the first feature and the second feature; obtain a semantic segmentation result of the first image; generate a fourth feature based on the third feature and the semantic segmentation result of the first image; and reconstruct the fourth feature to obtain a predicted image.

[0182] Step 1003: Obtain a first loss based on the second image in the training sample pair and the predicted image, where the first loss is used to describe the difference between the second image and the predicted image.

[0183] In this embodiment, after obtaining the predicted image, a first loss corresponding to the second image and the predicted image may be calculated based on a preset loss function to determine the difference between the second image and the predicted image.

[0184] In one possible implementation, the first loss corresponding to the second image and the predicted image can be obtained based on a reconstruction loss function and a gradient loss function to ensure that the enhanced image can meet objective and subjective indicator requirements.

[0185] For example, the reconstruction loss function may be a loss function that uses the L1 paradigm to obtain pixel-level loss between the predicted image and the second image. The reconstruction loss function may be as shown in Formula 1:

[0186]

[0187] Among them, L rec represents the reconstruction loss, || represents the L1 normal form, GT represents the true value, i.e., the value of the pixel in the second image, output represents the value of the pixel in the predicted image, and P is the number of pixels. The L1 normal form involves taking the difference between the pixel values ​​of the second image and the pixel values ​​of the predicted image, and then calculating the sum of the absolute values ​​of the differences corresponding to each pixel.

[0188] For example, the gradient loss function may be a loss representing the average gradient of the predicted image and the second image in the x / y direction. The gradient loss function may be as shown in Formula 2:

[0189] L grad =|grad(GT)-grad(output)| Formula 2

[0190] Among them, L grad Represents gradient loss, || represents L1 norm, GT represents the true value, that is, the value of the pixel of the second image, output represents the value of the pixel of the predicted image, and grad() represents the average gradient of the image in the x / y direction.

[0191] Based on the reconstruction loss function and the gradient loss function, the first loss corresponding to the second image and the predicted image can be obtained. For example, the function for calculating the first loss can be as shown in Formula 3:

[0192] L total =L rec +α*L grad Formula 3

[0193] Among them, L total It represents the first loss, and α is a hyperparameter used to adjust the weight of the gradient loss.

[0194] Step 1004: Update the model parameters of the image processing model to be trained at least according to the first loss until the model training conditions are met, thereby obtaining the image processing model.

[0195] The image processing model obtained after training in step 1004 can refer to Figure 4 The description in the corresponding embodiments will not be repeated here.

[0196] Optionally, in a possible implementation, the image processing model to be trained is further used to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

[0197] Optionally, in a possible implementation, the image processing model to be trained is also used to: obtain the true semantic segmentation result of the first image; obtain a second loss based on the semantic segmentation prediction result and the true semantic segmentation result, and the second loss is used to describe the difference between the semantic segmentation prediction result and the true semantic segmentation result; update the model parameters of the image processing model to be trained at least based on the first loss and the second loss until the model training conditions are met to obtain the image processing model.

[0198] That is to say, during the model training process, the semantic segmentation loss function can be used to constrain the semantic segmentation prediction result of the first image so that the semantic segmentation result generated by the model can be more accurate.

[0199] For example, the semantic segmentation loss function for obtaining the second loss between the semantic segmentation prediction result and the semantic segmentation true result may be a cross entropy loss function. The semantic segmentation loss function may be, for example, as shown in Formula 4:

[0200]

[0201] Among them, L seg is the second loss, p is the number of pixels in the image, s i It is used to represent the probability of semantic segmentation prediction results at pixel i for semantic category z. It is used to represent the probability of the true result of semantic segmentation for semantic category z at pixel i, and log() is used to represent the logarithm.

[0202] A third loss can be obtained based on the above-mentioned first loss and second loss. The model parameters of the image processing model to be trained are updated according to the third loss until the model training conditions are met to obtain the image processing model.

[0203] For example, the formula for obtaining the third loss may be as shown in Formula 5:

[0204] L total =L rec +α*L seg +β*L grad Formula 5

[0205] Among them, L total It is used to represent the third loss, α is the first hyperparameter, β is the second hyperparameter, and α and β are used to adjust the weights of semantic segmentation loss and gradient loss, respectively.

[0206] Optionally, in a possible implementation, the image processing model to be trained is also used to: perform feature fusion processing on the first feature and the second feature to obtain the third feature; and generate the fourth feature based on the third feature and the semantic segmentation result of the first image, including: performing feature fusion processing on the third feature and the semantic segmentation result of the first image to obtain the fourth feature.

[0207] Optionally, in a possible implementation, the feature fusion processing includes at least one of summation processing, multiplication processing, cascade processing and cascade convolution processing.

[0208] Optionally, in a possible implementation, the image processing model to be trained is also used to process the third feature to obtain a fifth feature; and generate a fourth feature based on the third feature and the semantic segmentation result of the first image.

[0209] Optionally, in a possible implementation, the image processing model to be trained is also used to preprocess the first image to obtain preprocessing features; downsample the preprocessing features to obtain downsampled features; process the downsampled features through the second network to obtain sixth features; and upsample the sixth features to obtain second features of the first image.

[0210] Optionally, in one possible implementation, the image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

[0211] See Figure 11 , Figure 11 This is a structural diagram of an image processing device provided in an embodiment of the present application. Figure 11 As shown, an image processing device provided by an embodiment of the present application includes: an acquisition unit 1101 and a processing unit 1102; the acquisition unit 1101 is used to acquire an image to be processed; the processing unit 1102 is used to process the image to be processed through a first network to obtain a first feature, and the first network is configured to at least extract features for image enhancement; the image to be processed is processed through a second network to obtain a second feature, and the second network is configured to at least extract semantic segmentation features; a third feature is generated based on the first feature and the second feature; the acquisition unit 1101 is also used to obtain a semantic segmentation result of the image to be processed; the processing unit 1102 is also used to generate a fourth feature based on the third feature and the semantic segmentation result of the image to be processed; and the fourth feature is reconstructed to obtain a target image.

[0212] Optionally, in a possible implementation, the processing unit 1102 is further configured to process the third feature through a third network to obtain a semantic segmentation result of the image to be processed.

[0213] Optionally, in a possible implementation, the processing unit 1102 is further used to perform feature fusion processing on the first feature and the second feature to obtain the third feature; and perform feature fusion processing on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature.

[0214] Optionally, in a possible implementation, the feature fusion processing includes at least one of summation processing, multiplication processing, cascade processing and cascade convolution processing.

[0215] Optionally, in a possible implementation, the processing unit 1102 is further configured to process the third feature to obtain a fifth feature; and generate a fourth feature based on the fifth feature and a semantic segmentation result of the image to be processed.

[0216] Optionally, in a possible implementation, the processing unit 1102 is further used to preprocess the image to be processed to obtain preprocessing features; downsample the preprocessing features to obtain downsampled features; process the downsampled features through the second network to obtain a sixth feature; and upsample the sixth feature to obtain a second feature of the image to be processed.

[0217] Optionally, in one possible implementation, the image processing device is used to perform at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image defogging, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

[0218] See Figure 12 , Figure 12 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. Figure 12 As shown, an embodiment of the present application provides a model training device, including: an acquisition unit 1201 and a training unit 1202; the acquisition unit 1201 is used to acquire a training sample pair, the training sample pair including a first image and a second image, the quality of the first image being lower than that of the second image; the prediction unit is used to process the first image through the image processing model to be trained to obtain a predicted image, wherein the image processing model to be trained is used to obtain the image to be processed; the first image is processed by a first network to obtain a first feature, and the first network is configured to at least extract features for image enhancement; the first image is processed by a second network to obtain a second feature, and the second network is configured to at least extract semantic segmentation features; a third feature is generated based on the first feature and the second feature; a semantic segmentation result of the first image is obtained; a fourth feature is generated based on the third feature and the semantic segmentation result of the first image; an image is reconstructed on the fourth feature to obtain a predicted image; a first loss is obtained based on the second image in the training sample pair and the predicted image, and the first loss is used to describe the difference between the second image and the predicted image; the model parameters of the image processing model to be trained are updated at least based on the first loss until the model training conditions are met to obtain an image processing model.

[0219] Optionally, in a possible implementation, the training unit 1202 is further configured to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

[0220] Optionally, in a possible implementation, the training unit 1202 is also used to obtain the true semantic segmentation result of the first image; obtain a second loss based on the semantic segmentation prediction result and the true semantic segmentation result, and the second loss is used to describe the difference between the semantic segmentation prediction result and the true semantic segmentation result; update the model parameters of the image processing model to be trained at least based on the first loss and the second loss until the model training conditions are met, and obtain the image processing model.

[0221] Optionally, in a possible implementation, the training unit 1202 is further used to perform feature fusion processing on the first feature and the second feature to obtain the third feature; and perform feature fusion processing on the third feature and the semantic segmentation result of the first image to obtain the fourth feature.

[0222] Optionally, in a possible implementation, the feature fusion processing includes at least one of summation processing, multiplication processing, cascade processing and cascade convolution processing.

[0223] Optionally, in a possible implementation, the training unit 1202 is further used to process the third feature to obtain a fifth feature; and generate a fourth feature based on the third feature and the semantic segmentation result of the first image.

[0224] Optionally, in a possible implementation, the training unit 1202 is also used to preprocess the first image to obtain preprocessing features; downsample the preprocessing features to obtain downsampled features; process the downsampled features through the second network to obtain a sixth feature; and upsample the sixth feature to obtain a second feature of the first image.

[0225] Optionally, in one possible implementation, the image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image dehazing, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

[0226] Next, we will introduce an execution device provided by the embodiment of the present application. Figure 13 , Figure 13This is a schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 1300 can be specifically manifested as a mobile phone, tablet, laptop, smart wearable device, server, etc., which is not limited here. Among them, the execution device 1300 can be deployed with Figure 13 The data processing device described in the corresponding embodiment is used to implement Figure 13 The data processing function in the corresponding embodiment. Specifically, the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (wherein the number of the processor 1303 in the execution device 1300 can be one or more, Figure 13 (taking one processor as an example), the processor 1303 may include an application processor 13031 and a communication processor 13032. In some embodiments of the present application, the receiver 1301, the transmitter 1302, the processor 1303 and the memory 1304 may be connected via a bus or other means.

[0227] Memory 1304 may include read-only memory and random access memory, and provides instructions and data to processor 1303. A portion of memory 1304 may also include non-volatile random access memory (NVRAM). Memory 1304 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0228] Processor 1303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0229] The methods disclosed in the above embodiments of the present application can be applied to or implemented by the processor 1303. The processor 1303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 1303. The above processor 1303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1303 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1304, and processor 1303 reads the information in memory 1304 and, in conjunction with its hardware, completes the steps of the above method.

[0230] Receiver 1301 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1302 can be used to output digital or character information through the first interface. Transmitter 1302 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1302 can also include a display device such as a display screen.

[0231] In one embodiment of the present application, the processor 1303 is configured to execute Figure 4 The image processing method executed by the execution device in the corresponding embodiment.

[0232] The present application also provides a training device. Figure 14 , Figure 14A structural diagram of a training device provided in an embodiment of the present application. Specifically, the training device 1400 is implemented by one or more servers. The training device 1400 may have relatively large differences due to different configurations or performances. It may include one or more central processing units (CPUs) 1414 (for example, one or more processors) and memory 1432, and one or more storage media 1430 (for example, one or more mass storage devices) storing application programs 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 can be short-term storage or persistent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1414 can be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the training device 1400.

[0233] The training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458; or one or more operating systems 1441, such as Windows Server 2003 or Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0234] Specifically, the training device can perform Figure 10 The steps in the corresponding embodiments.

[0235] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0236] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0237] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0238] For details, please refer to Figure 15 , Figure 15 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 1500. NPU 1500 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1503, which is controlled by controller 1504 to extract matrix data from memory and perform multiplication operations.

[0239] In some implementations, arithmetic circuit 1503 includes multiple processing units (PEs). In some implementations, arithmetic circuit 1503 is a two-dimensional systolic array. Arithmetic circuit 1503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1503 is a general-purpose matrix processor.

[0240] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1502 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1501 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1508.

[0241] Unified memory 1506 is used to store input and output data. Weight data is directly transferred to weight memory 1502 through the Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 through the DMAC.

[0242] BIU stands for Bus Interface Unit 1510 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1509 .

[0243] The bus interface unit 1510 (BIU) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0244] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or transfer weight data to the weight memory 1502 or transfer input data to the input memory 1501.

[0245] The vector calculation unit 1507 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0246] In some implementations, the vector calculation unit 1507 can store the processed output vector in the unified memory 1506. For example, the vector calculation unit 1507 can apply a linear function or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1507 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in subsequent layers in a neural network.

[0247] An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;

[0248] Unified memory 1506, input memory 1501, weight memory 1502, and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0249] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0250] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0251] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0252] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0253] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Processing the image to be processed by a first network to obtain a first feature, wherein the first network is configured to extract at least a feature for image enhancement; Processing the image to be processed by a second network to obtain a second feature, wherein the second network is configured to extract at least a semantic segmentation feature; Performing feature fusion processing on the first feature and the second feature to obtain a third feature; Obtaining a semantic segmentation result of the image to be processed; generating a fourth feature according to the third feature and a semantic segmentation result of the image to be processed; Perform image reconstruction on the fourth feature to obtain a target image.

2. The image processing method according to claim 1, wherein: The obtaining of the semantic segmentation result of the image to be processed includes: The third feature is processed by a third network to obtain a semantic segmentation result of the image to be processed.

3. The image processing method according to claim 1 or 2, characterized in that: Generating a fourth feature according to the third feature and a semantic segmentation result of the image to be processed includes: Feature fusion processing is performed on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature.

4. The image processing method according to claim 3, wherein: The feature fusion processing includes at least one of a summation processing, a multiplication processing, a cascade processing and a cascade convolution processing.

5. The image processing method according to any one of claims 1, 2 and 4, characterized in that: Before generating the fourth feature based on the third feature and the semantic segmentation result of the image to be processed, the method further includes: Processing the third feature to obtain a fifth feature; Generating the fourth feature based on the third feature and the semantic segmentation result of the image to be processed includes: generating the fourth feature based on the fifth feature and the semantic segmentation result of the image to be processed.

6. The image processing method according to any one of claims 1, 2 and 4, characterized in that: The processing of the image to be processed by the second network to obtain the second feature includes: Preprocessing the image to be processed to obtain preprocessing features; Performing downsampling processing on the preprocessed features to obtain downsampled features; Processing the downsampled features through the second network to obtain a sixth feature; Upsampling is performed on the sixth feature to obtain a second feature of the image to be processed.

7. The image processing method according to any one of claims 1, 2 and 4, characterized in that: The method is used to achieve at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image defogging, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

8. A model training method, characterized in that: include: Acquire a training sample pair, the training sample pair comprising a first image and a second image, wherein the quality of the first image is lower than that of the second image; Processing the first image using a to-be-trained image processing model to obtain a predicted image, wherein the to-be-trained image processing model is used to obtain the to-be-processed image; processing the first image using a first network to obtain a first feature, wherein the first network is configured to extract at least a feature for image enhancement; processing the first image using a second network to obtain a second feature, wherein the second network is configured to extract at least a semantic segmentation feature; and generating a third feature based on the first feature and the second feature. Obtaining a semantic segmentation result of the first image; generating a fourth feature based on the third feature and the semantic segmentation result of the first image; and performing image reconstruction on the fourth feature to obtain a predicted image; Obtaining a first loss according to the second image in the training sample pair and the predicted image, where the first loss is used to describe the difference between the second image and the predicted image; The model parameters of the image processing model to be trained are updated at least according to the first loss until the model training conditions are met to obtain the image processing model.

9. The model training method according to claim 8, characterized in that The image processing model to be trained is further used to process the third feature through a third network to obtain a semantic segmentation prediction result of the first image.

10. The model training method according to claim 9, characterized in that: The image processing model to be trained is also used for: Obtaining a true semantic segmentation result of the first image; Obtaining a second loss according to the semantic segmentation prediction result and the true semantic segmentation result, where the second loss is used to describe the difference between the semantic segmentation prediction result and the true semantic segmentation result; The model parameters of the image processing model to be trained are updated at least according to the first loss and the second loss until the model training conditions are met, thereby obtaining the image processing model.

11. The model training method according to any one of claims 8 to 10, characterized in that: The image processing model to be trained is also used for: performing feature fusion processing on the first feature and the second feature to obtain the third feature; Feature fusion processing is performed on the third feature and the semantic segmentation result of the image to be processed to obtain the fourth feature.

12. The model training method according to claim 11, characterized in that: The feature fusion processing includes at least one of a summation processing, a multiplication processing, a cascade processing and a cascade convolution processing.

13. The model training method according to any one of claims 8 to 10, characterized in that: The image processing model to be trained is also used to process the third feature to obtain a fifth feature; and generate a fourth feature based on the third feature and the semantic segmentation result of the first image.

14. The model training method according to any one of claims 8 to 10, characterized in that: The image processing model to be trained is further used to preprocess the first image to obtain preprocessing features; Performing downsampling processing on the preprocessed features to obtain downsampled features; Processing the downsampled features through the second network to obtain a sixth feature; Upsampling is performed on the sixth feature to obtain a second feature of the first image.

15. The model training method according to any one of claims 8 to 10, characterized in that: The image processing model is used to implement at least one of the following image enhancement tasks: image super-resolution reconstruction, image denoising, image defogging, image deblurring, image contrast enhancement, image demosaicing, image deraining, image color enhancement, image brightness enhancement, image detail enhancement, and image dynamic range enhancement.

16. An image processing device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the image processing device performs the method according to any one of claims 1 to 15.

17. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to implement the method of any one of claims 1 to 15.

Images