Image enhancement and denoising method and device, electronic equipment and medium
Through the U-shaped encoder-decoder structure and feature mixing unit, the problem of noise processing of infrared images in dark light environments is solved, efficient image enhancement and denoising is achieved, training costs are reduced, and it is suitable for the enhancement and de-extraction of infrared images. Noise task.
Patent Information
- Application Number
- CN202411081412.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, it is difficult to effectively remove noise and improve image quality when processing infrared images. Especially in dark light environments, existing algorithms usually require a large amount of paired data for training, and the correlation between denoising and enhancing tasks is not fully considered. It affects the enhancement and denoising effects of infrared images.
Using the U-shaped encoder-decoder structure, through jumping connection and feature mixing units, visual features are extracted and integrated, reducing computational burden and increasing receptive fields, and using different encoding strategies to extract image features, which is suitable for enhancement and denoising of infrared images.
While removing image noise, image structure features are retained, which improves the enhancement and denoising effects of infrared images, reduces training costs, and does not require a large amount of pairing data.
Smart Images

Figure CN118967488A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, in particular to the fields of image processing, computer vision and deep learning technology, and specifically to an image enhancement and denoising method, device, electronic device, computer-readable storage medium and computer program product. Background Art
[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0003] Infrared images are widely used in military, medical, security and other fields due to their anti-interference, high penetration, high sensitivity and high concealment. However, due to the characteristics of the imaging equipment itself and the influence of environmental noise, it is easy to introduce a lot of noise into the image, which makes it difficult to restore the image. Summary of the invention
[0004] The present disclosure provides an image enhancement and denoising method, apparatus, electronic device, computer-readable storage medium, and computer program product.
[0005] According to one aspect of the present disclosure, a model training method is provided, comprising: obtaining a sample image to be processed; inputting the sample image into a preset neural network model to obtain a first image feature, wherein the neural network model comprises an encoder and a decoder, and corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of a feature map unit by unit while increasing its number of channels unit by unit, and the size and number of channels of the feature maps processed between the jump-connected decoder and the corresponding units of the encoder are the same; based on the first image feature and a preset label image feature, determining a value of a preset loss function; and based on the value of the loss function, adjusting a parameter value of the neural network model, wherein the encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
[0006] According to another aspect of the present disclosure, there is provided an image enhancement and denoising method, comprising: acquiring an image to be enhanced and denoised; inputting the image into an encoder and a decoder in sequence to obtain a first image feature, wherein corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of a feature map unit by unit while increasing its number of channels unit by unit, and the size and number of channels of the feature maps processed between the jump-connected decoder and the corresponding units of the encoder are the same; and based on the first image feature, determining an enhanced and denoised image, wherein the encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
[0007] According to another aspect of the present disclosure, a model training device is provided, comprising: a first acquisition unit, configured to acquire a sample image to be processed; a first input unit, configured to input the sample image into a preset neural network model to obtain a first image feature, wherein the neural network model comprises an encoder and a decoder, and corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of a feature map unit by unit while increasing its number of channels unit by unit, and the size and number of channels of the feature map processed between the jump-connected decoder and the corresponding unit of the encoder are the same; a first determination unit, configured to determine the value of a preset loss function based on the first image feature and a preset label image feature; and an adjustment unit, configured to adjust the parameter value of the neural network model based on the value of the loss function, wherein the encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
[0008] According to another aspect of the present disclosure, an image enhancement and denoising device is provided, comprising: a second acquisition unit, configured to acquire an image to be enhanced and denoised; a second input unit, configured to input the image into an encoder and a decoder in sequence to obtain a first image feature, wherein corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of a feature map unit by unit while increasing its number of channels unit by unit, and the size and number of channels of the feature maps processed between the jump-connected decoder and the corresponding units of the encoder are the same; and a second determination unit, configured to determine the enhanced and denoised image based on the first image feature, wherein the encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the present disclosure.
[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method described in the present disclosure.
[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method described in the present disclosure when executed by a processor.
[0012] According to one or more embodiments of the present disclosure, a U-shaped encoder-decoder structure is followed with skip connections, which helps to reduce the computational burden and increase the receptive field; and, the feature mixing unit takes advantage of different encoding strategies to extract and integrate visual features, which can often remove image noise while retaining the structural features of the image as much as possible.
[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0015] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;
[0016] Figure 2 A flow chart of a model training method according to an embodiment of the present disclosure is shown;
[0017] Figure 3 A schematic diagram of a model training method according to an embodiment of the present disclosure is shown;
[0018] Figure 4 A schematic diagram of a feature mixing module according to an embodiment of the present disclosure is shown;
[0019] Figure 5A schematic diagram of a selection module according to an embodiment of the present disclosure is shown;
[0020] Figure 6 A flowchart of an image enhancement and denoising method according to an embodiment of the present disclosure is shown;
[0021] Figure 7 A structural block diagram of a model training device according to an embodiment of the present disclosure is shown;
[0022] Figure 8 A structural block diagram of an image enhancement and denoising device according to an embodiment of the present disclosure is shown; and
[0023] Fig. 9 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0025] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.
[0026] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.
[0027] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0028] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
[0029] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of model training methods, image enhancement and denoising methods.
[0030] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.
[0031] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.
[0032] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to obtain the corresponding image to be processed. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.
[0033] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.
[0034] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.
[0035] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.
[0036] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.
[0037] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.
[0038] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.
[0039] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as images. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.
[0040] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.
[0041] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.
[0042] The purpose of image enhancement technology is to improve the quality of images to achieve a pleasing effect. Usually, the work to be completed includes removing noise in the image, making the edges clear, and highlighting certain properties in the image. Image enhancement is a conversion of the original image to improve the clarity of the image or highlight certain features of the image to facilitate manual or computer interpretation. Although image enhancement processing does not increase the amount of information in the image, it improves the interpretation conditions and enhances the recognizability. It is an effective means of remote sensing image analysis research.
[0043] At present, image enhancement methods based on deep learning have achieved some good results. However, there is a problem that most of them belong to supervised learning, which means that a large amount of paired data is required for training. However, in real life, it is often difficult to obtain a large number of low-light and normal-light images of the same scene as data pairs. For example, for images of infrared scenes, it is often difficult to obtain their corresponding reference images. Researchers usually use different methods to synthesize paired data sets, which often cannot simulate real application scenarios, thus affecting the enhancement effect of infrared images.
[0044] Infrared images are very useful in all-day monitoring and mobile phone photography, but dark environments will introduce a lot of noise to the image, which makes image restoration difficult. For an infrared image with uneven noise distribution, current algorithms often design and train denoising network models and infrared enhancement network models separately to cascade or process in parallel. On the one hand, this increases the training cost of such tasks, and on the other hand, it does not fully consider the correlation between the two tasks, affecting the enhancement and denoising effects of infrared images.
[0045] Therefore, according to an embodiment of the present disclosure, a model training method is provided. Figure 2 A flow chart of the model training method according to the present disclosure is shown, as Figure 2 As shown, method 200 includes: obtaining a sample image to be processed (step 210); inputting the sample image into a preset neural network model to obtain a first image feature (step 220); determining a value of a preset loss function based on the first image feature and a preset label image feature (step 230); and adjusting a parameter value of the preset neural network based on the value of the loss function (step 240).
[0046] In the present disclosure, the neural network model includes an encoder and a decoder, and the corresponding units in the encoder and the decoder are jump-connected. The encoder reduces the size of the feature map unit by unit while increasing its number of channels unit by unit, and the size and number of channels of the feature map processed by the jump-connected decoder and the corresponding unit of the encoder are the same. The encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
[0047] According to the embodiments of the present disclosure, a U-shaped encoder-decoder structure is followed with skip connections, which helps to reduce the computational burden and increase the receptive field; and the feature mixing unit takes advantage of different encoding strategies to extract and integrate visual features, which can often remove image noise while retaining the structural features of the image as much as possible.
[0048] In some embodiments, the neural network model includes a feature extraction unit. The image input into the neural network model is converted into image features after passing through the feature extraction unit.
[0049] In the present disclosure, corresponding image features can be extracted from the original sample image by any suitable method. Image feature extraction refers to extracting representative features from the original image data for subsequent image analysis and processing tasks. For example, commonly used feature extraction methods include edge detection, corner detection, texture analysis, color histogram, etc. In the field of deep learning, convolutional neural networks (CNNs) are also widely used in image feature extraction tasks. Through CNN, representative features in the image can be automatically learned, thereby improving the accuracy of various tasks in the field of computer vision.
[0050] The embodiments of the present disclosure are applicable to the fields of image processing and video processing to enhance and denoise the corresponding images. Therefore, in some embodiments, the neural network model can be used to enhance and denoise infrared images. At this time, the neural network model can be called an infrared image enhancement and denoising network model (Adaptive infrared enhancement and denoising network model, AIEDNet).
[0051] According to some embodiments, the encoder includes at least one Transformer unit, and each Transformer unit in the at least one Transformer unit includes one or more Transformer modules connected in series. Multiple Transformer modules connected in series can enhance the feature extraction of the Transformer unit to enhance the subsequent image processing effect.
[0052] In some examples, the input unit of the encoder is a Transformer unit, that is, the sample image input to the neural network model is first input into the Transformer unit of the encoder. The Transformer unit may include one or more Transformer modules connected in series. The image input to the Transformer module can be converted into image features through the PatchEmbedding layer and input into subsequent network units. The Patch Embedding layer divides the input image into non-overlapping image patches of fixed size, and then embeds these image patches into a specific dimension to be input into the network layer of the Transformer module for processing.
[0053] According to some embodiments, the encoder and the decoder in the neural network model each include at least one feature mixing unit. Each of the at least one feature mixing unit includes one or more feature mixing modules connected in series, and the feature mixing module is used to mix image features extracted based on different encoding strategies. Multiple feature mixing modules connected in series can also enhance the feature extraction of the feature mixing unit to enhance the subsequent image processing effect.
[0054] For example, the different encoding strategies may be feature extraction modules implemented based on different algorithms, such as a convolution module, a multi-layer perception module, a multi-head self-attention module, a self-attention module, etc.
[0055] In some examples, multiple feature mixing units may be provided in the encoder. Since the encoder reduces the size of the feature map unit by unit while increasing the number of its channels unit by unit, the multiple feature mixing units may focus on extracting feature information of different dimensions respectively. For example, when two feature mixing units are included in the encoder, the first feature mixing unit focuses on extracting shallow feature information, and the second feature mixing unit focuses on extracting middle feature information.
[0056] It is understandable that the number of modules in each unit in the encoder may be the same or different, or partially the same, and this is not limited here. In addition, when the encoder includes both the Transformer unit and the feature mixing unit, the order of the Transformer unit and the feature mixing unit can be set according to actual conditions, and this is not limited here.
[0057] According to some embodiments, the decoder comprises a convolution unit comprising one or more convolution modules connected in series.
[0058] In some examples, the output unit of the decoder is the convolution unit. For example, the convolution module includes a 5x5 convolution kernel, which has a smaller number of parameters and can extract more features when restoring image feature information compared to other convolution kernel sizes.
[0059] According to some embodiments, the decoder comprises at least one Transformer unit, and each Transformer unit in the at least one Transformer unit comprises one or more Transformer modules connected in series.
[0060] In an embodiment of the present disclosure, the size and number of channels of the feature graphs processed by the corresponding units of the decoder and the encoder of the skip-connection are the same. That is, the decoder and the encoder include the same number of units, the encoder reduces the size of the feature graph unit by unit while increasing its number of channels unit by unit, the decoder increases the size of the feature graph unit by unit while reducing its number of channels unit by unit, and the size and number of channels of the feature graphs processed by the corresponding units in the encoder and the decoder are the same. For example, the encoder and decoder both include three units. The image feature parameter corresponding to the first unit in the encoder is H×W×C, the image feature parameter corresponding to the second unit is H / 2×W / 2×2C, and the image feature parameter corresponding to the third unit is H / 4×W / 4×4C; the image feature output by the third unit in the encoder is input into the first unit in the decoder, that is, the image feature parameter corresponding to the first unit in the decoder is H / 4×W / 4×4C. Therefore, the image feature parameter corresponding to the second unit in the decoder (jump-connected to the second unit in the encoder) is H / 2×W / 2×2C, and the image feature parameter corresponding to the third unit in the decoder (jump-connected to the first unit in the encoder) is H×W×C.
[0061] The jump connection, that is, the input of the corresponding unit in the decoder, includes not only the output image features of the previous unit, but also the output image features of the unit corresponding to the unit in the encoder. In the above example, for the second unit in the decoder, its input includes the output image features of the first unit in the decoder and the output image features of the second unit in the encoder.
[0062] It can be understood that the ratio or multiple by which the encoder / decoder reduces / increases the size of the feature map and increases / decreases the number of its channels unit by unit can be arbitrarily set according to actual conditions and is not limited here.
[0063] Figure 3 FIG. 1 is a schematic diagram showing a model training method according to an embodiment of the present disclosure. Figure 3As shown, the number of units in both the encoder and the decoder is 4, and each unit includes 4 corresponding network modules. Among them, the encoder includes a first Transformer unit (formed by connecting 4 Transformer Blocks in series), a first feature mixing unit (formed by connecting 4 Mixed Blocks in series), a second feature mixing unit (formed by connecting 4 Mixed Blocks in series), and a second Transformer unit (formed by connecting 4 Transformer Blocks in series). The decoder includes a third Transformer unit (formed by connecting 4 Transformer Blocks in series), a third feature mixing unit (formed by connecting 4 Mixed Blocks in series), a third feature mixing unit (formed by connecting 4 Mixed Blocks in series), and a convolutional unit (formed by connecting 4 5x5 Conv2d modules in series). Figure 3 The number of image feature channels C1 < C2 < C3 < C4 corresponding to each unit in Figure 3 , for example, C4 = 8C1, C3 = 4C1, C2 = 2C1.
[0064] Furthermore, this neural network model follows a U-shaped encoder-decoder structure and has skip-connections, which helps reduce the computational burden and increase the receptive field. When processing an infrared image through this neural network model, the input noisy infrared image is converted into low-level features by a Transformer unit, and then the features in the spatial dimension are fused by the feature mixing unit.
[0065] According to some embodiments, the feature mixing module includes: a selection module, and at least two feature extraction modules connected in parallel. Therefore, inputting the sample image into the neural network model to obtain the first image feature includes: when inputting the second image feature into the feature mixing module in the neural network model, inputting the second image feature into each of the at least two feature extraction modules respectively to obtain the third image features output by the at least two feature extraction modules, where the second image feature is the image feature input into the feature mixing module in the neural network model; inputting the third image features output by the at least two feature extraction modules into the mixing model to obtain the fourth image feature after mixing the third image features output by the at least two feature extraction modules; and fusing the second image feature and the fourth image feature to obtain the first image feature based on the fused image feature.
[0066] Figure 4 shows a schematic diagram of the feature mixing module according to an embodiment of the present disclosure. As Figure 4As shown in the figure, the feature mixing module includes three feature extraction modules connected in parallel: convolution module (Conv Block), multi-layer perception module (MLP), and multi-head self-attention module (MSA). Among them, the convolution module (Conv Block) is used to extract local visual features, MLP processes channel information, and MSA captures the global dependency of visual features. Then, the corresponding output image features are obtained through a selection module. Among them, ⊕ represents a fusion operation, such as addition, concat, pooling, etc.
[0067] In some examples, the convolution module (Conv Block) can include a depthwise separable convolution layer (DepthwiseConv layer) and a convolution layer (Conv layer). The number of the depthwise separable convolution layer and the convolution layer can be adaptively set, such as in Figure 4 In the example, there are 8 layers and 4 layers respectively, which are not limited here. The depthwise convolution layer can effectively reduce the number of parameters.
[0068] In some examples, the multi-layer perception module (MLP) can include a linear layer. The number of the linear layers can be adaptively set, such as in Figure 4 There are 2 layers in the middle, and there is no restriction here.
[0069] In the present disclosure, the number and type of feature extraction modules can be adaptively set to achieve the purpose of extracting image features based on different encoding strategies, which is not limited here.
[0070] According to some embodiments, the selection module includes: a global pooling layer, a multi-layer perception layer and a softmax layer. Inputting the third image features output by the at least two feature extraction modules into the hybrid model includes: for the third image features output by each feature extraction model in the at least two feature extraction modules, performing the following operations: sequentially inputting the third image features output by the feature extraction model into the global pooling layer, the multi-layer perception layer and the softmax layer corresponding to the feature extraction model in the selection module to obtain the fifth image features corresponding to the feature extraction model; and fusing the third image features output by the feature extraction model with the fifth image features corresponding to the feature extraction model to obtain the sixth image features corresponding to the feature extraction model; fusing the sixth image features corresponding to the at least two feature extraction modules to obtain the fourth image features.
[0071] Figure 5 FIG. 2 shows a schematic diagram of a selection module according to an embodiment of the present disclosure. Figure 5As shown in the figure, the features output by the convolution module (ConvBlock), the multi-layer perception module (MLP) and the multi-head self-attention module (MSA) are respectively passed through their corresponding global pooling layer, multi-layer perception layer and softmax layer, and then the mixed fourth image features are obtained through feature fusion. Among them, ⊕ represents a fusion operation, such as addition, concat, pooling, etc.
[0072] Specifically, the selection module enables the neural network model to adaptively adjust visual features according to different encoding strategies. Figure 5 First, define the image features output by different encoding strategies (i.e., each feature extraction module) as F Conv 、F MLP and F MSA . Therefore, the sum of features F can be expressed as:
[0073] F=F Conv +F MLP +F MSA
[0074] Then, feature F Conv 、F MLP and F MSA Passed through a selection module to adaptively aggregate features.
[0075] Then, the selection module aggregates the information of each channel through global average pooling to obtain the embedded global features. definition denote the number of channels, width, and height of the image features entering the feature mixing module and selecting the module. In particular, the cth element of g is obtained by The spatial downsampling F calculation in the dimension is:
[0076]
[0077] Next, in order to enable the model to infer the weights of different encoding strategies for each channel, the present invention further uses the corresponding MLP to encode g to obtain the hidden feature h, that is:
[0078]
[0079] Where n represents the number of visual encoding strategies (i.e., feature extraction modules). Figure 5 The example shown shows three types of feature extraction modules: Conv Block, MLP, and MSA. It is understood that the type and number of feature extraction modules are not limited here. Specifically, a softmax layer operation is applied to the channel dimension of the hidden feature h, that is:
[0080]
[0081] Among them, p represents the weight of different encoding strategies on each channel. The following features are weighted summed, namely:
[0082] V=p Conv ·F Conv +p MLP ·F MLP +p MSA ·F MSA
[0083] Among them, V is the mixed image feature output by the mixing module. Finally, the features of V are passed to the subsequent network layers layer by layer.
[0084] According to some embodiments, based on the second image feature and the preset label image feature, determining the value of the preset loss function includes: obtaining a seventh image feature corresponding to the sample image; fusing the first image feature and the seventh image feature to obtain an eighth image feature; and determining the value of the preset loss function based on the eighth image feature and the label image feature. The label image feature is an image feature corresponding to an image obtained by enhancing the sample image based on a contrast-constrained adaptive histogram equalization algorithm (CLAHE).
[0085] Among them, CLAHE (Contrast Limited Adaptive Histogram Equalization) is a method for enhancing image contrast by limiting the histogram contrast of each small block to prevent excessive noise amplification, while applying adaptive histogram equalization on the entire image. The CLAHE adaptive algorithm can enhance the contrast of the entire image, which plays an important role in feature extraction in the subsequent training of the neural network model and improves the image enhancement effect of the neural network model.
[0086] Continue to refer Figure 3 , after the first image feature output by the decoder is fused with the seventh image feature corresponding to the original sample image, the loss function value is calculated with the image feature enhanced based on the CLAHE algorithm. Wherein, ⊕ represents a fusion operation, such as addition, concat, pooling, etc.
[0087] According to some embodiments, the preset loss function is determined based on a weighted sum of a pixel value loss function and a color saturation loss function.
[0088] In some examples, the loss function Loss of the neural network model can be designed as follows:
[0089]
[0090] Where Y represents the sample image to be processed, N is the number of pixels in the sample image to be processed, Net(Y)(i,j) represents the pixel value corresponding to the (i,j)th pixel in the image obtained by reconstructing Y with the original sample image by the AIEDNet network. X(i,j) represents the pixel value corresponding to the (i,j)th pixel in the label image, S(·) represents the saturation value of the pixel in the predicted image, and α represents the weight ratio of the two loss functions.
[0091] The above loss functions include L2 pixel value loss function and color saturation loss function, that is, the mean square error and color saturation loss of the pixel value of the output image feature of the neural network model and the pixel value of the label image feature. In particular, in the image enhancement task, the color saturation loss function can help the model generate more vivid and natural images.
[0092] According to some embodiments, adjusting the parameter values of the neural network model based on the value of the loss function includes: adjusting the parameter values of the neural network model based on the value of the loss function, and the weight values in the preset loss function.
[0093] In some examples, the parameters to be trained include network weight parameters of the neural network model (AIEDNet) and a weight ratio for weighing the two loss functions. In the early stages of training, the weight ratio can be set to 0.5, for example.
[0094] In some examples, the neural network model (AIEDNet) is trained using the Adam optimizer with adaptability parameters beta1=0.8 and beta2=0.88, and the learning rate is set to 1×10e -5 In addition, an appropriate training batch size is selected, such as 128 sample images per batch, to achieve appropriate gradient updates in the initial stage of model training.
[0095] When the loss function value Loss is less than the empirical value C or the training is unified to the 500th epoch, the training of the network model is stopped and a trained neural network model (AIEDNet) is generated.
[0096] Epoch: When a complete data set passes through the neural network once and returns once, this process is called an epoch. In other words, all training samples undergo one forward propagation and one backward propagation in the neural network. An epoch is the process of training all training samples once.
[0097] In some embodiments, the trained neural network model (AIEDNet) can be used to enhance and denoise the infrared image to output and calculate the reconstructed image.
[0098] According to the embodiments of the present disclosure, Figure 6 As shown, an image enhancement and denoising method is also provided, including: obtaining an image to be enhanced and denoised (step 610); inputting the image into an encoder and a decoder in sequence to obtain a first image feature (step 620); and determining an enhanced and denoised image based on the first image feature (step 630).
[0099] In this embodiment, the corresponding units in the encoder and the decoder are jump-connected, and the encoder reduces the size of the feature map unit by unit while increasing the number of its channels unit by unit, and the size and number of channels of the feature map processed by the jump-connected decoder and the corresponding units of the encoder are the same. The encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix the image features extracted based on different encoding strategies.
[0100] In the present disclosure, the structure and / or operation of the encoder and the decoder may be as described in any one or more of the related embodiments in the model training described above, and will not be repeated here.
[0101] According to the embodiments of the present disclosure, Figure 7 As shown, a model training device 700 is also provided, including: a first acquisition unit 710, configured to acquire a sample image to be processed; a first input unit 720, configured to input the sample image into a preset neural network model to obtain a first image feature, wherein the neural network model includes an encoder and a decoder, and corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of the feature map unit by unit while increasing its number of channels unit by unit, and the size and number of channels of the feature map processed between the jump-connected decoder and the corresponding unit of the encoder are the same; a first determination unit 730, configured to determine the value of a preset loss function based on the first image feature and a preset label image feature; and an adjustment unit 740, configured to adjust the parameter value of the neural network model based on the value of the loss function, wherein the encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
[0102] Here, the operations of the above-mentioned units 710-740 of the model training device 700 are similar to the operations of the steps 210-240 described above, and will not be repeated here.
[0103] According to the embodiments of the present disclosure, Figure 8 As shown, an image enhancement and denoising device 800 is also provided, including: a second acquisition unit 810, configured to acquire an image to be enhanced and denoised; a second input unit 820, configured to input the image into an encoder and a decoder in sequence to obtain a first image feature, wherein corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of the feature map unit by unit while increasing its number of channels unit by unit, and the size and number of channels of the feature map processed between the jump-connected decoder and the corresponding unit of the encoder are the same; and a second determination unit 830, configured to determine the enhanced and denoised image based on the first image feature, wherein the encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
[0104] Here, the operations of the above-mentioned units 810-830 of the image enhancement and denoising device 800 are similar to the operations of the steps 610-630 described above, and will not be repeated here.
[0105] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0106] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.
[0107] refer to Fig. 9 , a block diagram of an electronic device 900 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0108] like Fig. 9As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0109] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the electronic device 900. The input unit 906 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 907 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 can include but is not limited to a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0110] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as method 200 or 600. For example, in some embodiments, method 200 or 600 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the method 200 or 600 described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the method 200 or 600 in any other appropriate manner (eg, by means of firmware).
[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0112] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0115] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0118] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.
Claims
1. A model training method, comprising: Obtaining a sample image to be processed; Input the sample image into a preset neural network model to obtain a first image feature, wherein the neural network model includes an encoder and a decoder, and corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of the feature map unit by unit while increasing the number of its channels unit by unit, and the size and number of channels of the feature map processed by the corresponding units of the jump-connected decoder and the encoder are the same; Determining a value of a preset loss function based on the first image feature and a preset label image feature; and Based on the value of the loss function, adjusting the parameter values of the neural network model, The encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
2. The method of claim 1, wherein: The encoder includes at least one Transformer unit, and each Transformer unit in the at least one Transformer unit includes one or more Transformer modules connected in series.
3. The method of claim 1, wherein: The encoder and the decoder in the neural network model each include at least one feature mixing unit, and each of the at least one feature mixing unit includes one or more feature mixing modules connected in series, and the feature mixing module is used to mix image features extracted based on different encoding strategies.
4. The method of claim 1, wherein: The decoder includes a convolution unit, and the convolution unit includes one or more convolution modules connected in series.
5. The method of claim 1, wherein: The decoder includes at least one Transformer unit, and each Transformer unit in the at least one Transformer unit includes one or more Transformer modules connected in series.
6. The method of claim 3, wherein: The feature mixing module includes: a selection module, and at least two feature extraction modules connected in parallel, and wherein inputting the sample image into a neural network model to obtain a first image feature includes: When the second image feature is input into the feature mixing module in the neural network model, the second image feature is respectively input into each of the at least two feature extraction modules to obtain a third image feature output by the at least two feature extraction modules, wherein the second image feature is an image feature in the neural network model that is input into the feature mixing module; Inputting the third image features output by the at least two feature extraction modules into the hybrid model to obtain a fourth image feature obtained by mixing the third image features output by the at least two feature extraction modules; and The second image feature and the fourth image feature are fused to obtain the first image feature based on the fused image feature.
7. The method of claim 6, wherein: The at least two feature extraction modules are determined based on the following modules: a convolution module, a multi-layer perception module, a multi-head self-attention module, and a self-attention module.
8. The method according to claim 6 or 7, wherein: The selection module includes: a global pooling layer, a multi-layer perception layer and a softmax layer, and wherein the third image features output by the at least two feature extraction modules are input into the hybrid model, comprising: For the third image feature output by each feature extraction model in the at least two feature extraction modules, the following operations are performed: inputting the third image feature output by the feature extraction model into the global pooling layer, the multi-layer perception layer and the softmax layer corresponding to the feature extraction model in the selection module in sequence to obtain a fifth image feature corresponding to the feature extraction model; and fusing the third image feature output by the feature extraction model with the fifth image feature corresponding to the feature extraction model to obtain a sixth image feature corresponding to the feature extraction model; The sixth image features respectively corresponding to the at least two feature extraction modules are fused to obtain the fourth image feature.
9. The method of claim 1, wherein: Determining a value of a preset loss function based on the second image feature and a preset label image feature includes: Acquire a seventh image feature corresponding to the sample image; fusing the first image feature and the seventh image feature to obtain an eighth image feature; Based on the eighth image feature and the label image feature, a value of a preset loss function is determined, wherein the label image feature is an image feature corresponding to an image obtained after enhancing the sample image based on a contrast-limited adaptive histogram equalization algorithm (CLAHE).
10. The method according to claim 1 or 9, wherein: The preset loss function is determined based on a weighted sum of a pixel value loss function and a color saturation loss function.
11. The method of claim 10, wherein: Adjusting the parameter value of the neural network model based on the value of the loss function includes: adjusting the parameter value of the neural network model based on the value of the loss function, and the weight value in the preset loss function.
12. An image enhancement and denoising method, comprising: Obtain an image to be enhanced and denoised; Inputting the image into an encoder and a decoder in sequence to obtain a first image feature, wherein corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of a feature map unit by unit while increasing the number of its channels unit by unit, and the size and number of channels of the feature map processed by the jump-connected decoder and the corresponding units of the encoder are the same; and Based on the first image feature, determining an enhanced and denoised image, The encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
13. The method of claim 12, wherein: The encoder includes at least one Transformer unit, and each Transformer unit in the at least one Transformer unit includes one or more Transformer modules connected in series.
14. The method of claim 12, wherein: The encoder and the decoder in the neural network model each include at least one feature mixing unit, and each of the at least one feature mixing unit includes one or more feature mixing modules connected in series, and the feature mixing module is used to mix image features extracted based on different encoding strategies.
15. The method of claim 12, wherein: The output unit of the decoder includes a convolution unit, and the convolution unit includes one or more convolution modules connected in series.
16. The method of claim 12, wherein: The decoder includes at least one Transformer unit, and each Transformer unit in the at least one Transformer unit includes one or more Transformer modules connected in series.
17. The method of claim 14, wherein: The feature mixing module includes: a selection module, and at least two feature extraction modules connected in parallel, and wherein inputting the first image feature into a preset neural network to obtain the first image feature includes: When the second image feature is input into the feature mixing module in the neural network model, the second image feature is respectively input into each of the at least two feature extraction modules to obtain a third image feature output by the at least two feature extraction modules, wherein the second image feature is an image feature in the preset neural network that is input into the feature mixing module; Inputting the third image features output by the at least two feature extraction modules into the hybrid model to obtain a fourth image feature obtained by mixing the third image features output by the at least two feature extraction modules; and The second image feature and the fourth image feature are fused to obtain the first image feature based on the fused image feature.
18. The method of claim 17, wherein: The at least two feature extraction modules are determined based on the following modules: a convolution module, a multi-layer perception module, a multi-head self-attention module, and a self-attention module.
19. The method according to claim 17 or 18, wherein: The selection module includes: a global pooling layer, a multi-layer perception layer and a softmax layer, and wherein the third image features output by the at least two feature extraction modules are input into the hybrid model, comprising: For the third image feature output by each feature extraction model in the at least two feature extraction modules, the following operations are performed: inputting the third image feature output by the feature extraction model into the global pooling layer, the multi-layer perception layer and the softmax layer corresponding to the feature extraction model in the selection module in sequence to obtain a fifth image feature corresponding to the feature extraction model; and fusing the third image feature output by the feature extraction model with the fifth image feature corresponding to the feature extraction model to obtain a sixth image feature corresponding to the feature extraction model; The sixth image features respectively corresponding to the at least two feature extraction modules are fused to obtain a fourth image feature obtained by mixing the third image features output by the at least two feature extraction modules.
20. A model training device, comprising: A first acquisition unit, configured to acquire a sample image to be processed; A first input unit is configured to input the sample image into a preset neural network model to obtain a first image feature, wherein the neural network model includes an encoder and a decoder, and corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of the feature map unit by unit while increasing the number of its channels unit by unit, and the size and number of channels of the feature map processed by the corresponding units of the jump-connected decoder and the encoder are the same; a first determining unit configured to determine a value of a preset loss function based on the first image feature and a preset label image feature; and an adjusting unit, configured to adjust a parameter value of the neural network model based on a value of the loss function, The encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
21. An image enhancement and denoising device, comprising: A second acquisition unit, configured to acquire an image to be enhanced and denoised; a second input unit configured to input the image into an encoder and a decoder in sequence to obtain a first image feature, wherein corresponding units in the encoder and the decoder are jump-connected, and wherein the encoder reduces the size of the feature map unit by unit while increasing the number of its channels unit by unit, and the size and number of channels of the feature map processed by the jump-connected decoder and the corresponding units of the encoder are the same; and a second determining unit configured to determine an enhanced and denoised image based on the first image feature, The encoder and the decoder include at least one feature mixing unit, and the feature mixing unit is used to mix image features extracted based on different encoding strategies.
22. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 19.
23. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-19.
24. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 19 is implemented.
