Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a to-be-processed image of a first domain; performing domain mapping processing on the to-be-processed image through a first generator module to obtain a target generated image of a second domain; performing coding processing on the target generation image to obtain coded data; and sending the coded data to a second device, the second device being used for decoding the coded data to obtain a target domain image of the second domain. According to the invention, the to-be-processed image of the first domain can be pre-mapped through the first generator module to obtain the target generated image of the second domain. And then, universal coding and decoding processing can be carried out on the target generated image. Therefore, the image processing steps can be simplified, the resource consumption can be reduced, the general standardization of image processing can be improved, and the image processing speed can be improved.

Description

Background Art

[0002] With the continuous development of artificial intelligence technology, in image processing algorithms in fields such as video transmission and face recognition, image encoding and decoding are required as underlying algorithm technologies.

[0003] However, images collected by different image acquisition devices may belong to different domains. For example, the image acquisition device can be a VIS (Visible) image acquisition device, but the quality of the VIS images it collects is low under extreme lighting conditions. Therefore, under extreme lighting conditions, an IR (Infra Red) image acquisition device can be selected for image acquisition to ensure the quality of image acquisition.

[0004] Since the methods for encoding and decoding images in different domains are different. For example, image encoding and decoding can be implemented for VIS images. At this time, non-VIS domain images can be collectively referred to as heterogeneous images. In related technologies, it is necessary to collect a heterogeneous image dataset and train the corresponding encoding and decoding system separately to achieve encoding and decoding. The steps are cumbersome and consume a large amount of resources, and the general standardization of image processing is low, reducing the speed of image processing.

[0005] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0006] The present disclosure provides an image processing method, apparatus, electronic device, and storage medium, which at least to some extent overcome the problems in the related technologies that the steps are cumbersome and consume a large amount of resources, the general standardization of image processing is low, and the speed of image processing is reduced.

[0007] Other features and advantages of the present disclosure will become apparent through the following detailed description, or will be partially learned through the practice of the present disclosure.

[0008] According to one aspect of the embodiments of the present disclosure, an image processing method is provided, which is executed by a first device and includes: obtaining a to-be-processed image in a first domain; performing domain mapping processing on the to-be-processed image through a first generator module to obtain a target generated image in a second domain; performing encoding processing on the target generated image to obtain encoded data; and sending the encoded data to a second device, where the second device is configured to perform decoding processing on the encoded data to obtain a target domain image in the second domain.

[0009] In some embodiments of the present disclosure, the image processing method provided by the embodiments of the present disclosure further includes: performing domain mapping processing on the target domain image through a second generator module to obtain an original domain image in the first domain.

[0010] In some embodiments of the present disclosure, encoding processing is performed on the target generated image to obtain encoded data, including: encoding the target generated image through an encoder pre-trained based on a second domain to obtain the encoded data;

[0011] Wherein, the encoded data is sent to a second device, and the second device is configured to perform decoding processing on the encoded data to obtain a target domain image in the second domain, including: sending the encoded data to the second device, and the second device is configured to perform decoding processing on the encoded data through a decoder pre-trained based on the second domain to obtain a target domain image in the second domain.

[0012] In some embodiments of the present disclosure, the image processing method provided by the embodiments of the present disclosure further includes: respectively constructing a first initial generator module and a second initial generator module; pre-training the first initial generator module and the second initial generator module through a training dataset to obtain the first generator module and the second generator module, wherein the training dataset includes a plurality of first training images in a first domain and a plurality of second training images in a second domain.

[0013] In some embodiments of the present disclosure, the image processing method provided by the embodiments of the present disclosure includes:

[0014] Iteratively perform the following operations until a preset end condition is met to obtain the first generator module and the second generator module:

[0015] Performing domain mapping processing on any first training image through the first initial generator module to obtain a first generated image in the second domain; performing domain mapping processing on the first generated image through the second initial generator module to obtain a first reconstructed image in the first domain; performing domain mapping processing on any second training image through the second initial generator module to obtain a second generated image in the first domain; performing domain mapping processing on the second generated image through the first initial generator module to obtain a second reconstructed image in the second domain; calculating a first loss value between the first training image and the first reconstructed image, and a second loss value between the second training image and the second reconstructed image; calculating a third loss value between the first training image and the second generated image, and a fourth loss value between the second training image and the first generated image; pre-training through the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the first generator module and the second generator module.

[0016] In some embodiments of the present disclosure, calculating a first loss value between the first training image and the first reconstructed image, and a second loss value between the second training image and the second reconstructed image includes: obtaining the first loss value between the first training image and the first reconstructed image, and the second loss value between the second training image and the second reconstructed image through a cycle-consistent loss algorithm;

[0017] Calculating a third loss value between the first training image and the second generated image, and a fourth loss value between the second training image and the first generated image includes: obtaining a first discriminator module, and scoring the first training image and the second generated image through the first discriminator module to obtain the third loss value; obtaining a second discriminator module, and scoring the second training image and the first generated image through the second discriminator module to obtain the fourth loss value.

[0018] In some embodiments of the present disclosure, the first domain is an infrared light image domain, and the second domain is a visible light image domain.

[0019] According to another aspect of the embodiments of the present disclosure, there is provided an image processing apparatus applied to a first device, including: a to-be-processed image acquisition unit configured to acquire a to-be-processed image in a first domain; a target generated image determination unit configured to perform domain mapping processing on the to-be-processed image through a first generator module to obtain a target generated image in a second domain; an encoding unit configured to perform encoding processing on the target generated image to obtain encoded data; a decoding unit configured to send the encoded data to a second device, and the second device is configured to perform decoding processing on the encoded data to obtain a target domain image in the second domain.

[0020] In some embodiments of the present disclosure, the image processing apparatus provided by the embodiments of the present disclosure further includes: an original domain image determination unit configured to perform domain mapping processing on the target domain image through a second generator module to obtain an original domain image in the first domain.

[0021] In some embodiments of the present disclosure, the encoding unit is configured to perform encoding processing on the target generated image through an encoder pre-trained based on the second domain to obtain the encoded data;

[0022] The decoding unit is configured to send the encoded data to the second device, and the second device is configured to perform decoding processing on the encoded data through a decoder pre-trained based on the second domain to obtain a target domain image in the second domain.

[0023] In some embodiments of the present disclosure, the image processing apparatus provided by the embodiments of the present disclosure further includes: a generator construction unit configured to construct a first initial generator module and a second initial generator module respectively; a pre-training unit configured to pre-train the first initial generator module and the second initial generator module through a training data set to obtain the first generator module and the second generator module, wherein the training data set includes a plurality of first training images in a first domain and a plurality of second training images in a second domain.

[0024] In some embodiments of the present disclosure, the pre-training unit is configured to iteratively perform the following operations until a preset end condition is met to obtain the first generator module and the second generator module: performing domain mapping processing on any first training image through the first initial generator module to obtain a first generated image in the second domain; performing domain mapping processing on the first generated image through the second initial generator module to obtain a first reconstructed image in the first domain; performing domain mapping processing on any second training image through the second initial generator module to obtain a second generated image in the first domain; performing domain mapping processing on the second generated image through the first initial generator module to obtain a second reconstructed image in the second domain; calculating a first loss value between the first training image and the first reconstructed image, and a second loss value between the second training image and the second reconstructed image; calculating a third loss value between the first training image and the second generated image, and a fourth loss value between the second training image and the first generated image; and pre-training through the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the first generator module and the second generator module.

[0025] In some embodiments of the present disclosure, the pre-training unit is configured to obtain the first loss value between the first training image and the first reconstructed image, and the second loss value between the second training image and the second reconstructed image through a cycle-consistent loss algorithm; obtain a first discriminator module, and score the first training image and the second generated image through the first discriminator module to obtain the third loss value; obtain a second discriminator module, and score the second training image and the first generated image through the second discriminator module to obtain the fourth loss value.

[0026] In some embodiments of the present disclosure, the type of the first domain includes an infrared light image domain and a visible light image domain, and the second domain is an image domain of a different type from the first domain.

[0027] According to another aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the above-mentioned image processing method by executing the executable instructions.

[0028] According to another aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program, which when executed by a processor implements the above-described image processing method.

[0029] According to another aspect of the present disclosure, there is provided a computer program product or a computer program, the computer program product or the computer program including computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method provided in any of the various alternative ways in the embodiments of the present disclosure.

[0030] The technical solution provided by the present disclosure can pre-map the image to be processed in the first domain through the first generator module to obtain the target generated image in the second domain. After that, general encoding and decoding processing can be performed on the target generated image. Therefore, the present disclosure can simplify the steps of image processing, reduce resource consumption, improve the general standardization of image processing, and improve the speed of image processing.

[0031] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0033] Figure 1 A schematic diagram showing a system architecture in an embodiment of the present disclosure;

[0034] Figure 2 A flowchart showing an image processing method in an embodiment of the present disclosure;

[0035] Figure 3 A schematic diagram showing a pre-training process in an embodiment of the present disclosure;

[0036] Figure 4 A schematic diagram showing an image processing process in an embodiment of the present disclosure;

[0037] Figure 5 A flowchart showing a pre-training method in an embodiment of the present disclosure;

[0038] Figure 6 A flowchart showing another image processing method in an embodiment of the present disclosure;

[0039] Figure 7 Schematic diagram of an image processing device in an embodiment of the present disclosure;

[0040] Figure 8 Block diagram showing the structure of an electronic device in an embodiment of the present disclosure. Detailed implementation manners

[0041] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments.

[0042] In addition, the drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] Figure 1 Schematic diagram showing an exemplary system architecture to which the image processing method or image processing device in an embodiment of the present disclosure can be applied.

[0044] As Figure 1 shown, the system architecture 100 may include a first device 101, a second device 102, and a network 103.

[0045] Among them, the first device 101 may obtain an image to be processed in the first domain, and may perform domain mapping processing on the image to be processed through a first generator module to obtain a target generated image in the second domain; perform encoding processing on the target generated image to obtain encoded data. Then, the first device 101 may send the encoded data to the second device 012. The second device 102 may receive the encoded data and perform decoding processing on the encoded data to obtain a target domain image in the second domain.

[0046] The network 103 is used to provide a medium for the communication link between the first device 101 and the second device 102, and may be a wired network or a wireless network.

[0047] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is usually the Internet, but can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network. In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent the data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some of the links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0048] The first device 101 can be a terminal device or a server, and the second device 102 can also be a terminal device or a server, which is not limited in this disclosure. Optionally, the terminal device can be various electronic devices, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0049] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0050] Those skilled in the art can know that Figure 1 the numbers of the first device, the second device and the network in

[0051] The following will describe the exemplary embodiments in detail in conjunction with the accompanying drawings and embodiments.

[0052] First, an image processing method is provided in an embodiment of the present disclosure. This method can be executed by a first device, and the first device can be any electronic device with computing and processing capabilities.

[0053] Figure 2 A flowchart of an image processing method in an embodiment of the present disclosure is shown. As Figure 2 shown, the image processing method provided in the embodiment of the present disclosure includes the following steps S202 to S208.

[0054] S202, obtain the image to be processed in the first domain.

[0055] The embodiment of the present disclosure does not limit the application scenario. Exemplarily, the present disclosure can be applied to fields such as video transmission, face recognition, and image compression.

[0056] The embodiment of the present disclosure also does not limit the content of the image to be processed. The content of the image to be processed can be determined based on the application scenario. Exemplarily, taking the video transmission field as an example, the image to be processed can be any frame image in the video to be transmitted. Or, taking the face recognition field as an example, the image to be processed can be an image containing a face.

[0057] The embodiment of the present disclosure also does not limit the method for the first device to obtain the image to be processed. The method for obtaining the image to be processed can be determined based on the first device. Exemplarily, the first device can be a smart phone, and the first device can obtain the image to be processed through the camera carried by the smart phone. Or, the first device can be a server, and the first device can obtain the image to be processed through an image acquisition device capable of communicating with the first device.

[0058] The embodiment of the present disclosure also does not limit the first domain. Exemplarily, the first domain can be the VIS domain, the IR domain, etc.

[0059] S204, through the first generator module, perform domain mapping processing on the image to be processed to obtain the target generated image in the second domain.

[0060] In some embodiments, the first generator module can be constructed by network structures such as U-Net (Unity-Networking) and ResNet (Residual Network), or the first generator module can also be constructed through other network structures. The present disclosure does not limit this.

[0061] Exemplarily, the type of the first domain may include an infrared light image domain, a visible light image domain, etc. The second domain is an image domain of a different type from the first domain. The embodiments of the present disclosure do not limit the types of the first domain and the second domain. In one possible implementation, the first domain may be an infrared light image domain, and the second domain may be a visible light image domain.

[0062] In this case, the image to be processed belonging to the infrared light image domain may be input into the first generator module for domain mapping processing, so as to obtain the corresponding target generated image belonging to the visible light image domain. The embodiments of the present disclosure do not limit the steps of the above domain mapping processing.

[0063] S206. Perform encoding processing on the target generated image to obtain encoded data.

[0064] In an exemplary embodiment, performing encoding processing on the target generated image to obtain encoded data includes: performing encoding processing on the target generated image through an encoder pre-trained based on the second domain to obtain encoded data.

[0065] In this case, sending the encoded data to a second device, where the second device is configured to perform decoding processing on the encoded data to obtain the target domain image of the second domain, includes: sending the encoded data to the second device, where the second device is configured to perform decoding processing on the encoded data through a decoder pre-trained based on the second domain to obtain the target domain image of the second domain.

[0066] Exemplarily, the above encoder and decoder may jointly constitute an image encoding and decoding system. Moreover, the above encoder and decoder may be pre-trained in the second domain in advance. Through pre-training, the accuracy of the image encoding and decoding system can be improved, and the effects of image encoding and decoding can be ensured.

[0067] In addition, the embodiments of the present disclosure do not limit the process of pre-training the above encoder and decoder. The process of pre-training the above encoder and decoder may be determined based on experience or application scenarios.

[0068] In some possible implementation manners, both the encoder and the decoder may be constructed through network structures such as ResNet. The embodiments of the present disclosure do not limit this.

[0069] S208. Send the encoded data to the second device, where the second device is configured to perform decoding processing on the encoded data to obtain the target domain image of the second domain.

[0070] In an exemplary embodiment, the graphics processing method provided by the embodiments of the present disclosure may further include: performing domain mapping processing on the target domain image through a second generator module to obtain the original domain image of the first domain.

[0071] Exemplarily, still taking the case where the first domain can be the infrared light image domain and the second domain can be the visible light image domain as an example, in this case, the target domain image belonging to the visible light image domain can be input into the second generator module for domain mapping processing, so as to obtain the corresponding original domain image belonging to the infrared light image domain. The embodiments of the present disclosure do not limit the steps of the above domain mapping processing.

[0072] It should be noted that in the exemplary embodiments, before obtaining the image to be processed in the first domain, it is necessary to determine the first generator module and the second generator module. A method for determining the first generator module and the second generator module can be as described below.

[0073] In an exemplary embodiment, the graphic processing method provided by the embodiments of the present disclosure may further include: respectively constructing a first initial generator module and a second initial generator module; pre-training the first initial generator module and the second initial generator module through a training data set to obtain the first generator module and the second generator module, where the training data set includes a plurality of first training images in the first domain and a plurality of second training images in the second domain.

[0074] Exemplarily, both the first initial generator module and the second initial generator module can be constructed through network structures such as U-Net and ResNet. In addition, the embodiments of the present disclosure do not limit the method for obtaining the above training data set.

[0075] In an exemplary embodiment, the graphic processing method provided by the embodiments of the present disclosure may further include: iteratively performing the following operations until a preset end condition is met to obtain the first generator module and the second generator module:

[0076] Performing domain mapping processing on any first training image through the first initial generator module to obtain a first generated image in the second domain; performing domain mapping processing on the first generated image through the second initial generator module to obtain a first reconstructed image in the first domain; performing domain mapping processing on any second training image through the second initial generator module to obtain a second generated image in the first domain; performing domain mapping processing on the second generated image through the first initial generator module to obtain a second reconstructed image in the second domain; calculating a first loss value between the first training image and the first reconstructed image, and a second loss value between the second training image and the second reconstructed image; calculating a third loss value between the first training image and the second generated image, and a fourth loss value between the second training image and the first generated image; pre-training through the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the first generator module and the second generator module.

[0077] The embodiments of the present disclosure do not limit the above-mentioned preset end conditions. Exemplarily, gradient calculations can be performed on the first loss value, the second loss value, the third loss value, and the fourth loss value. In this case, the preset end condition can be that the gradient value is less than a preset gradient threshold. The embodiments of the present disclosure do not limit the value of the preset gradient threshold, and the value of the preset gradient threshold can be determined according to experience or implementation scenarios.

[0078] Alternatively, the number of current pre-training times can be recorded. In this case, the preset end condition can be that the current training times are not less than a preset number threshold. The embodiments of the present disclosure do not limit the value of the preset number threshold, and the value of the preset number threshold can be determined according to experience or implementation scenarios.

[0079] In an exemplary embodiment, a generative adversarial network model can be constructed. The generative adversarial network model includes a first initial generator module, a second initial generator module, a first discriminator module, and a second discriminator module. Therefore, the first generator module and the second generator module can be obtained by pre-training the generative adversarial network model.

[0080] In this case, calculating the first loss value between the first training image and the first reconstructed image and the second loss value between the second training image and the second reconstructed image includes: obtaining the first loss value between the first training image and the first reconstructed image and the second loss value between the second training image and the second reconstructed image through a cycle-consistent loss algorithm.

[0081] In this case, calculating the third loss value between the first training image and the second generated image and the fourth loss value between the second training image and the first generated image includes: obtaining the first discriminator module, scoring the first training image and the second generated image through the first discriminator module to obtain the third loss value; obtaining the second discriminator module, scoring the second training image and the first generated image through the second discriminator module to obtain the fourth loss value. Exemplarily, before scoring, the first discriminator module and the second discriminator module can be respectively pre-trained correspondingly.

[0082] Exemplarily, the score obtained by the first discriminator module scoring the first training image and the second generated image can be a decimal between 0 and 1, and this score can be used to represent that the input image is a training image or a generated image.

[0083] In a possible implementation manner, a pre-training process can be as Figure 3 shown.

[0084] In Figure 3In [the method], a first training image in a first domain can be input into a first initial generator module. The first initial generator module can perform domain mapping processing on the first training image, so as to output a first generated image in a second domain. Then, the first generated image is input into a second initial generator module, so as to output a first reconstructed image in the first domain. Then, a first loss value between the first training image and the first reconstructed image can be calculated through a cycle-consistent loss algorithm.

[0085] Moreover, a second training image in a second domain can be input into the second initial generator module. The second initial generator module can perform domain mapping processing on the second training image, so as to output a second generated image in the first domain. Then, the second generated image is input into the first initial generator module, so as to output a second reconstructed image in the second domain. Then, a second loss value between the second training image and the second reconstructed image can be calculated through the cycle-consistent loss algorithm.

[0086] It should be noted that the present disclosure embodiments do not limit the sequence of obtaining the first reconstructed image and obtaining the second reconstructed image.

[0087] After that, the first discriminator module can score the first training image and the second generated image to obtain a third loss value. And the second discriminator module can score the second training image and the first generated image to obtain a fourth loss value. Finally, the first initial generator module, the second initial generator module, the first discriminator module, and the second discriminator module can be iteratively updated through pre-training using the first loss value, the second loss value, the third loss value, and the fourth loss value, and finally a first generator module, a second generator module, a pre-trained first discriminator module, and a pre-trained second discriminator module are obtained.

[0088] The method provided by the present disclosure embodiments can perform pre-mapping processing on an image to be processed in a first domain through the first generator module to obtain a target generated image in a second domain. After that, general encoding and decoding processing can be performed on the target generated image. Therefore, the present disclosure can simplify the steps of image processing, reduce resource consumption, improve the general standardization of image processing, and improve the speed of image processing.

[0089] In a possible implementation manner, a process of image processing can be as Figure 4 shown.

[0090] In Figure 4In [the method], the to-be-processed image obtained by the first device can be input into the first generator module to obtain a target generated image. Then, the target generated image can be input into an encoder to obtain encoded data. After that, the encoded data can be sent to the decoder included in the second device through a data transmission module. Subsequently, the target domain image can be obtained through decoding processing by the decoder. Optionally, after the decoder outputs the target domain image, the target domain image can also be input into the second generator module to obtain the original domain image.

[0091] Exemplarily, Figure 5 shows a flowchart of a pre-training method in an embodiment of the present disclosure. As Figure 5 shown, the pre-training method provided in the embodiment of the present disclosure includes the following steps S502 to S530.

[0092] S502, obtain a first training image.

[0093] S504, construct a first initial generator module.

[0094] S506, input the first training image into the first initial generator module to obtain a first generated image.

[0095] S508, construct a second initial generator module.

[0096] S510, input the first generated image into the second initial generator module to obtain a first reconstructed image.

[0097] S512, calculate the cycle consistency loss between the first training image and the first reconstructed image to obtain a first loss value.

[0098] S514, obtain a second training image.

[0099] S516, input the second training image into the second initial generator module to obtain a second generated image.

[0100] S518, input the second generated image into the first initial generator module to obtain a second reconstructed image.

[0101] S520, calculate the cycle consistency loss between the second training image and the second reconstructed image to obtain a second loss value.

[0102] S522, construct a first discriminator module.

[0103] S524, construct a second discriminator module.

[0104] S526, score the first training image and the second generated image through the first discriminator module to obtain a third loss value.

[0105] S528, use the second discriminator module to score the second training image and the first generated image to obtain a fourth loss value.

[0106] S530, perform iterative updates using the first loss value, the second loss value, the third loss value, and the fourth loss value, and finally obtain the first generator module, the second generator module, the first discriminator module, and the second discriminator module.

[0107] It should be noted that the implementation methods of the steps in S502 to S530 can refer to the above descriptions related to the pre-training process, and will not be elaborated here.

[0108] Exemplarily, Figure 6 shows a flowchart of an image processing method in an embodiment of the present disclosure. As Figure 6 shown, the image processing method provided in the embodiment of the present disclosure includes the following steps S602 to S620.

[0109] S602, obtain the image to be processed.

[0110] S604, obtain the first generator module.

[0111] S606, input the image to be processed into the first generator module to obtain a target generated image.

[0112] S608, construct the pre-trained encoder and the pre-trained decoder.

[0113] S610, input the target generated image into the pre-trained encoder to obtain encoded data.

[0114] S612, obtain the data transmission module.

[0115] The embodiment of the present disclosure does not limit the data transmission method of this data transmission module. Exemplarily, the data transmission method adopted by this data transmission module can be wired transmission or wireless transmission.

[0116] S614, send the encoded data to the pre-trained decoder through the data transmission module.

[0117] Exemplarily, after inputting the encoded data into the data transmission module, a pre-trained decoder can be constructed. Then, the encoded data is input into the decoder through the data transmission module.

[0118] S616, obtain the target domain image through the pre-trained decoder.

[0119] Optionally, after obtaining the target domain image, S618 and S620 can also be executed to obtain the original domain image.

[0120] S618, obtain the second generator module.

[0121] In S620, input the target domain image into the second generator module to obtain the original domain image.

[0122] It should be noted that the implementation manners of the steps from S602 to S620 can refer to the corresponding descriptions in S202 to S208 above, and will not be elaborated here.

[0123] The method provided by the embodiments of the present disclosure realizes the mapping between domains by introducing a generative adversarial network, thereby mapping heterogeneous image data onto the target domain (i.e., the second domain), and then using the image encoding and decoding system of the target domain for encoding and decoding. The encoding and decoding process of heterogeneous images can be processed by a single set of image encoding and decoding system, realizing the reuse of the model and reducing the computational cost. Moreover, the present disclosure does not require special equipment, has low requirements for the data set, and can be used in fields such as image and video compression and transmission, having commercial value for implementation.

[0124] Based on the same inventive concept, an image processing device is also provided in the embodiments of the present disclosure as described in the following embodiments. Since the principle of solving problems by this device embodiment is similar to that of the above method embodiment, the implementation of this device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be elaborated.

[0125] Figure 7 As shown in the schematic diagram of an image processing device in the embodiments of the present disclosure, as Figure 7 shown, the device is applied to the first device and includes:

[0126] A to-be-processed image acquisition unit 701, configured to acquire a to-be-processed image in the first domain;

[0127] A target generated image determination unit 702, configured to perform domain mapping processing on the to-be-processed image through the first generator module to obtain a target generated image in the second domain;

[0128] An encoding unit 703, configured to perform encoding processing on the target generated image to obtain encoded data;

[0129] A decoding unit 704, configured to send the encoded data to a second device, and the second device is configured to perform decoding processing on the encoded data to obtain a target domain image in the second domain.

[0130] In some embodiments of the present disclosure, the image processing device provided by the embodiments of the present disclosure further includes:

[0131] An original domain image determination unit, configured to perform domain mapping processing on the target domain image through the second generator module to obtain an original domain image in the first domain.

[0132] In some embodiments of the present disclosure, an encoding unit 703 is configured to perform encoding processing on the target generated image through an encoder pre-trained based on a second domain to obtain the encoded data;

[0133] A decoding unit 704 is configured to send the encoded data to the second device, and the second device is configured to perform decoding processing on the encoded data through a decoder pre-trained based on the second domain to obtain a target domain image in the second domain.

[0134] In some embodiments of the present disclosure, the image processing apparatus provided by the embodiments of the present disclosure further includes:

[0135] A generator construction unit is configured to construct a first initial generator module and a second initial generator module respectively;

[0136] A pre-training unit is configured to pre-train the first initial generator module and the second initial generator module through a training data set to obtain the first generator module and the second generator module, where the training data set includes a plurality of first training images in a first domain and a plurality of second training images in a second domain.

[0137] In some embodiments of the present disclosure, the pre-training unit is configured to iteratively perform the following operations until a preset end condition is met to obtain the first generator module and the second generator module: perform domain mapping processing on any first training image through the first initial generator module to obtain a first generated image in the second domain; perform domain mapping processing on the first generated image through the second initial generator module to obtain a first reconstructed image in the first domain; perform domain mapping processing on any second training image through the second initial generator module to obtain a second generated image in the first domain; perform domain mapping processing on the second generated image through the first initial generator module to obtain a second reconstructed image in the second domain; calculate a first loss value between the first training image and the first reconstructed image, and a second loss value between the second training image and the second reconstructed image; calculate a third loss value between the first training image and the second generated image, and a fourth loss value between the second training image and the first generated image; perform pre-training through the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the first generator module and the second generator module.

[0138] In some embodiments of the present disclosure, a pre-training unit is configured to obtain a first loss value between the first training image and the first reconstructed image, and a second loss value between the second training image and the second reconstructed image through a cycle-consistent loss algorithm; obtain a first discriminator module, and score the first training image and the second generated image through the first discriminator module to obtain the third loss value; obtain a second discriminator module, and score the second training image and the first generated image through the second discriminator module to obtain the fourth loss value.

[0139] In some embodiments of the present disclosure, the type of the first domain includes an infrared light image domain and a visible light image domain, and the second domain is an image domain of a different type from the first domain.

[0140] The device provided by the embodiments of the present disclosure can pre-map the image to be processed in the first domain through the first generator module to obtain the target generated image in the second domain. Thereafter, general encoding and decoding processing can be performed on the target generated image. Therefore, the present disclosure can simplify the image processing steps, reduce resource consumption, improve the general standardization of image processing, and improve the speed of image processing.

[0141] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method, or a program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuitry", "module", or "system" here.

[0142] Next, refer to Figure 8 to describe the electronic device 800 according to this embodiment of the present disclosure. Figure 8 The displayed electronic device 800 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0143] As Figure 8 shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one of the above-mentioned processing units 810, at least one of the above-mentioned storage units 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0144] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 810, so that the processing unit 810 executes the steps according to various exemplary embodiments of the present disclosure described in the above "Specific Embodiments" section of this specification.

[0145] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 8201 and / or a cache storage unit 8202, and may further include a read-only storage unit (ROM) 8203.

[0146] The storage unit 820 may also include a program / utilities 8204 having a set (at least one) of program modules 8205. Such program modules 8205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0147] The bus 830 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0148] The electronic device 800 may also communicate with one or more external devices 840 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or may communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be through an input / output (I / O) interface 850. Moreover, the electronic device 800 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 860. As shown in the figure, the network adapter 860 communicates with other modules of the electronic device 800 through the bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0149] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which may be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0150] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is further provided. The computer-readable storage medium may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementation manners, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Specific Embodiments" section of this specification.

[0151] More specific examples of the computer-readable storage medium in the present disclosure may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0152] In the present disclosure, the computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than the readable storage medium, and the readable medium may send, propagate, or transmit a program used by or in combination with an instruction execution system, apparatus, or device.

[0153] Optionally, the program code included on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0154] In specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as an independent software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (for example, by connecting through an Internet service provider via the Internet).

[0155] It should be noted that although several modules or units of a device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0156] In addition, although the steps of the methods in the present disclosure are described in a specific order in the drawings, this does not require or imply that these steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.

[0157] From the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the methods according to the embodiments of the present disclosure.

[0158] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present disclosure. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only regarded as exemplary, and the true scope of the present disclosure is pointed out by the appended claims.

Claims

1. An image processing method, characterized in that, Performed by a first device, including: Obtain a to-be-processed image of a first domain; Through a first generator module, perform domain mapping processing on the to-be-processed image to obtain a target generated image of a second domain; Perform encoding processing on the target generated image to obtain encoded data; Send the encoded data to a second device, where the second device is used to perform decoding processing on the encoded data to obtain a target domain image of the second domain.

2. The image processing method according to claim 1, wherein The method further includes: Through a second generator module, perform domain mapping processing on the target domain image to obtain an original domain image of the first domain.

3. The image processing method according to claim 1, wherein The performing encoding processing on the target generated image to obtain encoded data includes: Perform encoding processing on the target generated image through an encoder pre-trained based on the second domain to obtain the encoded data; Wherein, the sending the encoded data to the second device, where the second device is used to perform decoding processing on the encoded data to obtain a target domain image of the second domain, includes: Send the encoded data to the second device, where the second device is used to perform decoding processing on the encoded data through a decoder pre-trained based on the second domain to obtain a target domain image of the second domain.

4. The image processing method according to any one of claims 1 to 3, characterized in that, The method further includes: Construct a first initial generator module and a second initial generator module respectively; Pre-train the first initial generator module and the second initial generator module through a training data set to obtain the first generator module and the second generator module, where the training data set includes a plurality of first training images of the first domain and a plurality of second training images of the second domain.

5. The image processing method according to claim 4, characterized in that The method includes: Iteratively perform the following operations until a preset end condition is met to obtain the first generator module and the second generator module: Through the first initial generator module, perform domain mapping processing on any first training image to obtain a first generated image of the second domain; through the second initial generator module, perform domain mapping processing on the first generated image to obtain a first reconstructed image of the first domain; Through the second initial generator module, perform domain mapping processing on any second training image to obtain a second generated image of the first domain; through the first initial generator module, perform domain mapping processing on the second generated image to obtain a second reconstructed image of the second domain; Calculate a first loss value between the first training image and the first reconstructed image, and a second loss value between the second training image and the second reconstructed image; calculate a third loss value between the first training image and the second generated image, and a fourth loss value between the second training image and the first generated image; Perform pre-training through the first loss value, the second loss value, the third loss value, and the fourth loss value to obtain the first generator module and the second generator module.

6. The image processing method according to claim 5, wherein The calculating the first loss value between the first training image and the first reconstructed image, and the second loss value between the second training image and the second reconstructed image includes: Obtain a first loss value between the first training image and the first reconstructed image and a second loss value between the second training image and the second reconstructed image through a cycle-consistent loss algorithm; Calculating a third loss value between the first training image and the second generated image and a fourth loss value between the second training image and the first generated image includes: Obtain a first discriminator module, and score the first training image and the second generated image through the first discriminator module to obtain the third loss value; Obtain a second discriminator module, and score the second training image and the first generated image through the second discriminator module to obtain the fourth loss value.

7. The image processing method according to any one of claims 1 to 3, characterized in that The type of the first domain includes an infrared light image domain and a visible light image domain, and the second domain is an image domain of a different type from the first domain.

8. An image processing apparatus, characterized in that, Applied to a first device, it includes: A to-be-processed image acquisition unit for acquiring a to-be-processed image of the first domain; A target generated image determination unit for performing domain mapping processing on the to-be-processed image through a first generator module to obtain a target generated image of the second domain; An encoding unit for encoding the target generated image to obtain encoded data; A decoding unit for sending the encoded data to a second device, and the second device is used for decoding the encoded data to obtain a target domain image of the second domain.

9. An electronic device, characterized in that, Includes: A processor; And A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the image processing method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image processing method according to any one of claims 1 to 7.

Images