Image processing method, apparatus, medium, and device

Abstract

The present disclosure relates to an image processing method, device, medium and equipment. The method comprises: obtaining a target simulation image; dividing the target simulation image to obtain a plurality of target image blocks; generating a target real image corresponding to the target simulation image according to the target simulation image and the plurality of target image blocks through an image generation model, wherein the image generation model is a pre-trained model for converting a simulation image into a real image. In this way, the image generation model can not only learn the global semantic information of the target simulation image, but also learn the local detail information of different divided regions of the target simulation image, thereby better realizing the local detail generation conversion effect, making the simulation image more close to the real scene image while maintaining the original detail texture, and solving the related problems of low precision and poor effect of deep network training by simply using simulation data, and inability to meet the real complex scene requirements.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, medium and device. Background Technology

[0002] In recent years, deep neural network technology has become a major driving force and method for the development of artificial intelligence. Iteratively optimizing a good deep network model requires a large amount of high-quality data. However, in actual algorithm development, we find that collecting and labeling real-world images consumes a great deal of time and cost, and the quality of real-world data can be affected by the acquisition equipment. To address this, some researchers use simulated data such as CitySpaces and GTA V to replace real data in related academic research, such as using simulated data for depth estimation, scene segmentation, and semantic segmentation algorithms. However, simulated images still differ from real-world images. For example, high-frequency areas of the sky, surface lighting on objects, and skin color all differ from real-world data. Algorithms developed solely using simulated data are unlikely to meet the needs of real-world applications. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, medium, and device.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring a target simulation image; dividing the target simulation image into blocks to obtain a plurality of target image blocks; and generating a target realistic image corresponding to the target simulation image using an image generation model based on the target simulation image and the plurality of target image blocks, wherein the image generation model is a pre-trained model for converting simulation images into realistic images.

[0005] Optionally, the image generation model is trained by: acquiring a sample simulation image; dividing the sample simulation image into blocks to obtain multiple sample image blocks; generating a sample realization image corresponding to the sample simulation image using a target model based on the sample simulation image and the multiple sample image blocks; updating the model parameters of the target model based on the sample simulation image and the sample realization image; in response to not meeting the training cutoff condition, re-executing the steps from acquiring the sample simulation image to updating the model parameters of the target model based on the sample simulation image and the sample realization image; and obtaining the image generation model in response to meeting the training cutoff condition.

[0006] Optionally, generating a sample realization image corresponding to the sample simulation image using a target model based on the sample simulation image and the plurality of sample image blocks includes: generating a global sample image based on the sample simulation image; generating a local sample image corresponding to each of the plurality of sample image blocks; and generating a sample realization image corresponding to the sample simulation image based on the global sample image and all the local sample images.

[0007] Optionally, updating the model parameters of the target model based on the sample simulation image and the sample real image includes: generating a predicted simulation image based on the sample real image; determining whether the sample real image is true or false to obtain a first determination result; generating a predicted image block for each sample local image; determining whether the sample local image is true or false to obtain a second determination result for each sample local image; and updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, and all the second determination results.

[0008] Optionally, updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image patches, all the predicted image patches, the first determination result, and all the second determination results includes: calculating a global cycle consistency loss based on the sample simulation image and the predicted simulation image; calculating a global adversarial loss based on the first determination result; calculating a local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; calculating a local adversarial loss based on all the second determination results; and updating the model parameters of the target model based on the global cycle consistency loss, the global adversarial loss, the local cycle consistency loss, and the local adversarial loss.

[0009] Optionally, generating a sample realization image corresponding to the sample simulation image based on the sample global image and all the sample local images includes: stitching all the sample local images together to obtain a first stitched image; stitching the first stitched image and the sample global image together by channel to obtain a second stitched image; and generating a sample realization image corresponding to the sample simulation image based on the second stitched image.

[0010] Optionally, updating the model parameters of the target model based on the sample simulation image and the sample real image further includes: downsampling the sample real image at at least one sampling rate to obtain a first downsampled image at at least one scale; for each first downsampled image, determining whether the first downsampled image is true or false to obtain a third determination result; updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, and all the second determination results includes: updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, all the second determination results, and all the third determination results.

[0011] Optionally, updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image patches, all the predicted image patches, the first determination result, all the second determination results, and all the third determination results includes: calculating a global cycle consistency loss based on the sample simulation image and the predicted simulation image; calculating a global adversarial loss based on the first determination result and all the third determination results; calculating a local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; calculating a local adversarial loss based on all the second determination results; and updating the model parameters of the target model based on the global cycle consistency loss, the global adversarial loss, the local cycle consistency loss, and the local adversarial loss.

[0012] Optionally, updating the model parameters of the target model based on the sample simulation image and the sample real image further includes: for each sample local image, downsampling the sample local image at at least one sampling rate to obtain a second downsampled image of the sample local image at at least one scale; for each second downsampled image, determining whether the second downsampled image is real or fake to obtain a fourth determination result; updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, and all the second determination results includes: updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, all the second determination results, and all the fourth determination results.

[0013] Optionally, updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image patches, all the predicted image patches, the first determination result, all the second determination results, and all the fourth determination results includes: calculating a global cycle consistency loss based on the sample simulation image and the predicted simulation image; calculating a global adversarial loss based on the first determination result; calculating a local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; calculating a local adversarial loss based on all the second determination results and all the fourth determination results; and updating the model parameters of the target model based on the global cycle consistency loss, the global adversarial loss, the local cycle consistency loss, and the local adversarial loss.

[0014] Optionally, there may be overlapping areas among the plurality of sample image blocks.

[0015] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising: a first acquisition module configured to acquire a target simulation image; a first segmentation module configured to segment the target simulation image acquired by the first acquisition module into multiple target image blocks; and a first generation module configured to generate a target realistic image corresponding to the target simulation image based on the target simulation image acquired by the first acquisition module and the multiple target image blocks obtained by the first segmentation module, through an image generation model, wherein the image generation model is a pre-trained model for converting simulation images into realistic images.

[0016] According to a third aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the image processing method provided in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a memory having a computer program stored thereon; and a processor for executing the computer program in the memory to implement the steps of the image processing method provided in the first aspect of the present disclosure.

[0018] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: After obtaining the target simulation image, the target simulation image is divided into blocks to obtain multiple target image blocks; based on the target simulation image and the multiple target image blocks, a target realistic image corresponding to the target simulation image is generated by a pre-trained image generation model used to convert the simulation image into a realistic image. In this way, the image generation model can not only learn the global semantic information of the target simulation image, but also learn the local detail information of different block regions of the target simulation image, thereby achieving a better local detail generation and conversion effect. This makes the simulation image more closely resemble the real scene image while maintaining the original detail texture, solving the problems of low accuracy and poor effect of training deep networks using simulation data alone, and the inability to meet the needs of real complex scenes.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0022] Figure 2 This is a flowchart illustrating an image generation model training method according to an exemplary embodiment.

[0023] Figure 3 This is a schematic diagram of the structure of a target model according to an exemplary embodiment.

[0024] Figure 4 This is a schematic diagram of the structure of a target model according to another exemplary embodiment.

[0025] Figure 5 This is a schematic diagram of the structure of a target model according to another exemplary embodiment.

[0026] Figure 6 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0027] Figure 7 This is a block diagram illustrating an image generation model training apparatus according to an exemplary embodiment.

[0028] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0029] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0031] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0032] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment. The method can be applied to a terminal (e.g., a smartphone, tablet, etc.) and also to a server; this disclosure does not specifically limit its application. Figure 1 As shown, the image processing method may include the following steps S101 to S103.

[0033] In S101, the target simulation image is acquired.

[0034] In this disclosure, the target simulation image can be an image obtained through simulation software such as CitySpaces or GTA V.

[0035] In S102, the target simulation image is divided into blocks to obtain multiple target image blocks.

[0036] In this disclosure, multiple target image patches may or may not have overlapping regions; this disclosure does not impose any specific limitations.

[0037] In S103, based on the target simulation image and multiple target image blocks, a target realization image corresponding to the target simulation image is generated through an image generation model.

[0038] In this disclosure, the image generation model described above is a pre-trained model used to convert simulated images into realistic images. Specifically, a target simulated image and multiple target image patches can be input into the image generation model to obtain a target realistic image corresponding to the target simulated image.

[0039] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: After obtaining the target simulation image, the target simulation image is divided into blocks to obtain multiple target image blocks; based on the target simulation image and the multiple target image blocks, a target realistic image corresponding to the target simulation image is generated by a pre-trained image generation model used to convert the simulation image into a realistic image. In this way, the image generation model can not only learn the global semantic information of the target simulation image, but also learn the local detail information of different block regions of the target simulation image, thereby achieving a better local detail generation and conversion effect. This makes the simulation image more closely resemble the real scene image while maintaining the original detail texture, solving the problems of low accuracy and poor effect of training deep networks using simulation data alone, and the inability to meet the needs of real complex scenes.

[0040] The following is a detailed description of the specific implementation method for dividing the target simulation image into blocks to obtain multiple target image blocks in S102 above.

[0041] In one implementation, the target simulation image can be divided into blocks to obtain multiple target image blocks, wherein there are no overlapping areas between the multiple target image blocks.

[0042] For example, the target simulation image is segmented according to a grid, a nine-square grid, or other similar methods.

[0043] In another implementation, the target simulation image can be divided into blocks to obtain multiple target image blocks, wherein there are overlapping areas between the multiple target image blocks.

[0044] For example, the target simulation image can be divided into five image regions: upper left, upper right, center, lower left, and lower right, i.e., five target image blocks.

[0045] For example, let's establish a coordinate system with the top-left corner of the target simulation image as the origin, the width direction of the target simulation image as the x-axis, and the height direction of the target simulation image as the y-axis. Let the width and height of the target simulation image be w and h, respectively. Then, the diagonal coordinates of the top-left region are (0,0) and (w / 2+overlap,h / 2+overlap), the top-right region is (w / 2-overlap,0) and (w,h / 2+overlap), the center region is (w / 2-overlap,h / 2-overlap) and (w / 2+overlap,h / 2+overlap), the bottom-left region is (0,h / 2-overlap) and (w / 2+overlap,h), and the bottom-right region is (w / 2-overlap,h / 2-overlap) and (w,h). Here, `overlap` represents the width of the overlapping area between adjacent target image blocks.

[0046] Specifically, the top left and top right regions, cropped from the blocks, allow the image generation model to focus more on details such as the sky and clouds; the bottom left and bottom right regions allow the model to focus more on road surface details; and the central region allows the model to focus on the main subject of the input image. This reduces the gap between the simulated image and the real-world image, making the simulated image more closely resemble the real scene while retaining its original details and textures.

[0047] The training method for the aforementioned image generation model will be explained in detail below. Specifically, it can be achieved through... Figure 2 This is achieved through steps S201 to S206 shown in the diagram.

[0048] In S201, obtain the sample simulation image.

[0049] In this disclosure, the sample simulation image can be an image obtained through simulation software such as CitySpaces, GTA V, etc.

[0050] In S202, the sample simulation image is divided into blocks to obtain multiple sample image blocks.

[0051] In this disclosure, multiple sample image blocks may or may not have overlapping regions; this disclosure does not impose any specific limitations.

[0052] Alternatively, the sample simulation image can be divided into blocks in a manner similar to that used in S102 above for dividing the target simulation image, which will not be elaborated further in this disclosure.

[0053] In S203, based on the sample simulation image and multiple sample image blocks, a sample realization image corresponding to the sample simulation image is generated through the target model.

[0054] In S204, the model parameters of the target model are updated based on the sample simulation image and the sample real image.

[0055] In S205, it is determined whether the training cutoff condition is met.

[0056] In this disclosure, the training cutoff condition can be either the number of training iterations reaching a threshold for the number of prediction iterations, or the loss function of the target model stabilizing.

[0057] If the training cutoff condition is not met, return to S201 above and continue execution until the training cutoff condition is met; if the training cutoff condition is met, execute S206 below.

[0058] In S206, the image generation model is obtained.

[0059] The following is a detailed description of the specific implementation method for generating a sample realization image corresponding to the sample simulation image based on the sample simulation image and multiple sample image blocks in step S203 above. Specifically, it can be achieved through the following steps (1) to (3):

[0060] (1) Generate a global image of the sample based on the sample simulation image.

[0061] (2) For each sample image block in the multiple sample image blocks, generate the sample local image corresponding to the sample image block.

[0062] (3) Generate a sample realization image corresponding to the sample simulation image based on the global image of the sample and the local images of all samples.

[0063] In one implementation, such as Figure 3 As shown, the target model may include a first generator, a second generator, and a processing module. At this point, a global sample image can be generated using the first generator based on the sample simulation image; then, for each sample image block among multiple sample image blocks, a local sample image corresponding to that sample image block is generated using the second generator; finally, a realistic sample image corresponding to the sample simulation image is generated using the processing module based on the global sample image and all the local sample images.

[0064] For example, both the first and second generators can be deep convolutional neural networks.

[0065] The following is a detailed explanation of the specific implementation method for updating the model parameters of the target model based on the sample simulation image and the sample real image in S204 above. Specifically, it can be achieved through the following steps 1) to 5).

[0066] 1) Generate a predicted simulation image based on the real image of the sample.

[0067] 2) Determine whether the sample image is real or fake to obtain the first determination result.

[0068] 3) For each sample local image, generate a predicted image patch based on that sample local image.

[0069] 4) For each sample local image, determine whether the sample local image is true or false, and obtain the second determination result.

[0070] 5) Update the model parameters of the target model based on the sample simulation image, the predicted simulation image, multiple sample image blocks, all predicted image blocks, the first judgment result, and all second judgment results.

[0071] For example, such as Figure 4 As shown, the target model described above may further include a third generator, a fourth generator, a first discriminator, and a second discriminator. In this case, a predicted simulation image can be generated using the third generator based on the sample realistic image; the first discriminator determines the realism of the sample realistic image to obtain a first determination result. For each sample local image, a predicted image patch is generated using the fourth generator based on that sample local image; for each sample local image, the second discriminator determines the realism of that sample local image to obtain a second determination result; the model parameters of the first generator and the third generator are updated based on the sample simulation image, the predicted simulation image, and the first determination result; the model parameters of the second generator and the fourth generator are updated based on the sample simulation image, the predicted simulation image, multiple sample image patches, all predicted image patches, and the first determination result; the model parameters of the first discriminator are updated based on the first determination result; and the model parameters of the second discriminator are updated based on all second determination results.

[0072] In the above implementation, the target model is a recurrent generative adversarial network. This ensures that the real image of the sample after the realization conversion corresponds to the simulated image of the sample before the realization conversion, thereby improving the effect of the sample realization conversion.

[0073] The following is a detailed explanation of the specific implementation method for updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, multiple sample image blocks, all predicted image blocks, the first determination result, and all second determination results in step 5) above. Specifically, it can be achieved through the following steps 51) to 55).

[0074] 51) Calculate the global cycle consistency loss based on the sample simulation image and the predicted simulation image.

[0075] 52) Calculate the global adversarial loss based on the first judgment result.

[0076] 53) Calculate the local cycle consistency loss based on multiple sample image patches and all predicted image patches.

[0077] Specifically, for each sample image block, the cycle consistency sub-loss can be calculated based on the sample image block and the corresponding predicted image block; then, the sum of all cycle consistency sub-losses is determined as the local cycle consistency loss.

[0078] 54) Calculate the local adversarial loss based on all the second judgment results.

[0079] Specifically, for each second decision result, the first adversary loss can be calculated based on that second decision result; then, the sum of all first adversary losses can be determined as the local adversary loss.

[0080] 55) Update the model parameters of the target model based on the global cyclic consistency loss, global adversarial loss, local cyclic consistency loss, and local adversarial loss.

[0081] Specifically, such as Figure 4 As shown, the target model includes a first generator, a second generator, a third generator, a fourth generator, a first discriminator, a second discriminator, and a processing module. At this point, the model parameters of the first and third generators can be updated based on the global cycle consistency loss and the global adversarial loss, respectively; the model parameters of the second and fourth generators can be updated based on the global cycle consistency loss, the global adversarial loss, and the local cycle consistency loss, respectively; the model parameters of the first discriminator can be updated based on the global adversarial loss; and the model parameters of the second discriminator can be updated based on the local adversarial loss.

[0082] The following is a detailed description of the specific implementation method for generating a sample realization image corresponding to the sample simulation image based on the global image of the sample and all local images of the samples in step (3) above. Specifically, it can be implemented in a variety of ways. In one implementation method, step (3) above may include the following steps (31) and (32).

[0083] (31) Stitch together all the local images of the samples to obtain the first stitched image.

[0084] (32) The first stitched image and the sample simulation image are directly fused to obtain the sample real image corresponding to the sample simulation image. The pixel value of the pixel in the sample real image is equal to the mean or maximum (maximum or minimum) of the pixel value of the corresponding pixel in the first stitched image and the sample simulation image.

[0085] For example, the processing module in the target model described above may include a first stitching unit and a fusion unit. In this case, the first stitching unit can stitch together all local images of the samples to obtain a first stitched image; then, the fusion unit directly fuses the first stitched image and the sample simulation image to obtain a sample realization image corresponding to the sample simulation image.

[0086] In another embodiment, step (3) above may include steps (33) to (35).

[0087] (33) Stitch together the local images of all samples to obtain the first stitched image.

[0088] (34) The first stitched image and the global sample image are stitched together according to channels to obtain the second stitched image.

[0089] (35) Generate a sample realization image corresponding to the sample simulation image based on the second stitched image.

[0090] For example, such as Figure 5 As shown, the processing module in the target model can include a first stitching unit, a second stitching unit, and a fusion sub-model. At this point, the first stitching unit can stitch together all local images of the samples to obtain a first stitched image; then, the second stitching unit can stitch the first stitched image and the global image of the samples according to channels to obtain a second stitched image; finally, based on the second stitched image, the fusion sub-model generates a sample realization image corresponding to the sample simulation image.

[0091] For example, the fusion sub-model can be a shallow convolutional neural network.

[0092] In the above implementation, the fusion sub-model is used to fuse the second stitched image, which can eliminate the image block edge breaks caused by stitching and obtain a high-quality sample realistic image with complete details and textures.

[0093] Furthermore, preferably, there are overlapping areas among the multiple sample image blocks obtained by the above-mentioned S202 segmentation. This avoids edge breaks when stitching together all sample local images, making the first stitched image smoother, thereby improving the effect of subsequent sample realization conversion.

[0094] The following is a detailed description of the specific implementation method for stitching together all sample local images in steps (31) and (33) to obtain the first stitched image.

[0095] Specifically, each sample local image can be stitched together according to the position of its corresponding sample image block in the sample simulation image. In the first stitched image, the pixel value of the pixel point in the overlapping area formed by adjacent sample local images is equal to the mean or maximum (maximum or minimum) pixel value of the corresponding pixel point in each sample local image that constitutes the overlapping area.

[0096] Preferably, in the first stitched image, the pixel value of the pixel in the overlapping region formed by the local images of adjacent samples is equal to the maximum pixel value of the corresponding pixel in each of the local images of the samples constituting the overlapping region. This allows for the acquisition of the maximum image response, minimizing the loss of image information and thus improving the accuracy of the sample realism conversion.

[0097] When the aforementioned processing module includes a first stitching unit, a second stitching unit, and a fusion sub-model, step S204 can update the model parameters of the fusion sub-model while updating the model parameters of the target model based on the sample simulation image and the sample real image. Specifically, the model parameters of the fusion sub-model can be updated based on the global cycle consistency loss and the global adversarial loss.

[0098] In addition, the above-mentioned S204 may also include the following steps 6) and 7).

[0099] 6) Downsample the sampled image at at least one sampling rate to obtain a first downsampled image at at least one scale.

[0100] For example, the sample real image is downsampled at a sampling rate of 2 to obtain a first downsampled image A with a scale of 1 / 2 of the sample real image. At the same time, the sample real image is downsampled at a sampling rate of 4 to obtain a first downsampled image B with a scale of 1 / 4 of the sample real image.

[0101] 7) For each first downsampled image, determine whether it is true or false to obtain a third determination result.

[0102] Specifically, for each first downsampled image, a first discriminator can be used to determine whether the first downsampled image is real or fake, and a third determination result can be obtained.

[0103] At this point, step 5) above can be used to update the model parameters of the target model based on the sample simulation image, the predicted simulation image, multiple sample image blocks, all predicted image blocks, the first judgment result, all second judgment results, and all third judgment results.

[0104] For example, for Figure 4The target model structure shown can update the model parameters of the first generator and the third generator respectively based on the sample simulation image, the predicted simulation image, the first judgment result, and all the third judgment results; update the model parameters of the second generator and the fourth generator respectively based on the sample simulation image, the predicted simulation image, multiple sample image blocks, all predicted image blocks, the first judgment result, and all the third judgment results; update the model parameters of the first discriminator based on the first judgment result and all the third judgment results; and update the model parameters of the second discriminator based on all the second judgment results.

[0105] In the above embodiments, inputting images of different scales into the first discriminator enables supervised training of the first discriminator with multi-scale inputs, thereby enabling the first discriminator to distinguish between real and fake images at different scales, thus realizing the function of a multi-scale discriminator. In this way, it is possible to effectively distinguish image contour details in downsampled images obtained at lower sampling rates (e.g., 2), and also to distinguish more illumination semantic information in downsampled images obtained at higher sampling rates (e.g., 4), thereby further improving the realism conversion effect of simulated images.

[0106] The following provides a detailed explanation of the specific implementation method for updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, multiple sample image blocks, all predicted image blocks, the first judgment result, all second judgment results, and all third judgment results in step 5) above. Specifically, this can be achieved through the following steps 56) to 510):

[0107] 56) Calculate the global cycle consistency loss based on the sample simulation image and the predicted simulation image.

[0108] 57) Calculate the global adversarial loss based on the first judgment result and all third judgment results.

[0109] Specifically, the second adversary loss can be calculated based on the first decision result; at the same time, the third adversary loss can be calculated for each third decision result; finally, the sum of the second adversary loss and all third adversary losses is determined as the global adversary loss.

[0110] 58) Calculate the local cycle consistency loss based on multiple sample image patches and all predicted image patches.

[0111] 59) Calculate the local adversarial loss based on all the second judgment results.

[0112] 510) Update the model parameters of the target model based on the global cyclic consistency loss, global adversarial loss, local cyclic consistency loss, and local adversarial loss.

[0113] Additionally, S204 may also include steps 8) and 9).

[0114] 8) For each sample local image, downsample the sample local image at at least one sampling rate to obtain a second downsampled image of the sample local image at at least one scale.

[0115] For example, a sample local image is downsampled at a sampling rate of 2 to obtain a second downsampled image C with a scale of 1 / 2 of the sample local image. At the same time, the sample local image is downsampled at a sampling rate of 4 to obtain a second downsampled image D with a scale of 1 / 4 of the sample local image.

[0116] 9) For each second downsampled image, determine whether it is true or false to obtain the fourth determination result.

[0117] Specifically, for each second downsampled image, a second discriminator can be used to determine whether the second downsampled image is real or fake, thus obtaining a fourth determination result.

[0118] At this point, step 5) above can be used to update the model parameters of the target model based on the sample simulation image, the predicted simulation image, multiple sample image blocks, all predicted image blocks, the first judgment result, all second judgment results, and all fourth judgment results.

[0119] For example, for Figure 4 The target model structure shown can update the model parameters of the first generator and the third generator respectively based on the sample simulation image, the predicted simulation image and the first judgment result; update the model parameters of the second generator and the fourth generator respectively based on the sample simulation image, the predicted simulation image, multiple sample image blocks, all predicted image blocks and the first judgment result; update the model parameters of the first discriminator based on the first judgment result; and update the model parameters of the second discriminator based on all the second judgment results and all the fourth judgment results.

[0120] The following provides a detailed explanation of the specific implementation method for updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, multiple sample image blocks, all predicted image blocks, the first judgment result, all second judgment results, and all fourth judgment results in step 5) above. Specifically, this can be achieved through the following steps 511) to 515):

[0121] 511) Calculate the global cycle consistency loss based on the sample simulation image and the predicted simulation image.

[0122] 512) Calculate the global adversarial loss based on the first judgment result.

[0123] 513) Calculate the local cycle consistency loss based on multiple sample image patches and all predicted image patches.

[0124] 514) Calculate the local adversarial loss based on all second and fourth judgment results.

[0125] Specifically, for each second decision result, the first adversary loss can be calculated based on the second decision result; at the same time, for each fourth decision result, the fourth adversary loss can be calculated based on the fourth decision result; then, the sum of all first adversary losses and all fourth adversary losses is determined as the local adversary loss.

[0126] 515) Update the model parameters of the target model based on the global cyclic consistency loss, global adversarial loss, local cyclic consistency loss, and local adversarial loss.

[0127] In the above embodiments, inputting images of different scales into the second discriminator enables supervised training of the second discriminator across multiple scales. This allows the second discriminator to distinguish between real and fake images at different scales, thus achieving the function of a multi-scale discriminator. In this way, it is possible to effectively distinguish image contour details in downsampled images obtained at lower sampling rates (e.g., 2), and also to distinguish more illumination semantic information in downsampled images obtained at higher sampling rates (e.g., 4), thereby further improving the realism of the simulated image conversion.

[0128] Figure 6 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. Figure 6 As shown, the image processing device 600 may include: a first acquisition module 601, configured to acquire a target simulation image; a first segmentation module 602, configured to segment the target simulation image acquired by the first acquisition module 601 into multiple target image blocks; and a first generation module 603, configured to generate a target realistic image corresponding to the target simulation image based on the target simulation image acquired by the first acquisition module 601 and the multiple target image blocks obtained by the first segmentation module 602, using an image generation model, wherein the image generation model is a pre-trained model used to convert simulation images into realistic images.

[0129] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: After obtaining the target simulation image, the target simulation image is divided into blocks to obtain multiple target image blocks; based on the target simulation image and the multiple target image blocks, a target realistic image corresponding to the target simulation image is generated by a pre-trained image generation model used to convert the simulation image into a realistic image. In this way, the image generation model can not only learn the global semantic information of the target simulation image, but also learn the local detail information of different block regions of the target simulation image, thereby achieving a better local detail generation and conversion effect. This makes the simulation image more closely resemble the real scene image while maintaining the original detail texture, solving the problems of low accuracy and poor effect of training deep networks using simulation data alone, and the inability to meet the needs of real complex scenes.

[0130] The aforementioned image generation model can be trained using an image generation model training device, such as... Figure 7 As shown, the image generation model training device 700 may include: a second acquisition module 701, configured to acquire sample simulation images; a second segmentation module 702, configured to segment the sample simulation images acquired by the second acquisition module 701 into multiple sample image blocks; a second generation module 703, configured to generate a sample realization image corresponding to the sample simulation image through a target model based on the sample simulation images acquired by the second acquisition module 701 and the multiple sample image blocks obtained by the second segmentation module 702; an update module 704, configured to update the model parameters of the target model based on the sample simulation images acquired by the second acquisition module 701 and the sample realization image generated by the second generation module 703; and a trigger module 705, configured to: trigger the second acquisition module 701 to acquire sample simulation images in response to the failure to meet the training cutoff condition; and obtain an image generation model in response to meeting the training cutoff condition.

[0131] Optionally, the second generation module 703 includes: a first generation submodule configured to generate a global sample image based on the sample simulation image; a second generation submodule configured to generate a local sample image corresponding to each of the plurality of sample image blocks; and a third generation submodule configured to generate a realistic sample image corresponding to the sample simulation image based on the global sample image and all the local sample images.

[0132] Optionally, the update module 704 includes: a fourth generation submodule configured to generate a predicted simulation image based on the sample realization image; a first determination submodule configured to determine the authenticity of the sample realization image to obtain a first determination result; a fifth generation submodule configured to generate a predicted image block for each sample local image based on the sample local image; a second determination submodule configured to determine the authenticity of each sample local image based on the sample local image to obtain a second determination result; and a first update submodule configured to update the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, and all the second determination results.

[0133] Optionally, the first update submodule includes: a first calculation submodule configured to calculate a global cycle consistency loss based on the sample simulation image and the predicted simulation image; a second calculation submodule configured to calculate a global adversarial loss based on the first determination result; a third calculation submodule configured to calculate a local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; a fourth calculation submodule configured to calculate a local adversarial loss based on all the second determination results; and a second update submodule configured to update the model parameters of the target model based on the global cycle consistency loss, the global adversarial loss, the local cycle consistency loss, and the local adversarial loss.

[0134] The third generation submodule includes: a first stitching submodule, configured to stitch together all the sample local images to obtain a first stitched image; a second stitching submodule, configured to stitch the first stitched image and the sample global image according to channels to obtain a second stitched image; and a fusion submodule, configured to generate a sample realization image corresponding to the sample simulation image based on the second stitched image.

[0135] Optionally, the update module 704 further includes: a first downsampling module configured to downsample the sample real image at at least one sampling rate to obtain a first downsampled image at at least one scale; a third determination submodule configured to determine whether each first downsampled image is true or false to obtain a third determination result; the first update submodule is configured to update the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, all the second determination results, and all the third determination results.

[0136] Optionally, the first update submodule includes: a fifth calculation submodule configured to calculate a global cycle consistency loss based on the sample simulation image and the predicted simulation image; a sixth calculation submodule configured to calculate a global adversarial loss based on the first determination result and all the third determination results; a seventh calculation submodule configured to calculate a local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; an eighth calculation submodule configured to calculate a local adversarial loss based on all the second determination results; and a third update submodule configured to update the model parameters of the target model based on the global cycle consistency loss, the global adversarial loss, the local cycle consistency loss, and the local adversarial loss.

[0137] Optionally, the update module 704 further includes: a second downsampling module configured to downsample each of the sample local images at at least one sampling rate to obtain a second downsampled image of at least one scale of the sample local image; a fourth determination submodule configured to determine whether each of the second downsampled images is true or false to obtain a fourth determination result; and the first update submodule configured to update the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all of the predicted image blocks, the first determination result, all of the second determination results, and all of the fourth determination results.

[0138] Optionally, the first update submodule includes: a ninth calculation submodule configured to calculate a global cycle consistency loss based on the sample simulation image and the predicted simulation image; a tenth calculation submodule configured to calculate a global adversarial loss based on the first determination result; an eleventh calculation submodule configured to calculate a local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; a twelfth calculation submodule configured to calculate a local adversarial loss based on all the second determination results and all the fourth determination results; and a fourth update submodule configured to update the model parameters of the target model based on the global cycle consistency loss, the global adversarial loss, the local cycle consistency loss, and the local adversarial loss.

[0139] Optionally, there may be overlapping areas among the plurality of sample image blocks.

[0140] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0141] In addition, it should be noted that the image generation model training device 700 can be integrated into the image processing device 600 or can be independent of the image processing device 600. This disclosure does not make any specific limitations.

[0142] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the image processing method provided in this disclosure.

[0143] Figure 8 This is a block diagram illustrating an electronic device 800 according to an exemplary embodiment. For example, device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0144] Reference Figure 8 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0145] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the image processing method described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0146] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0147] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 800.

[0148] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0149] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0150] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0151] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0152] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0153] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method described above.

[0154] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to complete the image processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0155] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the image processing method described above when executed by the programmable device.

[0156] Figure 9 This is a block diagram illustrating an electronic device 900 according to an exemplary embodiment. For example, device 900 may be provided as a server. (Refer to...) Figure 9 The apparatus 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by memory 932 for storing instructions, such as application programs, that can be executed by the processing component 922. The application programs stored in memory 932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 922 is configured to execute instructions to perform the aforementioned image processing method.

[0157] Device 900 may also include a power supply component 926 configured to perform power management of device 900, a wired or wireless network interface 950 configured to connect device 900 to a network, and an input / output (I / O) interface 958. Device 900 can operate on an operating system, such as Windows Server, stored in memory 932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0158] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0159] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the target simulation image; The target simulation image is divided into blocks to obtain multiple target image blocks. There are overlapping areas between the multiple target image blocks. Different target image blocks correspond to different areas of the acquisition environment of the same simulation image. The target image blocks are used to enable the image generation model to generate an image of the real scene that retains the original details and textures and approximates the area corresponding to the target image block. Based on the target simulation image and the plurality of target image blocks, a target realistic image corresponding to the target simulation image is generated by an image generation model, wherein the image generation model is a pre-trained model used to convert simulation images into realistic images.

2. The method according to claim 1, characterized in that, The image generation model is trained in the following way: Obtain sample simulation images; The simulated sample image is divided into blocks to obtain multiple sample image blocks; Based on the sample simulation image and the plurality of sample image blocks, a sample realization image corresponding to the sample simulation image is generated through the target model; The model parameters of the target model are updated based on the simulated sample image and the real sample image. In response to the failure to meet the training cutoff condition, the step of obtaining the sample simulation image is re-executed up to the step of updating the model parameters of the target model based on the sample simulation image and the sample real image; In response to the satisfaction of the training cutoff condition, an image generation model is obtained.

3. The method according to claim 2, characterized in that, The step of generating a realistic sample image corresponding to the sample simulation image through a target model based on the sample simulation image and the plurality of sample image blocks includes: Generate a global image of the sample based on the simulated sample image; For each of the plurality of sample image blocks, a sample local image corresponding to the sample image block is generated based on the sample image block; Based on the global image of the sample and all the local images of the sample, a realistic image of the sample corresponding to the simulated image of the sample is generated.

4. The method according to claim 3, characterized in that, The step of updating the model parameters of the target model based on the sample simulation image and the sample realization image includes: Based on the real images of the samples, generate predicted simulation images; The realism of the sample image is determined to be either real or fake, and a first determination result is obtained. For each of the sample local images, a predicted image patch is generated based on the sample local image; For each of the sample local images, the sample local image is judged to be true or false, and a second judgment result is obtained; The target model is updated based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, and all the second determination results.

5. The method according to claim 4, characterized in that, The step of updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, and all the second determination results includes: Calculate the global cycle consistency loss based on the sample simulation image and the predicted simulation image; Calculate the global adversarial loss based on the first determination result; Calculate the local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; Calculate the local adversarial loss based on all the second determination results; The target model parameters are updated based on the global cyclic consistency loss, the global adversarial loss, the local cyclic consistency loss, and the local adversarial loss.

6. The method according to any one of claims 3-5, characterized in that, The step of generating a sample realization image corresponding to the sample simulation image based on the sample global image and all the sample local images includes: All the local images of the samples are stitched together to obtain the first stitched image; The first stitched image is stitched together with the sample global image according to channels to obtain the second stitched image; Based on the second stitched image, a sample realization image corresponding to the sample simulation image is generated.

7. The method according to claim 4 or 5, characterized in that, The step of updating the model parameters of the target model based on the sample simulation image and the sample realization image further includes: The sampled image is downsampled at at least one sampling rate to obtain a first downsampled image at at least one scale; For each of the first downsampled images, a true or false determination is made to obtain a third determination result; The step of updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, and all the second determination results includes: The target model is updated based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, all the second determination results, and all the third determination results.

8. The method according to claim 7, characterized in that, The step of updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, all the second determination results, and all the third determination results includes: Calculate the global cycle consistency loss based on the sample simulation image and the predicted simulation image; Calculate the global adversarial loss based on the first determination result and all the third determination results; Calculate the local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; Calculate the local adversarial loss based on all the second determination results; The target model parameters are updated based on the global cyclic consistency loss, the global adversarial loss, the local cyclic consistency loss, and the local adversarial loss.

9. The method according to claim 4 or 5, characterized in that, The step of updating the model parameters of the target model based on the sample simulation image and the sample realization image further includes: For each of the sample local images, the sample local image is downsampled at at least one sampling rate to obtain a second downsampled image of the sample local image at at least one scale; For each of the second downsampled images, a true or false determination is made to obtain a fourth determination result; The step of updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, and all the second determination results includes: The target model is updated based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, all the second determination results, and all the fourth determination results.

10. The method according to claim 9, characterized in that, The step of updating the model parameters of the target model based on the sample simulation image, the predicted simulation image, the plurality of sample image blocks, all the predicted image blocks, the first determination result, all the second determination results, and all the fourth determination results includes: Calculate the global cycle consistency loss based on the sample simulation image and the predicted simulation image; Calculate the global adversarial loss based on the first determination result; Calculate the local cycle consistency loss based on the plurality of sample image patches and all the predicted image patches; Based on all the second determination results and all the fourth determination results, calculate the local adversarial loss; The target model parameters are updated based on the global cyclic consistency loss, the global adversarial loss, the local cyclic consistency loss, and the local adversarial loss.

11. The method according to any one of claims 2-5, characterized in that, There are overlapping areas among the multiple sample image blocks.

12. An image processing apparatus, characterized in that, include: The first acquisition module is configured to acquire the target simulation image; The first segmentation module is configured to segment the target simulation image obtained by the first acquisition module into multiple target image blocks, wherein there are overlapping areas between the multiple target image blocks, and different target image blocks correspond to different areas of the acquisition environment of the same simulation image. The target image blocks are used to enable the image generation model to generate an image that retains the original details and textures and approximates the real scene of the area corresponding to the target image block. The first generation module is configured to generate a target realistic image corresponding to the target simulation image based on the target simulation image obtained by the first acquisition module and the plurality of target image blocks obtained by the first segmentation module, through an image generation model, wherein the image generation model is a pre-trained model used to convert simulation images into realistic images.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 11.

14. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1 to 11.

Images