Model training method, terminal device, and computer-readable storage medium

Abstract

The application is suitable for the technical field of image processing, and provides a model training method, a terminal device and a computer readable storage medium, comprising: acquiring at least one group of sample images; generating a synthetic image corresponding to the sample image through a generator; calculating a first difference value between the sample image and the synthetic image through a discriminator; calculating a second difference value between a mask image of the sample image and a mask image of the synthetic image; wherein the mask image is used to distinguish a shadow area and a non-shadow area in the image; training the image optimization model according to the first difference value and the second difference value to obtain a trained image optimization model; wherein the generator in the trained image optimization model is used for light compensation processing on an input image. The image optimization model trained by the above method can improve the robustness of image light compensation processing, thereby facilitating the improvement of the picture quality of a display device.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a model training method, a terminal device, and a computer-readable storage medium. Background Technology

[0002] With the popularization of ultra-high-definition display technology and the continuous expansion of smart TVs and home audio-visual scenarios, users' requirements for display device picture quality have upgraded from focusing on static parameters such as brightness, contrast, and color gamut to pursuing a picture quality experience that adapts to all environments. In some application scenarios, there may be obvious bright and dark areas in the displayed image. For example, in the scenario of live sports broadcasts, due to factors such as stadium lighting layout, backlighting, venue light and shadow obstruction, mixed night / day light sources, and shadow projection from large stands, the displayed image is prone to having both overly bright and overly dark areas. In this case, it is necessary to supplement the lighting in the dark areas of the displayed image to make the adjusted image uniform in brightness. However, the picture quality compensation strategies in related technologies have poor robustness and cannot adaptively supplement the lighting according to the content of the image, which easily leads to over-compensation or under-compensation, making it difficult to meet the requirements of high-quality, all-scenario, and intelligent picture quality adaptation. Summary of the Invention

[0003] This application provides a model training method, a terminal device, and a computer-readable storage medium, which can improve the robustness of image illumination processing, thereby improving the image quality of display devices.

[0004] In a first aspect, embodiments of this application provide a model training method applied to an image optimization model, the image optimization model including a generator and a discriminator; the model training method includes: Acquire at least one set of sample images; wherein each set of sample images includes a real shadow image and a real fill light image corresponding to the real shadow image; The generator generates a composite image corresponding to the sample image; wherein each composite image includes a composite shadow image and a composite fill light image; The discriminator calculates a first difference value; wherein the first difference value is the difference in the overall pixel distribution between the sample image and the synthesized image; Calculate a second difference value; wherein the second difference value is the difference in the mask region between the mask image of the sample image and the mask image of the synthesized image; the mask image is used to distinguish between shadowed and non-shadowed regions in the image; The image optimization model is trained based on the first difference value and the second difference value to obtain the trained image optimization model; wherein, the generator in the trained image optimization model is used to perform supplementary lighting processing on the input image.

[0005] In this embodiment, an image optimization model is trained based on a first difference value and a second difference value. The first difference value is the difference between the sample image and the synthesized image, and the second difference value is the difference between the mask image of the sample image and the mask image of the synthesized image. It can be understood that the first difference value is the difference in the overall pixel distribution, measuring whether the entire image looks similar, but without distinguishing between bright and dark areas. The second difference value is the difference in the mask area, measuring whether the bright and dark areas look similar, thus focusing the optimization on the objects the model truly needs to focus on (i.e., bright and dark areas). Therefore, by introducing the second difference value, this application achieves differentiated supervised training for shadow / non-shadow areas. This allows the model to more specifically focus on the light and shadow boundaries and detailed features of bright and dark areas in the image during training, resulting in a more natural transition of light and shadow in the image after supplemental lighting, which is beneficial for improving image quality.

[0006] Furthermore, the first difference value implements global image-level adversarial constraints, ensuring overall visual consistency between the synthesized image and the real sample; the second difference value implements refined constraints on local lighting and shadow regions. Joint supervision training using the first and second difference values ​​enables the model to learn both global and local lighting and shadow features simultaneously, significantly improving the model's adaptability and robustness in complex scenarios such as mixed light sources, backlighting, and partial occlusion, avoiding the problem of insufficient learning of local lighting and shadow features under a single global constraint.

[0007] In one possible implementation of the first aspect, each set of sample images includes a real shadow image and a real fill light image corresponding to the real shadow image; each set of synthesized images includes a synthesized shadow image and a synthesized fill light image; The calculation of the first difference value by the discriminator includes: The discriminator calculates a first loss value between the real shadow image and the synthetic shadow image; The discriminator calculates a second loss value between the real supplementary lighting image and the synthetic supplementary lighting image; wherein the first difference value includes the first loss value and the second loss value.

[0008] The above approach considers the differences between real and synthetic shadow images, as well as the differences between real and synthetic lighting images. This allows the discriminator to supervise the generator in both domains simultaneously, ensuring that the generator both "understands the distribution characteristics of the shadow image" and learns to "generate outputs that conform to the distribution of the real lighting image." This prevents the model from being biased towards only one side, leading to distortion in content or lighting. The bidirectional adversarial constraint makes the model's cycle consistency more reliable, which is beneficial for improving model training performance.

[0009] In one possible implementation of the first aspect, each set of synthesized images further includes a reconstructed shadow image and a reconstructed fill light image; wherein the reconstructed shadow image is an image generated by the generator based on the synthesized fill light image, and the reconstructed fill light image is an image generated by the generator based on the synthesized shadow image; The calculation of the first difference value by the discriminator includes: The discriminator calculates a third loss value between the reconstructed supplemental lighting image and the real supplemental lighting image. The discriminator calculates a fourth loss value between the reconstructed shadow image and the real shadow image; wherein the first difference value includes the third loss value and the fourth loss value.

[0010] The above approach considers the differences between the real shadow image and the reconstructed shadow image, as well as the differences between the real supplementary lighting image and the reconstructed supplementary lighting image. This is equivalent to supervising the reverse reconstruction process, making the reconstructed image as similar to the real image as possible. This, to a certain extent, constrains the range of content generated by the generator, reduces the possibility of the generator arbitrarily modifying the image content, ensures the consistency of the image content before and after supplementary lighting, and helps to ensure the stability of model training.

[0011] In one possible implementation of the first aspect, calculating the second difference value includes: The non-shadow area is segmented from the real supplementary lighting image to obtain the first mask image; The non-shadow areas are segmented from the synthesized supplementary lighting image to obtain a second mask image; The non-shadow region is segmented from the reconstructed supplementary lighting image to obtain the third mask image; A fifth loss value is calculated based on the difference between the first mask image and the second mask image; A sixth loss value is calculated based on the difference between the first mask image and the third mask image; wherein the second difference value includes the fifth loss value and the sixth loss value.

[0012] In the above implementation, the mask differences in non-shadow areas are introduced, enabling differentiated supervised training of these areas. Furthermore, the synthesized supplementary lighting image is the direct output of the generator. By comparing its non-shadow areas with those of the real supplementary lighting image, the generator can be directly constrained to reduce overexposure and detail blurring in non-shadow areas. The reconstructed supplementary lighting image is the result of the "original image → supplementary lighting → restoration" process. Comparing its non-shadow areas with those of the real supplementary lighting image provides "double insurance" for the non-shadow areas during the bidirectional conversion process. This simultaneously constrains the handling of non-shadow areas during both forward supplementary lighting and reverse restoration, reducing the possibility of the model incorrectly modifying the content of non-shadow areas to meet cyclic loss requirements. Therefore, by comprehensively considering the fifth and sixth loss values, dual supervision of the non-shadow areas is achieved, forming a complementary constraint system. This ensures both the local controllability of the supplementary lighting operation and the overall consistency of the bidirectional conversion, thus improving the model training effect.

[0013] In one possible implementation of the first aspect, calculating the second difference value includes: The shadow region is segmented from the real shadow image to obtain the fourth mask image; The shadow region is segmented from the synthesized shadow image to obtain the fifth mask image; The shadow region is segmented from the reconstructed shadow image to obtain the sixth mask image; The seventh loss value is calculated based on the difference between the fourth mask image and the fifth mask image; An eighth loss value is calculated based on the difference between the fourth mask image and the sixth mask image; wherein the second difference value includes the seventh loss value and the eighth loss value.

[0014] In the above implementation, the mask differences in the shadow regions are introduced, achieving differentiated supervised training of the shadow regions. Furthermore, the synthesized shadow image is the direct output of the generator. By comparing its shadow region differences with the real shadow image, the generator can be directly constrained to reduce problems such as overexposure and blurred details in the shadow regions. The reconstructed shadow image is the result of "original image → restoration → reconstruction." Comparing its shadow region differences with the real shadow image is equivalent to adding "double insurance" to the shadow regions during the bidirectional conversion process. This simultaneously constrains the handling of shadow regions during forward restoration and backward reconstruction, reducing the possibility of the model incorrectly modifying the content of shadow regions to meet cyclic loss requirements. Therefore, by comprehensively considering the seventh and eighth loss values, dual supervision of the shadow regions is achieved, forming a complementary constraint system. This ensures both the local controllability of the lighting operation and the overall consistency of the bidirectional conversion, which is beneficial for improving the model training effect.

[0015] In one possible implementation of the first aspect, training the image optimization model based on the first difference value and the second difference value to obtain the trained image optimization model includes: The first difference value and the second difference value are weighted and summed to obtain the total loss value; The model parameters of the image optimization model are adjusted based on the total loss to obtain the adjusted image optimization model; If the preset conditions are met, the adjusted image optimization model is recorded as the trained image optimization model. If the preset conditions are not met, the next model training will continue until the trained image optimization model is obtained.

[0016] In the above approach, the first difference value achieves adversarial constraints at the global image level, ensuring overall visual consistency between the synthesized image and the real sample; the second difference value achieves refined constraints on local lighting and shadow regions. Joint supervised training using both the first and second difference values ​​enables the model to learn both global and local lighting and shadow features simultaneously, significantly improving its adaptability and robustness in complex scenarios such as mixed light sources, backlighting, and partial occlusion, avoiding the problem of insufficient learning of local lighting and shadow features under a single global constraint.

[0017] In one possible implementation of the first aspect, after obtaining the trained image optimization model, the method further includes: Obtain the image to be processed; The generator of the trained image optimization model generates the target illumination image corresponding to the image to be processed.

[0018] In one possible implementation of the first aspect, generating the target illumination image corresponding to the image to be processed by the generator of the trained image optimization model includes: Acquire ambient light data; The image to be processed and the ambient light data are input into the generator of the trained image optimization model, and the target supplementary lighting image is output.

[0019] In the above method, ambient light data is used as input, enabling the generator to accurately perceive the real lighting conditions of the current scene. This allows it to output different intensities and styles of supplementary lighting effects for different environments (such as night stadiums, backlit stands, and mixed light sources), achieving "scene-adaptive intelligent supplementary lighting." This avoids the failure of fixed supplementary lighting algorithms in complex lighting scenarios and helps improve the supplementary lighting effect.

[0020] Secondly, embodiments of this application provide a model training apparatus, including: Acquisition unit, used to acquire at least one set of sample images; A generation unit is used to generate a synthetic image corresponding to the sample image through the generator; A first calculation unit is configured to calculate a first difference value using the discriminator; wherein the first difference value is the difference between the sample image and the synthesized image; The second calculation unit is used to calculate a second difference value; wherein the second difference value is the difference between the mask image of the sample image and the mask image of the synthesized image; the mask image is used to distinguish between shadowed and non-shadowed areas in the image; The training unit is used to train the image optimization model based on the first difference value and the second difference value to obtain the trained image optimization model; wherein, the generator in the trained image optimization model is used to perform supplementary lighting processing on the input image.

[0021] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method as described in any one of the first aspects above.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model training method as described in any one of the first aspects above.

[0023] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the model training method described in any one of the first aspects.

[0024] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the light and dark image provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of an image optimization model provided in an embodiment of this application; Figure 3 This is a schematic flowchart of the model training method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the processing flow of the image optimization model provided in the embodiments of this application; Figure 5 This is a schematic diagram of another processing flow of the image optimization model provided in the embodiments of this application; Figure 6 This is a structural block diagram of the model training device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0033] With the popularization of ultra-high-definition display technology and the continuous expansion of smart TVs and home entertainment scenarios, users' requirements for display device picture quality have upgraded from focusing on static parameters such as brightness, contrast, and color gamut to pursuing a picture quality experience that adapts to all environments. In some application scenarios, there may be obvious bright and dark areas in the displayed image. For example, in the scenario of live sports broadcasts, due to factors such as stadium lighting layout, backlighting, venue light and shadow obstruction, mixed night / day light sources, and shadow projection from large stands, the displayed image is very likely to have both overly bright and overly dark areas.

[0034] For example, see Figure 1 This is a schematic diagram of the light and dark image provided in the embodiments of this application. For example... Figure 1 As shown, this is a photograph of the stadium, which contains a distinct bright area 11 and a distinct dark area 12.

[0035] In the above situations, it is necessary to supplement the lighting in the dark areas of the displayed image to make the brightness and darkness of the adjusted image consistent. However, the image quality compensation strategies in related technologies have poor robustness and cannot adaptively supplement the lighting according to the content of the image. They are prone to over-compensation or under-compensation, making it difficult to meet the requirements of high-quality, all-scene, and intelligent image quality adaptation.

[0036] Based on this, this application provides a model training method. In this application embodiment, an image optimization model is trained based on a first difference value and a second difference value, wherein the first difference value is the difference between the sample image and the synthesized image, and the second difference value is the difference between the mask image of the sample image and the mask image of the synthesized image. It can be understood that the first difference value is the difference in the overall pixel distribution, which measures whether the whole image looks like it, but does not distinguish between bright and dark areas in the image; the second difference value is the difference in the mask area, which measures whether the bright and dark areas look like it, thus focusing the optimization on the objects that the model truly needs to focus on (i.e., bright and dark areas). Therefore, by introducing the second difference value, this application achieves differentiated supervised training for shadow / non-shadow areas, which allows the model to more specifically focus on the light and shadow boundaries and detailed features of bright and dark areas in the image during training, thereby making the light and shadow transitions of the image after supplementary lighting natural and improving image quality.

[0037] Furthermore, the first difference value implements global image-level adversarial constraints, ensuring overall visual consistency between the synthesized image and the real sample; the second difference value implements refined constraints on local lighting and shadow regions. Joint supervision training using the first and second difference values ​​enables the model to learn both global and local lighting and shadow features simultaneously, significantly improving the model's adaptability and robustness in complex scenarios such as mixed light sources, backlighting, and partial occlusion, avoiding the problem of insufficient learning of local lighting and shadow features under a single global constraint.

[0038] For ease of explanation, the image optimization model involved in the embodiments of this application will be introduced first. See [link to relevant documentation]. Figure 2 This is a schematic diagram of the structure of an image optimization model provided in an embodiment of this application. Figure 2 As shown, the image optimization model includes a first generator 21, a second generator 22, a first discriminator 23, and a second discriminator 24.

[0039] Specifically, the first generator 21 is used to convert an image in domain X into an image in domain Y; the second generator 22 is used to convert an image in domain Y into an image in domain X. The first discriminator 23 is used to determine the difference between the input image and the real image in domain X; the second discriminator 24 is used to determine the difference between the input image and the real image in domain Y.

[0040] Understandably, during the training of the image optimization model, the first generator 21, the second generator 22, the first discriminator 23, and the second discriminator 24 need to be trained together. The training objective is to enable the generator to produce sufficiently realistic images so that the discriminator can accurately detect fake images. During training, the generator and discriminator evolve against each other; that is, the images generated by the generator become increasingly realistic, making it impossible for the discriminator to distinguish between real and fake images, thus driving the discriminator to continuously upgrade; while the discriminator becomes increasingly "picky," thus driving the generator to continuously upgrade.

[0041] In the application of image optimization models, only a generator is needed. For example, in the application of image illumination processing, if the image to be processed belongs to neighborhood X and the target illuminated image after illumination processing belongs to neighborhood Y, then only the first generator 21 is used, that is, the image to be processed is input into the trained first generator 21 to obtain the target illuminated image. If the image to be processed belongs to neighborhood Y and the target illuminated image after illumination processing belongs to neighborhood X, then only the second generator 22 is used, that is, the image to be processed is input into the trained second generator 22 to obtain the target illuminated image.

[0042] based on Figure 2 The image optimization model shown below illustrates the model training method of this application embodiment.

[0043] See Figure 3 This is a flowchart illustrating the model training method provided in the embodiments of this application. It is intended as an example and not a limitation. The method may include steps S301-S305, and each step will be described in detail below.

[0044] S301, acquire at least one set of sample images.

[0045] Each set of sample images includes a real shadow image and a real fill light image corresponding to the real shadow image.

[0046] S302, the generator generates a synthetic image corresponding to the sample image.

[0047] In one implementation, each composite image includes a composite shadow image and a composite fill light image. Accordingly, S302 includes: generating a composite fill light image corresponding to the real shadow image using a generator; and generating a composite shadow image corresponding to the real fill light image using a generator.

[0048] For example, with Figure 2 For example, see the image optimization model shown below. Figure 4 This is a schematic diagram of the processing flow of the image optimization model provided in the embodiments of this application. Figure 4As shown, the real shadow image from the sample image is input into the first generator 21, and a synthesized supplementary lighting image is output; the real supplementary lighting image from the sample image is input into the second generator 22, and a synthesized shadow image is output. In other words, in Figure 4 In the example, the image without supplemental lighting belongs to domain X, and the image with supplemental lighting belongs to domain Y.

[0049] In another implementation, each composite image includes a reconstructed shadow image and a reconstructed illumination image. Accordingly, S302 includes: generating a reconstructed shadow image corresponding to the composite illumination image using a generator; and generating a reconstructed illumination image corresponding to the composite shadow image using a generator.

[0050] For example, with Figure 2 For example, see the image optimization model shown below. Figure 5 This is a schematic diagram of another processing flow of the image optimization model provided in the embodiments of this application. For example... Figure 5 As shown, the composite lighting image generated by the first generator 21 is input into the second generator 22, and the reconstructed shadow image is output; the composite shadow image generated by the second generator 22 is input into the first generator 21, and the reconstructed lighting image is output.

[0051] In another implementation, each composite image includes a composite shadow image, a composite fill light image, a reconstructed shadow image, and a reconstructed fill light image.

[0052] S303, calculate the first difference value using the discriminator.

[0053] Wherein, the first difference value is the difference in the overall pixel distribution between the sample image and the synthesized image.

[0054] In one implementation, if each composite image includes a composite shadow image and a composite fill light image, then S303 includes: calculating a first loss value between the real shadow image and the composite shadow image using a discriminator; calculating a second loss value between the real fill light image and the composite fill light image using a discriminator; wherein the first difference value includes the first loss value and the second loss value.

[0055] For example, such as Figure 4 As shown, the real shadow image and the synthesized shadow image are input into the first discriminator 23, and a first loss value is output. The real supplementary lighting image and the synthesized supplementary lighting image are input into the second discriminator 24, and a second loss value is output. It can be seen that in... Figure 4In the example, the real shadow image without supplemental lighting belongs to domain X, and the composite supplemental lighting image after supplemental lighting belongs to domain Y; the real supplemental lighting image belongs to domain Y, and the composite shadow image belongs to domain X; the first discriminator 23 is used to determine the difference between the input image and the real image in domain X; the second discriminator 24 is used to determine the difference between the input image and the real image in domain Y.

[0056] The above approach considers the differences between real and synthetic shadow images, as well as the differences between real and synthetic lighting images. This allows the discriminator to supervise the generator in both domains simultaneously, ensuring that the generator both "understands the distribution characteristics of the shadow image" and learns to "generate outputs that conform to the distribution of the real lighting image." This prevents the model from being biased towards only one side, leading to distortion in content or lighting. The bidirectional adversarial constraint makes the model's cycle consistency more reliable, which is beneficial for improving model training performance.

[0057] In another implementation, if each assembled image includes a reconstructed shadow image and a reconstructed supplementary lighting image, then S303 includes: calculating a third loss value between the reconstructed supplementary lighting image and the real supplementary lighting image through a discriminator; calculating a fourth loss value between the reconstructed shadow image and the real shadow image through a discriminator; wherein the first difference value includes the third loss value and the fourth loss value.

[0058] For example, such as Figure 5 As shown, the real shadow image and the reconstructed shadow image are input into the first discriminator 23, and a fourth loss value is output. The real supplementary lighting image and the reconstructed supplementary lighting image are input into the second discriminator 24, and a third loss value is output. It can be seen that in... Figure 5 In the example, the reconstructed shadow image belongs to domain X, and the reconstructed fill light image belongs to domain Y; the first discriminator 23 is used to determine the difference between the input image and the real image in domain X; the second discriminator 24 is used to determine the difference between the input image and the real image in domain Y.

[0059] The above approach considers the differences between the real shadow image and the reconstructed shadow image, as well as the differences between the real supplementary lighting image and the reconstructed supplementary lighting image. This is equivalent to supervising the reverse reconstruction process, making the reconstructed image as similar to the real image as possible. This, to a certain extent, constrains the range of content generated by the generator, reduces the possibility of the generator arbitrarily modifying the image content, ensures the consistency of the image content before and after supplementary lighting, and helps to ensure the stability of model training.

[0060] In another implementation, if each composite image includes a composite shadow image, a composite supplementary lighting image, a reconstructed shadow image, and a reconstructed supplementary lighting image, then S303 includes: calculating a first loss value between the real shadow image and the composite shadow image using a discriminator; calculating a second loss value between the real supplementary lighting image and the composite supplementary lighting image using a discriminator; calculating a third loss value between the reconstructed supplementary lighting image and the real supplementary lighting image using a discriminator; and calculating a fourth loss value between the reconstructed shadow image and the real shadow image using a discriminator; wherein the first difference value includes the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0061] The above approach is equivalent to bidirectional supervision in both the forward generation chain (from real image to synthetic image) and the reverse reconstruction chain (from synthetic image to reconstructed image), which makes the model's bidirectional conversion capability stronger, reduces the situation of "good forward lighting effect but failure in reverse reconstruction", ensures the consistency of the model's cycle, and helps to improve the model training effect.

[0062] S304, Calculate the second difference value.

[0063] The second difference value is the difference in the masked regions between the masked image of the sample image and the masked image of the synthesized image. The masked image is used to distinguish between shadowed and non-shadowed regions in the image.

[0064] In some implementations, the second difference value includes a fifth loss value and a sixth loss value. Accordingly, S304 includes: The non-shadow region is segmented from the real illuminated image to obtain the first mask image; The non-shadow areas are segmented from the synthesized supplementary lighting image to obtain the second mask image; The non-shadow region is segmented from the reconstructed illuminated image to obtain the third mask image; The fifth loss value is calculated based on the difference between the first mask image and the second mask image; The sixth loss value is calculated based on the difference between the first mask image and the third mask image.

[0065] In the above implementation, the mask differences in non-shadow areas are introduced, enabling differentiated supervised training of these areas. Furthermore, the synthesized supplementary lighting image is the direct output of the generator. By comparing its non-shadow areas with those of the real supplementary lighting image, the generator can be directly constrained to reduce overexposure and detail blurring in non-shadow areas. The reconstructed supplementary lighting image is the result of the "original image → supplementary lighting → restoration" process. Comparing its non-shadow areas with those of the real supplementary lighting image provides "double insurance" for the non-shadow areas during the bidirectional conversion process. This simultaneously constrains the handling of non-shadow areas during both forward supplementary lighting and reverse restoration, reducing the possibility of the model incorrectly modifying the content of non-shadow areas to meet cyclic loss requirements. Therefore, by comprehensively considering the fifth and sixth loss values, dual supervision of the non-shadow areas is achieved, forming a complementary constraint system. This ensures both the local controllability of the supplementary lighting operation and the overall consistency of the bidirectional conversion, thus improving the model training effect.

[0066] In other implementations, the second difference value includes a seventh loss value and an eighth loss value. Accordingly, S304 includes: The shadow region is segmented from the real shadow image to obtain the fourth mask image; The shadow region is segmented from the synthesized shadow image to obtain the fifth mask image; The shadow region is segmented from the reconstructed shadow image to obtain the sixth mask image; The seventh loss value is calculated based on the difference between the fourth and fifth mask images; The eighth loss value is calculated based on the difference between the fourth and sixth mask images.

[0067] In the above implementation, the mask differences in the shadow regions are introduced, achieving differentiated supervised training of the shadow regions. Furthermore, the synthesized shadow image is the direct output of the generator. By comparing its shadow region differences with the real shadow image, the generator can be directly constrained to reduce problems such as overexposure and blurred details in the shadow regions. The reconstructed shadow image is the result of "original image → restoration → reconstruction." Comparing its shadow region differences with the real shadow image is equivalent to adding "double insurance" to the shadow regions during the bidirectional conversion process. This simultaneously constrains the handling of shadow regions during forward restoration and backward reconstruction, reducing the possibility of the model incorrectly modifying the content of shadow regions to meet cyclic loss requirements. Therefore, by comprehensively considering the seventh and eighth loss values, dual supervision of the shadow regions is achieved, forming a complementary constraint system. This ensures both the local controllability of the lighting operation and the overall consistency of the bidirectional conversion, which is beneficial for improving the model training effect.

[0068] In some implementations, the second difference value includes the fifth, sixth, seventh, and eighth loss values. Correspondingly, S304 includes: The non-shadow region is segmented from the real supplementary lighting image to obtain the first mask image; the non-shadow region is segmented from the synthetic supplementary lighting image to obtain the second mask image; the non-shadow region is segmented from the reconstructed supplementary lighting image to obtain the third mask image; a fifth loss value is calculated based on the difference between the first mask image and the second mask image; a sixth loss value is calculated based on the difference between the first mask image and the third mask image. The shadow region is segmented from the real shadow image to obtain the fourth mask image; the shadow region is segmented from the synthetic shadow image to obtain the fifth mask image; the shadow region is segmented from the reconstructed shadow image to obtain the sixth mask image; the seventh loss value is calculated based on the difference between the fourth mask image and the fifth mask image; the eighth loss value is calculated based on the difference between the fourth mask image and the sixth mask image.

[0069] The above implementation introduces mask differences for both shadow and non-shadow regions, achieving dual-supervised training that differentiates between shadow and non-shadow regions. Furthermore, calculating the loss separately for shadow and non-shadow regions decouples the two objectives, allowing for targeted training of the model's ability to handle shadow and non-shadow regions respectively, thus improving training effectiveness.

[0070] In one embodiment, a trained segmentation model can be used to obtain the mask image. Specifically, the sample image is input into the segmentation model to obtain the mask image of the sample image; the synthesized image is input into the segmentation model to obtain the mask image of the synthesized image. Optionally, the segmentation model can be a DenseNet model. For example, a DenseNet model can be used to obtain the first to sixth mask images mentioned above.

[0071] S305, Train the image optimization model based on the first difference value and the second difference value to obtain the trained image optimization model.

[0072] In some implementations, S305 includes: The total loss value is obtained by weighted summation of the first and second difference values. The image optimization model is trained based on the total loss to obtain the trained image optimization model.

[0073] Optionally, one way to train an image optimization model based on the total loss includes: The model parameters of the image optimization model are adjusted based on the total loss to obtain the adjusted image optimization model; If the preset conditions are met, the adjusted image optimization model will be recorded as the trained image optimization model. If the preset conditions are not met, the model training will continue until the trained image optimization model is obtained.

[0074] The preset condition can be that the number of iterations has reached a preset number. For example, after obtaining the adjusted image optimization model, it is determined whether the current number of iterations has reached the preset number; if it has, the adjusted image optimization model is recorded as the trained image optimization model; if it has not, the next model training is performed until the trained image optimization model is obtained.

[0075] Alternatively, another way to train the image optimization model based on the total loss includes: The total loss value is obtained by weighted summation of the first and second difference values. If the preset conditions are met, the adjusted image optimization model will be recorded as the trained image optimization model. If the preset conditions are not met, the model parameters of the image optimization model are adjusted according to the total loss to obtain the adjusted image optimization model; and the next model training continues until the trained image optimization model is obtained.

[0076] The preset condition can be that the total loss value is less than a preset threshold. For example, after calculating the total loss value, if the current total loss value is less than the preset threshold, i.e., the preset condition is met, then the adjusted image optimization model is recorded as the trained image optimization model; if the current total loss value is not less than the preset threshold, i.e., the preset condition is not met, then the model parameters of the image optimization model are adjusted according to the total loss to obtain the adjusted image optimization model; and the next model training continues until the trained image optimization model is obtained.

[0077] An exemplary complete process of the model training method of this application is presented. The model training method includes the following steps: S601, acquire at least one set of sample images.

[0078] S602, the synthesized image corresponding to the sample image is generated by the generator in the image optimization model. Specifically: the first generator generates a synthesized supplementary lighting image corresponding to the real shadow image; the second generator generates a synthesized shadow image corresponding to the real supplementary lighting image; the second generator generates a reconstructed shadow image corresponding to the synthesized supplementary lighting image; and the first generator generates a reconstructed supplementary lighting image corresponding to the synthesized shadow image.

[0079] S603, the discriminator in the image optimization model calculates the first difference value between the sample image and the synthesized image. Specifically: the first discriminator calculates the first loss value between the real shadow image and the synthesized shadow image; the second discriminator calculates the second loss value between the real supplementary lighting image and the synthesized supplementary lighting image; the second discriminator calculates the third loss value between the reconstructed supplementary lighting image and the real supplementary lighting image; and the first discriminator calculates the fourth loss value between the reconstructed shadow image and the real shadow image.

[0080] S604 calculates the second difference value between the mask image of the sample image and the mask image of the synthesized image. Specifically: The non-shadow region is segmented from the real supplementary lighting image to obtain the first mask image; the non-shadow region is segmented from the synthetic supplementary lighting image to obtain the second mask image; the non-shadow region is segmented from the reconstructed supplementary lighting image to obtain the third mask image; a fifth loss value is calculated based on the difference between the first mask image and the second mask image; a sixth loss value is calculated based on the difference between the first mask image and the third mask image. The shadow region is segmented from the real shadow image to obtain the fourth mask image; the shadow region is segmented from the synthetic shadow image to obtain the fifth mask image; the shadow region is segmented from the reconstructed shadow image to obtain the sixth mask image; the seventh loss value is calculated based on the difference between the fourth mask image and the fifth mask image; the eighth loss value is calculated based on the difference between the fourth mask image and the sixth mask image.

[0081] S605, Train the image optimization model based on the first difference value and the second difference value to obtain the trained image optimization model. Specifically: Perform a weighted summation of the first difference value and the second difference value to obtain the total loss value; Train the image optimization model based on the total loss to obtain the trained image optimization model.

[0082] In this embodiment, an image optimization model is trained based on a first difference value and a second difference value. The first difference value is the difference between the sample image and the synthesized image, and the second difference value is the difference between the mask image of the sample image and the mask image of the synthesized image. It can be understood that the first difference value is the difference in the overall pixel distribution, measuring whether the entire image looks similar, but without distinguishing between bright and dark areas. The second difference value is the difference in the mask area, measuring whether the bright and dark areas look similar, thus focusing the optimization on the objects the model truly needs to focus on (i.e., bright and dark areas). Therefore, by introducing the second difference value, this application achieves differentiated supervised training for shadow / non-shadow areas. This allows the model to more specifically focus on the light and shadow boundaries and detailed features of bright and dark areas in the image during training, resulting in a more natural transition of light and shadow in the image after supplemental lighting, which is beneficial for improving image quality.

[0083] Furthermore, the first difference value implements global image-level adversarial constraints, ensuring overall visual consistency between the synthesized image and the real sample; the second difference value implements refined constraints on local lighting and shadow regions. Joint supervision training using the first and second difference values ​​enables the model to learn both global and local lighting and shadow features simultaneously, significantly improving the model's adaptability and robustness in complex scenarios such as mixed light sources, backlighting, and partial occlusion, avoiding the problem of insufficient learning of local lighting and shadow features under a single global constraint.

[0084] In some embodiments, after obtaining the trained image optimization model, the method further includes: acquiring an image to be processed; and generating a target illumination image corresponding to the image to be processed using a generator of the trained image optimization model.

[0085] In some implementations, the steps for acquiring the target illumination image may include: Acquire ambient light data; input the image to be processed and the ambient light data into the generator of the trained image optimization model, and output the target supplementary lighting image.

[0086] In the above method, ambient light data is used as input, enabling the generator to accurately perceive the real lighting conditions of the current scene. This allows it to output different intensities and styles of supplementary lighting effects for different environments (such as night stadiums, backlit stands, and mixed light sources), achieving "scene-adaptive intelligent supplementary lighting." This avoids the failure of fixed supplementary lighting algorithms in complex lighting scenarios and helps improve the supplementary lighting effect.

[0087] Understandably, ambient light data can also be used as input to assist in model training. For example, in S302, ambient light data is input into the generator along with sample images to obtain a synthetic image. Specifically, ambient light data is input into the first generator along with real shadow images to obtain a synthetic supplementary lighting image.

[0088] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0089] Corresponding to the model training method described in the above embodiments, Figure 6 This is a structural block diagram of the model training device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0090] Reference Figure 6 The model training device 7 includes: The acquisition unit 71 is used to acquire at least one set of sample images; wherein each set of sample images includes a real shadow image and a real fill light image corresponding to the real shadow image.

[0091] The generation unit 72 is used to generate a synthetic image corresponding to the sample image through the generator; wherein each group of synthetic images includes a synthetic shadow image and a synthetic fill light image.

[0092] The first calculation unit 73 is used to calculate a first difference value through the discriminator; wherein the first difference value is the difference in the overall pixel distribution between the sample image and the synthesized image.

[0093] The second calculation unit 74 is used to calculate a second difference value; wherein the second difference value is the difference in the mask region between the mask image of the sample image and the mask image of the synthesized image; the mask image is used to distinguish between shadow regions and non-shadow regions in the image.

[0094] Training unit 75 is used to train the image optimization model based on the first difference value and the second difference value to obtain the trained image optimization model; wherein, the generator in the trained image optimization model is used to perform supplementary lighting processing on the input image.

[0095] Optionally, the first computing unit 73 is also used for: The discriminator calculates a first loss value between the real shadow image and the synthetic shadow image; the discriminator also calculates a second loss value between the real fill light image and the synthetic fill light image; wherein the first difference value includes the first loss value and the second loss value.

[0096] Optionally, the first computing unit 73 is also used for: The discriminator calculates a third loss value between the reconstructed supplemental lighting image and the real supplemental lighting image; the discriminator also calculates a fourth loss value between the reconstructed shadow image and the real shadow image; wherein the first difference value includes the third loss value and the fourth loss value.

[0097] Optionally, the second computing unit 74 is also used for: The non-shadow area is segmented from the real supplementary lighting image to obtain the first mask image; The non-shadow areas are segmented from the synthesized supplementary lighting image to obtain a second mask image; The non-shadow region is segmented from the reconstructed supplementary lighting image to obtain the third mask image; A fifth loss value is calculated based on the difference between the first mask image and the second mask image; A sixth loss value is calculated based on the difference between the first mask image and the third mask image; wherein the second difference value includes the fifth loss value and the sixth loss value.

[0098] Optionally, the second computing unit 74 is also used for: The shadow region is segmented from the real shadow image to obtain the fourth mask image; The shadow region is segmented from the synthesized shadow image to obtain the fifth mask image; The shadow region is segmented from the reconstructed shadow image to obtain the sixth mask image; The seventh loss value is calculated based on the difference between the fourth mask image and the fifth mask image; An eighth loss value is calculated based on the difference between the fourth mask image and the sixth mask image; wherein the second difference value includes the seventh loss value and the eighth loss value.

[0099] Optionally, training unit 75 is also used for: The first difference value and the second difference value are weighted and summed to obtain the total loss value; The model parameters of the image optimization model are adjusted based on the total loss to obtain the adjusted image optimization model; If the preset conditions are met, the adjusted image optimization model is recorded as the trained image optimization model. If the preset conditions are not met, the next model training will continue until the trained image optimization model is obtained.

[0100] Optionally, the model training device 7 also includes: The processing unit 76 is used to acquire the image to be processed and generate a target supplementary lighting image corresponding to the image to be processed through the generator of the trained image optimization model.

[0101] Optionally, the processing unit 76 is further configured to: acquire ambient light data; input the image to be processed and the ambient light data into the generator of the trained image optimization model, and output the target supplementary lighting image.

[0102] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0103] in addition, Figure 6 The model training device shown can be a software unit, hardware unit, or a combination of software and hardware built into an existing terminal device, or it can be integrated into the terminal device as an independent component, or it can exist as an independent terminal device.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0105] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 7 As shown, the terminal device 8 of this embodiment includes: at least one processor 80 ( Figure 7 (Only one is shown in the diagram) a processor, a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, wherein the processor 80 executes the computer program 82 to implement the steps in any of the above-described model training method embodiments.

[0106] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 7 This is merely an example of terminal device 8 and does not constitute a limitation on terminal device 8. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0107] The processor 80 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0108] In some embodiments, the memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. In other embodiments, the memory 81 may be an external storage device of the terminal device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 8. Furthermore, the memory 81 may include both internal and external storage units of the terminal device 8. The memory 81 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0109] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0110] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, Applied to an image optimization model, the image optimization model includes a generator and a discriminator; the model training method includes: Acquire at least one set of sample images; wherein each set of sample images includes a real shadow image and a real fill light image corresponding to the real shadow image; The generator generates a composite image corresponding to the sample image; wherein each composite image includes a composite shadow image and a composite fill light image; the composite shadow image is obtained by inputting the real fill light image from the sample image into the generator; the composite fill light image is obtained by inputting the real shadow image from the sample image into the generator; The discriminator calculates a first difference value; wherein the first difference value is the difference between the sample image and the synthesized image; Calculate a second difference value; wherein the second difference value is the difference between the mask image of the sample image and the mask image of the synthesized image; the mask image is used to distinguish between shadowed and non-shadowed areas in the image; The image optimization model is trained based on the first difference value and the second difference value to obtain the trained image optimization model; wherein, the generator in the trained image optimization model is used to perform supplementary lighting processing on the input image; The step of calculating the first difference value through the discriminator includes: The discriminator calculates a first loss value between the real shadow image and the synthetic shadow image; The discriminator calculates a second loss value between the real supplementary lighting image and the synthetic supplementary lighting image; wherein the first difference value includes the first loss value and the second loss value.

2. The model training method as described in claim 1, characterized in that, Each set of synthesized images further includes a reconstructed shadow image and a reconstructed fill light image; wherein, the reconstructed shadow image is an image generated by the generator based on the synthesized fill light image, and the reconstructed fill light image is an image generated by the generator based on the synthesized shadow image; The calculation of the first difference value by the discriminator includes: The discriminator calculates a third loss value between the reconstructed supplemental lighting image and the real supplemental lighting image. The discriminator calculates a fourth loss value between the reconstructed shadow image and the real shadow image; wherein the first difference value includes the third loss value and the fourth loss value.

3. The model training method as described in claim 2, characterized in that, The calculation of the second difference value includes: The non-shadow area is segmented from the real supplementary lighting image to obtain the first mask image; The non-shadow areas are segmented from the synthesized supplementary lighting image to obtain a second mask image; The non-shadow region is segmented from the reconstructed supplementary lighting image to obtain the third mask image; A fifth loss value is calculated based on the difference between the first mask image and the second mask image; A sixth loss value is calculated based on the difference between the first mask image and the third mask image; wherein the second difference value includes the fifth loss value and the sixth loss value.

4. The model training method as described in claim 2, characterized in that, The calculation of the second difference value includes: The shadow region is segmented from the real shadow image to obtain the fourth mask image; The shadow region is segmented from the synthesized shadow image to obtain the fifth mask image; The shadow region is segmented from the reconstructed shadow image to obtain the sixth mask image; The seventh loss value is calculated based on the difference between the fourth mask image and the fifth mask image; An eighth loss value is calculated based on the difference between the fourth mask image and the sixth mask image; wherein the second difference value includes the seventh loss value and the eighth loss value.

5. The model training method as described in claim 1, characterized in that, The step of training the image optimization model based on the first difference value and the second difference value to obtain the trained image optimization model includes: The first difference value and the second difference value are weighted and summed to obtain the total loss value; The model parameters of the image optimization model are adjusted based on the total loss to obtain the adjusted image optimization model; If the preset conditions are met, the adjusted image optimization model is recorded as the trained image optimization model. If the preset conditions are not met, the next model training will continue until the trained image optimization model is obtained.

6. The model training method according to any one of claims 1 to 5, characterized in that, After obtaining the trained image optimization model, the method further includes: Obtain the image to be processed; The generator of the trained image optimization model generates the target illumination image corresponding to the image to be processed.

7. The model training method as described in claim 6, characterized in that, The step of generating the target illumination image corresponding to the image to be processed by the generator of the trained image optimization model includes: Acquire ambient light data; The image to be processed and the ambient light data are input into the generator of the trained image optimization model, and the target supplementary lighting image is output.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images