Image conversion method and image conversion device

Abstract

This invention proposes an image conversion method applicable to image conversion networks including an image generator, a conversion discriminator, and a focus discriminator. The method includes: the image generator generating a converted image based on an unconverted image and focus information of the unconverted image; the conversion discriminator generating a conversion discrimination value by operating on the converted image; the image generator calculating a first generation loss function value based on the conversion discrimination value and updating the parameters of the image generator based on the first generation loss function value; the focus discriminator performing an operation based on the unconverted image, the converted image, and the focus information to generate a focus discrimination value; and the image generator calculating a second generation loss function value based on the focus discrimination value and updating the image generator based on the second generation loss function value.

Description

Technical Field

[0001] The invention relates to an image conversion method and an image conversion network. Background Art

[0002] With the popularity and booming development of mobile devices and social platforms, users have more diverse demands for photo quality and image effects, and their requirements for image quality are relatively strict. Therefore, in order to meet market expectations, the current mobile device photo image function has been developed towards professional SLR effects.

[0003] The design of improving the image quality of mobile device photography can be roughly divided into two technical solutions: hardware and software. Hardware solutions, such as using high-end photosensitive components or incorporating multiple lenses, are prone to cost burdens due to the increase in hardware and are often limited by volume considerations. Software solutions use software image processing or learning algorithms for post-production to achieve a special image style. In the current software solution, taking the image conversion technology of converting the target object as an example, there will be cases of misjudgment of the target area, inconsistent tones before and after the conversion, and changes in the color of the target object. Although the expected special effects can be achieved, there are still concerns about distortion in color and tone. Summary of the invention

[0004] The invention provides an image conversion method and an image conversion network.

[0005] In an exemplary embodiment, an image conversion method according to the present invention is applicable to an image conversion network, and the image conversion network is connected to a first database and includes an image generator, a conversion identifier, and a focus identifier, and the first database stores multiple unconverted images. The image conversion method includes: the image generator performs an operation based on one of the multiple unconverted images and at least one focus information of one of the multiple unconverted images to generate a conversion image; the conversion identifier performs an operation on the conversion image to generate a conversion identification value; the image generator calculates a first generation loss function value based on the conversion identification value, and updates the parameters of the image generator based on the first generation loss function value; the focus identifier performs an operation based on one of the multiple unconverted images, the conversion image and the focus information to generate at least one focus identification value; and the image generator calculates a second generation loss function value based on the at least one focus identification value, and updates the parameters of the image generator based on the second generation loss function value.

[0006] In an exemplary embodiment, an image conversion network according to the present invention is connected to a first database storing multiple unconverted images, and the image conversion network includes: an image generator, configured to perform operations and generate a converted image based on one of the multiple unconverted images and at least one focus information of one of the multiple unconverted images; a conversion identifier, a signal connected to the image generator, configured to perform operations and generate a conversion identification value based on the converted image; and a focus identifier signal connected to the image generator, configured to perform operations based on one of the multiple unconverted images, the converted image and the at least one focus information to generate at least one focus identification value, wherein the image generator is configured to also perform: calculating a first generation loss function value based on the conversion identification value, and updating the parameters of the image generator based on the first generation loss function value, and calculating a second generation loss function value based on the at least one focus identification value, and updating the parameters of the image generator based on the second generation loss function value.

[0007] Based on the above, in the image conversion method and image conversion network proposed in the embodiments of the present invention, the image generator, the conversion identifier and the focus identifier use the unconverted image, focus information and the converted image to jointly perform machine learning to update the parameters of the image generator and improve the image conversion efficiency of the image generator.

[0008] In order to make the above features of the present invention more clearly understood, embodiments are given below with reference to the accompanying drawings for detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an image conversion network and a first database according to an embodiment of the present invention.

[0010] Figure 2 is a flow chart of an image conversion method according to an embodiment of the present invention.

[0011] Figure 3 FIG. 4 is a block diagram of an image conversion network, a first database and a second database according to another embodiment of the present invention.

[0012] Figure 4 is a flow chart of an image conversion method according to another embodiment of the present invention.

[0013] Figure 5 FIG. 4 is a flowchart of partial steps of an image conversion method according to another embodiment of the present invention.

[0014] Figure 6 FIG. 4 is a flowchart of partial steps of an image conversion method according to another embodiment of the present invention.

[0015] Figure 7 FIG. 4 is a flowchart of partial steps of an image conversion method according to another embodiment of the present invention.

[0016] Figure 8 FIG. 4 is a flowchart of partial steps of an image conversion method according to another embodiment of the present invention.

[0017] Fig. 9 is a flow chart of an image conversion method according to another embodiment of the present invention. DETAILED DESCRIPTION

[0018] Components / members / steps with the same reference numerals in the drawings and embodiments of the present specification represent the same or similar parts. Components / members / steps with the same reference numerals or the same terms in different embodiments can refer to the relevant descriptions of each other.

[0019] Figure 1 1 is a block diagram of an image conversion network 11 and a first database 2 according to an embodiment of the present invention. The first database 2 stores a plurality of unconverted images, each of which has at least one focus information.

[0020] The image conversion network 1 is connected to the first database 2 and can access the first database 2. The image conversion network 1 can perform an image conversion method and includes an image generator 11 (Generator), a conversion discriminator 12 (Discriminator) whose signal is connected to the image generator 11, and a focus discriminator 13 (Discriminator) whose signal is connected to the image generator 11. The image generator 11 can perform image conversion and can perform Generative Adversarial Network (GAN)-like learning together with the conversion discriminator 12 and the focus discriminator 13 to update parameters.

[0021] Further references Figure 2 , which is a flow chart of an embodiment of the image conversion method of the present invention. Before the image conversion method executed by the image conversion network 1 is executed, the image generator 11, the conversion discriminator 12 and the focus discriminator 13 are each preset with parameters, that is, the initialization parameter setting is performed first, and the image conversion method uses a learning operation to update the parameters of the image generator 11, the conversion discriminator 12 and the focus discriminator 13 to optimize the conversion performance of the image conversion network 1.

[0022] The steps of the image conversion method of this embodiment are as follows: Step S1 is executed, the image generator 11 is configured to perform an operation based on one of the plurality of unconverted images and at least one focus information of one of the plurality of unconverted images to generate a converted image, wherein the one of the plurality of unconverted images is I A Represents, and the unconverted image I A The converted image generated by the image generator 11 is represented by a mathematical formula G(I A ). The image generator 11 of this embodiment selects an unconverted image from the first database 2, namely, the unconverted image I A Then, according to the selected unconverted image I A and its focus information, for the selected unconverted image I A Perform image conversion to generate the converted image G(I A ). In this embodiment, the unconverted image I A It has one focus information, but it can also be multiple, which is not limited to this embodiment; it should be supplemented that the image generator 11 of this embodiment performs shallow depth of field processing and conversion on the image, but it is not limited to this, and other image effect conversions can also be performed, such as image brightness conversion, color optimization and other effects.

[0023] After obtaining the converted image G(I A ), then step S21 is executed, and the conversion discriminator 12 performs the conversion image G(I A ) performs an operation to generate a conversion identification value, and the conversion identification value is expressed in a mathematical formula as D(G(I A )). The conversion identifier 12 of the present embodiment is used to identify the similarity or difference between the input images, and the conversion identification value is information indicating the degree of similarity or difference between the images. Specifically, the conversion identification value can be a value between 0 and 1. The larger the conversion identification value, the more similar it is. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0024] Then, step S22 is executed, the image generator 11 generates a signal according to the conversion identification value D(G(I A )) calculates a first generation loss function value, and updates the parameters of the image generator 11 according to the first generation loss function value. The image generator 11 calculates the first generation loss function value based on the identification result of the conversion identifier 12 and based on a first generation loss function, and further adjusts and updates the parameters based on the first generation loss function value. The first generation loss function can be expressed as

[0025]

[0026] Wherein, L(G) represents the first generation loss function, is the expected value, I A is the untransformed image, G(I A ) is the transformed image, p A is the probability distribution of the first database 2, and I A ~p A is an unconverted image I randomly selected from the first database 2 A , D(G(I A )) is the conversion identification value. The first generated loss function value of this embodiment can be a value between 0 and 1, and the lower the first generated loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0027] Step S31 is executed, the focus identifier 13 performs an operation based on one of the plurality of unconverted images, the converted image generated by the image generator 11 and the focus information to generate a focus identification value. The converted image of step S31 of this embodiment is the unconverted image I selected from the first database 2. A , generated by the image generator 11, that is, generated by the image generator 11 after parameter update. The focus identifier 13 of this embodiment is used to identify the similarity or difference between the input images, and the focus identification value is information indicating the degree of similarity or difference between the images. Specifically, the focus identification value can be a value between 0 and 1, and the larger the focus identification value, the more similar it is. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0028] Next, step S32 is executed, the image generator 11 calculates a second generation loss function value according to the generated focus identification value, and updates the parameters of the image generator 11 according to the second generation loss function value. The image generator 11 calculates the second generation loss function value based on the identification result of the focus identifier 13 and based on a second generation loss function, and further adjusts and updates the parameters of the image generator 11 based on the second generation loss function value. The second generation loss function value of this embodiment can be a value between 0 and 1, and the lower the second generation loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0029] In steps S21, S22 and steps S31, S32 of the present embodiment, the image generator 11 updates parameters according to different loss function values, and the execution order of steps S21, S22 and steps S31, S32 is interchangeable, that is, steps S31, S32 may be executed in sequence first and then steps S21, S22 may be executed in sequence, and the execution order is not limited to the present embodiment.

[0030] Execute step S71 to determine whether the first generation loss function value and the second generation loss function value are both less than or equal to a preset threshold value, and when it is determined that either the first generation loss function value or the second generation loss function value is greater than the preset threshold value, continue to repeat the aforementioned steps S1, S21, S22, S31, S32 and S71 to perform learning and updating on the image generator 11.

[0031] On the other hand, when it is determined that both the first generated loss function value and the second generated loss function value are less than or equal to the preset threshold value, the image generator 11 completes the learning and executes step S8, wherein the image generator 11 receives a test image and corresponding at least one test focus information, and performs image conversion on the test image according to the updated parameters and the at least one test focus information to generate a converted test image. It is further explained that the image generator 11 in step S8 of this embodiment performs image conversion on the test image using the parameters updated in the last execution. In other words, the image conversion method of this embodiment uses the process of minimizing the first and second loss generation function values ​​to learn and optimize the image conversion network 1. The number of test focus information for each test image in this embodiment is one, but it can also be multiple, and is not limited to this embodiment.

[0032] In other embodiments, the judgment condition of step S71 for determining whether the image generator 11 has completed learning and updating may also be to determine whether the first generation loss function value and the second generation loss function value have converged to determine whether the image generator 11 has completed learning and updating. That is, when the first and second generation loss function values ​​have converged, it is determined that the image generator 11 has completed learning, and the judgment condition is not limited to the threshold value of this embodiment.

[0033] It is further explained that the image generator 11 can be integrated into a mobile device (not shown), and the test image and the test focus information can be obtained via the mobile device and its lens (not shown), wherein the test focus information of the test image can be determined by the user when shooting, such as information of focusing by clicking on the screen of the mobile device when shooting. In this embodiment, the image generator 11 performs image conversion on the image obtained by the mobile device to generate the converted test image with a shallow depth of field effect, but it is not limited to the shallow depth of field effect, and can also be other image effect conversions, such as image brightness conversion, color optimization and other effects.

[0034] In the image conversion method of the present embodiment, the image generator 11 uses the conversion identification value of the conversion identifier 12 to calculate the first generation loss function value to update the parameters. On the other hand, it also uses the focus information and the focus identification value of the focus identifier 13 to calculate the second generation loss function value to update the parameters. That is, the parameters of the image generator 11 of the present embodiment are completed after at least two stages of learning operations.

[0035] Figure 3 1 is a block diagram of an image conversion network 1, a first database 2 and a second database 3 according to another embodiment of the present invention. The first database 2 stores a plurality of unconverted images, each of which has at least one focus information. The second database 3 stores a plurality of conversion template images. In this embodiment, the number of focus information of each unconverted image is one, but it can also be a plurality of focus information, which is not limited to this embodiment.

[0036] The image conversion network 1 is connected to the first database 2 and the second database 3, and can access the first database 2 and the second database 3. The image conversion network 1 can perform an image conversion method, and includes an image generator 11 (Generator), a conversion discriminator 12 (Discriminator) connected to the image generator 11, and a focus discriminator 13 (Discriminator) connected to the image generator 11. The image generator 11 can perform image conversion, and can perform a Generative Adversarial Network (GAN)-like learning operation with the conversion discriminator 12 and the focus discriminator 13 to update parameters.

[0037] Figure 4 , 5 6 is a flowchart of another embodiment of the image conversion method of the present invention, which is applicable to Figure 3The image conversion network 1, the first database 2 and the second database 3 are shown. Before the image conversion method executed by the image conversion network 1 is executed, the image generator 11, the conversion identifier 12 and the focus identifier 13 are respectively preset with parameters, that is, the initialization parameter setting is performed first, and the image conversion method utilizes the learning operation to update the image generator 11, the conversion identifier 12 and the focus identifier 13 to optimize the conversion performance of the image conversion network 1.

[0038] Reference Figure 3 and 4 , the steps of the image conversion method of this embodiment are described as follows: Execute step S1, the image generator 11 is configured to perform an operation based on one of the multiple unconverted images and at least one focus information of one of the multiple unconverted images to generate a converted image. After the image generator 11 of this embodiment selects an unconverted image from the first database 2, it performs image conversion on the selected unconverted image based on the selected unconverted image and its focus information to generate the converted image. In this embodiment, the number of focus information of the selected unconverted image is one, but it can also be multiple focus information; it is supplemented that the image generator 11 of this embodiment performs shallow depth of field processing and conversion on the image, but it is not limited to this, and it can also be other image effect conversions, such as image brightness conversion, color optimization and other effects.

[0039] Step S21 is executed, and the conversion identifier 12 performs an operation on the conversion image to generate a conversion identification value. The conversion identifier 12 of the present embodiment is used to identify the similarity or difference between the input images, and the conversion identification value is information indicating the degree of similarity or difference between the images. Specifically, the conversion identification value can be a value between 0 and 1, and the larger the conversion identification value, the more similar it is. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0040] Then, step S22 is executed, the image generator 11 calculates a first generation loss function value according to the conversion identification value, and updates the parameters of the image generator 11 according to the first generation loss function value. In other words, the image generator 11 calculates the first generation loss function value based on the identification result of the conversion identification device 12 and a first generation loss function, and then adjusts and updates the parameters based on the first generation loss function value. The first generation loss function can be mathematically expressed as

[0041]

[0042] Wherein, L(G) represents the first generation loss function, is the expected value, I A is the unconverted image, p Ais the probability distribution of the first database 2, and I A ~p A is an unconverted image I randomly selected from the first database 2 A , G(I A ) is the transformed image, D(G(I A )) is the conversion identification value. The first generated loss function value of this embodiment can be a value between 0 and 1, and the lower the first generated loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0043] Step S41 is executed, and the conversion identifier 12 calculates another conversion identification value according to one of the plurality of conversion template images. In step S41, the conversion identifier 12 randomly selects a conversion template image from the second database 3, which is I B Represents, and for the selected conversion template image I B The operation is performed to obtain the other conversion identification value, which is D(I B In addition, when the parameters of the image generator 11 have been updated in step S22, steps S1 and S21 are performed again, that is, the parameters corresponding to the unconverted image I are updated first. A The transformed image G(I A ), and according to the updated conversion image G(I A ), execute step S21 again to update the conversion identification value D(G(I A )).

[0044] Then, step S42 is performed, and the conversion identification device 12 generates a plurality of conversion identification values ​​D(I B )、D(G(I A )) calculates a conversion identification loss function value, and updates the parameters of the conversion identifier 12 according to the conversion identification loss function value, wherein the conversion identification loss function value is calculated based on a conversion identification loss function, and the conversion identification loss function can be mathematically expressed as

[0045]

[0046] Among them, L(D) represents the conversion identification loss function, is the expected value, I A is the unconverted image, p A is the probability distribution of the first database 2, I A ~p A is used to represent the unconverted image I randomly selected from the first database 2 A , G(I A ) is the transformed image, IB To transform the sample image, p B is the probability distribution of the second database 3, I B ~p B is used to represent the conversion sample image I randomly selected from the second database 3 B , and D(G(I A )) and D(I B ) is the conversion identification value. The conversion identification device 12 in step S42 of this embodiment substitutes the multiple conversion identification values ​​obtained by calculating the conversion image and the conversion template image into the conversion identification loss function, that is, the above mathematical formula, to calculate the conversion identification loss function value, and then update the parameters of the conversion identification device 12 accordingly. The conversion identification loss function value of this embodiment can be a value between 0 and 1, and the lower the conversion identification loss function value, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0047] Step S51 is executed, and the focus identifier 13 calculates a focus identification loss function value according to two of the plurality of unconverted images. In step S51, the focus identifier 13 randomly selects two unconverted images from the first database 2, and performs an operation on the selected unconverted images. The focus identifier 13 of this embodiment is used to identify the similarity or difference between the input images, and the focus identification value is information indicating the degree of similarity or difference between the images. Specifically, the focus identification value is a value between 0 and 1, and the larger the focus identification value is, the more similar it is. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0048] For the sake of clarity, the two unconverted images selected in step S51 are respectively represented by unconverted image I and A , unconverted image I A′ In step S51 of this embodiment, the focus identifier 13 is based on the unconverted image I A and the unconverted image I A′ Calculate a focus identification value, which is expressed in mathematical form as D s (I A ,I A′ ), and based on the unconverted image I A Calculate another focus identification value, which is expressed in mathematical form as D s (I A ,I A ), and then according to the unconverted image I A′ Calculate another focus identification value, which is expressed in mathematical form as D s (I A′ ,I A′ ).

[0049] After obtaining the focus identification value D s (I A ,I A′ )、D s (I A ,I A ) and D s (I A′ ,I A′ ), the focus discriminator 13 is based on a focus discriminator loss function and according to the focus discriminator value D s (I A ,I A′ )、D s (I A ,I A ) and D s (I A′ ,I A′ ) calculates the focus identification loss function value, and the focus identification loss function can be mathematically expressed as

[0050]

[0051] Among them, L(D s ) represents the focal identification loss function, is the expected value, I A ,I A ' is the unconverted image, p A is the probability distribution of the first database 2, and I A ,I A '~p A is used to represent the unconverted image I randomly selected from the first database 2 A ,I A ', D s (I A ,I A '),D s (I A ,I A )、D s (I A ',I A ') is the focus identification value.

[0052] Then, step S52 is executed, and the focus identifier 13 updates the parameters of the focus identifier 13 according to the focus identification loss function value. The focus identification loss function value of this embodiment can be a value between 0 and 1, and the lower the focus identification loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0053] Then, step S33 is executed, wherein the focus identifier 13 performs calculations based on the converted image, one of the plurality of unconverted images, and the at least one focus information to generate a plurality of focus identification values. The number of focus information of each unconverted image may be one or more, and this embodiment is described by taking one focus information as an example. The one of the plurality of unconverted images may be the unconverted image I selected in step S1. A , and the plurality of focus identification values ​​include a first focus identification value and at least one second focus identification value. Figure 5 and 6 is a flow chart of step S33 of the image conversion method according to the present invention.

[0054] Further references Figure 5 The details of step S33 of this embodiment are as follows: Step S331 is executed, the focus identifier 13 selects the unconverted image I according to one of the plurality of unconverted images, that is, the selected unconverted image I A , and the converted image G(I A ) performs an operation to generate the first focus identification value, which can be mathematically expressed as D s (I A ,G(I A )); and executing step S332, the focus identifier 13 is based on one of the plurality of unconverted images, that is, the selected unconverted image I A , and the at least one focus information is operated to generate the at least one second focus identification value. In this embodiment, the number of focus information is one, and the number of the second focus identification value is also one, but it is not limited to this. The details of step S332 are as follows Figure 6 As shown, it will be explained in detail in the next paragraph.

[0055] Further references Figure 6 , for clarity, corresponds to the unconverted image I A The focus information is represented by (x, y), and the details of step S332 of this embodiment are as follows: Step S333 is executed, the focus identifier 13 is based on the at least one focus information, that is, the focus information (x, y), and one of the plurality of unconverted images (that is, the unconverted image I A ), generating an unconverted sub-image; and according to the at least one focus information, namely the focus information (x, y), and the converted image G(I A ) generates at least one converted sub-image. In this embodiment, the number of the focus information is one, so the number of the converted sub-images is also one, but this is not limited to this embodiment, and the number can also be greater than one. On the other hand, the focus identifier 13 of this embodiment takes the unconverted image I AThe block corresponding to the focal information (x, y) is taken as the unconverted sub-image, which can be mathematically expressed as S(I A ,x,y), and the transformed image G(I A ) is used as the conversion sub-image, which can be mathematically expressed as S(G(I A ),x,y).

[0056] Then, step S334 is executed, and the focus identifier 13 determines the focus of the non-converted sub-image S(I A , x, y) and the at least one conversion sub-image, namely the conversion sub-image S(G(I A ), x, y), perform operations to generate the at least one second focus identification value. The number of the second focus identification value in this embodiment is one, which can be expressed as D s (S(I A ,x,y),S(G(I A ),x,y)).

[0057] After obtaining the first and second focus identification values, step S32 is executed, the image generator 11 calculates a second generation loss function value according to the plurality of focus identification values, and updates the parameters of the image generator 11 according to the second generation loss function value. The image generator 11 is based on the identification result of the focus identifier 13, that is, the first focus identification value D s (I A ,G(I A )) and the second focal point identification value D s (S(I A ,x,y),S(G(I A ),x,y)), and based on a second generation loss function to calculate the second generation loss function value, and then adjust and update the parameters according to the second generation loss function value. The second generation loss function can be mathematically expressed as

[0058]

[0059] Among them, L(G s ) represents the second generation loss function, is the expected value, I A is the untransformed image, G(I A ) is the transformed image, p A is the probability distribution of the first database 2, and I A ~p A is used to represent the unconverted image I randomly selected from the first database 2 A , (x, y) is the focus information, S(I A ,x,y) represents the unconverted sub-image, S(G(IA ),x,y) represents the transformed sub-image, D s (I A ,G(I A )) is the first focal point identification value, D s (S(I A ,x,y),S(G(I A ), x, y)) is the second focus identification value. The second generation loss function value of this embodiment can be a value between 0 and 1, and the lower the second generation loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0060] Execute step S72 to determine whether the first generation loss function value, the second generation loss function value, the conversion identification loss function value and the focus identification loss function value are all less than or equal to a preset threshold value, and when it is determined that any one of the first generation loss function value, the second generation loss function value, the conversion identification loss function value and the focus identification loss function value is greater than the preset threshold value, continue to repeat the aforementioned steps S1, S21, S22, S41, S42, S51, S52, S33, S32 and S72 to perform learning and updating on the image generator 11, the conversion identifier 12 and the focus identifier 13.

[0061] On the other hand, when it is determined that the first generation loss function value, the second generation loss function value, the conversion identification loss function value and the focus identification loss function value are all less than or equal to the preset threshold value, it is determined that the image generator 11 has completed learning; and step S8 is executed, the image generator 11 receives a test image and the corresponding at least one test focus information, and performs image conversion on the test image according to the updated parameters and the at least one test focus information to generate a conversion test image. It is further explained that the image generator 11 in step S8 of this embodiment uses the parameters updated for the last time to perform image conversion on the test image. In other words, the image conversion method of this embodiment is to learn and optimize the image conversion network 1 by minimizing the first and second loss generation function values, the conversion identification loss function value and the focus identification loss function value. The number of test focus information for each test image in this embodiment is one, but it can also be multiple, not limited to this embodiment.

[0062] In other embodiments, the judgment condition of step S72 for judging whether the image generator 11, the conversion identifier 12 and the focus identifier 13 have completed learning and updating can also be based on whether the first generation loss function value, the second generation loss function value, the conversion identification loss function value and the focus identification loss function value have respectively converged to determine whether the image generator 11 has completed learning and updating. That is, when the first and second generation loss function values, the conversion identification loss function value and the focus identification loss function value have all converged, it is judged that the image generator 11 has completed learning, and therefore it is not limited to the judgment condition of the threshold of this embodiment.

[0063] It is worth noting that the execution order between steps S21, S22 and steps S41, S42 of this embodiment is interchangeable, that is, in other implementations, steps S41, S42 can be executed in sequence after executing S1, and after executing step S42, steps S21, S22 can be executed in sequence, so the execution order of the steps is not limited to this embodiment.

[0064] It is further explained that the image generator 11 can be integrated into a mobile device, and the test image and the corresponding test focus information can be obtained through the lens of the mobile device, wherein the test focus information of the test image can be determined when the user shoots, such as the information that the user clicks the screen of the mobile device to focus when shooting. In this embodiment, the image generator 11 performs image conversion on the image obtained by the mobile device to generate the converted test image with a shallow depth of field effect, but it is not limited to the shallow depth of field effect, and can also be other image effect conversions, such as image brightness conversion, color optimization and other effects.

[0065] In the image conversion method of the present embodiment, the image generator 11 uses the conversion identification value of the conversion identifier 12 to calculate the first generation loss function value to update the parameters, and calculates the conversion loss function value to update the conversion identifier 12 accordingly; on the other hand, the focus identifier 13 is updated according to the focus loss function value, and the focus identifier 13 uses the focus information to calculate the focus identification value, and calculates the second generation loss function value to update the parameters of the image generator 11. That is, the image generator 11 of the present embodiment learns against the conversion identifier 12 and the focus identifier 13, and the parameters of the image generator 11 are completed after at least two stages of learning operations.

[0066] Reference Figure 4 , 7 and 8, which is a flowchart of another embodiment of the image conversion method of the present invention, the image conversion method of this embodiment is Figure 4 , 5 Similar to the embodiment shown in 6, it is also applicable to Figure 3The image conversion network 1 connected to the first database 2 and the second database 3 is shown, and the image conversion method of this embodiment is similar to Figure 4 , 5 The difference from the embodiment 6 is that the unconverted image of this embodiment has multiple focus information. Here, two focus information are used as an example. The focus information of the unconverted image of this embodiment includes a first focus information (x1, y1) and a second focus information (x2, y2).

[0067] This embodiment and Figure 4 , 5 The differences in the execution steps of the embodiment shown in FIG. 6 are as follows: Figure 7 is a detailed flow chart of step S33 of this embodiment, and further reference is made to Figure 7 In this embodiment, step S33 is to execute step S331, wherein the focus identifier 13 is based on one of the plurality of unconverted images (i.e., the unconverted image I A ) and the converted image G(I A ) performs an operation to generate the first focus identification value; and executes step S335, the focus identifier 13 performs an operation to generate the first focus identification value according to one of the plurality of unconverted images (ie, the unconverted image I A ), the first focus information and the second focus information are operated to generate a plurality of second focus identification values.

[0068] Figure 8 is a detailed flow chart of step S335 of this embodiment, refer to Figure 8 The details of step S335 of this embodiment are as follows: Step S336 is executed, the focus identifier 13 is based on the first focus information (x1, y1) and one of the plurality of unconverted images, that is, the unconverted image I A , generate a first unconverted sub-image, and according to the second focus information (x2, y2) and the unconverted image I A A second unconverted sub-image is generated, and then the first focus information (x1, y1) and the converted image G (I A ) generates a first conversion sub-image, and according to the second focus information (x2, y2) and the conversion image G(I A ) generates a second converted sub-image. In this embodiment, the focus identifier 13 takes the unconverted image I A The blocks corresponding to the first focus information (x1, y1) and the second focus information (x2, y2) are respectively used as the first unconverted sub-image and the second unconverted sub-image, which can be expressed as S(I A ,x1,y1),S(I A ,x2,y2), and take the transformed image G(I A) in which the blocks corresponding to the first focus information (x1, y1) and the second focus information (x2, y2) are respectively used as the first conversion sub-image and the second conversion sub-image, which can be respectively expressed as S(G(I A ),x1,y1)、S(G(I A ),x2,y2).

[0069] Then, step S337 is executed, and the focus identifier 13 determines the focus of the first unconverted sub-image S(I A , x1, y1), the second unconverted sub-image S(I A , x2, y2), the first converted sub-image S(G(I A ), x1, y1) and the second converted sub-image S(G(I A ), x2, y2) are operated to generate two second focus identification values, which can be expressed as D s (S(I A ,x1,y1),S(G(I A ),x1,y1)) and D s (S(I A ,x2,y2),S(G(I A ),x2,y2)).

[0070] After obtaining the first and second focus identification values, step S32 is executed, the image generator 11 calculates a second generation loss function value according to the first focus identification value and the plurality of second focus identification values, and updates the parameters of the image generator 11 according to the second generation loss function value. The second generation loss function of the present embodiment can be mathematically expressed as

[0071]

[0072] Among them, L(G s ) represents the second generation loss function, is the expected value, I A is the untransformed image, G(I A ) is the transformed image, p A is the probability distribution of the first database 2, I A ~p A Represents the unconverted image I A is randomly selected from the first database 2, (x1, y1) and (x2, y2) are the focus information, S(I A ,x1,y1),S(I A ,x2,y2) is the unconverted sub-image, S(G(I A ),x1,y1)、S(G(I A ),x2,y2) is the transformed sub-image, and D s(I A ,G(I A )) is the first focal point identification value, D s (S(I A ,x1,y1),S(G(I A ),x1,y1))、D s (S(I A ,x2,y2),S(G(I A ),x2,y2)) is the second focus identification value.

[0073] The second generated loss function value of this embodiment can be a value between 0 and 1, and the lower the second generated loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0074] After executing S337, similar to the previous embodiment, step S32 is executed to update the parameters of the image generator 11, and after determining that the learning is completed in step S72, step S8 is executed, the image generator 11 receives a test image and at least one test focus information corresponding to the test image, and the image generator 11 performs a conversion on the test image according to the updated parameters and the at least one test focus information to generate a conversion test image. It is further explained that the image generator 11 in step S8 of this embodiment uses the parameters updated for the last time to perform image conversion on the test image. It is worth mentioning that each test image of this embodiment has two test focus information, but this is not limited to this, and the number of the multiple test focus information can also be greater than two.

[0075] Reference Figure 5 , 6 and 9, which are flowcharts of another embodiment of the image conversion method of the present invention. The image conversion method of this embodiment is applicable to Figure 3 The image conversion network 1, the first database 2 and the second database 3 are shown. The first database 2 stores a plurality of unconverted images, each of which has at least one focus information. The second database 3 stores a plurality of conversion template images. In the present embodiment, the number of focus information of each unconverted image is one, but it can also be a plurality of focus information, which is not limited to the present embodiment.

[0076] Further references Figure 3 Before the image conversion method performed by the image conversion network 1 is executed, the image generator 11, the conversion identifier 12 and the focus identifier 13 are each preset with parameters, that is, the initialization parameter setting is performed first, and the image conversion method uses a learning operation to update the image generator 11, the conversion identifier 12 and the focus identifier 13 to optimize the conversion performance of the image conversion network 1.

[0077] The steps of the image conversion method of this embodiment are as follows: Step S1 is executed, and the image generator 11 is configured to perform an operation based on one of the multiple unconverted images and at least one focus information of one of the multiple unconverted images to generate a converted image. After the image generator 11 selects an unconverted image from the first database 2, it performs image conversion on the selected unconverted image based on the selected unconverted image and its focus information to generate the converted image. In this embodiment, the image generator 11 performs shallow depth of field processing and conversion on the image, but it is not limited to this, and other image effect conversions can also be performed, such as image brightness conversion, color optimization and other effects.

[0078] After obtaining the converted image, step S51 is executed, and the focus identifier 13 calculates a focus identification loss function value according to two of the plurality of unconverted images. In step S51, the focus identifier 13 randomly selects two unconverted images from the first database 2 and performs calculations on the selected images. The focus identifier 13 of this embodiment is used to identify the similarity or difference between the input images, and the focus identification value is information indicating the degree of similarity or difference between the images. Specifically, the focus identification value is a value between 0 and 1, and the larger the focus identification value is, the more similar it is. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0079] For the sake of clarity, the two unconverted images selected in step S51 are respectively represented by unconverted image I and A , unconverted image I A′ In step S51 of this embodiment, the focus identifier 13 is based on the unconverted image I A and the unconverted image I A′ Calculate a focus identification value, which is expressed in mathematical form as D s (I A ,I A′ ), and based on the unconverted image I A Calculate another focus identification value, which is expressed in mathematical form as D s (I A ,I A ), and then according to the unconverted image I A′ Calculate another focus identification value, which is expressed in mathematical form as D s (I A′ ,I A′ ).

[0080] After obtaining the plurality of focus identification values ​​D s (I A ,I A′ )、D s (IA ,I A ) and D s (I A′ ,I A′ ), the focus discriminator 13 is based on a focus discriminator loss function and according to the plurality of focus discriminator values ​​D s (I A ,I A′ )、D s (I A ,I A ) and D s (I A′ ,I A′ ) calculates the focus identification loss function value, and the focus identification loss function can be mathematically expressed as

[0081]

[0082] Among them, L(D s ) represents the focus identification loss function, is the expected value, I A ,I A ' is the unconverted image, p A is the probability distribution of the first database 2, and I A ,I A '~p A is used to represent the unconverted image I randomly selected from the first database 2 A ,I A ', D s (I A ,I A '),D s (I A ,I A )、D s (I A ',I A ') is the focus identification value. Then, step S52 is executed, and the focus identifier 13 updates the parameters of the focus identifier 13 according to the focus identification loss function value. The focus identification loss function value of this embodiment can be a value between 0 and 1, and the lower the focus identification loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0083] Then, step S33 is performed, wherein the focus identifier 13 performs calculations according to the converted image, one of the plurality of unconverted images, and the at least one focus information to generate a plurality of focus identification values. The one of the plurality of unconverted images may be the unconverted image I selected in step S1. A , and the plurality of focus identification values ​​include a first focus identification value and at least one second focus identification value. Figure 5 , 6 is a flow chart of step S33 of the image conversion method according to the present invention. Figure 5 The details of step S33 of this embodiment are as follows: Step S331 is executed, the focus identifier 13 is based on one of the plurality of unconverted images, that is, the unconverted image I A , and the converted image G(I A ) performs an operation to generate the first focus identification value, which can be mathematically expressed as D s (I A ,G(I A )); and executing step S332, the focus identifier 13 is based on one of the plurality of unconverted images, ie, the unconverted image I A , and the at least one focus information is operated to generate the at least one second focus identification value. The number of focus information in this embodiment is one, but it is not limited to this. The details of step S332 are as follows Figure 6 As shown, it will be explained in detail in the next paragraph.

[0084] Further references Figure 6 , for clarity, corresponds to the unconverted image I A The focus information is represented by (x, y), and the details of step S332 of this embodiment are as follows: Step S333 is executed, the focus identifier 13 is based on the at least one focus information, that is, the focus information (x, y), and one of the plurality of unconverted images (that is, the unconverted image I A ) generates at least one unconverted sub-image, and according to the at least one focus information, namely the focus information (x, y), and the converted image G(I A ) generates at least one converted sub-image. In this embodiment, the number of the focus information is one, so the number of the converted sub-image and the unconverted sub-image is also one, but it is not limited to this embodiment, and the number can be greater than one. On the other hand, the focus identifier 13 of this embodiment takes the block corresponding to the focus information (x, y) in the unconverted image as the unconverted sub-image, which can be mathematically expressed as S(I A ,x,y), and the conversion sub-image is obtained from the conversion image, which can be mathematically expressed as S(G(I A ), x, y). Then, step S334 is executed, and the focus identifier 13 is based on (I A ,x,y) and the at least one converted sub-image S(G(I A ), x, y), perform operations to generate the at least one second focus identification value. The number of the second focus identification value in this embodiment is one, which can be expressed as D s (S(I A,x,y),S(G(I A ),x,y)).

[0085] After obtaining the first and second focus identification values, step S32 is executed, the image generator 11 calculates a second generation loss function value according to the focus identification value, and updates the parameters of the image generator 11 according to the second generation loss function value. The image generator 11 is based on the identification result of the focus identifier 13, that is, the first focus identification value D s (I A ,G(I A )) and the second focal point identification value D s (S(I A ,x,y),S(G(I A ),x,y)), and based on a second generation loss function to calculate the second generation loss function value, and then adjust and update the parameters according to the second generation loss function value. The second generation loss function can be mathematically expressed as

[0086]

[0087] Among them, L(G s ) represents the second generation loss function, is the expected value, I A is the untransformed image, G(I A ) is the transformed image, p A is the probability distribution of the first database 2, and I A ~p A is used to represent the unconverted image I randomly selected from the first database 2 A , (x, y) is the focus information, S(I A ,x,y) represents the unconverted sub-image, S(G(I A ),x,y) represents the transformed sub-image, D s (I A ,G(I A )) is the first focal point identification value, D s (S(I A ,x,y),S(G(I A ), x, y)) is the second focus identification value. The second generation loss function value of this embodiment can be a value between 0 and 1, and the lower the second generation loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0088] Then, step S43 is executed, the conversion identifier 12 calculates a conversion identification value according to one of the plurality of conversion template images, and calculates another conversion identification value according to the conversion image. In step S43, the conversion identifier 12 randomly selects a conversion template image from the second database 3, which is I B Represents, and for the selected conversion template image I B The conversion identification value is obtained by performing an operation, and D(I B ) represents. In addition, since the parameters of the image generator 11 have been updated in step S32, step S1 is executed again, that is, the parameters corresponding to the unconverted image I are updated first. A The transformed image G(I A ), and then according to the updated transformation image G(I A ) calculates the other conversion identification value, which can be D(G(I A ))express.

[0089] Then, step S42 is performed, and the conversion identification device 12 generates a plurality of conversion identification values ​​D(I B )、D(G(I A )) calculates a conversion identification loss function value, and updates the parameters of the conversion identifier 12 according to the conversion identification loss function value, wherein the conversion identification loss function value is calculated based on a conversion identification loss function, and the conversion identification loss function can be mathematically expressed as

[0090]

[0091] Among them, L(D) represents the conversion identification loss function, is the expected value, I A is the unconverted image, p A is the probability distribution of the first database 2, I A ~p A is used to represent the unconverted image I randomly selected from the first database 2 A , G(I A ) is the transformed image, I B To transform the sample image, p B is the probability distribution of the second database 3, I B ~p B is used to represent the conversion sample image I randomly selected from the second database 3 B , and D(G(I A )) and D(I B ) is the conversion identification value.

[0092] The conversion identification loss function value of this embodiment can be a value between 0 and 1, and the lower the conversion identification loss function value, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements. The conversion identifier 12 in step S42 of this embodiment substitutes the multiple conversion identification values ​​obtained by calculating the conversion image and the conversion template image into the conversion identification loss function, that is, the above mathematical formula, to calculate the conversion identification loss function value, and then update the parameters of the conversion identifier 12 accordingly.

[0093] Step S21 is executed, and the conversion discriminator 12 performs an operation on the conversion image to generate a conversion discrimination value. Since the parameters of the conversion discriminator 12 have been updated in step S42, the conversion discriminator 12 in step S21 of this embodiment is to perform an operation on the conversion image G(I A ) performs the operation and updates the corresponding conversion identification value D(G(I A )). The conversion identifier 12 of the present embodiment is used to identify the similarity or difference between the input images, and the conversion identification value is information indicating the degree of similarity or difference between the images. Specifically, the conversion identification value can be a value between 0 and 1. The larger the conversion identification value, the more similar it is. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0094] Then, step S22 is executed, the image generator 11 calculates a first generation loss function value according to the conversion identification value, and updates the parameters of the image generator 11 according to the first generation loss function value. In other words, the image generator 11 calculates the first generation loss function value based on the identification result of the conversion identification device 12 and a first generation loss function, and adjusts and updates the parameters according to the first generation loss function value. The first generation loss function can be mathematically expressed as

[0095]

[0096] Wherein, L(G) represents the first generation loss function, is the expected value, I A is the unconverted image, p A is the probability distribution of the first database 2, and I A ~p A is used to represent the unconverted image I randomly selected from the first database 2 A , G(I A ) is the transformed image, D(G(I A)) is the conversion identification value. The first generated loss function value of this embodiment can be a value between 0 and 1, and the lower the first generated loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0097] Execute step S72 to determine whether the first generation loss function value, the second generation loss function value, the conversion identification loss function value and the focus identification loss function value are all less than or equal to a preset threshold value, and when it is determined that any one of the first generation loss function value, the second generation loss function value, the conversion identification loss function value and the focus identification loss function value is greater than the preset threshold value, continue to repeat the aforementioned steps S1, S51, S52, S33, S32, S43, S42, S21, S22 and S72 to perform learning and updating on the image generator 11, the conversion identifier 12 and the focus identifier 13.

[0098] On the other hand, when it is determined that the first generation loss function value, the second generation loss function value, the conversion identification loss function value and the focus identification loss function value are all less than or equal to the preset threshold value, it is determined that the image generator 11 has completed learning, and step S8 is executed, the image generator 11 receives a test image and the corresponding at least one test focus information, and the image generator 11 performs image conversion on the test image according to the updated parameters and the at least one focus test information to generate a conversion test image. It is further explained that the image generator 11 in step S8 of this embodiment uses the parameters updated for the last time to perform image conversion on the test image. In other words, the image conversion method of this embodiment is to learn and optimize the image conversion network 1 by minimizing the first and second loss generation function values, the conversion identification loss function value and the focus identification loss function value. The number of test focus information for each test image in this embodiment is one, but it can also be multiple, not limited to this embodiment.

[0099] In other embodiments, the judgment condition of step S72 for judging whether the image generator 11, the conversion identifier 12 and the focus identifier 13 have completed learning and updating may also be based on whether the first generation loss function value, the second generation loss function value, the conversion identification loss function value and the focus identification loss function value have respectively converged to determine whether the image generator 11 has completed learning and updating. That is, when the first and second generation loss function values, the conversion identification loss function value and the focus identification loss function value have all converged, it is judged that the image generator 11 has completed learning, and therefore it is not limited to the judgment condition of the threshold of this embodiment.

[0100] It is worth noting that the execution order of steps S51, S52 and steps S33, S32 of this embodiment is interchangeable, that is, in other implementations, after executing S1, steps S33, S32 can be executed in sequence, and after executing step S32, steps S51, S52 can be executed in sequence. The execution order of the steps is not limited to this embodiment.

[0101] It is further explained that the image generator 11 can be integrated into a mobile device, and the test image and the corresponding test focus information can be obtained through the lens of the mobile device, wherein the test focus information of the test image can be determined when the user shoots, such as the information that the user clicks the screen of the mobile device to focus when shooting. In this embodiment, the image generator 11 performs image conversion on the image obtained by the mobile device to generate the converted test image with a shallow depth of field effect, but it is not limited to the shallow depth of field effect, and can also be other image effect conversions, such as image brightness conversion, color optimization and other effects.

[0102] In the image conversion method of the present embodiment, the image generator 11 uses the conversion identification value of the conversion identifier 12 to calculate the first generation loss function value to update the parameters, and calculates the conversion loss function value to update the conversion identifier 12 accordingly; on the other hand, the focus identifier 13 is updated according to the focus loss function value, and the focus identifier 13 uses the focus information to calculate the focus identification value, and calculates the second generation loss function value to update the parameters of the image generator 11. That is, the image generator 11 of the present embodiment learns against the conversion identifier 12 and the focus identifier 13, and the parameters of the image generator 11 are completed after at least two stages of learning operations.

[0103] Reference Figure 7 , 8 and 9, which are flowcharts of another embodiment of the image conversion method of the present invention. The image conversion method of this embodiment is Figure 5 , 6 Similar to the embodiment of 9, it is also applicable to Figure 3 The image conversion network 1 connected to the first database 2 and the second database 3 is shown, and the image conversion method of this embodiment is similar to Figure 5 , 6 The difference from the embodiment of 9 is that each unconverted image has multiple focus information. Here, two focus information are used as an example. The focus information of the unconverted image in this embodiment includes a first focus information (x1, y1) and a second focus information (x2, y2).

[0104] This embodiment and Figure 5 , 6 The differences in the execution steps of the embodiment shown in 9 are as follows: Figure 7is a detailed flow chart of step S33 of this embodiment, and further reference is made to Figure 7 In this embodiment, step S33 is to execute step S331, wherein the focus identifier 13 is based on one of the plurality of unconverted images (i.e., the unconverted image I A ) and the converted image perform an operation to generate the first focus identification value; and execute step S335, the focus identifier 13 performs an operation according to one of the plurality of unconverted images (ie, the unconverted image I A ), the first focus information and the second focus information are operated to generate a plurality of second focus identification values.

[0105] Figure 8 is a detailed flow chart of step S335 of this embodiment, refer to Figure 8 The details of step S335 of this embodiment are as follows: Step S336 is executed, the focus identifier 13 is based on the first focus information (x1, y1) and one of the plurality of unconverted images, that is, the unconverted image I A , generating a first unconverted sub-image, and generating a second unconverted sub-image according to the second focus information (x2, y2) and the unconverted image, and then generating a first converted sub-image according to the first focus information (x1, y1) and the converted image, and generating a second converted sub-image according to the second focus information (x2, y2) and the converted image. In this embodiment, the focus identifier 13 takes the unconverted image I A The blocks corresponding to the first focus information (x1, y1) and the second focus information (x2, y2) are respectively used as the first unconverted sub-image and the second unconverted sub-image, which can be expressed as S(I A ,x1,y1),S(I A ,x2,y2); and take the transformed image G(I A ) in which the blocks corresponding to the first focus information (x1, y1) and the second focus information (x2, y2) are respectively used as the first conversion sub-image and the second conversion sub-image, which can be respectively expressed as S(G(I A ),x1,y1)、S(G(I A ),x2,y2).

[0106] Then, step S337 is executed, and the focus identifier 13 determines the focus of the first unconverted sub-image S(I A , x1, y1), the second unconverted sub-image S(I A , x2, y2), the first converted sub-image S(G(I A ), x1, y1) and the second converted sub-image S(G(I A ), x2, y2) are operated to generate two second focus identification values, which can be expressed as D s(S(I A ,x1,y1),S(G(I A ),x1,y1)) and D s (S(I A ,x2,y2),S(G(I A ),x2,y2)).

[0107] After obtaining the first and second focus identification values, step S32 is executed, the image generator 11 calculates a second generation loss function value according to the first focus identification value and the plurality of second focus identification values, and updates the parameters of the image generator 11 according to the second generation loss function value. The second generation loss function of the present embodiment can be mathematically expressed as

[0108]

[0109] Among them, L(G s ) represents the second generation loss function, is the expected value, I A is the untransformed image, G(I A ) is the transformed image, p A is the probability distribution of the first database 2, I A ~p A It is used to represent the unconverted image I A is randomly selected from the first database 2, (x1, y1) and (x2, y2) are the focus information, S(I A ,x1,y1),S(I A ,x2,y2) is the unconverted sub-image, S(G(I A ),x1,y1)、S(G(I A ),x2,y2) represents the transformed sub-image, and D s (I A ,G(I A )) is the first focal point identification value, D s (S(I A ,x1,y1),S(G(I A ),x1,y1))、D s (S(I A ,x2,y2),S(G(I A ), x2, y2)) is the second focus identification value. The second generation loss function value of this embodiment can be a value between 0 and 1, and the lower the second generation loss function value is, the closer the image conversion network 1 is to the ideal state. However, this is not limited to this, and the definition corresponding to the value can be adjusted according to the calculation requirements.

[0110] It is worth mentioning that, similar to the previous embodiment, in step S8, the image generator 11 receives a test image and at least one test focus information corresponding to the test image, and the image generator 11 performs a conversion on the test image according to the updated parameters and the at least one test focus information to generate a conversion test image. However, in this embodiment, the number of test focus information for each test image is two, but it is not limited to this, and the number of the multiple test focus information can also be greater than two. It is further explained that the image generator 11 in step S8 of this embodiment uses the parameters updated for the last time to perform image conversion on the test image.

[0111] It should be noted that the image conversion network 1 of the present invention can be a plurality of software or firmware programs similar to a Generative Adversarial Network (GAN), or a circuit module. In addition, the image conversion method in the present invention can be in the form of a computer program, which executes the image conversion method through a processor and a memory, and the steps of the image conversion method of the present invention, in addition to the sequence clearly described, can be adjusted according to the needs of implementation, and can be executed simultaneously or partially simultaneously.

[0112] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person having ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the invention shall be determined by the scope of the attached patent application.

[0113]

Explanation of symbols

[0114] 1: Image Conversion Network

[0115] 11: Image Generator

[0116] 12: Transformation Discriminator

[0117] 13: Focus Discriminator

[0118] 2: First database

[0119] 3: Second database

[0120] S1, S8: Steps

[0121] S21, S22, S31~S33, S41~S43, S51, S52, S71, S72: Steps

[0122] S331~S337: Steps

Claims

1. An image conversion method, applicable to an image conversion network, wherein the image conversion network is connected to a first database and comprises an image generator, a conversion discriminator, and a focus discriminator, and the first database stores a plurality of unconverted images, the image conversion method comprising: The image generator performs an operation based on one of the plurality of unconverted images and at least one focus information (x, y) of one of the plurality of unconverted images to generate a converted image; The transformation discriminator performs an operation on one of the plurality of untransformed images and the transformed image to generate a transformation discriminant value; The image generator calculates a first generation loss function value according to the conversion identification value, and updates a parameter of the image generator according to the first generation loss function value; The focus identifier performs an operation based on one of the plurality of unconverted images, the converted image and the at least one focus information (x, y) to generate at least one focus identification value, the at least one focus identification value includes at least one second focus identification value, and the step of the focus identifier performing an operation based on one of the plurality of unconverted images and the at least one focus information (x, y) to generate the at least one second focus identification value comprises: The focus identifier generates at least one unconverted sub-image according to the at least one focus information (x, y) and one of the plurality of unconverted images; The focus identifier generates at least one conversion sub-image according to the at least one focus information (x, y) and the conversion image; and The focus identifier performs an operation according to the at least one unconverted sub-image and the at least one converted sub-image to generate the at least one second focus identification value; and The image generator calculates a second generation loss function value according to the at least one focus identification value, and updates the parameters of the image generator according to the second generation loss function value. The conversion identification value and the at least one focus identification value are image difference degree information or image similarity degree information.

2. The image conversion method according to claim 1, wherein: The image conversion network is also connected to a second database storing a plurality of conversion template images, and the image conversion method further comprises: The transformation identifier calculates another transformation identification value based on one of the plurality of untransformed images and one of the plurality of transformed template images; and The conversion discriminator calculates a conversion identification loss function value according to a plurality of conversion identification values, and updates a parameter of the conversion discriminator according to the conversion identification loss function value.

3. The image conversion method according to claim 1, further comprising: The focus discriminator calculates a focus discrimination loss function value based on two of the plurality of unconverted images, wherein the focus discrimination loss function value indicates how close the image conversion network is to an ideal state; as well as The focus discriminator updates the parameters of the focus discriminator according to the focus discriminator loss function value.

4. The image conversion method according to claim 1, further comprising: The image generator receives a test image and at least one test focus information (x, y); as well as The image generator performs conversion on the test image according to the updated parameters and the at least one test focus information (x, y) to generate a converted test image.

5. The image conversion method according to claim 1, wherein: The step of performing an operation according to one of the plurality of unconverted images, the converted image and the at least one focus information (x, y) to generate the at least one focus identification value comprises: The focus identifier performs an operation based on one of the plurality of unconverted images and the converted image to generate a first focus identification value; and The focus identifier performs an operation according to one of the plurality of unconverted images and the at least one focus information (x, y) to generate the at least one second focus identification value, and The image generator calculates the second generation loss function value according to the at least one focus identification value, including: The image generator calculates the second generation loss function value according to the first focus identification value and the at least one second focus identification value.

6. The image conversion method according to claim 1, further comprising determining whether the first generation loss function value and the second generation loss function value are both less than or equal to a preset threshold value, and if either of the first generation loss function value and the second generation loss function value is greater than the preset threshold value, performing the steps: The image generator performs an operation according to one of the plurality of unconverted images and at least one focus information (x, y) of one of the plurality of unconverted images to generate a converted image; The transformation discriminator performs an operation on one of the plurality of untransformed images and the transformed image to generate a transformation discriminant value; The image generator calculates a first generation loss function value according to the conversion identification value, and updates a parameter of the image generator according to the first generation loss function value; The focus identifier performs an operation according to one of the plurality of unconverted images, the converted image and the at least one focus information (x, y) to generate at least one focus identification value; and The image generator calculates a second generation loss function value according to the at least one focus identification value, and updates the parameters of the image generator according to the second generation loss function value.

7. The image conversion method according to claim 2, further comprising: The focus discriminator calculates a focus discriminator loss function value based on two of the plurality of unconverted images; as well as The focus discriminator updates the parameters of the focus discriminator according to the focus discriminator loss function value.

8. The image conversion method according to claim 7, further comprising determining whether the first generation loss function value, the second generation loss function value, the conversion identification loss function value, and the focus identification loss function value are all less than or equal to a preset threshold value, and when any one of the first generation loss function value, the second generation loss function value, the conversion identification loss function value, and the focus identification loss function value is greater than the preset threshold value, performing the steps: The focus discriminator calculates a focus discriminator loss function value based on two of the plurality of unconverted images; and The focus discriminator updates the parameters of the focus discriminator according to the focus discriminator loss function value.

9. An image conversion device connected to a first database storing a plurality of unconverted images, the image conversion device comprising: an image generator configured to perform an operation and generate a converted image according to one of the plurality of unconverted images and at least one focus information (x, y) of one of the plurality of unconverted images; a transformation discriminator, signally connected to the image generator and configured to perform an operation and generate a transformation identification value based on one of the plurality of untransformed images and the transformed image; as well as A focus discriminator is signally connected to the image generator and is configured to perform an operation based on one of the plurality of unconverted images, the converted image and the at least one focus information (x, y) to generate at least one focus discrimination value, the at least one focus discrimination value including at least one second focus discrimination value, and wherein the focus discriminator is configured to further perform: generating at least one unconverted sub-image according to the at least one focus information (x, y) and one of the plurality of unconverted images; generating at least one conversion sub-image according to the at least one focus information (x, y) and the conversion image; as well as performing an operation according to the at least one unconverted sub-image and the at least one converted sub-image to generate the at least one second focus identification value, Among other things, the image generator is configured to also perform: Calculating a first generation loss function value according to the conversion identification value, and updating the parameters of the image generator according to the first generation loss function value, and calculating a second generation loss function value according to the at least one focus identification value, and updating the parameters of the image generator according to the second generation loss function value, The conversion identification value and the at least one focus identification value are image difference degree information or image similarity degree information.

10. The image conversion device according to claim 9, further connected to a second database storing a plurality of conversion template images, wherein: The conversion identifier of the image conversion device is configured to further perform: calculating another conversion identification value based on one of the plurality of unconverted images and one of the plurality of converted template images; and A conversion identification loss function value is calculated based on the plurality of conversion identification values, and a parameter of the conversion identifier is updated based on the conversion identification loss function value.

11. The image conversion apparatus of claim 9, wherein the focus identifier is configured to further perform: Calculating a focus discrimination loss function value based on two of the plurality of unconverted images, wherein the focus discrimination loss function value indicates how close the image conversion device is to an ideal condition; and The parameters of the focus discriminator are updated according to the focus discriminator loss function value.

12. The image conversion apparatus of claim 9, wherein the focus identifier is configured to further perform: performing an operation based on one of the plurality of unconverted images and the converted image to generate a first focus identification value; and performing an operation according to one of the plurality of unconverted images and the at least one focus information (x, y) to generate the at least one second focus identification value, in, The image generator calculates the second generation loss function value according to the first focus identification value and the at least one second focus identification value.

13. The image conversion device according to claim 9, wherein the image generator is configured to further execute: receiving a test image and at least one test focus information (x, y); and According to the updated parameters of the image generator and the at least one test focus information (x, y), the test image is transformed to generate a transformed test image.

Images