Image reconstruction model training method, device and electronic device

Abstract

The disclosed embodiments disclose a training method, device and electronic device for an image reconstruction model. A specific implementation of the method includes: obtaining an initial panoramic training sample image, determining at least one enhanced panoramic training sample image based on the initial panoramic training sample image, and obtaining a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; extracting an initial object sample training image from the initial panoramic training sample image, and obtaining an object sample image training sample set; based on a preset loss function, using the panoramic image training sample set and the object sample image training sample set to train a first image processing model and a second image processing model. The first image processing model and the second image processing model can be trained together, and based on the output of the converged target image processing model, a more realistic reconstructed image can be obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a training method, device and electronic device for an image reconstruction model. Background Art

[0002] Image reconstruction technology is a technology that obtains the shape information of three-dimensional objects through digital processing of external measured data of objects. It has been widely used in various fields such as medical imaging and industrial non-destructive testing. In the process of training the image reconstruction model, it is often necessary to determine the corresponding loss function to facilitate the reconstruction of images close to the real object. Summary of the invention

[0003] This disclosure section is provided to introduce concepts in a brief form, which will be described in detail in the detailed description section below. This disclosure section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] The embodiments of the present disclosure provide a method, device and electronic device for training an image reconstruction model.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for training an image reconstruction model, the method comprising: acquiring an initial panoramic training sample image, determining at least one enhanced panoramic training sample image based on the initial panoramic training sample image, and obtaining a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; extracting an initial object sample training image from the initial panoramic training sample image, and obtaining an object sample image training sample set; based on a preset loss function, using the panoramic image training sample set and the object sample image training sample set to train a first image processing model and a second image processing model, and obtaining a trained first target image processing model and a second target image processing model.

[0006] In a second aspect, an embodiment of the present disclosure provides a training device for an image reconstruction model, the device comprising: a determination module, used to acquire an initial panoramic training sample image, determine at least one enhanced panoramic training sample image based on the initial panoramic training sample image, and obtain a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; an extraction module, used to extract an initial object sample training image from the initial panoramic training sample image, and obtain an object sample image training sample set; a training module, used to train a first image processing model and a second image processing model based on a preset loss function using the panoramic image training sample set and the object sample image training sample set, and obtain a trained first target image processing model and a second target image processing model.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the image reconstruction model described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image reconstruction model training method described in the first aspect above.

[0009] The training method, device and electronic device of the image reconstruction model provided by the embodiments of the present disclosure obtain an initial panoramic training sample image, determine at least one enhanced panoramic training sample image according to the initial panoramic training sample image, and obtain a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; extract an initial object sample training image from the initial panoramic training sample image to obtain an object sample image training sample set; based on a preset loss function, train a first image processing model and a second image processing model using the panoramic image training sample set and the object sample image training sample set to obtain a trained first target image processing model and a second target image processing model. The first image processing model and the second image processing model can be trained together, and a more realistic reconstructed image can be obtained based on the output of the converged target image processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 A flowchart of an embodiment of a method for training an image reconstruction model according to the present disclosure;

[0012] Figure 2 A schematic diagram of a flow chart of a training image processing model involved in the present disclosure;

[0013] Figure 3 A schematic structural diagram of an embodiment of a training device for an image reconstruction model according to the present disclosure;

[0014] Figure 4 An exemplary system architecture in which the training method of the image reconstruction model of one embodiment of the present disclosure can be applied;

[0015] Figure 5The diagram is a schematic diagram of a basic structure of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0018] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0023] Please refer to Figure 1 , which shows a flowchart of an embodiment of a training method for an image reconstruction model according to the present disclosure, such as Figure 1 As shown, the training method of the image reconstruction model includes the following steps 101 to 103.

[0024] Step 101, obtaining an initial panoramic training sample image, determining at least one enhanced panoramic training sample image according to the initial panoramic training sample image, and obtaining a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image;

[0025] In some application scenarios, for example, the above-mentioned initial panoramic training sample images can be collected by a panoramic camera, a mobile phone, or other device that can actually take panoramic images. The initial panoramic training sample images here may include, for example, landscape panoramic images, indoor panoramic images, and the like. Each initial panoramic training sample image may include multiple object images. For example, the initial landscape panoramic training sample image may include object images corresponding to objects such as mountains, houses, and trees; the initial indoor panoramic training sample image may include object images corresponding to objects such as televisions, sofas, and coffee tables. In these application scenarios, the initial panoramic training sample images may, for example, be low dynamic range images (Low-Dynamic Range, referred to as LDR images).

[0026] In these application scenarios, at least one enhanced panoramic training sample image may be determined according to the initial panoramic training sample image.

[0027] The enhanced panoramic training sample image may include, for example, an image obtained by performing a data enhancement operation on the initial panoramic training sample image. The data enhancement operation here may include, for example, distorting, decolorizing or deilluminating the initial panoramic training sample image, and other operations that may substantially change part of the data of the initial panoramic training sample image.

[0028] After determining a plurality of enhanced panoramic training sample images, the plurality of enhanced panoramic training sample images can be sorted to obtain a panoramic image training sample set. Afterwards, a panoramic image training sample can be obtained from the panoramic image training sample set to train an image processing model for processing panoramic images. In this way, a corresponding enhanced panoramic training sample image can be determined by a certain initial panoramic training sample image, so that the sample images in the panoramic image training sample set are more abundant.

[0029] Step 102, extracting initial object sample training images from the initial panoramic training sample images to obtain an object sample image training sample set;

[0030] For each initial panoramic training sample image, object images corresponding to each object can be extracted based on the objects included in the initial panoramic training sample image, and the extracted object images can be regarded as the initial object sample training images mentioned above. For example, for an initial panoramic training sample image including a television, a sofa, and a coffee table, an initial object sample training image including only the television, an initial object sample training image including only the sofa, and an initial object sample training image including only the coffee table can be extracted.

[0031] After obtaining a plurality of initial object sample training images, the plurality of initial object sample training images may be sorted to obtain an object sample image training sample set. Thereafter, an initial object sample training image may be obtained from the object sample image training sample set to train an image processing model for processing object images.

[0032] Step 103, based on a preset loss function, the panoramic image training sample set and the object sample image training sample set are used to train the first image processing model and the second image processing model to obtain the trained first target image processing model and the second target image processing model.

[0033] The preset loss function may include, for example, a mean absolute error loss function (also called an L1 loss function). The first image processing model and the second image processing model may include, for example, an image segmentation network U-net.

[0034] In some application scenarios, the first image processing model and the second image processing model can be trained based on a preset loss function to make the two image processing models converge and obtain the corresponding first target image processing model and second target image processing model. After that, the first target image processing model can be used to process the panoramic image to reconstruct the target panoramic image with less loss; the second target image processing model can be used to process the object image to reconstruct the target object image with less loss.

[0035] In this embodiment, an initial panoramic training sample image is first obtained, and at least one enhanced panoramic training sample image is determined according to the initial panoramic training sample image to obtain a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; then, an initial object sample training image is extracted from the initial panoramic training sample image to obtain an object sample image training sample set; finally, based on a preset loss function, the panoramic image training sample set and the object sample image training sample set are used to train the first image processing model and the second image processing model to obtain the trained first target image processing model and the second target image processing model. The first image processing model and the second image processing model can be trained together, and a more realistic reconstructed image can be obtained based on the output of the converged target image processing model.

[0036] In some optional implementations, the preset loss function includes a first preset loss sub-function and a second preset loss sub-function. The first loss sub-function can be used in the training process of the first image processing model to calculate the loss between the initial panoramic training sample image or the enhanced panoramic training sample image and the corresponding output result. The second loss sub-function can be used in the training process of the second image processing model to calculate the loss between the initial object sample training image and the corresponding output result.

[0037] Please refer to Figure 2 , which shows a flow chart of the training image processing model involved in the present disclosure, such as Figure 2 As shown, the above step 103 may include the following sub-steps:

[0038] Sub-step 1031, based on the first preset loss sub-function, using the panoramic image training sample set and the output results obtained by inputting the initial panoramic training sample image and the enhanced panoramic training sample image into the first image processing model respectively, determining the first loss sub-function value;

[0039] In some application scenarios, the first loss sub-function value may be determined based on the output result obtained by inputting the initial panoramic training sample image into the first image processing model and the output result obtained by inputting the enhanced panoramic training sample image into the first image processing model. For example, the output result obtained by inputting the initial panoramic training sample image A and the enhanced panoramic training sample image B into the first image processing model may be determined first, and then the loss calculation may be performed based on the output result and the initial panoramic training sample image A and the enhanced panoramic training sample image B using the first preset loss sub-function to determine the above-mentioned first loss sub-function value.

[0040] Sub-step 1032: adjusting the model parameters of the first image processing model based on the first loss sub-function value to obtain the first target image processing model;

[0041] In the above application scenario, the model parameters of the first image processing model can be adjusted in the direction of reducing the value of the first loss sub-function by inputting multiple initial panoramic training samples or enhancing panoramic training sample images to obtain the first target image processing model.

[0042] Sub-step 1033, based on the second preset loss sub-function, using the panoramic image training sample set, the object sample image training sample set, and the output results of inputting the initial object sample training image into the second image processing model and inputting the initial panoramic training sample image into the first image processing model, respectively, to determine the second loss sub-function value;

[0043] In some application scenarios, the second loss sub-function value may be determined based on the output results obtained by inputting the initial object sample training image into the second image processing model and the initial panoramic training sample image into the first image processing model. For example, the output result a' obtained by inputting the initial object sample training image a into the second image processing model and the output result A' obtained by inputting the initial panoramic training sample image A into the first image processing model may be determined first, and then the output result a' and the initial object sample training image a, the output result A' and the initial panoramic training sample image A may be used to perform a loss function using a second preset loss function to determine the above-mentioned second loss sub-function value.

[0044] Sub-step 1034, based on the second loss sub-function value, adjust the model parameters of the second image processing model to obtain the second target image processing model.

[0045] In the above application scenario, the model parameters of the second image processing model can also be adjusted in the direction of reducing the value of the second loss sub-function by inputting multiple initial panoramic training sample images into the first image processing model and correspondingly inputting the same number of initial object sample training images into the second image processing model to obtain the second target image processing model.

[0046] In some optional implementations, the step 101 of determining at least one enhanced panoramic training sample image based on the initial panoramic training sample image to obtain a panoramic image training sample set includes: using the first depth data corresponding to the initial panoramic training sample image as a distortion reference, distorting the initial panoramic training sample image with the first target object as a reference point, to obtain a first enhanced panoramic training sample image with the second target object as a reference point; wherein the first depth data is the output result obtained by the first image processing model based on the initial panoramic training sample image.

[0047] In some application scenarios, the initial panoramic training sample image can be input into the first image processing model to obtain the first depth data. Then, the initial panoramic training sample image can be distorted according to the first depth data, so that the distorted initial panoramic training sample image with the first target object as the reference point can become the first enhanced panoramic training sample image with the second target object as the reference point. Here, the panoramic training sample image with a certain target object as the reference point can be regarded as a panoramic image obtained by taking a photo with the target object as the base point. For example, the initial panoramic training sample image includes a TV, a coffee table and a sofa, and the initial panoramic training sample image is a panoramic image taken with the camera placed on the coffee table; after the initial panoramic training sample image is input into the first image processing model, the first depth data based on the coffee table can be obtained. Then, the initial panoramic training sample image can be distorted according to the first depth data to obtain the first enhanced panoramic training sample image regarded as being taken with the camera placed on the sofa.

[0048] By distorting the initial panoramic training sample image, a first enhanced panoramic training sample image can be obtained, which increases the number of training sample images and enriches the panoramic image training sample set.

[0049] In some optional implementations, the step 101 of determining at least one enhanced panoramic training sample image based on the initial panoramic training sample image to obtain a panoramic image training sample set also includes: using the second depth data corresponding to the first enhanced panoramic training sample image as a distortion reference, warping the first enhanced panoramic training sample image to obtain a second enhanced panoramic training sample image with a third target object as a reference point; wherein the third target object is the same object as the first target object; and the second depth data is the output result obtained by the first image processing model based on the first enhanced panoramic training sample image.

[0050] In some application scenarios, the first enhanced panoramic training sample image can be input into the first image processing model to obtain the second depth data. Then, the first enhanced panoramic training sample image can be distorted according to the second depth data, so that the distorted first enhanced panoramic training sample image with the second target object as the reference point can become the second enhanced panoramic training sample image with the third target object as the reference point. In these application scenarios, the third target object can be the same object as the first target object. For example, in the above-mentioned first enhanced panoramic training sample image including a TV, a coffee table and a sofa, the first enhanced panoramic training sample image with the sofa as the reference point can be distorted back to the image with the coffee table as the reference point (that is, the second enhanced panoramic training sample image) with the second depth data as the distortion reference. In this way, in the subsequent training process of the image processing model, the training can be carried out in the direction of making the depth data of each pixel point of the second enhanced panoramic training sample image and the initial panoramic training sample image the same.

[0051] By distorting the first enhanced panoramic training sample image, a second enhanced panoramic training sample image can be obtained, thereby increasing the number of training sample images and further enriching the panoramic image training sample set.

[0052] In some optional implementations, the above step 1031 may include the following steps 201 to 202.

[0053] Step 201, inputting the initial panoramic training sample image, the first enhanced panoramic training sample image and the second enhanced panoramic training sample image into the first image processing model respectively, to obtain corresponding first output result, second output result and third output result;

[0054] The first output result may include first depth data, first illumination data, first color data and / or first normal data. That is, the initial panoramic training sample image may be input into the first image processing model, and the first image processing model may output the first depth data, first illumination data, first color data and / or first normal data corresponding to the initial panoramic training sample image.

[0055] The second output result may include second depth data, second illumination data, second color data and / or second normal data. That is, the first enhanced panoramic training sample image may be input into the first image processing model, and the first image processing model may output the second depth data, second illumination data, second color data and / or second normal data corresponding to the first enhanced panoramic training sample image.

[0056] The third output result may include third depth data, third illumination data, third color data and / or third normal data. That is, the second enhanced panoramic training sample image may be input into the first image processing model, and the first image processing model may output the third depth data, third illumination data, third color data and / or third normal data corresponding to the second enhanced panoramic training sample image.

[0057] Step 202: determine the first loss sub-function value based on the first output result, the second output result and / or the third output result.

[0058] After the first image processing model outputs corresponding output results, the first loss sub-function value can be determined based on these output results.

[0059] In this embodiment, the first image processing model is trained using the initial panoramic training sample image and the first enhanced panoramic training sample image and the second enhanced panoramic training sample image obtained based on the initial panoramic training sample image. Based on the converged first target image processing model, a reconstructed image that is closer to the input image can be output.

[0060] In some optional implementations, the first output result includes first depth data, first normal data, and first color data; the third output result includes third depth data, third normal data, and third color data; and the above-mentioned step 202 may include step A: performing loss calculation on the first depth data, first normal data, first color data, third depth data, third normal data, and third color data respectively to determine the first loss sub-function value.

[0061] In some application scenarios, the first depth data and the third depth data can be subjected to loss calculation; the first normal data and the third normal data can be subjected to loss calculation; the first color data and the third color data can be subjected to loss calculation to determine the corresponding first loss sub-function value. In these application scenarios, since the third output result is obtained based on the second enhanced panoramic training sample image, the third target object in the second enhanced panoramic training sample image is the same as the first target object. Therefore, it is hoped that the first image processing model can be trained in the direction of making the depth data of each pixel point of the second enhanced panoramic training sample image and the initial panoramic training sample image as similar as possible. Subsequently, this can be regarded as a constraint on the depth data of the first image processing model in the process of processing panoramic images. At the same time, the normal data and color data are constrained so that consistency can be maintained between different data.

[0062] In some optional implementations, the first output result includes first normal data and first color data; the second output result includes second illumination data; and the above step 202 may include:

[0063] Step B1, rendering a predicted second target object image based on the first normal data, the first color data, and the second illumination data;

[0064] The first image processing model can render a new three-dimensional image of the second target object (ie, predict the second target object image) through the first normal data, the first color data, and the second illumination data.

[0065] Step B2: determining the first loss sub-function value based on the predicted second target object image and the second target object image.

[0066] After the first image processing model renders the predicted second target object image, the predicted second target object image can be compared with the real second target object image to determine the difference between the two, and then the first loss sub-function value can be determined.

[0067] In this way, through the above steps B1 and B2, the normal loss data, color loss data and illumination loss data of the first image processing model can be constrained, which is conducive to outputting a reconstructed image that is closer to a real panoramic image.

[0068] In some optional implementations, the first output result includes first illumination data, and the above step 202 may include step C: performing loss calculation on the first illumination data and the real illumination data corresponding to the initial panoramic training sample image to determine the first loss sub-function value.

[0069] After the first image processing model outputs the first illumination data based on the input initial panoramic training sample image, the loss of the first illumination data and the real illumination data corresponding to the initial panoramic training sample image can be calculated to determine the loss of the first illumination data relative to the real illumination data, so as to facilitate training the first image processing model in a direction where the output approaches the real illumination data. In some application scenarios, the above-mentioned real illumination data can be obtained through a high dynamic range image (High-Dynamic Range, referred to as HDR image) of the initial panoramic training sample image.

[0070] In some optional implementations, the first output result includes first normal data, and the above step 202 may include step D: performing loss calculation on the first normal data and the real normal data corresponding to the initial panoramic training sample image to determine the first loss sub-function value.

[0071] After the first image processing model outputs the first normal data based on the input initial panoramic training sample image, the loss of the first normal data and the real normal data corresponding to the initial panoramic training sample image can be calculated to determine the loss of the first normal data relative to the real normal data, so as to facilitate training the first image processing model in a direction where the output approaches the real normal data. In some application scenarios, the above-mentioned real normal data can also be obtained, for example, through a high dynamic range image of the initial panoramic training sample image.

[0072] In some application scenarios, the above-mentioned multiple steps of determining the value of the first loss sub-function (for example, the above-mentioned step A, step C, step D, etc.) can also be performed. By constraining the depth loss data, normal loss data, color loss data, etc. for multiple times, the first image processing model can be trained in the direction in which the value of the first loss sub-function becomes smaller. In these application scenarios, the obtained depth loss data, normal loss data, color loss data, etc. can be assigned corresponding weights according to actual conditions to obtain the corresponding weighted sums. For example, a heavier weight of 35% can be assigned to the depth loss data.

[0073] In some optional implementations, the above step 1033 may include the following steps 301 to 303.

[0074] Step 301, inputting the initial panoramic training sample image and the first enhanced panoramic training sample image into the first image processing model respectively to obtain a corresponding first output result and a second output result;

[0075] The implementation process and effect of the above step 301 may be similar to or the same as the relevant parts of the above step 201, and will not be repeated here.

[0076] Step 302, inputting the initial object sample training image into the second image processing model to obtain a fourth output result;

[0077] The fourth output result may include fourth depth data, fourth illumination data, fourth color data and / or fourth normal data. That is, the initial object sample training image may be input into the second image processing model, and the second image processing model may output fourth depth data, fourth illumination data, fourth color data and / or fourth normal data corresponding to the initial object training sample image.

[0078] Step 303: determine the second loss sub-function value based on the first output result, the second output result and / or the fourth output result.

[0079] After the first image processing model and the second image processing model output corresponding output results respectively, the second loss sub-function value can be determined based on these output results.

[0080] In this embodiment, the second image processing model is trained by using the output results between the two image processing models, and a reconstructed image that is closer to the input image can be output based on the converged second target image processing model.

[0081] In some optional implementations, the first output result includes first depth data, first normal data, and first color data; the second output result includes second illumination data; the fourth output result includes fourth depth data, fourth normal data, fourth color data, and fourth illumination data; and the above-mentioned step 303 may include step a: performing loss calculation on the first depth data, first normal data, first color data, second illumination data, fourth depth data, fourth normal data, fourth color data, and fourth illumination data, respectively, to determine the second loss sub-function value.

[0082] In some application scenarios, the first depth data and the fourth depth data may be subjected to loss calculation; the first normal data and the fourth normal data may be subjected to loss calculation; the first color data and the fourth color data may be subjected to loss calculation; the second illumination data and the fourth illumination data may be subjected to loss calculation to determine the corresponding second loss sub-function value. In these application scenarios, it should be noted that the first depth data, the first normal data, and the first color data should be data corresponding to the corresponding object image included in the initial panoramic training sample image, so that they can be consistent with the illumination data of the corresponding object image, so that the first output result and the second output result here can correspond to the same target object image. For example, if the fourth depth data, the fourth normal data, and the fourth color data correspond to house A, then the first depth data, the first normal data, and the first color data may correspond to house A in the initial panoramic training sample image.

[0083] Through the above step a, the depth loss data, normal loss data, color loss data and illumination loss data of the second image processing model can be constrained, which is conducive to outputting a reconstructed object image that is closer to the target object image.

[0084] In some optional implementations, the fourth output result includes fourth normal data, fourth color data, and fourth illumination data, and the above step 303 may include:

[0085] Step b1, rendering a predicted first target object image based on the fourth normal data, the fourth color data, and the fourth illumination data;

[0086] The second image processing model can render a new three-dimensional image of the first target object (ie, predict the first target object image) through the fourth normal data, the fourth color data, and the fourth illumination data.

[0087] Step b2: determining the second loss sub-function value based on the predicted first target object image and the first target object image.

[0088] After the second image processing model renders the predicted first target object image, the predicted first target object image can be compared with the actual first target object image to determine the difference between the two, and then the second loss sub-function value can be determined.

[0089] In this way, through the above steps b1 and b2, the normal loss data, color loss data and illumination loss data of the second image processing model can be constrained, which is conducive to outputting a reconstructed object image that is closer to the real image of the target object.

[0090] In some optional implementations, the fourth output result includes fourth illumination data; and the above-mentioned step 303 may include step c: performing loss calculation on the fourth illumination data and the real illumination data corresponding to the initial object sample training image to determine the second loss sub-function value.

[0091] After the second image processing model outputs the fourth illumination data based on the input initial object sample training image, the loss of the fourth illumination data and the real illumination data corresponding to the initial object sample training image can be calculated to determine the loss of the fourth illumination data relative to the real illumination data, so as to facilitate training the second image processing model in a direction where the output approaches the real illumination data. In some application scenarios, the above-mentioned real illumination data can also be obtained through a high dynamic range image of the initial object sample training image.

[0092] In some optional implementations, the fourth output result includes fourth normal data; and the above-mentioned step 303 may include step d: performing loss calculation on the fourth normal data and the real normal data corresponding to the initial object sample training image to determine the second loss sub-function value.

[0093] After the second image processing model outputs the fourth normal data based on the input initial object sample training image, the fourth normal data can be subjected to loss calculation with respect to the real normal data corresponding to the initial object sample training image to determine the loss of the fourth normal data relative to the real normal data, so as to facilitate training the second image processing model in a direction where the output approaches the real normal data. In some application scenarios, the real normal data can also be obtained, for example, through a high dynamic range image of the initial object sample training image.

[0094] In some application scenarios, after executing multiple steps to determine the value of the second loss sub-function (for example, steps a, c, d, etc. above), corresponding weights can be assigned to the depth loss data, normal loss data, or color loss data according to actual conditions, so that the determined second loss sub-function is more suitable for image processing in the actual situation.

[0095] Please refer to Figure 3 , which shows a schematic diagram of the structure of an embodiment of a training device for an image reconstruction model according to the present disclosure, such as Figure 3 As shown, the training device of the image reconstruction model includes a determination module, an extraction module and a training module. Among them, the determination module 301 is used to obtain an initial panoramic training sample image, determine at least one enhanced panoramic training sample image according to the initial panoramic training sample image, and obtain a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; the extraction module 302 is used to extract an initial object sample training image from the initial panoramic training sample image to obtain an object sample image training sample set; the training module 303 is used to train the first image processing model and the second image processing model using the panoramic image training sample set and the object sample image training sample set based on a preset loss function to obtain a trained first target image processing model and a second target image processing model.

[0096] It should be noted that the specific processing of the determination module 301, the extraction module 302 and the training module 303 of the training device for the image reconstruction model and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of steps 101 to 103 in the corresponding embodiment are not repeated here.

[0097] In some optional implementations of the present embodiment, the training module 303 is further used to: determine a first loss sub-function value based on a first preset loss sub-function, using a panoramic image training sample set and the output results obtained by respectively inputting the initial panoramic training sample image and the enhanced panoramic training sample image into the first image processing model; adjust the model parameters of the first image processing model based on the first loss sub-function value to obtain the first target image processing model; determine a second loss sub-function value based on a second preset loss sub-function, using a panoramic image training sample set, an object sample image training sample set, and the output results obtained by inputting the initial object sample training image into the second image processing model and inputting the initial panoramic training sample image into the first image processing model; adjust the model parameters of the second image processing model based on the second loss sub-function value to obtain the second target image processing model.

[0098] In some optional implementations of the present embodiment, the determination module 301 is further used to: use the first depth data corresponding to the initial panoramic training sample image as a distortion reference, distort the initial panoramic training sample image with the first target object as a reference point, and obtain a first enhanced panoramic training sample image with the second target object as a reference point; wherein the first depth data is the output result obtained by the first image processing model based on the initial panoramic training sample image.

[0099] In some optional implementations of the present embodiment, the determination module 301 is further used to: use the second depth data corresponding to the first enhanced panoramic training sample image as a distortion reference, distort the first enhanced panoramic training sample image, and obtain a second enhanced panoramic training sample image with a third target object as a reference point; wherein the third target object is the same object as the first target object; and the second depth data is the output result obtained by the first image processing model based on the first enhanced panoramic training sample image.

[0100] In some optional implementations of the present embodiment, the training module 303 is further used to: input the initial panoramic training sample image, the first enhanced panoramic training sample image, and the second enhanced panoramic training sample image into the first image processing model, respectively, to obtain the corresponding first output result, second output result, and third output result; determine the first loss sub-function value based on the first output result, the second output result and / or the third output result.

[0101] In some optional implementations of this embodiment, the first output result includes first depth data, first normal data, and first color data; the third output result includes third depth data, third normal data, and third color data; and the training module 303 is further used to: perform loss calculations on the first depth data, first normal data, first color data, third depth data, third normal data, and third color data, respectively, to determine the first loss sub-function value.

[0102] In some optional implementations of this embodiment, the first output result includes first normal data and first color data; the second output result includes second illumination data; and the training module 303 is further used to: render a predicted second target object image based on the first normal data, the first color data, and the second illumination data; and determine the first loss sub-function value based on the predicted second target object image and the second target object image.

[0103] In some optional implementations of this embodiment, the first output result includes first illumination data, and the training module 303 is further used to: perform loss calculation on the first illumination data and the real illumination data corresponding to the initial panoramic training sample image to determine the first loss sub-function value.

[0104] In some optional implementations of this embodiment, the first output result includes first normal data, and the training module 303 is further used to: perform loss calculation on the first normal data and the real normal data corresponding to the initial panoramic training sample image to determine the first loss sub-function value.

[0105] In some optional implementations of the present embodiment, the training module 303 is further used to: input the initial panoramic training sample image and the first enhanced panoramic training sample image into the first image processing model respectively to obtain the corresponding first output result and second output result; input the initial object sample training image into the second image processing model to obtain a fourth output result; and determine the second loss sub-function value based on the first output result, the second output result and / or the fourth output result.

[0106] In some optional implementations of this embodiment, the first output result includes first depth data, first normal data, and first color data; the second output result includes second illumination data; the fourth output result includes fourth depth data, fourth normal data, fourth color data, and fourth illumination data; and the training module 303 is further used to: perform loss calculations on the first depth data, first normal data, first color data, second illumination data, fourth depth data, fourth normal data, fourth color data, and fourth illumination data, respectively, to determine the second loss sub-function value.

[0107] In some optional implementations of this embodiment, the fourth output result includes fourth normal data, fourth color data, and fourth illumination data, and the training module 303 is further used to: render a predicted first target object image based on the fourth normal data, fourth color data, and fourth illumination data; and determine the second loss sub-function value based on the predicted first target object image and the first target object image.

[0108] In some optional implementations of this embodiment, the fourth output result includes fourth illumination data; and the training module 303 is further used to: perform loss calculation on the fourth illumination data and the real illumination data corresponding to the initial object sample training image to determine the second loss sub-function value.

[0109] In some optional implementations of this embodiment, the fourth output result includes fourth normal data; and the training module 303 is further used to: perform loss calculation on the fourth normal data and the real normal data corresponding to the initial object sample training image to determine the second loss sub-function value.

[0110] Please refer to Figure 4 , which shows an exemplary system architecture in which the training method of the image reconstruction model of an embodiment of the present disclosure can be applied.

[0111] like Figure 4 As shown, the system architecture may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 is used to provide a medium for a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The above-mentioned terminal devices and servers may communicate using any currently known or future developed network protocols such as HTTP (HyperText Transfer Protocol), and may be interconnected with digital data communications (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., Ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0112] The terminal devices 401, 402, 403 can interact with the server 405 through the network 404 to receive or send messages, etc. Various client applications can be installed on the terminal devices 401, 402, 403, such as video publishing applications, search applications, and news information applications.

[0113] Terminal devices 401, 402, 403 can be hardware or software. When terminal devices 401, 402, 403 are hardware, they can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 401, 402, 403 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0114] Server 405 can be a server that can provide various services, such as receiving image acquisition requests sent by terminal devices 401, 402, 403, analyzing and processing the image acquisition requests, and sending the analysis and processing results (such as image data corresponding to the above acquisition requests) to terminal devices 401, 402, 403.

[0115] It should be noted that the training method of the image reconstruction model provided in the embodiment of the present disclosure can be executed by a server or by a terminal device. Accordingly, the training device of the image reconstruction model can be set in a server or in a terminal device.

[0116] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0117] Reference below Figure 5 , which shows an electronic device (eg, Figure 4 A schematic diagram of the structure of the server in FIG. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0118] like Figure 5As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0119] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0120] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0121] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0122] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0123] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains an initial panoramic training sample image, determines at least one enhanced panoramic training sample image based on the initial panoramic training sample image, and obtains a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; extracts an initial object sample training image from the initial panoramic training sample image to obtain an object sample image training sample set; and based on a preset loss function, uses the panoramic image training sample set and the object sample image training sample set to train a first image processing model and a second image processing model to obtain a trained first target image processing model and a second target image processing model.

[0124] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0125] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the unit itself in some cases. For example, the determination module 301 may also be described as "a module for obtaining an initial panoramic training sample image, determining at least one enhanced panoramic training sample image based on the initial panoramic training sample image, and obtaining a panoramic image training sample set".

[0127] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0128] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0130] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0131] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A training method for an image reconstruction model, characterized in that: include: Acquire an initial panoramic training sample image, determine at least one enhanced panoramic training sample image according to the initial panoramic training sample image, and obtain a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; wherein the enhanced panoramic training sample image includes an image obtained after performing a data enhancement operation on the initial panoramic training sample image; Extracting initial object sample training images from initial panoramic training sample images to obtain an object sample image training sample set; Based on a preset loss function, the first image processing model and the second image processing model are trained using the panoramic image training sample set and the object sample image training sample set to obtain a trained first target image processing model and a trained second target image processing model; The preset loss function includes a first preset loss sub-function and a second preset loss sub-function, and Based on a preset loss function, the first image processing model and the second image processing model are trained using the panoramic image training sample set and the object sample image training sample set to obtain the trained first target image processing model and the second target image processing model, including: Inputting the initial panoramic training sample image, the first enhanced panoramic training sample image and the second enhanced panoramic training sample image into the first image processing model respectively to obtain a corresponding first output result, a second output result and a third output result; Inputting the initial object sample training image into the second image processing model to obtain a fourth output result; Determine a first loss sub-function value based on the first output result, the second output result and / or the third output result; Determine a second loss sub-function value based on the first output result, the second output result and / or the fourth output result; Based on the first loss sub-function value, adjusting the model parameters of the first image processing model to obtain the first target image processing model; Based on the second loss sub-function value, the model parameters of the second image processing model are adjusted to obtain the second target image processing model.

2. The method according to claim 1, characterized in that The step of determining at least one enhanced panoramic training sample image according to the initial panoramic training sample image to obtain a panoramic image training sample set comprises: Using the first depth data corresponding to the initial panoramic training sample image as a distortion reference, the initial panoramic training sample image with the first target object as a reference point is distorted to obtain a first enhanced panoramic training sample image with the second target object as a reference point; wherein the first depth data is the output result obtained by the first image processing model based on the initial panoramic training sample image.

3. The method according to claim 2, characterized in that The step of determining at least one enhanced panoramic training sample image according to the initial panoramic training sample image to obtain a panoramic image training sample set further comprises: Taking the second depth data corresponding to the first enhanced panoramic training sample image as a distortion reference, the first enhanced panoramic training sample image is distorted to obtain a second enhanced panoramic training sample image with a third target object as a reference point; wherein the third target object is the same object as the first target object; and the second depth data is the output result obtained by the first image processing model based on the first enhanced panoramic training sample image.

4. The method according to claim 1, characterized in that: The first output result includes first depth data, first normal data, and first color data; the third output result includes third depth data, third normal data, and third color data; as well as The determining the first loss sub-function value based on the first output result, the second output result and / or the third output result comprises: Losses are calculated for the first depth data, the first normal data, the first color data, the third depth data, the third normal data, and the third color data, respectively, to determine the first loss sub-function value.

5. The method according to claim 1, characterized in that The first output result includes first normal data and first color data; the second output result includes second illumination data; as well as The determining the first loss sub-function value based on the first output result, the second output result and / or the third output result comprises: Rendering a predicted second target object image based on the first normal data, the first color data, and the second illumination data; Based on the predicted second target object image and the second target object image, the first loss sub-function value is determined.

6. The method according to claim 1, characterized in that The first output result includes first illumination data, and The determining the first loss sub-function value based on the first output result, the second output result and / or the third output result comprises: A loss calculation is performed on the first illumination data and the real illumination data corresponding to the initial panoramic training sample image to determine the first loss sub-function value.

7. The method according to claim 1, characterized in that The first output result includes first normal data, and The determining the first loss sub-function value based on the first output result, the second output result and / or the third output result comprises: A loss calculation is performed on the first normal data and the real normal data corresponding to the initial panoramic training sample image to determine the first loss sub-function value.

8. The method according to claim 1, characterized in that The first output result includes first depth data, first normal data, and first color data; the second output result includes second illumination data; the fourth output result includes fourth depth data, fourth normal data, fourth color data, and fourth illumination data; as well as The determining the second loss sub-function value based on the first output result, the second output result and / or the fourth output result includes: Perform loss calculations on the first depth data, the first normal data, the first color data, the second illumination data, the fourth depth data, the fourth normal data, the fourth color data, and the fourth illumination data respectively to determine the second loss sub-function value.

9. The method according to claim 1, characterized in that: The fourth output result includes fourth normal data, fourth color data, fourth illumination data, and The determining the second loss sub-function value based on the first output result, the second output result and / or the fourth output result includes: Rendering a predicted first target object image based on the fourth normal data, the fourth color data, and the fourth illumination data; Based on the predicted first target object image and the first target object image, the second loss sub-function value is determined.

10. The method according to claim 1, characterized in that The fourth output result includes fourth illumination data; as well as The determining the second loss sub-function value based on the first output result, the second output result and / or the fourth output result includes: A loss calculation is performed on the fourth illumination data and the real illumination data corresponding to the initial object sample training image to determine the second loss sub-function value.

11. The method according to claim 1, characterized in that: The fourth output result includes fourth normal data; as well as The determining the second loss sub-function value based on the first output result, the second output result and / or the fourth output result includes: A loss calculation is performed on the fourth normal data and the real normal data corresponding to the initial object sample training image to determine the second loss sub-function value.

12. A training device for an image reconstruction model, characterized in that: include: A determination module is used to obtain an initial panoramic training sample image, determine at least one enhanced panoramic training sample image according to the initial panoramic training sample image, and obtain a panoramic image training sample set, wherein the initial panoramic training sample image includes at least one object image; wherein the enhanced panoramic training sample image includes an image obtained after performing a data enhancement operation on the initial panoramic training sample image; An extraction module is used to extract an initial object sample training image from the initial panoramic training sample image to obtain an object sample image training sample set; A training module, configured to train the first image processing model and the second image processing model using the panoramic image training sample set and the object sample image training sample set based on a preset loss function to obtain a trained first target image processing model and a trained second target image processing model; The preset loss function includes a first preset loss sub-function and a second preset loss sub-function, and The training module is used to input the initial panoramic training sample image, the first enhanced panoramic training sample image and the second enhanced panoramic training sample image into the first image processing model respectively to obtain the corresponding first output result, second output result and third output result; and input the initial object sample training image into the second image processing model to obtain a fourth output result; determine a first loss sub-function value based on the first output result, the second output result and / or the third output result; determine a second loss sub-function value based on the first output result, the second output result and / or the fourth output result; adjust the model parameters of the first image processing model based on the first loss sub-function value to obtain the first target image processing model; adjust the model parameters of the second image processing model based on the second loss sub-function value to obtain the second target image processing model.

13. An electronic device, characterized in that: include: one or more processors; A storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of claims 1-11.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images