Image processing method, device and electronic device

Abstract

The disclosed embodiment discloses an image processing method, device and electronic device. A specific implementation of the method includes: obtaining a panoramic image to be processed; generating a target panoramic image based on a predetermined brightness influence matrix, target illumination information and a color panoramic image corresponding to the panoramic image to be processed; wherein the color panoramic image is obtained by removing illumination from the panoramic image to be processed. A target panoramic image with a target illumination effect can be reconstructed to enhance the user's viewing experience.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to an image processing method, device and electronic device. Background Art

[0002] Image reconstruction technology is a technology that obtains the shape information of three-dimensional objects through digital processing of data measured outside the object. It has been widely used in various fields such as medical imaging and industrial non-destructive testing. In some application scenarios, after reconstructing a panoramic image using an image reconstruction model, it is often necessary to perform processing on the panoramic image, such as changing the color tone and adding shadows, to improve the user's viewing experience. Summary of the invention

[0003] This disclosure section is provided to introduce concepts in a brief form, which will be described in detail in the detailed description section below. This disclosure section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] Embodiments of the present disclosure provide an image processing method, apparatus, and electronic device.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, the method comprising: acquiring a panoramic image to be processed; generating a target panoramic image based on a predetermined brightness influence matrix, target lighting information, and a color panoramic image corresponding to the panoramic image to be processed; wherein the color panoramic image is obtained by removing lighting from the panoramic image to be processed.

[0006] In a second aspect, an embodiment of the present disclosure provides an image processing device, which includes: an acquisition module for acquiring a panoramic image to be processed; a generation module for generating a target panoramic image based on a predetermined brightness influence matrix, target lighting information and a color panoramic image corresponding to the panoramic image to be processed; wherein the color panoramic image is obtained by removing lighting from the panoramic image to be processed.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image processing method described in the first aspect above.

[0009] The image processing method, device and electronic device provided by the embodiments of the present disclosure acquire a panoramic image to be processed; based on a predetermined brightness influence matrix, target illumination information and a color panoramic image corresponding to the panoramic image to be processed, generate a target panoramic image; wherein the color panoramic image is obtained by removing illumination from the panoramic image to be processed. A target panoramic image with a target illumination effect can be reconstructed to improve the viewing experience of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 is a flowchart of an embodiment of an image processing method according to the present disclosure;

[0012] Figure 2 A flowchart of an embodiment of a training target image processing model involved in the present disclosure;

[0013] Figure 3 A flowchart of another embodiment of a training target image processing model according to the present disclosure;

[0014] Figure 4 is a structural schematic diagram of an embodiment of an image processing device according to the present disclosure;

[0015] Figure 5 An exemplary system architecture in which the image processing method of one embodiment of the present disclosure can be applied;

[0016] Figure 6 The diagram is a schematic diagram of a basic structure of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0019] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0024] Please refer to Figure 1 , which shows a flowchart of an embodiment of an image processing method according to the present disclosure, such as Figure 1 As shown, the image processing method includes the following steps 101 to 102.

[0025] Step 101, obtaining a panoramic image to be processed;

[0026] In some application scenarios, after an image is captured by a panoramic camera, a mobile phone or other device that can actually capture an image, the above-mentioned panoramic image to be processed can be generated based on the captured image. The panoramic image to be processed here can include, for example, a landscape panoramic image, an indoor panoramic image, etc.

[0027] Step 102, generating a target panoramic image based on a predetermined brightness influence matrix, target illumination information, and a color panoramic image corresponding to the panoramic image to be processed; wherein the color panoramic image is obtained by removing illumination from the panoramic image to be processed.

[0028] The above-mentioned target illumination information may include, for example, the brightness information and the incident angle of the target light source. In some application scenarios, the target illumination information may be predetermined so that the generated target panoramic image may have the target illumination effect. The target illumination effect here may be reflected, for example, by the shadows in the scene; the above-mentioned color panoramic image may include a panoramic image with color information and without illumination information. The color information here may include, for example, the color value of a pixel. The above-mentioned brightness influence matrix may include a precomput radiance transfer matrix (PRT matrix for short), which may be used to characterize the influence of the light source and / or color value on each pixel.

[0029] In some application scenarios, the panoramic image to be processed can be re-illuminated to obtain a target panoramic image with a target illumination effect. Specifically, the panoramic image to be processed can be given target illumination information based on a predetermined PRT matrix and a color panoramic image to generate a target panoramic image with a target illumination effect. In these application scenarios, multiple target illumination information can be preset, and these target illumination information can, for example, represent the illumination information when the same light source illuminates the scene reflected by the panoramic image to be processed at different positions. In this way, through the predetermined PRT matrix, the color panoramic image and the above-mentioned multiple target illumination information, multiple target panoramic images with target illumination effects can be generated, and the dynamic process in which the scene reflected by the panoramic image to be processed changes with the position change of the same light source is realized. For example, the predetermined target illumination information a1, the target illumination information a2 and the target illumination information a3 correspond to the illumination information in the morning, noon and evening respectively, then the corresponding target panoramic image A1, the target panoramic image A2 and the target panoramic image A3 can be generated respectively based on the brightness influence matrix and the color panoramic image. In this way, when the three target panoramic images are played continuously, the scene of the scene where the panoramic image to be processed is located in a day can be obtained, which is convenient for users to experience the lighting conditions of the scene in different time periods.

[0030] In this embodiment, by first acquiring a panoramic image to be processed, and then generating a target panoramic image based on a predetermined brightness influence matrix, target illumination information, and a color panoramic image corresponding to the panoramic image to be processed, wherein the color panoramic image is obtained by removing illumination from the panoramic image to be processed, a target panoramic image with a target illumination effect can be reconstructed, thereby improving the viewing experience of users.

[0031] In some optional implementations, the brightness impact matrix and the de-illuminated color panoramic image predetermined in the above step 102 are obtained based on the following steps: at least one reference panoramic image to be processed that is in the same scene as the panoramic image to be processed is input into a pre-trained target image processing model.

[0032] The above-mentioned target image processing model may include, for example, an image segmentation network U-net.

[0033] The scene where the reference panoramic image to be processed is located can be consistent with the scene where the panoramic image to be processed is located, so that the brightness influence matrix and the color panoramic image corresponding to the reference panoramic image to be processed are consistent with those of the panoramic image to be processed. In some application scenarios, for example, two panoramic images can be obtained in the same scene, one of which is used as the panoramic image to be processed and the other as the reference panoramic image to be processed.

[0034] After the reference panoramic image to be processed is input into the target image processing model, the brightness influence matrix and the color panoramic image corresponding to the reference panoramic image to be processed can be obtained through the target image processing model. In this way, based on the brightness influence matrix, the color panoramic image and the target lighting information, the target panoramic image with the target lighting effect in the scene can be obtained.

[0035] Please refer to Figure 2 , which shows a flow chart of an embodiment of a training target image processing model involved in the present disclosure, such as Figure 2 As shown, the target image processing model is obtained based on the following steps:

[0036] Step 201, obtaining a training sample image set, wherein the training sample image set includes at least one initial panoramic training sample image; the initial panoramic training sample image corresponds to preset initial illumination information;

[0037] In some application scenarios, panoramic images without illumination information can be generated first, and then known illumination information can be added to these panoramic images without illumination to obtain the above-mentioned initial panoramic training sample images. In this way, the initial illumination information can be used to perform comparative training on the relevant illumination parameters of the target image processing model to promote the convergence of the target image processing model.

[0038] In some optional implementations, the above step 201 may include the following sub-steps:

[0039] First, obtain at least one initial environment map in the same scene;

[0040] The above-mentioned initial environment map can be used to simulate the mapping effect of the smooth surface existing in the scene on the surrounding scene, and the lighting information corresponding to the scene can be determined based on the mapping effect. In some application scenarios, the initial lighting information can be pre-marked in the above-mentioned initial environment map, so that the lighting information in the initial panoramic training sample image generated by the initial environment map can be known. Multiple initial environment maps correspond to the same scene, and the PRT matrix and color panoramic image obtained based on these initial environment maps can be the same, which is convenient for constraining the PRT matrix and image color information during the training process of the image processing model.

[0041] Then, based on the at least one initial environment map, at least one initial panoramic training sample image of the training sample image set is generated.

[0042] After obtaining a plurality of initial environment maps, a corresponding initial panoramic training sample image may be generated for each initial environment map, and the plurality of initial panoramic training sample images may be sorted to obtain a training sample image set.

[0043] Step 202: Perform the following model training operations using the at least one initial panoramic training sample image:

[0044] Sub-step 2021, inputting the initial panoramic training sample image into the initial image processing model to obtain an output result;

[0045] In some application scenarios, after obtaining a training sample image set, the initial panoramic training sample image therein may be input into an initial image processing model to obtain a corresponding output result.

[0046] Sub-step 2022, performing panoramic image reconstruction according to the output result to obtain a reconstructed panoramic sample image corresponding to the initial panoramic sample image;

[0047] After obtaining an output result corresponding to the initial panoramic training sample image using the initial image processing model, the output result can be used to reconstruct a reconstructed panoramic sample image corresponding to the initial panoramic training sample image.

[0048] In some optional implementations, the output result includes a predicted brightness influence matrix, predicted illumination information, and a predicted color panoramic image after de-illumination, and the predicted illumination information is determined based on a predicted environment map. Thus, step 2022 may include: reconstructing a panoramic image according to the predicted brightness influence matrix, the predicted environment map, and the predicted color panoramic image after de-illumination to obtain the reconstructed panoramic sample image;

[0049] In some application scenarios, the initial image processing model can output a predicted brightness influence matrix, a predicted color panoramic image, and a predicted environment map. Then, the corresponding reconstructed panoramic sample image can be rendered based on these three.

[0050] Sub-step 2023, determining whether a loss value of a preset loss function meets a preset condition according to the reconstructed panoramic sample image and the initial panoramic sample image;

[0051] The above-mentioned preset loss function may include, for example, a mean absolute error loss function (also called an L1 loss function).

[0052] In some application scenarios, after obtaining the reconstructed panoramic sample image, the reconstructed panoramic sample image and the initial panoramic sample image can be calculated based on a preset loss function to determine the loss value of the preset loss function. In these application scenarios, it can be further determined whether the loss value of the preset loss function meets the preset conditions, and the above preset conditions can include, for example, that the loss value is less than a preset loss threshold. That is, when the loss value obtained by calculating the reconstructed panoramic sample image and the initial panoramic sample image based on the preset loss function is less than the preset threshold, it can be regarded that the loss value of the preset loss function meets the preset conditions.

[0053] Sub-step 2024, if yes, stop the training, and determine the initial image processing model when the training is stopped as the target image processing model;

[0054] If it is determined that the loss value meets the preset conditions, the training operation of the initial image processing model can be stopped, and the initial image processing model at this time can be determined as the target image processing model. After that, the brightness influence matrix and the color panoramic image can be determined using the target image processing model.

[0055] Sub-step 2025, otherwise, adjust the model parameters of the initial image processing model according to the loss value of the preset loss function, and re-execute the above-mentioned model training operation.

[0056] If it is determined that the loss value does not meet the preset conditions, it can be considered that the current initial image processing model cannot output a relatively accurate brightness influence matrix and a color panoramic image corresponding to the reference panoramic image to be processed. Then, the model parameters of the initial image processing model can be adjusted in the direction of making the loss value meet the preset conditions. The model training operation is re-executed to make the initial image processing model converge and obtain the target image processing model.

[0057] In this implementation, the initial image processing model is trained by using multiple initial panoramic training sample images in the training sample image set, so as to obtain a target image processing model that can output a relatively accurate brightness influence matrix and a color panoramic image corresponding to the reference panoramic image to be processed.

[0058] The corresponding PRT matrix and color panoramic image in the same scene should be the same. In some application scenarios, based on the same PRT matrix, multiple panoramic images acquired in the same scene can be re-illuminated. That is, based on the PRT matrix and color panoramic image in the same scene, the corresponding reconstructed panoramic sample images can be reconstructed respectively under the target illumination information that changes according to a certain rule, and then the dynamic change effect of the scene under different illumination can be achieved. The certain rule here can include, for example, the law of the sun rising in the east and setting in the west. Accordingly, the reconstructed reconstructed panoramic sample image can show the illumination change of the scene within a day. Therefore, an initial panoramic training sample image A can be randomly selected from the training sample image set collected in the same scene as the calibrated initial panoramic training sample image. Then, the predicted PRT matrix b, the predicted color panoramic image b, and the calibrated initial environment map a corresponding to the calibrated initial panoramic training sample image A can be rendered to obtain the calibrated reconstructed panoramic sample image A'. In the desired state, the calibrated reconstructed panoramic sample image A' should be consistent with the calibrated initial panoramic training sample image A. Then, the calibration process of the predicted PRT matrix and the predicted color panoramic image can be realized.

[0059] Please refer to Figure 3 , which shows a flow chart of another embodiment of the training target image processing model involved in the present disclosure, such as Figure 3 As shown, the target image processing model is obtained based on the following steps:

[0060] Step 301, obtaining a training sample image set, wherein the training sample image set includes an initial blurred panoramic training sample image, wherein the initial blurred panoramic training sample image is determined by an initial blurred environment map, and the initial blurred environment map is generated based on an initial spherical harmonic function corresponding to the initial environment map;

[0061] In some application scenarios, after obtaining the initial environment map, the initial spherical harmonic function corresponding to the initial environment map can be calculated, and the initial blurred environment map can be generated by the initial spherical harmonic function, and then the initial blurred panoramic training sample image can be generated by the initial blurred environment map. Here, in the process of generating the initial blurred environment map by the initial spherical harmonic function, the calculation of many parameters can be reduced, and then the convergence speed of the initial image processing model can be accelerated.

[0062] In these application scenarios, after obtaining a plurality of initial blurred panoramic training sample images, they can be sorted to obtain a training sample image set.

[0063] Step 302: Perform the following model training operations using the at least one initial blurred panoramic training sample image:

[0064] Sub-step 3021, inputting the initial blurred panoramic training sample image into the initial image processing model to obtain an output result;

[0065] After the initial blurred panoramic training sample image is obtained, the initial blurred panoramic training sample image can be input into the initial image processing model to obtain the output result of the initial image processing model.

[0066] Sub-step 3022, performing panoramic image reconstruction according to the output result to obtain a reconstructed blurred panoramic sample image corresponding to the initial blurred panoramic training sample image;

[0067] In some application scenarios, the output results may include a predicted brightness influence matrix, a predicted spherical harmonic function, and a predicted color panoramic image after de-illumination.

[0068] In this way, after obtaining the preset brightness influence matrix, the predicted spherical harmonics and the predicted color panoramic image using the initial image processing model, a reconstructed blurred panoramic sample image corresponding to the initial blurred panoramic training sample image can be reconstructed.

[0069] Sub-step 3023, determining whether a loss value of the preset loss function meets a preset condition according to the reconstructed blurred panoramic sample image and the initial blurred panoramic sample image;

[0070] The above-mentioned preset loss function may also include, for example, a mean absolute error loss function (also called an L1 loss function).

[0071] After obtaining the reconstructed blurred panoramic sample image, the reconstructed blurred panoramic sample image and the initial blurred panoramic training sample image can be calculated based on a preset loss function to determine whether the loss value of the preset loss function meets the preset condition. In some application scenarios, the preset condition may include, for example, that the loss value is less than a preset loss threshold. That is, when the loss value obtained by calculating the reconstructed panoramic sample image and the initial panoramic sample image based on the preset loss function is less than the preset threshold, it can be considered that the loss value of the preset loss function meets the preset condition.

[0072] Sub-step 3024, if yes, stop the training, and determine the initial image processing model when the training is stopped as the target image processing model;

[0073] Sub-step 3025, otherwise, adjust the model parameters of the initial image processing model according to the loss value of the preset loss function, and re-execute the above-mentioned model training operation.

[0074] The implementation process of the above sub-steps 3024 to 3025 and the technical effects obtained can be compared with Figure 2 Sub-steps 2024 and 2025 are the same or similar and are not described in detail here.

[0075] In this implementation, the step of training the initial image processing model based on the initial blurred panoramic training sample images is highlighted, which accelerates the convergence speed of the initial image processing model.

[0076] Please refer to Figure 4 , which shows a schematic structural diagram of an embodiment of an image processing device according to the present disclosure, such as Figure 4 As shown, the image processing device includes an acquisition module 401 and a generation module 402. The acquisition module 401 is used to acquire a panoramic image to be processed; the generation module 402 is used to generate a target panoramic image based on a predetermined brightness influence matrix, target illumination information, and a color panoramic image corresponding to the panoramic image to be processed; wherein the color panoramic image is obtained by removing illumination from the panoramic image to be processed.

[0077] It should be noted that the specific processing of the acquisition module 401 and the generation module 402 of the image processing device and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101 to step 102 in the corresponding embodiment are not repeated here.

[0078] In some optional implementations of the present embodiment, the predetermined brightness impact matrix and the de-illuminated color panoramic image are obtained based on the following steps: at least one reference panoramic image to be processed that is in the same scene as the panoramic image to be processed is input into a pre-trained target image processing model.

[0079] In some optional implementations of the present embodiment, the target image processing model is obtained based on the following steps: obtaining a training sample image set, the training sample image set including at least one initial panoramic training sample image; the initial panoramic training sample image corresponds to preset initial illumination information; using the at least one initial panoramic training sample image to perform the following model training operations: inputting the initial panoramic training sample image into the initial image processing model to obtain an output result; performing panoramic image reconstruction based on the output result to obtain a reconstructed panoramic sample image corresponding to the initial panoramic sample image; determining whether the loss value of a preset loss function meets a preset condition based on the reconstructed panoramic sample image and the initial panoramic sample image; if so, stopping the training, and determining the initial image processing model at the time of stopping the training as the target image processing model; otherwise, adjusting the model parameters of the initial image processing model according to the loss value of the preset loss function, and re-executing the model training operation.

[0080] In some optional implementations of this embodiment, obtaining the training sample image set includes: obtaining at least one initial environment map under the same scene; and generating at least one initial panoramic training sample image of the training sample image set based on the at least one initial environment map.

[0081] In some optional implementations of the present embodiment, the output result includes a predicted brightness influence matrix, predicted lighting information, and a predicted color panoramic image after removal of lighting; and the predicted lighting information is determined based on a predicted environment map, and the panoramic image is reconstructed according to the output result to obtain a reconstructed panoramic sample image corresponding to the initial panoramic sample image, including: panoramic image reconstruction is performed according to the predicted brightness influence matrix, the predicted environment map, and the predicted color panoramic image after removal of lighting to obtain the reconstructed panoramic sample image.

[0082] In some optional implementations of the present embodiment, the target image processing model is obtained based on the following steps: obtaining a training sample image set, the training sample image set including an initial blurred panoramic training sample image, the initial blurred panoramic training sample image is determined by an initial blurred environment map, and the initial blurred environment map is generated based on an initial spherical harmonic function corresponding to the initial environment map; using the at least one initial blurred panoramic training sample image to perform the following model training operation: inputting the initial blurred panoramic training sample image into the initial image processing model to obtain an output result; performing panoramic image reconstruction according to the output result to obtain a reconstructed blurred panoramic sample image corresponding to the initial blurred panoramic training sample image; determining whether the loss value of the preset loss function meets a preset condition based on the reconstructed blurred panoramic sample image and the initial blurred panoramic sample image; if so, stopping the training, and determining the initial image processing model at the time of stopping the training as the target image processing model; otherwise, adjusting the model parameters of the initial image processing model according to the loss value of the preset loss function, and re-executing the above model training operation.

[0083] Please refer to Figure 5 , which shows an exemplary system architecture in which the image processing method of an embodiment of the present disclosure can be applied.

[0084] like Figure 5As shown, the system architecture may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is used to provide a medium for a communication link between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The above-mentioned terminal devices and servers may communicate using any currently known or future developed network protocols such as HTTP (HyperText Transfer Protocol), and may be interconnected with digital data communications (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., Ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0085] The terminal devices 501, 502, 503 can interact with the server 505 through the network 504 to receive or send messages, etc. Various client applications can be installed on the terminal devices 501, 502, 503, such as video publishing applications, search applications, and news information applications.

[0086] Terminal devices 501, 502, 503 can be hardware or software. When terminal devices 501, 502, 503 are hardware, they can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 501, 502, 503 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0087] Server 505 can be a server that can provide various services, such as receiving image acquisition requests sent by terminal devices 501, 502, 503, analyzing and processing the image acquisition requests, and sending the analysis and processing results (such as image data corresponding to the above-mentioned acquisition requests) to terminal devices 501, 502, 503.

[0088] It should be noted that the image processing method provided in the embodiment of the present disclosure may be executed by a server, and accordingly, the image processing device may be arranged in the server.

[0089] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0090] Reference below Figure 6 , which shows an electronic device (eg, Figure 5 A schematic diagram of the structure of the server in FIG. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of the electronic device are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0092] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0093] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0094] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0095] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0096] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains a panoramic image to be processed; generates a target panoramic image based on a predetermined brightness influence matrix, target lighting information and a color panoramic image corresponding to the panoramic image to be processed; wherein the color panoramic image is obtained based on removing lighting from the panoramic image to be processed.

[0097] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0098] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the unit itself in some cases. For example, the acquisition module 401 may also be described as a "module for acquiring a panoramic image to be processed".

[0100] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0101] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0103] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0104] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, It is characterized in that include: Obtaining a panoramic image to be processed; Generate a target panoramic image based on a predetermined brightness influence matrix, target illumination information, and a color panoramic image corresponding to the panoramic image to be processed; wherein the color panoramic image is obtained by removing illumination from the panoramic image to be processed; The predetermined brightness influence matrix and the de-illuminated color panoramic image are obtained based on the following steps: Inputting at least one reference panoramic image to be processed having the same scene as the panoramic image to be processed into a pre-trained target image processing model to obtain; The target image processing model is obtained based on the following steps: Acquire a training sample image set, wherein the training sample image set includes an initial blurred panoramic training sample image, wherein the initial blurred panoramic training sample image is determined by an initial blurred environment map, and the initial blurred environment map is generated based on an initial spherical harmonic function corresponding to the initial environment map; The following model training operation is performed using the at least one initial blurred panoramic training sample image: Inputting the initial blurred panoramic training sample image into the initial image processing model to obtain an output result; Perform panoramic image reconstruction according to the output result to obtain a reconstructed blurred panoramic sample image corresponding to the initial blurred panoramic training sample image; Determining whether a loss value of a preset loss function meets a preset condition according to the reconstructed blurred panoramic sample image and the initial blurred panoramic training sample image; If yes, stop the training, and determine the initial image processing model when the training is stopped as the target image processing model; Otherwise, the model parameters of the initial image processing model are adjusted according to the loss value of the preset loss function, and the above model training operation is re-executed.

2. The method according to claim 1, It is characterized in that The target image processing model is obtained based on the following steps: Acquire a training sample image set, wherein the training sample image set includes at least one initial panoramic training sample image; the initial panoramic training sample image corresponds to preset initial illumination information; The following model training operation is performed using the at least one initial panoramic training sample image: Inputting the initial panoramic training sample image into the initial image processing model to obtain an output result; Reconstructing a panoramic image according to the output result to obtain a reconstructed panoramic sample image corresponding to the initial panoramic training sample image; Determining whether a loss value of a preset loss function meets a preset condition according to the reconstructed panoramic sample image and the initial panoramic training sample image; If yes, stop the training, and determine the initial image processing model when the training is stopped as the target image processing model; Otherwise, the model parameters of the initial image processing model are adjusted according to the loss value of the preset loss function, and the model training operation is re-executed.

3. The method according to claim 2, It is characterized in that The step of obtaining a training sample image set comprises: Get at least one initial environment map for the same scene; At least one initial panoramic training sample image of the training sample image set is generated based on the at least one initial environment map.

4. The method according to claim 2, It is characterized in that The output results include a predicted brightness influence matrix, predicted illumination information, and a predicted color panoramic image after de-illumination; and The predicted lighting information is determined based on a predicted environment map, and The reconstructing the panoramic image according to the output result to obtain a reconstructed panoramic sample image corresponding to the initial panoramic training sample image includes: The panoramic image is reconstructed according to the predicted brightness influence matrix, the predicted environment map and the predicted color panoramic image after de-illumination to obtain the reconstructed panoramic sample image.

5. An image processing device, It is characterized in that include: An acquisition module, used for acquiring the panoramic image to be processed; A generating module, configured to generate a target panoramic image based on a predetermined brightness influence matrix, target illumination information, and a color panoramic image corresponding to the panoramic image to be processed; wherein the color panoramic image is obtained by removing illumination from the panoramic image to be processed; The predetermined brightness influence matrix and the de-illuminated color panoramic image are obtained based on the following steps: Inputting at least one reference panoramic image to be processed having the same scene as the panoramic image to be processed into a pre-trained target image processing model to obtain; The target image processing model is obtained based on the following steps: Acquire a training sample image set, wherein the training sample image set includes an initial blurred panoramic training sample image, wherein the initial blurred panoramic training sample image is determined by an initial blurred environment map, and the initial blurred environment map is generated based on an initial spherical harmonic function corresponding to the initial environment map; The following model training operation is performed using the at least one initial blurred panoramic training sample image: Inputting the initial blurred panoramic training sample image into the initial image processing model to obtain an output result; Perform panoramic image reconstruction according to the output result to obtain a reconstructed blurred panoramic sample image corresponding to the initial blurred panoramic training sample image; Determining whether a loss value of a preset loss function meets a preset condition according to the reconstructed blurred panoramic sample image and the initial blurred panoramic training sample image; If yes, stop the training, and determine the initial image processing model when the training is stopped as the target image processing model; Otherwise, the model parameters of the initial image processing model are adjusted according to the loss value of the preset loss function, and the above model training operation is re-executed.

6. An electronic device, It is characterized in that include: one or more processors; A storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of claims 1 to 4.

7. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images