Panoramic image generation method and apparatus, device, storage medium and program product

Abstract

Disclosed in the embodiments of the present disclosure are a panoramic image generation method and apparatus, a device, a storage medium and a program product. The method comprises: in at least one collection point, collecting depth information and image information of a target scene; on the basis of the depth information and the image information, generating a three-dimensional spatial model of the target scene; on the basis of the image information, generating at least one original panoramic image, an area to be filled being present in the at least one original panoramic image; on the basis of the three-dimensional spatial model and the at least one collection point, generating at least one rendered panoramic image; and, on the basis of the at least one rendered panoramic image, filling the area to be filled in the original panoramic image at the corresponding collection point, so as to obtain a stitched panoramic image at the at least one collection point. The embodiments of the present disclosure use the rendered panoramic images at each collection point generated by the three-dimensional spatial model to fill corresponding original panoramic images, thereby completing the image of the real scene.

Description

Method, device, equipment, storage medium and program product for generating panoramic image

[0001] This disclosure claims priority to a Chinese patent application filed with the Patent Office of China on July 21, 2023, with application number CN202310906728.X and invention name “Method, device and storage medium for generating panoramic images,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to panoramic image processing technology in three-dimensional space, and more particularly to a method, apparatus, device, computer-readable storage medium, and computer program product for generating a panoramic image. Background Art

[0003] When capturing and generating panoramic images of indoor or outdoor three-dimensional spaces, it's often impossible to capture a complete panorama, requiring the scene to be completed. Existing image completion algorithms typically use the image itself to complete the panorama, which can result in the resulting panorama not matching the actual scene.

[0004] Summary of the Invention

[0005] A technical problem to be solved by the embodiments of the present disclosure is to provide a method, apparatus, device, storage medium and program product for generating a panoramic image.

[0006] According to one aspect of an embodiment of the present disclosure, a method for generating a panoramic image is provided, the method comprising:

[0007] At at least one acquisition point, collecting depth information and image information of the target scene;

[0008] generating a three-dimensional spatial model of the target scene based on the depth information and the image information;

[0009] generating at least one original panoramic image based on the image information, wherein the at least one original panoramic image includes an area to be filled;

[0010] Rendering and generating at least one rendered panoramic image based on the three-dimensional space model and the at least one acquisition point;

[0011] Based on the at least one rendered panoramic image, the to-be-filled area in the original panoramic image corresponding to the acquisition point is filled to obtain a stitched panoramic image of the at least one acquisition point.

[0012] In one embodiment of the present disclosure, the method further includes:

[0013] performing a Gaussian kernel convolution operation on the image to be Gaussian processed for the acquisition points to obtain a Gaussian convolution image, wherein the image to be Gaussian processed is the rendered panoramic image or the original panoramic image;

[0014] Determining at least one target pixel block in the image to be Gaussian processed based on a pixel value difference between at least one pixel block in the image to be Gaussian processed and a corresponding pixel block in the Gaussian convolution image, wherein an absolute value of the pixel value difference corresponding to the target pixel block is less than a set threshold;

[0015] According to the position coordinates of the at least one target pixel block in the image to be Gaussian processed, traverse the position range corresponding to the image to be matched to obtain a corresponding matching pixel block, wherein the image to be matched is the original panoramic image or the rendered panoramic image;

[0016] Based on the color difference between the at least one target pixel block and the corresponding matching pixel block, the image color of the to-be-filled area of ​​the stitched panorama is adjusted to obtain a panorama with consistent tones.

[0017] In yet another embodiment of the present disclosure, adjusting the image color of the to-be-filled area of ​​the stitched panorama based on the color difference between the at least one target pixel block and the corresponding matching pixel block to obtain a tonally consistent panorama includes:

[0018] determining a white balance correction matrix based on an image intensity difference between the at least one target pixel block and a corresponding matching pixel block;

[0019] Using the white balance correction matrix to adjust the image intensity value of the to-be-filled area of ​​the stitched panorama to obtain a white-balanced panorama;

[0020] Using a color mapping function, adjusting the contrast and brightness of the area to be filled in the white-balanced panoramic image so that the contrast of the area to be filled after adjustment is consistent with the contrast of other areas of the white-balanced panoramic image, and the brightness of the area to be filled after adjustment is consistent with the brightness of other areas of the white-balanced panoramic image, thereby obtaining a color adjustment area; the other area being the area of ​​the white-balanced panoramic image excluding the area to be filled;

[0021] The color-adjusted panoramic image region and the other regions are fused using a Poisson fusion algorithm to obtain the tonal consistent panoramic image.

[0022] In another embodiment of the present disclosure, determining at least one target pixel block in the image to be Gaussian processed based on a pixel value difference between at least one pixel block in the image to be Gaussian processed and a corresponding pixel block in the Gaussian convolution image includes:

[0023] Dividing the image to be Gaussian processed and the Gaussian convolution image into at least one pixel block according to the same division standard;

[0024] Calculating the pixel mean of at least one pixel block in the image to be Gaussian processed and the pixel mean of at least one pixel block in the Gaussian convolution image respectively to obtain at least one first pixel mean and at least one second pixel mean;

[0025] determining a pixel value difference between the at least one first pixel mean and the corresponding second pixel mean;

[0026] At least one pixel block in the Gaussian convolution image is determined as the at least one target pixel block according to the pixel value difference.

[0027] In yet another embodiment of the present disclosure, the method further includes:

[0028] Obtaining a gradient map of the at least one rendered panoramic image and a corresponding gradient map of the stitched panoramic image;

[0029] Based on the gradient map of the at least one rendered panoramic image, performing line optimization on the corresponding gradient map of the stitched panoramic image to obtain an optimized gradient map of the stitched panoramic image;

[0030] The optimized gradient map of the stitched panorama is used to guide a preset network model to optimize the texture information of the tonally consistent panorama to obtain a final panorama.

[0031] In another embodiment of the present disclosure, the optimizing gradient map of the stitched panorama to guide a preset network model to optimize the texture information of the tonally consistent panorama to obtain a final panorama includes:

[0032] Merging the optimized gradient map and the tonal consistent panoramic image using a multi-channel merging function to obtain a multi-channel feature map;

[0033] Using a pre-trained preset network model to perform feature extraction and grid sampling on the multi-channel feature map to obtain a spline after grid sampling;

[0034] The texture information in the tonally consistent panorama is optimized by the spline after the grid sampling to obtain the final panorama.

[0035] According to another aspect of an embodiment of the present disclosure, a device for generating a panoramic image is provided, the device comprising:

[0036] An acquisition module, configured to acquire depth information and image information of a target scene at at least one acquisition point;

[0037] A model generation module, configured to generate a three-dimensional spatial model of the target scene based on the depth information and the image information;

[0038] An original image generation module, configured to generate at least one original panoramic image based on the image information, wherein the at least one original panoramic image includes an area to be filled;

[0039] A rendering image generation module, configured to generate at least one rendered panoramic image based on the three-dimensional space model and the at least one acquisition point;

[0040] A filling module is configured to fill, based on the at least one rendered panoramic image, an area to be filled in the original panoramic image corresponding to the acquisition point, to obtain a stitched panoramic image of the at least one acquisition point.

[0041] In one embodiment of the present disclosure, the apparatus further comprises:

[0042] a Gaussian processing module, configured to perform a Gaussian kernel convolution operation on the image to be Gaussian processed for the acquisition points to obtain a Gaussian convolution image, wherein the image to be Gaussian processed is the rendered panoramic image or the original panoramic image;

[0043] A first determining module is configured to determine at least one target pixel block in the image to be Gaussian processed based on a pixel value difference between at least one pixel block in the image to be Gaussian processed and a corresponding pixel block in the Gaussian convolution image, wherein an absolute value of the pixel value difference corresponding to the at least one target pixel block is less than a set threshold;

[0044] a second determining module, configured to traverse, according to the position coordinates of the at least one target pixel block in the image to be Gaussian processed, a position range corresponding to the image to be matched to obtain a corresponding matching pixel block, wherein when the image to be Gaussian processed is the rendered panoramic image, the image to be matched is the original panoramic image; and when the image to be Gaussian processed is the original panoramic image, the image to be matched is the rendered panoramic image;

[0045] The color adjustment module is configured to adjust the image color of the to-be-filled area of ​​the stitched panorama based on the color difference between the at least one target pixel block and the corresponding matching pixel block, so as to obtain a panorama with consistent tones.

[0046] In another embodiment of the present disclosure, the color adjustment module includes:

[0047] A first determination submodule, configured to determine a white balance correction matrix based on an image intensity difference between the at least one target pixel block and a corresponding matching pixel block;

[0048] a first adjustment submodule, configured to adjust the image intensity value of the to-be-filled area of ​​the stitched panorama using the white balance correction matrix to obtain a white-balanced panorama;

[0049] a second adjustment submodule, configured to adjust the contrast and brightness of the area to be filled in the white-balanced panoramic image using a color mapping function, so that the contrast of the area to be filled after adjustment is consistent with the contrast of other areas of the white-balanced panoramic image, and the brightness of the area to be filled after adjustment is consistent with the brightness of other areas of the white-balanced panoramic image, thereby obtaining a color-adjusted area; the other areas being areas of the white-balanced panoramic image excluding the area to be filled;

[0050] The Poisson fusion submodule is used to fuse the color-adjusted panoramic image region with the other regions using a Poisson fusion algorithm to obtain the tonal consistent panoramic image.

[0051] In yet another embodiment of the present disclosure, the first determining module includes:

[0052] A pixel block division submodule, configured to divide the image to be Gaussian processed and the Gaussian convolution image into at least one pixel block according to the same division standard;

[0053] a pixel value calculation submodule, configured to respectively calculate the pixel mean of at least one pixel block in the image to be Gaussian processed and the pixel mean of at least one pixel block in the Gaussian convolution image, to obtain at least one first pixel mean and at least one second pixel mean;

[0054] a difference calculation submodule, configured to determine a pixel value difference between the at least one first pixel mean and the corresponding second pixel mean;

[0055] A determination submodule is configured to determine, based on the pixel value difference, at least one pixel block in the Gaussian convolution image as the at least one target pixel block.

[0056] In yet another embodiment of the present disclosure, the apparatus further comprises:

[0057] A gradient image acquisition module, configured to acquire a gradient image of the at least one rendered panoramic image and a corresponding gradient image of the stitched panoramic image;

[0058] a gradient optimization module, configured to optimize the lines of the gradient map of the corresponding stitched panorama based on the gradient map of the at least one rendered panorama, to obtain an optimized gradient map of the stitched panorama;

[0059] The image texture optimization module is used to guide a preset network model to optimize the texture information of the tonal consistent panorama through the optimized gradient map of the stitched panorama to obtain a final panorama.

[0060] In another embodiment of the present disclosure, the image texture optimization module includes:

[0061] a channel merging submodule, configured to merge the optimized gradient map and the tone-consistent panoramic image using a multi-channel merging function to obtain a multi-channel feature map;

[0062] A feature extraction submodule, configured to perform feature extraction and grid sampling on the multi-channel feature map using a pre-trained preset network model to obtain a spline after grid sampling;

[0063] The image optimization submodule is used to optimize the texture information in the tonal consistent panorama by using the spline after the grid sampling to obtain the final panorama.

[0064] According to another aspect of the present disclosure, an electronic device is provided, the electronic device including:

[0065] memory for storing computer programs;

[0066] The processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the method for generating the panoramic image is implemented.

[0067] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method for generating a panoramic image is implemented.

[0068] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the above-mentioned method for generating a panoramic image when executed by a processor.

[0069] Based on the panoramic image generation method, device, equipment, storage medium and program product provided by the above-mentioned embodiments of the present disclosure, when it is necessary to generate a panoramic image of a target scene, the depth information and image information of the target scene can be first collected, and then a three-dimensional space model of the target scene can be generated based on the collected depth information and image information; at least one original panoramic image is generated based on the image information, and there is an area to be filled in the at least one original panoramic image, wherein the original panoramic image is obtained by splicing image information collected by the same collection point; then, based on the three-dimensional space model and at least one collection point, at least one rendered panoramic image is rendered and generated; using the at least one rendered panoramic image, the area to be filled in the original panoramic image of the corresponding collection point is filled to obtain a spliced ​​panoramic image of the at least one collection point. The technical solution provided by the embodiments of the present disclosure can complete the image of the real scene by using the rendered panoramic image of at least one collection point generated by the three-dimensional space model to fill the original panoramic image of the corresponding collection point, thereby reducing the problem of image misalignment caused by directly completing the image through the original panoramic image, and improving the display effect of the panoramic image in the three-dimensional scene.

[0070] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0072] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0073] FIG1 is a flow chart of an embodiment of a method for generating a panoramic image according to the present disclosure;

[0074] FIG2A is a flow chart of adjusting the color of a stitched panorama according to the present disclosure;

[0075] FIG2B is a schematic diagram of a rendered panoramic image in the panoramic image generation method of the present disclosure;

[0076] FIG2C is a schematic diagram of an original panoramic image in the method for generating a panoramic image of the present disclosure;

[0077] FIG2D is a schematic diagram of a stitched panoramic image in the panoramic image generation method of the present disclosure;

[0078] FIG2E is a schematic diagram of a panorama with consistent tones after color adjustment is performed on the stitched panorama in the panorama generation method of the present disclosure;

[0079] FIG2F is a flowchart illustrating a specific implementation of step 202 in the process of adjusting the color of the stitched panorama according to the present disclosure;

[0080] FIG2G is a flowchart illustrating a specific implementation of step 204 in the process of adjusting the color of the stitched panorama according to the present disclosure;

[0081] FIG3A is a flowchart of the present disclosure for adjusting texture of a stitched panorama;

[0082] FIG3B is a diagram showing the effect of inconsistent textures in the panoramic image generation method of the present disclosure;

[0083] FIG3C is a panoramic image after texture optimization in the panoramic image generation method of the present disclosure;

[0084] FIG3D is a schematic diagram of using an optimized gradient map to guide deformation of a graphic according to the present disclosure;

[0085] FIG4 is a schematic structural diagram of an embodiment of a device for generating a panoramic image according to the present disclosure;

[0086] FIG5 is a schematic structural diagram of another embodiment of a device for generating a panoramic image according to the present disclosure;

[0087] FIG6 is a structural diagram of an electronic device provided in accordance with an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0088] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0089] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0090] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0091] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0092] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0093] The disclosed embodiments may be applied to a computer system / server that is operable with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the foregoing.

[0094] Computer systems / servers may be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and the like, that perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located on local or remote computer system storage media, including storage devices.

[0095] FIG1 is a flow chart of an embodiment of a method for generating a panoramic image according to the present disclosure. The method for generating a panoramic image can be applied to an electronic device (such as a computer system or a server). As shown in FIG1 , the method for generating a panoramic image includes the following steps:

[0096] Step 101: Collect depth information and image information of a target scene at at least one collection point.

[0097] In an optional embodiment, in order to generate a three-dimensional spatial model of the target scene, multiple images can be collected at at least one collection point, and depth information can be collected at the corresponding collection point, or depth information can be continuously collected in the target scene.

[0098] When collecting image information, a group of images can be collected at a collection point, and then the group of images collected at the collection point are spliced ​​to obtain the original panoramic image of the collection point. For example, a group of images can be collected at collection point 1, and the original panoramic image of collection point 1 can be generated based on the images collected at collection point 1.

[0099] In an optional embodiment, the collection points of the image information can be pre-set. For example, positions at preset distances in directions such as 120 degrees in the horizontal direction, 180 degrees in the horizontal direction, 240 degrees in the horizontal direction, 300 degrees in the horizontal direction, and 0 degrees in the horizontal direction of the origin are used as collection points, and a group of collection images are collected at these collection points respectively.

[0100] In an optional embodiment, after the depth information and the image information are collected, step 102 or step 103 may be performed.

[0101] In an optional example, step 101 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the acquisition module 41 executed by the processor.

[0102] Step 102: Generate a three-dimensional spatial model of the target scene based on the depth information and the image information.

[0103] In an alternative embodiment, an existing model construction algorithm can be used to construct a three-dimensional spatial model of the target scene based on the depth information and image information. Alternatively, a spatial model of the target scene can be generated based on the acquired depth information. The generated spatial model can then be mapped using a texture mapping algorithm based on the image information acquisition points to obtain a three-dimensional spatial model of the target scene.

[0104] In an optional example, step 102 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the model generation module 42 executed by the processor.

[0105] Step 103: Generate at least one original panoramic image based on the image information, wherein the at least one original panoramic image has an area to be filled.

[0106] The original panoramic image may be determined by stitching together image information collected at the same collection point. For example, each original panoramic image may be determined by stitching together image information collected at the same collection point.

[0107] In an optional embodiment, based on a group of image information collected at a collection point, the group of image information includes at least one image information, and an original panoramic image of the collection point can be generated by performing a stitching operation on the at least one image information collected at the collection point.

[0108] In an optional embodiment, the area to be filled refers to an area in the original panoramic image generated based on the collected image information where no target scene information is collected.

[0109] In an optional embodiment, the original panoramic image obtained by stitching together a set of image information collected at a single acquisition point may contain areas where the target scene information is not captured (i.e., areas to be filled). For example, the area indicated by the mosaic area in FIG2C is the area to be filled in the original panoramic image. Due to the limitations of the imaging parameters of the image acquisition device (e.g., camera intrinsic parameters, extrinsic parameters, etc.), when capturing images at a single acquisition point, positions slightly above (such as the mosaic area near the upper edge in FIG2C ) or slightly below (such as the mosaic area near the lower edge in FIG2C ) may not be captured, resulting in areas to be filled.

[0110] In an optional example, step S103 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the original image generation module 43 executed by the processor.

[0111] Step 104 : Render and generate at least one rendered panoramic image based on the three-dimensional space model and the at least one acquisition point.

[0112] In an optional embodiment, for a single acquisition point, a corresponding rendered panorama may be generated by rendering in the three-dimensional space model. The rendered panorama shown in FIG2B and the original panorama shown in FIG2C correspond to the same acquisition point.

[0113] Alternatively, an existing panorama rendering method may be used to render a panorama at a specific acquisition point in a three-dimensional space model, for example, using three-dimensional modeling rendering and production software to render and obtain the rendered panorama.

[0114] In an optional example, step S104 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the rendering image generation module 44 executed by the processor.

[0115] Step 105 : Based on the at least one rendered panoramic image, fill the to-be-filled area in the original panoramic image corresponding to the acquisition point to obtain a stitched panoramic image of the at least one acquisition point.

[0116] In an optional embodiment, for the acquisition points, an image stitching algorithm can be used to directly fill the area to be filled in the original panorama based on the image information in the rendered panorama to obtain a stitched panorama corresponding to the acquisition points. For example, the area to be filled in the original panorama shown in FIG2C is directly filled based on the image information in the rendered panorama shown in FIG2B. The stitched panorama can be seen in FIG2D. Among them, the stitched panorama may have problems such as inconsistent color or inconsistent texture (stitching occurs). For example, as shown in FIG2D, it can be seen that the color of the area to be filled (the area indicated by labels 21 and 22) is inconsistent with the color of the original area of ​​the original panorama (the area indicated by label 23). Therefore, it is necessary to adjust the color and / or texture of the stitched panorama. For example, the color of the stitched panorama can be adjusted later by the embodiment shown in FIG2A, and / or the texture of the stitched panorama can be adjusted by the embodiment shown in FIG3A. The details are not described here. Please refer to the detailed description of the embodiment shown in FIG2A and the embodiment shown in FIG3A below for understanding.

[0117] In an optional example, step S105 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the filling module 45 executed by the processor.

[0118] The method for generating a panoramic image provided by the above-mentioned embodiment of the present disclosure can, when it is necessary to generate a panoramic image of a target scene, first collect depth information and image information of the target scene, and then generate a three-dimensional space model of the target scene based on the collected depth information and image information; generate at least one original panoramic image based on the image information, wherein there is an area to be filled in the original panoramic image; then render and generate at least one rendered panoramic image based on the three-dimensional space model and at least one acquisition point; based on the at least one rendered panoramic image, fill the area to be filled in the original panoramic image corresponding to the acquisition point to obtain a spliced ​​panoramic image of at least one acquisition point. The technical solution provided by the embodiment of the present disclosure can complete the image of the real scene by using the rendered panoramic image of at least one acquisition point generated by the three-dimensional space model to fill the original panoramic image of the corresponding acquisition point, thereby reducing the problem of possible image misalignment caused by directly completing the image through the original panoramic image, and improving the display effect of the panoramic image in the three-dimensional scene.

[0119] In order to better illustrate the panoramic image generation solution of the present disclosure, another embodiment is listed below for illustration.

[0120] Figure 2A is a flowchart of adjusting the color of a stitched panorama according to the present disclosure. Figure 2B is a schematic diagram of a rendered panorama according to the present disclosure. Figure 2C is a schematic diagram of an original panorama according to the present disclosure. Figure 2D is a schematic diagram of a stitched panorama according to the present disclosure. Figure 2E is a schematic diagram of a panorama with consistent tones after color adjustment of the stitched panorama according to the present disclosure.

[0121] This embodiment takes how to adjust the color of a stitched panorama as an example for illustrative description, as shown in FIG2A , including the following steps:

[0122] In step 201 , a Gaussian kernel convolution operation is performed on the Gaussian processed image for each acquisition point to obtain a Gaussian convolution image.

[0123] The collection point in this embodiment may be any one of the at least one collection point.

[0124] In an optional embodiment, the image to be Gaussian processed can be a rendered panoramic image or an original panoramic image. By performing a Gaussian kernel convolution operation on the rendered panoramic image or the original panoramic image, relatively smooth pixel blocks in the image to be Gaussian processed can be determined, wherein the smaller the pixel point in the convolution result, the smoother the image.

[0125] Optionally, a linear smoothing filter process may be performed on the image to be Gaussian processed through a Gaussian kernel convolution operation, thereby eliminating Gaussian noise in the image to be Gaussian processed and obtaining a Gaussian convolution image.

[0126] In an optional example, step 201 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the Gaussian processing module 46 executed by the processor.

[0127] Step 202 : determining at least one target pixel block in the image to be Gaussian processed based on a pixel value difference between at least one pixel block in the image to be Gaussian processed and a corresponding pixel block in the Gaussian convolution image.

[0128] The absolute value of the pixel value difference corresponding to the target pixel block is less than the set threshold.

[0129] In an optional embodiment, the specific implementation of determining at least one target pixel block can be seen in FIG2F , which is a flowchart of the specific implementation of step 202 in the process of adjusting the color of the stitched panorama of the present disclosure. As shown in FIG2F , the following steps 221-224 are included:

[0130] Step 221 : Divide the image to be Gaussian processed and the Gaussian convolution image into at least one pixel block according to the same division standard.

[0131] In an optional embodiment, the division criterion refers to a division method of pixel blocks. For example, both the image to be Gaussian processed and the Gaussian convolution image are divided into pixel blocks of size 5*5, and the resulting pixel blocks are of size 5 pixels*5 pixels. The pixel blocks in the image to be Gaussian processed and the pixel blocks in the Gaussian convolution image divided based on the same division criterion are one-to-one corresponding.

[0132] In an optional embodiment, when dividing the image, only the valid area of ​​the image may be divided, that is, the portion of the to-be-filled area in the original panorama is not processed, and the difference in color (for example, color temperature, brightness, contrast, etc.) between the rendered panorama and the original panorama is determined only by using pixels in the valid areas of the rendered panorama and the original panorama.

[0133] In an optional example, step 221 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the pixel block division submodule 471 executed by the processor.

[0134] Step 222 , respectively calculating pixel means of at least one pixel block in the image to be Gaussian processed and at least one pixel block in the Gaussian convolution image, to obtain at least one first pixel mean and at least one second pixel mean.

[0135] In an optional example, step 222 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the pixel value calculation submodule 472 executed by the processor.

[0136] Step 223: Determine a pixel value difference between at least one first pixel mean and at least one second pixel mean.

[0137] In steps 222 and 223, the pixel mean of the pixel block in the image to be Gaussian processed is calculated to obtain a first pixel mean, and the pixel mean of the pixel block in the Gaussian convolution image is calculated to obtain a second pixel mean. Since the pixel blocks in the image to be Gaussian processed and the pixel blocks in the Gaussian convolution image are one-to-one corresponding, the first pixel mean and the second pixel mean are one-to-one corresponding. The difference between the first pixel mean and the corresponding second pixel mean is calculated to obtain the pixel value difference between the pixel blocks in the image to be Gaussian processed and the Gaussian convolution image (i.e., the change in pixel value before and after Gaussian processing).

[0138] In an optional example, step 223 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the difference calculation submodule 473 executed by the processor.

[0139] Step 224: Determine at least one pixel block in the Gaussian convolution image as at least one target pixel block based on the pixel value difference.

[0140] Optionally, a pixel block in the Gaussian convolution image whose absolute value of the pixel value difference is less than a set threshold is determined as a target pixel block.

[0141] In an optional example, step 224 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the determination submodule 474 executed by the processor.

[0142] In an optional embodiment, the set threshold value may be a pre-set value. By determining whether the absolute value of the pixel value difference is less than the set threshold value, it can be determined whether the pixel block is a relatively smooth pixel block (a pixel block with a lower resolution). If the pixel value difference is less than the set threshold value, it indicates that the pixel block does not have obvious feature changes and can be determined as a target pixel block. For example, the image corresponding to the pixel block may be part of a floor.

[0143] In an optional example, step S202 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by the first determining module 47 executed by the processor.

[0144] Step 203: According to the position coordinates of at least one target pixel block in the image to be Gaussian processed, traverse the position range corresponding to the image to be matched to obtain the corresponding matching pixel block.

[0145] The image to be matched is an original panoramic image or a rendered panoramic image.

[0146] In an optional embodiment, when the image to be Gaussian processed is a rendered panoramic image, the image to be matched is an original panoramic image; when the image to be Gaussian processed is an original panoramic image, the image to be matched is a rendered panoramic image.

[0147] In an optional embodiment, the position range corresponding to the image to be matched may be a position range including the position coordinates of the target pixel block and at least one pixel adjacent to the position coordinates. For example, the position coordinates of the target pixel block in the image to be Gaussian processed and a range of 10 pixels * 10 pixels centered on the position coordinates. After traversing to obtain a matching pixel block, the position coordinates of the matching pixel block are recorded.

[0148] In an optional example, step S203 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by the second determining module 48 executed by the processor.

[0149] Step 204 adjusts the image color of the to-be-filled area of ​​the stitched panorama based on the color difference between at least one target pixel block and the corresponding matching pixel block, thereby obtaining a panorama with consistent tones. Tone refers to light and shadow, where light and shadow are the relationship between light and dark, and tones are the relationship between colors. Tone addresses the relationship between black and white layers, the contrast between virtual and real, and the hue purity of colors in the visual image.

[0150] In an optional embodiment, the color difference between the at least one target pixel block and the corresponding matching pixel block in the to-be-matched image may include a color temperature difference, a brightness difference, a contrast difference, and the like.

[0151] In an optional embodiment, since there is at least one target pixel block in the valid area of ​​the image to be Gaussian processed, the color difference between the at least one target pixel block and the corresponding matching pixel block can be calculated, and then the color difference between the rendered panorama and the original panorama is obtained by calculating the average of the color differences between the at least one target pixel block and the corresponding matching pixel block, that is, the color difference between the area to be filled and the other area (the valid area of ​​the original panorama) in the stitched panorama is obtained.

[0152] In an optional example, step S204 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the color adjustment module 49 executed by the processor.

[0153] In one embodiment, the method for adjusting the color of the image in the to-be-filled area of ​​the stitched panorama can be seen in FIG2G , which is a flowchart of the specific implementation of step 204 in the process of adjusting the color of the stitched panorama of the present disclosure. As shown in FIG2G , the following steps 241 to 244 are included:

[0154] Step 241 : determining a white balance correction matrix based on an image intensity difference between at least one target pixel block and a corresponding matching pixel block.

[0155] In one embodiment, by rendering the image intensity difference between the panorama and the original panorama and using a white balance consistency algorithm, a white balance correction matrix, that is, a mapping matrix from RGB three channels to RGB three channels (3x3 white balance correction matrix), can be obtained.

[0156] In an optional example, step 241 may be executed by the processor calling a corresponding instruction stored in a memory, or may be executed by the first determining submodule 491 executed by the processor.

[0157] Step 242 : Using the white balance correction matrix, adjust the image intensity value of the to-be-filled area of ​​the stitched panorama to obtain a white-balanced panorama.

[0158] In an optional example, step 242 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first adjustment submodule 492 executed by the processor.

[0159] Step 243: Use a color mapping function to adjust the contrast and brightness of the area to be filled in the white balanced panoramic image so that the contrast of the area to be filled after adjustment is consistent with the contrast of other areas of the white balanced panoramic image, and the brightness of the area to be filled after adjustment is consistent with the brightness of other areas of the white balanced panoramic image; and obtain a color adjustment area.

[0160] The other areas are areas in the white balance panorama except the area to be filled.

[0161] Optionally, the histogram of the rendered panorama and the histogram of the original panorama can be calculated, the color mapping function can be determined based on the histogram of the rendered panorama and the histogram of the original panorama, and the contrast and brightness of the area to be filled in the white balanced panorama can be adjusted using the color adjustment function. The contrast of the adjusted area to be filled in the obtained white balanced panorama is consistent with the brightness of other areas of the white balanced panorama, and the brightness of the adjusted area to be filled in the obtained white balanced panorama is consistent with the brightness of other areas of the white balanced panorama.

[0162] It should be noted that the color temperature, brightness, and contrast adjustment algorithms in the existing technology can be used to adjust the color temperature, brightness, and contrast of the area to be filled in the stitched panorama by the rendered panorama, so that the color temperature, brightness, and contrast of the entire stitched panorama remain consistent. The implementation process of using the white balance consistency algorithm and color mapping function to adjust the image color is not introduced in detail here.

[0163] In an optional example, step 243 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the second adjustment submodule 493 executed by the processor.

[0164] Step 244 : Using a Poisson fusion algorithm, the color adjustment area is fused with other areas to obtain a panorama with consistent tones.

[0165] In an optional embodiment, because the human eye is often sensitive to color differences, a Poisson blending algorithm can be applied to the color-adjusted panorama to achieve a more natural transition between gradient and grayscale values ​​at the stitching point of the stitched panorama. The effect is shown in Figure 2E , where the processed panorama maintains overall color consistency. The Poisson blending algorithm is an image fusion method whose basic principle is to find a mapping function based on the Poisson equation so that the pixel values ​​and gradient values ​​within the fusion region meet given conditions as closely as possible, thereby achieving seamless fusion of the two images.

[0166] In the above embodiment, the color of the stitched panorama can be adjusted by white balance consistency, color adjustment functions, etc., to ensure that after the original panorama is filled with the rendered panorama, the filled area of ​​the original panorama is consistent in color with other areas of the original panorama, thereby improving the image effect of the panorama of the target scene. White balance is an indicator that describes the accuracy of white after the three primary colors of red, green and blue are mixed in the image. Color adjustment functions may include but are not limited to: grb function, hsv function, color palettes function, etc. The grb function creates colors by setting the color saturation of the three given basic colors red, green and blue. The hsv function is a function that converts colors from RGB format to HSV (hue, saturation, value) format. The color palettes function is a powerful function in the R language that can be used to create a custom color palette.

[0167] In an optional example, step 244 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the Poisson fusion submodule 494 executed by the processor.

[0168] FIG3A is a flowchart of the present disclosure for adjusting the texture of a stitched panorama. FIG3B is a diagram illustrating the effect of texture inconsistency in the present disclosure's method for generating a panorama. FIG3C is a panorama after texture optimization in the present disclosure's method for generating a panorama. FIG3D is a schematic diagram of the present disclosure for using an optimized gradient map to guide graphic deformation. The embodiment uses an example of how to adjust the texture of a stitched panorama as an example, as shown in FIG3A , and includes the following steps:

[0169] Step 301: Obtain a gradient image of at least one rendered panoramic image and a corresponding gradient image of a stitched panoramic image.

[0170] In an optional embodiment, the stitched panorama may have inconsistent textures, for example, the floor position indicated by number 31 in FIG3B (which is not the same acquisition point as the rendering used in the embodiment shown in FIG2A ). Therefore, this embodiment calculates the gradient map of the stitched panorama to obtain the structural lines of the stitched panorama, and compares the gradient map of the stitched panorama with the gradient map of the rendered panorama to obtain the gradient information of the area to be filled.

[0171] In an optional example, step 301 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the gradient map acquisition module 50 executed by the processor.

[0172] Step 302 : Based on the gradient map of at least one rendered panoramic image, line optimization is performed on the gradient map of the corresponding stitched panoramic image to obtain an optimized gradient map of the stitched panoramic image.

[0173] In an optional embodiment, after extracting the gradient information of the area to be filled, a structure line (structure_line_fillarea) of the area to be filled can be fitted using singular value decomposition (SVD). This structure line of the area to be filled is optimized to be consistent with the structure lines of other areas of the stitched panorama. Singular value decomposition is a method for extracting information that simplifies data and removes noise and redundant data.

[0174] In an optional embodiment, the method of optimizing the structural lines of the area to be filled may include, but is not limited to, image deformation based on the moving least squares method. After optimizing the lines of the gradient map of the stitched panorama, the optimized gradient map of the stitched panorama can be obtained. Image deformation based on the moving least squares method controls the deformation through a set of source control points and target control points. For each point whose position after deformation is to be determined, a least squares optimization objective function is solved according to a preset deformation type (such as affine transformation, similarity transformation, rigid transformation, etc.) to estimate a local coordinate transformation matrix. The coordinate transformation matrix is ​​applied to the current point to calculate the position of the point after deformation.

[0175] In an optional example, step 302 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the optimization gradient module 51 executed by the processor.

[0176] Step 303 : The optimized gradient map of the stitched panorama is used to guide a preset network model to optimize the texture information of the panorama with consistent tones, thereby obtaining a final panorama.

[0177] In an optional embodiment, FIG. 3D illustrates a process of optimizing a color-adjusted stitched panorama (tonally consistent panorama) using an optimized gradient map.

[0178] Among them, the optimization process can include: merging the optimized gradient map and the tonally consistent panorama through a multi-channel merging function to obtain a multi-channel feature map; using a pre-trained preset network model to perform feature extraction and grid sampling on the multi-channel feature map to obtain a spline after grid sampling; optimizing the texture information in the tonally consistent panorama through the spline after grid sampling to obtain the final panorama.

[0179] Optionally, the optimized gradient map and the tonal consistent panorama can be spliced ​​on the channel through a multi-channel merging (concat) function to obtain a multi-channel feature map, and then the multi-channel feature map is down-sampled, feature extracted and grid sampled to obtain a spline curve. The lines and color affine transformation algorithm obtained based on the grid sampling can guide the tonal consistent panorama to deform, complete the texture adjustment, and obtain the final effect map, see Figure 3C. After the texture optimization, the edge lines of the floor tiles are consistent, which solves the problem of texture misalignment. The concat function is used to connect two arrays. The preset network model in this embodiment can be a deep neural network model of any structure. The present disclosure does not limit the structure of the preset network model, and it can achieve feature extraction and grid sampling.

[0180] In an optional example, step 303 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the image texture optimization module 52 executed by the processor.

[0181] Through the above embodiment, the problem of inconsistent texture between the area to be filled and the original image in the stitched panorama can be repaired, so that the final panorama of the three-dimensional real scene obtained is not only complete but also has consistent texture and good image effect.

[0182] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0183] Corresponding to the aforementioned embodiments of the method for generating a panoramic image, the present disclosure also provides corresponding embodiments of an apparatus for generating a panoramic image.

[0184] FIG4 is a schematic diagram of the structure of an embodiment of a device for generating a panoramic image according to the present disclosure. The device is applied to an electronic device (such as a computer system or a server). As shown in FIG4 , the device includes:

[0185] The acquisition module 41 is configured to acquire depth information and image information of a target scene at at least one acquisition point.

[0186] The model generation module 42 is used to generate a three-dimensional space model of the target scene based on the depth information and the image information.

[0187] The original image generating module 43 is configured to generate at least one original panoramic image based on the image information, wherein the at least one original panoramic image includes an area to be filled.

[0188] The original panoramic image is obtained by stitching image information collected from the same collection points.

[0189] The rendering image generation module 44 is used to render and generate at least one rendered panoramic image based on the three-dimensional space model and at least one acquisition point.

[0190] The filling module 45 is configured to fill the to-be-filled area in the original panoramic image corresponding to the acquisition point based on the at least one rendered panoramic image, so as to obtain a stitched panoramic image of the at least one acquisition point.

[0191] FIG5 is a schematic diagram of the structure of another embodiment of the panoramic image generation device disclosed in the present invention. As shown in FIG5, based on the embodiment shown in FIG4, in one embodiment, the device of this embodiment further includes:

[0192] The Gaussian processing module 46 is used to perform a Gaussian kernel convolution operation on the Gaussian processed image for the acquisition points to obtain a Gaussian convolution image.

[0193] The first determination module 47 is configured to determine at least one target pixel block in the image to be Gaussian processed based on a pixel value difference between at least one pixel block in the image to be Gaussian processed and a corresponding pixel block in the Gaussian convolution image.

[0194] The absolute value of the pixel value difference corresponding to at least one target pixel block is smaller than a set threshold.

[0195] The second determining module 48 is configured to traverse the position range corresponding to the image to be matched to obtain a corresponding matching pixel block according to the position coordinates of at least one target pixel block in the image to be Gaussian processed.

[0196] The color adjustment module 49 is configured to adjust the image color of the to-be-filled area of ​​the stitched panorama based on the color difference between at least one target pixel block and the corresponding matching pixel block, so as to obtain a panorama with consistent tones.

[0197] The image to be matched is an original panorama or a rendered panorama. When the image to be processed by Gaussian is a rendered panorama, the image to be matched is an original panorama; when the image to be processed by Gaussian is an original panorama, the image to be matched is a rendered panorama.

[0198] In an optional embodiment, the color adjustment module 49 includes:

[0199] The first determination submodule 491 is used to determine a white balance correction matrix based on an image intensity difference between at least one target pixel block and a corresponding matching pixel block;

[0200] A first adjustment submodule 492 is configured to adjust the image intensity value of the to-be-filled area of ​​the stitched panorama using a white balance correction matrix to obtain a white-balanced panorama;

[0201] The second adjustment submodule 493 is configured to use a color mapping function to adjust the contrast and brightness of the area to be filled in the white-balanced panoramic image so that the contrast of the area to be filled after adjustment is consistent with the contrast of other areas of the white-balanced panoramic image, and the brightness of the area to be filled after adjustment is consistent with the brightness of other areas of the white-balanced panoramic image, thereby obtaining a color-adjusted area; the other areas are areas in the white-balanced panoramic image excluding the area to be filled;

[0202] The Poisson fusion submodule 494 is used to fuse the color-adjusted panoramic image region with other regions using a Poisson fusion algorithm to obtain a panoramic image with consistent tones.

[0203] In an optional embodiment, the first determining module 47 includes:

[0204] The pixel block division submodule 471 is used to divide the image to be Gaussian processed and the Gaussian convolution image into at least one pixel block according to the same division standard;

[0205] a pixel value calculation submodule 472 for respectively calculating the pixel mean of at least one pixel block in the image to be Gaussian processed and the pixel mean of at least one pixel block in the Gaussian convolution image, to obtain at least one first pixel mean and at least one second pixel mean;

[0206] a difference calculation submodule 473 for determining a pixel value difference between at least one first pixel mean and a corresponding second pixel mean;

[0207] The determination submodule 474 is configured to determine at least one pixel block in the Gaussian convolution image as at least one target pixel block according to the pixel value difference.

[0208] In an optional embodiment, the providing device may further include:

[0209] A gradient map acquisition module 50 is used to respectively acquire a gradient map of at least one rendered panoramic image and a gradient map of a corresponding stitched panoramic image;

[0210] A gradient optimization module 51 is configured to optimize the gradient map of the corresponding stitched panorama based on the gradient map of at least one rendered panorama, thereby obtaining an optimized gradient map of the stitched panorama;

[0211] The image texture optimization module 52 is configured to guide a preset network model to optimize the texture information of the tonally consistent panorama through the optimized gradient map of the stitched panorama to obtain a final panorama.

[0212] In yet another embodiment of the present disclosure, the image texture optimization module 52 includes:

[0213] A channel merging submodule 521 is configured to merge the optimized gradient map and the tonal consistent panoramic image using a multi-channel merging function to obtain a multi-channel feature map;

[0214] A feature extraction submodule 522 is used to perform feature extraction and grid sampling on the multi-channel feature map using a pre-trained preset network model to obtain a spline after grid sampling;

[0215] The image optimization submodule 523 is used to optimize the texture information in the tonal consistent panorama by using the spline after grid sampling to obtain the final panorama.

[0216] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0217] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0218] Below, with reference to Figure 6, an electronic device according to an embodiment of the present disclosure is described, in which an apparatus for implementing the method according to an embodiment of the present disclosure may be integrated. Figure 6 is a structural diagram of an electronic device provided by an illustrative embodiment of the present disclosure. As shown in Figure 6, the electronic device includes one or more processors 61, one or more computer-readable storage media memory 62, and a computer program stored in the memory and executable on the processor. When the program in memory 62 is executed, the above-mentioned method for generating a panoramic image can be implemented.

[0219] Specifically, in actual applications, the electronic device may further include components such as an input device 63 and an output device 64, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Those skilled in the art will appreciate that the structure of the electronic device shown in FIG6 does not limit the electronic device, and may include more or fewer components than shown, or certain components, or different component arrangements.

[0220] The processor 61 can be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 62, and calling data stored in the memory 62, thereby monitoring the electronic device as a whole.

[0221] The memory 62 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 61 may execute the program instructions to implement the panoramic image generation method of each embodiment of the present disclosure above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0222] The input device 63 may be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0223] The output device 64 can output various information to the outside, including the determined distance information, direction information, etc. The output device 64 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0224] The electronic device may also include a power supply for supplying power to various components, which may be logically connected to the processor 61 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply may also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0225] Of course, for simplicity, FIG6 only shows some of the components of the electronic device related to the present disclosure, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may further include any other appropriate components depending on the specific application.

[0226] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method for generating a panoramic image according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0227] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0228] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method for generating a panoramic image according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0229] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0230] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0231] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0232] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0233] The description of the present disclosure is provided for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the disclosed form. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure and to enable those skilled in the art to understand the present disclosure and design various embodiments with various modifications suitable for specific applications.

Claims

1. A method for generating a panoramic image, characterized in that: The method comprises: At at least one acquisition point, collecting depth information and image information of the target scene; Based on the depth information and the image information, generating a three-dimensional space model of the target scene; Based on the image information, generating at least one original panoramic image, wherein the at least one original panoramic image has an area to be filled; Based on the three-dimensional space model and the at least one acquisition point, rendering and generating at least one rendered panoramic image; Based on the at least one rendered panoramic image, the to-be-filled area in the original panoramic image corresponding to the acquisition point is filled to obtain a stitched panoramic image of the at least one acquisition point.

2. The method according to claim 1, characterized in that The method further comprises: For the acquisition points, performing a Gaussian kernel convolution operation on the image to be Gaussian processed to obtain a Gaussian convolution image, wherein the image to be Gaussian processed is the rendered panoramic image or the original panoramic image; Based on the pixel value difference between at least one pixel block in the image to be Gaussian processed and the corresponding pixel block in the Gaussian convolution image, determining at least one target pixel block in the image to be Gaussian processed, wherein the absolute value of the pixel value difference corresponding to the target pixel block is less than a set threshold; According to the position coordinates of the at least one target pixel block in the image to be Gaussian processed, traverse the position range corresponding to the image to be matched to obtain the corresponding matching pixel block, wherein the image to be matched is the original panoramic image or the rendered panoramic image; Based on the color difference between the at least one target pixel block and the corresponding matching pixel block, the image color of the to-be-filled area of ​​the stitched panorama is adjusted to obtain a panorama with consistent tones.

3. The method according to claim 2, characterized in that The step of adjusting the image color of the to-be-filled area of ​​the stitched panorama based on the color difference between the at least one target pixel block and the corresponding matching pixel block to obtain a panorama with consistent tones includes: determining a white balance correction matrix based on an image intensity difference between the at least one target pixel block and a corresponding matching pixel block; Using the white balance correction matrix to adjust the image intensity value of the to-be-filled area of ​​the stitched panorama to obtain a white balanced panorama; Using a color mapping function, adjusting the contrast and brightness of the area to be filled in the white balance panorama, so that the contrast of the area to be filled after adjustment is consistent with the contrast of other areas of the white balance panorama, and the brightness of the area to be filled after adjustment is consistent with the brightness of other areas of the white balance panorama, to obtain a color adjustment area; the other area is the area in the white balance panorama except the area to be filled; The color adjustment area and the other areas are fused using a Poisson fusion algorithm to obtain the panorama with consistent tones.

4. The method according to claim 2 or 3, characterized in that: The determining at least one target pixel block in the image to be Gaussian processed based on a pixel value difference between at least one pixel block in the image to be Gaussian processed and a corresponding pixel block in the Gaussian convolution image comprises: Dividing the image to be Gaussian processed and the Gaussian convolution image into at least one pixel block according to the same division standard; Respectively calculating a pixel mean of at least one pixel block in the to-be-Gaussian-processed image and a pixel mean of at least one pixel block in the Gaussian convolution image to obtain at least one first pixel mean and at least one second pixel mean; determining a pixel value difference between the at least one first pixel mean and the corresponding second pixel mean; At least one pixel block in the Gaussian convolution image is determined as the at least one target pixel block according to the pixel value difference.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: Obtaining a gradient map of the at least one rendered panoramic image and a corresponding gradient map of the stitched panoramic image; Based on the gradient map of the at least one rendered panoramic image, performing line optimization on the corresponding gradient map of the stitched panoramic image to obtain an optimized gradient map of the stitched panoramic image; The preset network model is guided by the optimized gradient map of the stitched panorama to optimize the texture information of the panorama with consistent tones to obtain a final panorama.

6. The method according to claim 5, characterized in that The step of guiding a preset network model to optimize the texture information of the panorama with consistent tones through the optimized gradient map of the stitched panorama to obtain a final panorama includes: Merging the optimized gradient map and the tonal consistent panoramic map through a multi-channel merging function to obtain a multi-channel feature map; Using a pre-trained preset network model to perform feature extraction and grid sampling on the multi-channel feature map to obtain a spline after grid sampling; The texture information in the tonal consistent panorama is optimized by the spline after the grid sampling to obtain the final panorama.

7. A device for generating a panoramic image, characterized in that: The device comprises: A collection module, used to collect depth information and image information of a target scene at at least one collection point; A model generation module, used to generate a three-dimensional space model of the target scene based on the depth information and the image information; An original image generation module, used to generate at least one original panoramic image based on the image information, wherein the at least one original panoramic image has an area to be filled; A rendering image generation module, used for rendering and generating at least one rendering panoramic image based on the three-dimensional space model and the at least one acquisition point; A filling module is used to fill the to-be-filled area in the original panorama corresponding to the acquisition point based on the at least one rendered panorama to obtain a stitched panorama of the at least one acquisition point.

8. The device according to claim 7, characterized in that The device also includes: A Gaussian processing module, for performing a Gaussian kernel convolution operation on the Gaussian processing image for the acquisition point to obtain a Gaussian convolution image, wherein the Gaussian processing image is the rendered panoramic image or the original panoramic image; A first determination module is used to determine at least one target pixel block in the image to be Gaussian processed based on a pixel value difference between at least one pixel block in the image to be Gaussian processed and a corresponding pixel block in the Gaussian convolution image, wherein an absolute value of the pixel value difference corresponding to the at least one target pixel block is less than a set threshold; A second determination module is used to traverse and obtain corresponding matching pixel blocks within a position range corresponding to the image to be matched according to the position coordinates of the at least one target pixel block in the image to be Gaussian processed, wherein when the image to be Gaussian processed is the rendered panoramic image, the image to be matched is the original panoramic image, and when the image to be Gaussian processed is the original panoramic image, the image to be matched is the rendered panoramic image; The color adjustment module is used to adjust the image color of the to-be-filled area of ​​the stitched panorama based on the color difference between the at least one target pixel block and the corresponding matching pixel block to obtain a panorama with consistent tones.

9. An electronic device, characterized in that: include: Memory, used to store computer products; A processor is used to execute the computer product stored in the memory, and when the computer product is executed, the method for generating a panoramic image described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method for generating a square panoramic image described in any one of claims 1 to 6 is implemented.

11. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method for generating a panoramic image described in any one of claims 1 to 6 is implemented.

Images