Image processing method and device, electronic device and storage medium

Abstract

The present invention provides an image processing method and device, an electronic device and a storage medium, wherein the method comprises: obtaining depth information of a panoramic picture and depth information of an embedded model; wherein the depth information of the panoramic picture includes depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes depth information of each pixel of the embedded model finally drawn on the screen; comparing the depth information of the panoramic picture with the depth information of the embedded model, confirming the occlusion relationship between the embedded model and the panoramic picture in the scene; setting the pixel values ​​of the panoramic picture and the embedded model in the scene according to the occlusion relationship, so as to fuse the panoramic picture with the embedded model. The method enables the panoramic picture and the embedded model to form a correct occlusion relationship, thereby improving the comprehensive expression ability of the panoramic picture.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and device, electronic equipment and storage medium. Background Art

[0002] Currently, due to the limitations of hardware performance and bandwidth on the web page, it is impossible to display high-precision scene models. Usually, high-precision models are rendered as panoramic images and displayed on the web page. In addition, based on high-precision models, interactive pictures or videos can be embedded in the panoramic images, which increases interactivity and provides more powerful information expression capabilities.

[0003] However, since panoramic images are two-dimensional images, the relative position relationship of objects in the scene model is lost during the rendering process, resulting in an incorrect occlusion relationship between the embedded image or video and the panoramic image. Summary of the invention

[0004] The present invention provides an image processing method, device and electronic device, which are used to solve the defect of incorrect occlusion of panoramic pictures and embedded models in the prior art and achieve correct fusion of the two.

[0005] The present invention provides an image processing method, comprising:

[0006] Acquire depth information of the panoramic picture and depth information of the embedded model; wherein the depth information of the panoramic picture includes depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes depth information of each pixel of the embedded model finally drawn on the screen;

[0007] Comparing the depth information of the panoramic picture with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic picture in the scene;

[0008] Pixel values ​​of the panoramic image and the embedded model in the scene are set according to the occlusion relationship to fuse the panoramic image with the embedded model.

[0009] According to an image processing method provided by the present invention, the step of obtaining depth information of a panoramic image and depth information of an embedded model includes:

[0010] Rendering a panoramic picture and calculating the depth information corresponding to each pixel, and saving the depth information of the panoramic picture;

[0011] Paste a panoramic picture in the scene and determine the camera position and orientation;

[0012] Obtain the embedding model to which the image and / or video needs to be attached, and determine the size and position of the embedding model;

[0013] Calculate the depth information of the embedded model under the current camera.

[0014] According to an image processing method provided by the present invention, comparing the depth information of the panoramic image with the depth information of the embedded model, and confirming the occlusion relationship between the embedded model and the panoramic image in the scene includes:

[0015] Compare the depth information of the embedded model pixels with the depth information of the panoramic image pixels;

[0016] If the depth information of the embedded model pixel is greater than the depth information of the panoramic picture pixel, determining that the panoramic picture pixel occludes the embedded model pixel;

[0017] If the depth information of the panoramic picture pixel is greater than the depth information of the embedded model pixel, it is determined that the embedded model pixel occludes the panoramic picture pixel.

[0018] According to an image processing method provided by the present invention, setting the pixel values ​​of the panoramic image and the embedded model in the scene according to the occlusion relationship to fuse the panoramic image with the embedded model includes:

[0019] According to the occlusion relationship, the transparency of the occluded panoramic picture pixels or embedded model pixels is set to 0, so as to merge the panoramic picture with the embedded model.

[0020] The present invention also provides an image processing device, comprising:

[0021] A depth information acquisition module, used to acquire depth information of the panoramic picture and depth information of the embedded model; wherein the depth information of the panoramic picture includes depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes depth information of each pixel of the embedded model finally drawn on the screen;

[0022] A fusion module is used to compare the depth information of the panoramic image with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic image in the scene; and set the pixel values ​​of the panoramic image and the embedded model in the scene according to the occlusion relationship to fuse the panoramic image with the embedded model.

[0023] According to an image processing device provided by the present invention, the depth information acquisition module is further used for:

[0024] Rendering a panoramic picture and calculating the depth information corresponding to each pixel, and saving the depth information of the panoramic picture;

[0025] Paste a panoramic picture in the scene and determine the camera position and orientation;

[0026] Obtain the embedding model to which the image and / or video needs to be attached, and determine the size and position of the embedding model;

[0027] Calculate the depth information of the embedded model under the current camera.

[0028] According to an image processing device provided by the present invention, the fusion module is further used for:

[0029] Compare the depth information of the embedded model pixels with the depth information of the panoramic image pixels;

[0030] If the depth information of the embedded model pixel is greater than the depth information of the panoramic picture pixel, determining that the panoramic picture pixel occludes the embedded model pixel;

[0031] If the depth information of the panoramic picture pixel is greater than the depth information of the embedded model pixel, it is determined that the embedded model pixel occludes the panoramic picture pixel.

[0032] According to an image processing device provided by the present invention, the fusion module is also used for:

[0033] According to the occlusion relationship, the transparency of the occluded panoramic picture pixels or embedded model pixels is set to 0, so as to merge the panoramic picture with the embedded model.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned image processing methods when executing the program.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned image processing methods.

[0036] The image processing method and device provided by the present invention obtain the depth information of the panoramic picture and the depth information of the embedded model in the process of rendering the panoramic picture to save the relative position relationship of the objects in the scene model, and then set the pixel values ​​of the embedded model and the panoramic picture in the scene according to the depth information of the panoramic picture and the depth information of the embedded model, and merge the two to ensure that the panoramic picture and the embedded model can form a correct occlusion relationship, thereby improving the comprehensive expression ability of the panoramic picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present invention;

[0039] Figure 2 It is a panoramic picture embedded in a picture video provided by an embodiment of the present invention;

[0040] Figure 3 It is a schematic diagram of a process of obtaining depth information of a panoramic image and depth information of an embedded model provided by an embodiment of the present invention;

[0041] Figure 4 It is a flow chart of comparing the depth information of a panoramic picture with the depth information of an embedded model to confirm the occlusion relationship between the embedded model and the panoramic picture in a scene, provided by an embodiment of the present invention;

[0042] Figure 5 is a structural schematic diagram of an image processing device provided by an embodiment of the present invention;

[0043] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0046] Step 110, obtaining depth information of the panoramic image and depth information of the embedded model.

[0047] The depth information of the panoramic picture includes the depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes the depth information of each pixel of the embedded model finally drawn on the screen.

[0048] Among them, the panoramic picture is stitched together by one or more groups of photos taken by the camera in a 360-degree circle, which fully displays all the scenery within the 360-degree spherical range, retains the authenticity of the scene to the maximum extent, and the flat photos are processed by software to obtain a three-dimensional sense of space. The embedded model refers to a model embedded in the scene to attach pictures and videos. The depth information represents the distance from the image collector to each point in the scene. The depth information of the panoramic picture and the embedded model is usually obtained by rendering the parameters of the camera and performing a projection transformation to obtain the depth corresponding to each pixel.

[0049] Step 120: compare the depth information of the panoramic image with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic image in the scene.

[0050] Specifically, the pixel area with a deeper depth is farther away from the image collector, and will be blocked by the pixel area with a lighter depth and cannot be presented in the current field of view.

[0051] Step 130: setting pixel values ​​of the panoramic image and the embedded model in the scene according to the occlusion relationship to fuse the panoramic image with the embedded model.

[0052] In order to display the correct occlusion relationship between the panoramic image and the embedded model, the occluded area needs to be set to transparent. The way to set transparency is usually to change the alpha channel value of the occluded pixels.

[0053] Specifically, Figure 2 For panoramic images embedded in image videos, such as Figure 2 As shown, there are many embedding methods, such as embedding the model above the panoramic picture, or inserting the panoramic picture at a certain angle. Different embedding positions may cause errors in the occlusion relationship between the embedded model and the panoramic picture, affecting the viewing effect. This method first obtains the depth information of the panoramic picture as the background and the embedded model to which the picture or video needs to be attached, compares the two, and then confirms the occlusion relationship, so as to display all or part of the attached picture or video, and at the same time allow users to obtain a three-dimensional viewing experience.

[0054] The method provided by the embodiment of the present invention obtains the depth information of the panoramic picture and the depth information of the embedded model during the rendering of the panoramic picture to save the relative position relationship of objects in the scene model, and then sets the pixel values ​​of the embedded model and the panoramic picture in the scene according to the depth information of the panoramic picture and the depth information of the embedded model, and merges the two to ensure that the panoramic picture and the embedded model can form a correct occlusion relationship, thereby improving the comprehensive expression ability of the panoramic picture.

[0055] According to one embodiment of the present invention, Figure 3FIG. 1 is a flow chart of obtaining the depth information of the panoramic image and the depth information of the embedded model in step 110. Figure 3 As shown, step 110 includes:

[0056] Step 310 , rendering the panoramic picture and calculating the depth information corresponding to each pixel, and saving the depth information of the panoramic picture.

[0057] Specifically, in the process of rendering the panoramic picture, the depth information corresponding to each pixel of the panoramic picture is calculated by rendering the camera parameters and performing projection transformation, and is saved in the alpha channel of the panoramic picture or saved separately as an independent depth image.

[0058] Among them, the α channel refers to a special channel or a "non-color" channel. Its value ranges from 0 to 1, and is used to store the contribution of the current pixel to the image, where 0 represents transparency and 1 represents opaqueness. Therefore, the α channel is mainly used to save and edit selections. When generating an image file, it is not necessary to generate an α channel. It is usually generated artificially during the image processing process, and the selection area information is read from it. Therefore, when outputting the plate, the α channel will be deleted because it is irrelevant to the final generated image. But sometimes, such as when the three-dimensional software is finally rendered and output, an α channel will be generated for later synthesis in the plane processing software. Therefore, in an embodiment of the invention, the depth information can be saved in the α channel.

[0059] Among them, the depth image refers to an image that uses the depth from the image collector to each point in the scene as the pixel value, which can directly reflect the geometric shape of the visible surface of the object in the scene.

[0060] Step 320, attaching a panoramic image to the scene, and determining the camera position and orientation.

[0061] Step 330: Obtain an embedded model to which the image and / or video needs to be attached, and determine the size and position of the embedded model.

[0062] Among them, posture represents the spatial position of an object, which is the collective name of position (x, y, z) and posture (RX, RY, RZ).

[0063] Step 340: Calculate the depth information of the embedded model under the current camera.

[0064] According to one embodiment of the present invention, Figure 4 FIG. 1 is a flow chart of comparing the depth information of the panoramic image with the depth information of the embedded model in step 120 to confirm the occlusion relationship between the embedded model and the panoramic image in the scene, as shown in FIG. Figure 4 As shown, step 120 includes:

[0065] Step 410 , comparing the depth information of the embedded model pixels with the depth information of the panoramic image pixels.

[0066] Step 420: If the depth information of the embedded model pixel is greater than the depth information of the panoramic picture pixel, it is determined that the panoramic picture pixel occludes the embedded model pixel.

[0067] Step 430: If the depth information of the panoramic picture pixel is greater than the depth information of the embedded model pixel, it is determined that the embedded model pixel occludes the panoramic picture pixel.

[0068] Specifically, pixel areas with deeper depths are farther away from the image collector and will be blocked by pixel areas with shallower depths and cannot be presented in the current field of view.

[0069] According to one embodiment of the present invention, setting the pixel values ​​of the panoramic image and the embedded model in the scene according to the occlusion relationship to fuse the panoramic image with the embedded model includes: setting the transparency of the occluded panoramic image pixels or embedded model pixels to 0 according to the occlusion relationship to fuse the panoramic image with the embedded model.

[0070] It should be noted that in the process of converting the imaging of objects in three-dimensional space into two-dimensional images in the image collector, objects closer to the image collector occupy a larger area in the field of view and may block objects farther away from the image collector. In the generated two-dimensional image, the geometric shape of the blocked object will not be displayed, so the blocked area needs to be set to transparent. The way to set transparency is usually to change the alpha channel value of the blocked pixel.

[0071] The image processing device provided by the present invention is described below. The image processing device described below and the image processing method described above can be referred to each other.

[0072] Figure 5 This is a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. As shown in the figure, the device includes: a depth information acquisition module 510 and a fusion module 520.

[0073] A depth information acquisition module 510 is used to acquire depth information of the panoramic image and depth information of the embedded model;

[0074] The depth information of the panoramic picture includes the depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes the depth information of each pixel of the embedded model finally drawn on the screen;

[0075] The fusion module 520 is used to compare the depth information of the panoramic image with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic image in the scene; and set the pixel values ​​of the panoramic image and the embedded model in the scene according to the occlusion relationship to fuse the panoramic image with the embedded model.

[0076] According to an embodiment of the present invention, the depth information acquisition module 510 is further used to:

[0077] Rendering a panoramic picture and calculating the depth information corresponding to each pixel, and saving the depth information of the panoramic picture;

[0078] Paste a panoramic picture in the scene and determine the camera position and orientation;

[0079] Obtain the embedding model to which the image and / or video needs to be attached, and determine the size and position of the embedding model;

[0080] Calculate the depth information of the embedded model under the current camera.

[0081] According to one embodiment of the present invention, the fusion module 520 is further configured to:

[0082] Compare the depth information of the embedded model pixels with the depth information of the panoramic image pixels;

[0083] If the depth information of the embedded model pixel is greater than the depth information of the panoramic picture pixel, determining that the panoramic picture pixel occludes the embedded model pixel;

[0084] If the depth information of the panoramic picture pixel is greater than the depth information of the embedded model pixel, it is determined that the embedded model pixel occludes the panoramic picture pixel.

[0085] According to one embodiment of the present invention, the fusion module 520 is further configured to:

[0086] According to the occlusion relationship, the transparency of the occluded panoramic picture pixels or embedded model pixels is set to 0, so as to merge the panoramic picture with the embedded model.

[0087] The image processing method and device provided by the present invention obtain the depth information of the panoramic picture and the depth information of the embedded model in the process of rendering the panoramic picture to save the relative position relationship of the objects in the scene model, and then set the pixel values ​​of the embedded model and the panoramic picture in the scene according to the depth information of the panoramic picture and the depth information of the embedded model, and merge the two to ensure that the panoramic picture and the embedded model can form a correct occlusion relationship, thereby improving the comprehensive expression ability of the panoramic picture.

[0088] Figure 6An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image processing method, which includes: obtaining the depth information of the panoramic picture and the depth information of the embedded model; wherein the depth information of the panoramic picture includes the depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes the depth information of each pixel of the embedded model finally drawn on the screen; comparing the depth information of the panoramic picture with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic picture in the scene; setting the pixel values ​​of the panoramic picture and the embedded model in the scene according to the occlusion relationship to merge the panoramic picture with the embedded model.

[0089] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image processing method provided by the above-mentioned methods, and the method includes: obtaining depth information of a panoramic picture and depth information of an embedded model; wherein the depth information of the panoramic picture includes depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes depth information of each pixel of the embedded model finally drawn on the screen; comparing the depth information of the panoramic picture with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic picture in the scene; setting the pixel values ​​of the panoramic picture and the embedded model in the scene according to the occlusion relationship to fuse the panoramic picture with the embedded model.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-mentioned image processing methods, the methods comprising: obtaining depth information of a panoramic picture and depth information of an embedded model; wherein the depth information of the panoramic picture includes depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes depth information of each pixel of the embedded model; comparing the depth information of the panoramic picture with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic picture in the scene; and setting the pixel value of the overlapping area of ​​the panoramic picture and the embedded model in the scene according to the occlusion relationship to fuse the panoramic picture with the embedded model.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized in that: include: Acquire depth information of the panoramic picture and depth information of the embedded model; wherein the depth information of the panoramic picture includes depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes depth information of each pixel of the embedded model finally drawn on the screen; Comparing the depth information of the panoramic picture with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic picture in the scene; Setting pixel values ​​of the panoramic image and the embedded model in the scene according to the occlusion relationship to fuse the panoramic image with the embedded model; The step of acquiring the depth information of the panoramic image and the depth information of the embedded model includes: Rendering a panoramic picture and calculating the depth information corresponding to each pixel, and saving the depth information of the panoramic picture; Paste a panoramic picture in the scene and determine the camera position and orientation; Obtain the embedding model to which the image and / or video needs to be attached, and determine the size and position of the embedding model; Calculate the depth information of the embedded model under the current camera.

2. The image processing method according to claim 1, characterized in that: Comparing the depth information of the panoramic picture with the depth information of the embedded model to confirm the occlusion relationship between the embedded model and the panoramic picture in the scene includes: Compare the depth information of the embedded model pixels with the depth information of the panoramic image pixels; If the depth information of the embedded model pixel is greater than the depth information of the panoramic picture pixel, determining that the panoramic picture pixel occludes the embedded model pixel; If the depth information of the panoramic picture pixel is greater than the depth information of the embedded model pixel, it is determined that the embedded model pixel occludes the panoramic picture pixel.

3. The image processing method according to claim 2, characterized in that: Setting pixel values ​​of the panoramic image and the embedded model in the scene according to the occlusion relationship to fuse the panoramic image with the embedded model includes: According to the occlusion relationship, the transparency of the occluded panoramic picture pixels or embedded model pixels is set to 0, so as to merge the panoramic picture with the embedded model.

4. An image processing device, characterized in that: include: A depth information acquisition module, used to acquire depth information of the panoramic picture and depth information of the embedded model; wherein the depth information of the panoramic picture includes depth information of each pixel of the panoramic picture, and the depth information of the embedded model includes depth information of each pixel of the embedded model finally drawn on the screen; A fusion module, used to compare the depth information of the panoramic picture with the depth information of the embedded model, confirm the occlusion relationship between the embedded model and the panoramic picture in the scene; set the pixel values ​​of the panoramic picture and the embedded model in the scene according to the occlusion relationship, so as to fuse the panoramic picture with the embedded model; Wherein, the depth information acquisition module is also used for: Rendering a panoramic picture and calculating the depth information corresponding to each pixel, and saving the depth information of the panoramic picture; Paste a panoramic picture in the scene and determine the camera position and orientation; Obtain the embedding model to which the image and / or video needs to be attached, and determine the size and position of the embedding model; Calculate the depth information of the embedded model under the current camera.

5. The image processing device according to claim 4, characterized in that: The fusion module is also used for: Compare the depth information of the embedded model pixels with the depth information of the panoramic image pixels; If the depth information of the embedded model pixel is greater than the depth information of the panoramic picture pixel, determining that the panoramic picture pixel occludes the embedded model pixel; If the depth information of the panoramic picture pixel is greater than the depth information of the embedded model pixel, it is determined that the embedded model pixel occludes the panoramic picture pixel.

6. The image processing device according to claim 5, characterized in that: The fusion module is also used for: According to the occlusion relationship, the transparency of the occluded panoramic picture pixels or embedded model pixels is set to 0, so as to merge the panoramic picture with the embedded model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the image processing method according to any one of claims 1 to 3 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 3 are implemented.

Images