Image processing method and device, electronic equipment and storage medium

Abstract

According to the image processing method and device, the electronic equipment and the storage medium, the pose of a virtual object associated with a target object is determined based on the pose of the target object in a real scene, and a virtual image corresponding to the virtual object is generated based on the pose of the virtual object. According to the embodiment of the invention, the first target image is obtained based on the virtual image and the real scene image corresponding to the real scene, and the first target image is smoothed to obtain the second target image, so that the virtual object can be added to the target object in the real scene image, and image distortion and image noise caused by image addition can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Virtual reality technology can present an interactive virtual reality space to users. The image see-through function it provides can display real-time images of the real environment, so that users can perceive the real environment around them and interact with the outside world. The implementation methods of image see-through mainly include optical see-through (OST) and video see-through (VST). The former relies on a translucent optical synthesizer to present the image of the real environment to the user, and the latter captures the real-time image of the real environment through a camera and presents it on the display. Among them, video see-through allows complete occlusion between virtual objects and real objects. However, due to the limited computing power of virtual reality devices, the image see-through function they provide often cannot accurately reconstruct the three-dimensional geometric information of the real environment, making it difficult to accurately add virtual objects on its basis. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, according to one or more embodiments of the present disclosure, there is provided an image processing method, comprising:

[0005] Determining a position and posture of a virtual object associated with the target object based on the position and posture of the target object in the real scene;

[0006] generating a virtual image corresponding to the virtual object based on the position and posture of the virtual object;

[0007] Obtaining a first target image based on the virtual image and a real scene image corresponding to the real scene;

[0008] The first target image is smoothed to obtain a second target image.

[0009] In a second aspect, according to one or more embodiments of the present disclosure, there is provided an image processing apparatus, comprising:

[0010] A posture determination unit, used to determine the posture of a virtual object associated with the target object based on the posture of the target object in the real scene;

[0011] A first image processing unit, configured to generate a virtual image corresponding to the virtual object based on the position and posture of the virtual object;

[0012] A second image processing unit, configured to obtain a first target image based on the virtual image and a real scene image corresponding to the real scene;

[0013] The third image processing unit is used to perform smoothing on the first target image to obtain a second target image.

[0014] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory so that the electronic device executes the image processing method provided according to one or more embodiments of the present disclosure.

[0015] In a fourth aspect, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, wherein the non-transitory computer storage medium stores a program code, and when the program code is executed by a computer device, the computer device executes the image processing method provided according to one or more embodiments of the present disclosure.

[0016] According to one or more embodiments of the present disclosure, by determining the posture of a virtual object based on the posture of a target object, generating a virtual image based on the posture of the virtual object, obtaining a first target image based on the virtual image and a real scene image, and smoothing the first target image to finally obtain a second target image, it is possible to add a virtual object to a target object in a real scene image and reduce image distortion and image noise caused by image addition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 A flowchart of an image processing method provided according to an embodiment of the present disclosure;

[0019] Figure 2 A schematic diagram of a region to be smoothed according to an embodiment of the present disclosure;

[0020] Figure 3 A flowchart of an image processing method provided according to another embodiment of the present disclosure;

[0021] Figure 4 A schematic diagram of the structure of an image processing device provided according to an embodiment of the present disclosure;

[0022] Figure 5 The figure is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be understood that the steps described in the embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0025] As used herein, the term "including" and its variations are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The term "responsive to" and related terms refer to a signal or event being affected to a certain extent by another signal or event, but not necessarily completely or directly. If event x occurs "responsive to" event y, x may be directly or indirectly responsive to y. For example, the occurrence of y may ultimately lead to the occurrence of x, but there may be other intermediate events and / or conditions. In other cases, y may not necessarily lead to the occurrence of x, and x may occur even if y has not yet occurred. In addition, the term "responsive to" may also mean "at least partially responsive to".

[0026] The term "determine" broadly covers a variety of actions, which may include obtaining, calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and similar actions, and may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and similar actions, and may also include receiving, generating, creating, establishing, and similar actions, as well as parsing, selecting, choosing, and similar actions, etc. The relevant definitions of other terms will be given in the following description. The relevant definitions of other terms will be given in the following description.

[0027] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B).

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0031] refer to Figure 1 , which shows an image processing method provided by an embodiment of the present disclosure, including steps S110 to S140.

[0032] Step S110: determining the position and posture of a virtual object associated with the target object based on the position and posture of the target object in the real scene.

[0033] In some embodiments, the target object is a real object that exists in the real scene image presented to the user and needs to add a virtual object. It can be a part of the human body, such as limbs, hands, neck or head, or an animal, plant or object existing in the real scene, but the present disclosure is not limited to this. The virtual object can be a virtual item added to the target object. In a specific application scenario, the target object can be the user's hand and / or the control handle of the virtual reality device held by the user, and its associated virtual object can be a glove, a game prop or other given virtual item, so that the display effect of the user wearing gloves and holding virtual items can be presented in the virtual reality space.

[0034] In some embodiments, the pose of an object may include the position and / or attitude (e.g., rotation angle or direction) of the object. In a specific embodiment, the pose of an object includes position information and attitude information, wherein the position information indicates the amount of movement along the three rectangular coordinate axes of x, y, and z, and the attitude information indicates the amount of rotation around the three coordinate axes, and the position information and attitude information involve a total of 6 degrees of freedom (i.e., 6DOF).

[0035] In a specific implementation, a preset target tracking algorithm may be used to track the target object to obtain the position and posture of the target object.

[0036] In a specific embodiment, one or more of vision-based, laser-based, or sensor-based methods may be used to determine the position and posture of the target object, and the present disclosure is not limited thereto.

[0037] Step S120: generating a virtual image corresponding to the virtual object based on the position and posture of the virtual object.

[0038] In some embodiments, a virtual model corresponding to a virtual object may be pre-set, and after obtaining the position and posture of the virtual object, the virtual model may be adjusted based on the position and posture of the virtual object to obtain a virtual image containing the virtual object. Exemplarily, the virtual model may be loaded into memory in advance, the position and posture of the virtual model may be obtained by tracking the target object, and the virtual model may be rasterized based on the position and posture of the virtual object and the pre-loaded model to obtain a depth map (i.e., a virtual image) corresponding to the virtual object. Exemplarily, the depth map corresponding to the virtual object has a corresponding depth only in the virtual model part, and no depth exists in the rest of the part.

[0039] Step S130: Obtain a first target image based on the virtual image and the real scene image corresponding to the real scene.

[0040] The real scene image can be an image captured by a camera of a virtual reality device, or an image obtained after the images are processed by a preset processing step. In a specific embodiment, the real scene image can be obtained based on video perspective (VST).

[0041] In some embodiments, the real scene image may be a depth map of the real scene. Exemplarily, the depth map of the virtual object may be superimposed on the depth map of the real scene to obtain a first target image, thereby achieving a rough fusion of the real scene and the virtual object. In the first target image, the virtual object may replace, cover the target object, or be located at a preset position of the target object.

[0042] Step S140: performing smoothing processing on the first target image to obtain a second target image.

[0043] In some embodiments, the smoothing process may specifically include local smoothing, that is, smoothing the boundary of the virtual object in the first target image, and may further include overall smoothing, that is, smoothing the entire first target image after the local smoothing process. In a specific embodiment, the local smoothing process includes Laplace smoothing; the overall smoothing process includes Gaussian filtering, for example, the first target image may be subjected to Gaussian filtering with a filter kernel of 3×3, but the present disclosure is not limited thereto.

[0044] In this embodiment, the distortion caused by adding the virtual object to the real scene image can be reduced by locally smoothing the boundary of the virtual object. In addition, the image noise and distortion caused by local smoothing can be reduced by overall smoothing the overall image after local smoothing.

[0045] In some embodiments, the second target image may be a depth image, and three-dimensional reconstruction and texture mapping may be performed based on the second target image, and finally rendered on the screen. Three-dimensional reconstruction is used to obtain three-dimensional geometric information of the scene, and texture mapping uses images, functions, or other data sources to change the appearance of the edges of objects. Exemplarily, a mesh model may be generated based on the depth data in the second target image, a texture map and texture coordinates may be generated based on the mesh model and the corresponding RGB image, and rendering may be performed based on information such as the mesh model, texture map, and texture coordinates, and projected to two eye coordinate systems for display.

[0046] According to one or more embodiments of the present disclosure, by determining the posture of a virtual object based on the posture of a target object, generating a virtual image based on the posture of the virtual object, obtaining a first target image based on the virtual image and a real scene image, and smoothing the first target image, a second target image is finally obtained, thereby making it possible to add a virtual object to a target object in a real scene image and reduce image distortion and image noise caused by image addition.

[0047] In some embodiments, the specific steps of locally smoothing the boundary of the virtual object in the first target image include:

[0048] Step A1: based on the boundary of the virtual image in the first target image, determining a region to be smoothed including the boundary;

[0049] For example, a ring-shaped smoothing area with a width of preset pixels can be determined with the boundary of the virtual image as the center. Figure 2 , the boundary 20 of the virtual object in the first target image 10 is as follows Figure 2 As shown, assuming that the preset number of pixels is 8 pixels, the boundary 20 can be used as a reference, and an inner contour 31 with the same shape and 4 pixels away from the boundary 20 is determined inwardly, and an outer contour 32 with the same shape and 4 pixels away from the boundary 20 is determined outward. The boundary 20 is located between the inner contour 31 and the outer contour 32, and the inner contour 31 and the outer contour 32 form an area to be smoothed 30 with a width of 8 pixels.

[0050] Step A2: determining a target depth value of each pixel in the area to be smoothed, wherein the determined target depth value of each pixel is as close as possible to its original depth value and the target depth values ​​of its adjacent pixels as a whole.

[0051] In this embodiment, the target depth values ​​between adjacent pixels in the area to be smoothed are as close as possible, so that the image is smoother, and the target depth value of each pixel does not deviate from its original depth value as much as possible, so as to prevent image distortion.

[0052] In some embodiments, the adjacent pixels include a pixel adjacent in a first direction and a pixel adjacent in a second direction, and the first direction and the second direction are perpendicular to each other. For example, the pixels adjacent to pixel A may include a pixel directly above or below the pixel, and a pixel to the left or right of pixel A. In this way, since the target depth value of each pixel in the area to be smoothed is a pixel adjacent to it in the first direction (e.g., directly above) and a pixel adjacent to it in the second direction (e.g., to the right), the target depth value of each pixel is finally associated with the four pixels adjacent to it (e.g., the pixels above, below, left, and right), without repeated calculations.

[0053] In some specific implementations, an energy function may be constructed based on the cumulative value of the square value of the difference between the target depth value of each pixel and its original depth value, and the cumulative value of the square value of the difference between the target depth value of each pixel and the target depth value of its adjacent pixels, and the target depth value of each pixel is obtained by minimizing the energy function. For example, the energy function E may be constructed as follows:

[0054]

[0055] Among them, d ij represents the target depth value to be solved for the (i, j)th pixel in the area to be smoothed. Similarly, d i+1,j Indicates the target depth value to be solved for the (i+1,j)th pixel, d i,j+1 represents the target depth value to be solved for the (i,j+1)th pixel, represents the depth value (i.e., original depth value) of the (i, j)th pixel in the first target image, and λ is a smoothing coefficient. For example, i corresponds to the number in the horizontal direction, and j corresponds to the number in the vertical direction.

[0056] Thus, as shown in the following formula, by minimizing the energy function, the target depth value of each pixel in the area to be smoothed can be obtained:

[0057]

[0058] In some embodiments, the (i+1, j)th pixel may be a neighboring pixel located on the right side of the (i, j)th pixel, and the (i, j+1)th pixel may be a neighboring pixel located above the (i, j)th pixel.

[0059] In some embodiments, the smoothing coefficient (i.e., the weight value) can be a fixed constant (e.g., 0.3). The larger the smoothing coefficient, the greater the weight of the accumulated value of the square of the difference between the target depth value of each pixel and the target depth value of its adjacent pixels in the energy function, so that the target depth values ​​of each pixel are closer and the smoothing strength is higher.

[0060] In some embodiments, in the energy function, the square value of the difference between the target depth values ​​of adjacent pixels may correspond to a corresponding weight value, and the weight value may be determined based on the color deviation between the adjacent pixels. For example, it may be determined based on the square of the difference between the color values ​​(e.g., RGB values) of the adjacent pixels. In a specific embodiment, the energy function E may be constructed as follows:

[0061]

[0062] Among them, λ ij =f(c ij -c i+1,j ), λ′ ij =f(c ij -c i,j+1 );c ij represents the color value of the (i, j)th pixel in the area to be smoothed. Similarly, c i+1,j represents the color value of the (i+1,j)th pixel, c i,j+1 represents the color value of the (i, j+1)th pixel. Exemplarily, the function f(x) may be an exponential function, such as a natural exponential function, but the present disclosure is not limited thereto.

[0063] In this embodiment, the weight value corresponding to the square value of the difference between the target depth values ​​of the pair of adjacent pixels in the energy function is determined based on the color deviation between the adjacent pixels, so that the smoothing effect of the boundary can be further optimized.

[0064] In some embodiments, step S110 includes: determining the position and state of the virtual object based on the position and state of the target object in the real scene; step S120 includes: generating a virtual image corresponding to the virtual object based on the position and state of the virtual object.

[0065] Among them, the state of the object may include the form, posture or shape of the object. The state of the virtual object is related to the type of the virtual object. In a specific embodiment, when the target object is a real hand, and the virtual object is a virtual hand or a glove, the gesture or hand shape of the virtual object can be determined based on the gesture or hand shape of the target object, and according to the virtual model corresponding to the different gestures or hand shapes of the virtual object set in advance, so that the gesture or hand shape of the virtual object finally presented can be consistent with the gesture or hand shape of the target object in the real scene. In another specific embodiment, when the target object is a real hand, and the virtual object is a virtual pistol prop, the firing state (e.g., trigger, safety state) of the virtual pistol prop can be determined based on the gesture or hand shape of the target object. For example, when the user presents a bent index finger gesture or hand shape, it can be determined that the virtual prop pistol is in the firing state of pulling the trigger, but the present disclosure is not limited to this.

[0066] refer to Figure 3 , which shows an image processing method provided by an embodiment of the present disclosure, including steps S301 to S307:

[0067] Step S301: obtaining a depth map of a real scene;

[0068] Step S302: determining the position and posture of a virtual object associated with the target object based on the position and posture of the target object in the real scene;

[0069] Step S303: rasterizing the virtual model of the virtual object based on the position and posture of the virtual object to obtain a depth map of the virtual object;

[0070] Step S304: obtaining a first target image based on the depth map of the virtual object and the depth map of the real scene;

[0071] Step S305: performing local smoothing processing on the boundary of the virtual object in the first target image.

[0072] Step S306: performing overall smoothing processing on the first target image after the local smoothing processing to obtain a second target image.

[0073] Step S307: Perform three-dimensional reconstruction and texture mapping based on the second target image, and render and display. Three-dimensional reconstruction is used to obtain three-dimensional geometric information of the scene, and texture mapping uses images, functions or other data sources to change the appearance of the object edge. Exemplarily, a mesh model can be generated based on the depth data in the second target image, and a texture map and texture coordinates can be generated based on the mesh model and the corresponding RGB image. Rendering is performed based on information such as the mesh model, texture map, and texture coordinates, and projected to two eye coordinate systems for display.

[0074] According to one or more embodiments of the present disclosure, by determining the posture of a virtual object based on the posture of a target object, generating a virtual image based on the posture of the virtual object, obtaining a first target image based on the virtual image and a real scene image, and smoothing the first target image, a second target image is finally obtained, thereby making it possible to add a virtual object to a target object in a real scene image and reduce image distortion and image noise caused by image addition.

[0075] Accordingly, refer to Figure 4 According to an embodiment of the present disclosure, an image processing device 400 is provided, comprising:

[0076] A posture determination unit 401, configured to determine the posture of a virtual object associated with the target object based on the posture of the target object in the real scene;

[0077] A first image processing unit 402, configured to generate a virtual image corresponding to the virtual object based on the position and posture of the virtual object;

[0078] A second image processing unit 403 is used to obtain a first target image based on the virtual image and a real scene image corresponding to the real scene;

[0079] The third image processing unit 404 is configured to perform smoothing on the first target image to obtain a second target image.

[0080] In some embodiments, the third image processing unit includes:

[0081] A local smoothing unit is used to perform local smoothing processing on the boundary of the virtual object in the first target image.

[0082] In some embodiments, the third image processing unit further includes:

[0083] The overall smoothing unit is used to perform overall smoothing processing on the first target image after the local smoothing.

[0084] In some embodiments, the local smoothing unit includes:

[0085] A region determining unit, configured to determine, based on a boundary of the virtual image in the first target image, a region to be smoothed including the boundary;

[0086] The depth determination unit is used to determine a target depth value of each pixel in the area to be smoothed, wherein the determined target depth value of each pixel is as close as possible to its original depth value and the target depth values ​​of its adjacent pixels as a whole.

[0087] In some embodiments, the depth determination unit is used to construct an energy function based on the accumulated value of the square value of the difference between the target depth value of each pixel and its original depth value, and the accumulated value of the square value of the difference between the target depth value of each pixel and the target depth value of its adjacent pixels, and obtain the target depth value of each pixel by minimizing the energy function.

[0088] In some embodiments, the adjacent pixels include a pixel adjacent in a first direction and a pixel adjacent in a second direction, and the first direction and the second direction are perpendicular to each other.

[0089] In some embodiments, in the energy function, the square value of the difference between the target depth values ​​of adjacent pixels corresponds to a corresponding weight value, and the weight value is determined based on the color deviation between the adjacent pixels.

[0090] In some embodiments, the overall smoothing process includes a Gaussian filtering process.

[0091] In some embodiments, the posture determination unit is used to determine the posture and state of the virtual object based on the posture and state of the target object in the real scene; the first image processing unit is used to generate a virtual image corresponding to the virtual object based on the posture and state of the virtual object.

[0092] In some embodiments, the image processing apparatus further comprises:

[0093] A rendering unit is used to perform three-dimensional reconstruction and texture mapping based on the second target image, and render and display it.

[0094] For the embodiments of the device, since they basically correspond to the method embodiments, the relevant parts refer to the partial description of the method embodiments. The device embodiments described above are merely illustrative, and the modules described as separation modules may or may not be separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present embodiment. Those of ordinary skill in the art can understand and implement without paying creative work.

[0095] Accordingly, according to one or more embodiments of the present disclosure, there is provided an electronic device, including:

[0096] at least one memory and at least one processor;

[0097] The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to enable the electronic device to execute the image processing method provided according to one or more embodiments of the present disclosure.

[0098] Accordingly, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, which stores program code, and the program code can be executed by a computer device to enable the computer device to perform the image processing method provided according to one or more embodiments of the present disclosure.

[0099] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 800 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0100] like Figure 5 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0101] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0102] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0103] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0104] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0105] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0106] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method of the present disclosure.

[0107] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0110] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] According to one or more embodiments of the present disclosure, there is provided an image processing method, comprising: determining a posture of a virtual object associated with a target object based on the posture of the target object in a real scene; generating a virtual image corresponding to the virtual object based on the posture of the virtual object; obtaining a first target image based on the virtual image and a real scene image corresponding to the real scene; and smoothing the first target image to obtain a second target image.

[0113] According to one or more embodiments of the present disclosure, the smoothing of the first target image includes: locally smoothing the boundary of the virtual object in the first target image.

[0114] According to one or more embodiments of the present disclosure, the smoothing of the first target image further includes: performing overall smoothing on the first target image after the local smoothing.

[0115] According to one or more embodiments of the present disclosure, the locally smoothing the boundary of the virtual object in the first target image includes: determining, based on the boundary of the virtual image in the first target image, an area to be smoothed that includes the boundary; determining a target depth value for each pixel in the area to be smoothed, wherein the determined target depth value for each pixel is as close as possible to its original depth value and the target depth values ​​of its adjacent pixels as a whole.

[0116] According to one or more embodiments of the present disclosure, determining the target depth value of each pixel in the area to be smoothed includes: constructing an energy function based on the accumulated value of the square value of the difference between the target depth value of each pixel and its original depth value, and the accumulated value of the square value of the difference between the target depth value of each pixel and the target depth values ​​of its adjacent pixels; and obtaining the target depth value of each pixel by minimizing the energy function.

[0117] According to one or more embodiments of the present disclosure, the adjacent pixels include a pixel adjacent in a first direction and a pixel adjacent in a second direction, and the first direction and the second direction are perpendicular to each other.

[0118] According to one or more embodiments of the present disclosure, in the energy function, the square value of the difference between the target depth values ​​between adjacent pixels corresponds to a corresponding weight value, and the weight value is determined based on the color deviation between the adjacent pixels.

[0119] According to one or more embodiments of the present disclosure, the overall smoothing process includes a Gaussian filtering process.

[0120] According to one or more embodiments of the present disclosure, determining the posture of a virtual object associated with the target object based on the posture of the target object in a real scene includes: determining the posture and state of the virtual object based on the posture and state of the target object in the real scene; generating a virtual image corresponding to the virtual object based on the posture of the virtual object includes: generating a virtual image corresponding to the virtual object based on the posture and state of the virtual object.

[0121] The method provided according to one or more embodiments of the present disclosure further includes: performing three-dimensional reconstruction and texture mapping based on the second target image, and rendering and displaying.

[0122] According to one or more embodiments of the present disclosure, there is provided an image processing device, comprising: a posture determination unit, for determining the posture of a virtual object associated with a target object based on the posture of the target object in a real scene; a first image processing unit, for generating a virtual image corresponding to the virtual object based on the posture of the virtual object; a second image processing unit, for obtaining a first target image based on the virtual image and a real scene image corresponding to the real scene; and a third image processing unit, for smoothing the first target image to obtain a second target image.

[0123] According to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory so that the electronic device executes the image processing method provided according to one or more embodiments of the present disclosure.

[0124] According to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, wherein the non-transitory computer storage medium stores a program code, and when the program code is executed by a computer device, the computer device executes an image processing method provided according to one or more embodiments of the present disclosure.

[0125] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0126] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0127] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Determining a position and posture of a virtual object associated with the target object based on the position and posture of the target object in the real scene; generating a virtual image corresponding to the virtual object based on the position and posture of the virtual object; Obtaining a first target image based on the virtual image and a real scene image corresponding to the real scene; The first target image is smoothed to obtain a second target image.

2. The method according to claim 1, characterized in that: The step of smoothing the first target image comprises: Locally smoothing the boundary of the virtual object in the first target image.

3. The method according to claim 2, characterized in that The smoothing process of the first target image further includes: Perform overall smoothing processing on the first target image after the local smoothing.

4. The method according to claim 2, characterized in that: The locally smoothing the boundary of the virtual object in the first target image includes: Based on the boundary of the virtual image in the first target image, determining a region to be smoothed including the boundary; A target depth value of each pixel in the area to be smoothed is determined, wherein the determined target depth value of each pixel is as close as possible to its original depth value and the target depth values ​​of its adjacent pixels as a whole.

5. The method according to claim 4, characterized in that The determining of a target depth value of each pixel in the area to be smoothed comprises: An energy function is constructed based on the accumulated value of the square value of the difference between the target depth value of each pixel and its original depth value, and the accumulated value of the square value of the difference between the target depth value of each pixel and the target depth value of its adjacent pixels; The target depth value of each pixel is obtained by minimizing the energy function.

6. The method according to claim 4, characterized in that The adjacent pixels include a pixel adjacent in a first direction and a pixel adjacent in a second direction, and the first direction and the second direction are perpendicular to each other.

7. The method according to claim 4, characterized in that In the energy function, the square value of the difference between the target depth values ​​of adjacent pixels corresponds to a corresponding weight value, and the weight value is determined based on the color deviation between the adjacent pixels.

8. The method according to claim 3, characterized in that The overall smoothing process includes Gaussian filtering.

9. The method according to claim 1, characterized in that: The determining the position and posture of the virtual object associated with the target object based on the position and posture of the target object in the real scene comprises: determining the position and posture and state of the virtual object based on the position and posture of the target object in the real scene; The generating a virtual image corresponding to the virtual object based on the position and posture of the virtual object includes: generating a virtual image corresponding to the virtual object based on the position and posture and state of the virtual object.

10. The method according to claim 1, characterized in that Also includes: Three-dimensional reconstruction and texture mapping are performed based on the second target image, and then rendered and displayed.

11. An image processing device, characterized in that: include: A posture determination unit, used to determine the posture of a virtual object associated with the target object based on the posture of the target object in the real scene; A first image processing unit, configured to generate a virtual image corresponding to the virtual object based on the position and posture of the virtual object; A second image processing unit, configured to obtain a first target image based on the virtual image and a real scene image corresponding to the real scene; The third image processing unit is used to perform smoothing on the first target image to obtain a second target image.

12. An electronic device, characterized in that: include: at least one memory and at least one processor; The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to enable the electronic device to execute the method according to any one of claims 1 to 10.

13. A non-transitory computer storage medium, characterized in that The non-transitory computer storage medium stores a program code, and when the program code is executed by a computer device, the computer device executes the method according to any one of claims 1 to 10.

Images