Image processing method and related device

Abstract

The invention provides an image processing method and a related device, in the method, after an electronic device obtains an image needing to be processed, downsampling is carried out on a depth map corresponding to the image to obtain depth maps corresponding to different levels, and the depth map of each level corresponds to a depth change value. Therefore, in the image processing process, the depth map of the corresponding level is used for processing the area with the dramatic depth change, the area obtained through processing is smooth, the depth maps of different levels are used, the area with serious deformation can be corrected, and the use experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method and related devices. Background Art

[0002] With the development of technology, electronic devices (such as mobile phones, tablets, etc.) can display three-dimensional (3D) wallpapers. 3D wallpaper is an image with a three-dimensional effect. Through specific image processing technology, the two-dimensional image presents a sense of three-dimensional space, thereby providing users with a more vivid and realistic visual effect.

[0003] Currently, depth map-based three-dimensional (3D) synthesis technology is a common image processing technology for converting two-dimensional (2D) images into 3D images. In the process of converting 2D to 3D using this technology, it is necessary to rely on the depth map to convert the 2D plane grid into a 3D solid grid. However, current technology may cause the converted solid grid in the edge area of ​​the foreground to appear stretched and deformed, affecting the display effect and giving users a poor user experience. Summary of the Invention

[0004] An image processing method and related device provided in the embodiments of the present application can improve the stretching deformation occurring on the image and enhance the image processing effect.

[0005] In a first aspect, the present application provides an image processing method, applied to an electronic device, the method comprising:

[0006] Get the image to be processed;

[0007] Generate a depth map set corresponding to the image to be processed, the depth map set including N depth maps, each depth map in the N depth maps corresponding to a depth threshold range, where N is a positive integer greater than or equal to 2;

[0008] The target area of ​​the image to be processed is processed according to the target depth map in the depth map set to obtain a target image, wherein the depth change value corresponding to the target area is within the depth threshold range corresponding to the target depth map, and the depth change value is the maximum difference between the depth values ​​of two adjacent pixels in the target area.

[0009] In an embodiment of the present application, the electronic device can generate multiple depth maps corresponding to the image to be processed during the process of processing the image to be processed. In this way, when processing the image to be processed, a matching depth map can be selected from the multiple depth maps based on the size of the depth change value of each area in the image to be processed, and then the area can be processed using the targeted depth map, which can greatly reduce the impact of the depth change value of the area on the target image to be processed and improve the processing effect.

[0010] In a possible implementation of the first aspect, the image to be processed includes a two-dimensional image serving as a lock screen wallpaper, and the target image includes a three-dimensional image serving as the lock screen wallpaper.

[0011] In an embodiment of the present application, in a target image obtained by performing a displacement transformation on each area in a two-dimensional image based on a targeted depth map, the stretching deformation of the intersection area of ​​the foreground and background is optimized, thereby improving user experience.

[0012] In a possible implementation of the first aspect, the image to be processed includes a map based on a caustic effect, and the target image is an image rendered based on the image to be processed.

[0013] It is understandable that when a texture with a discrete effect is rendered onto a flat surface, the edges of the texture may be stretched and deformed. In the embodiments of the present application, the target image obtained by rendering each area of ​​the texture with a discrete effect based on a targeted depth map has optimized stretching and deformation of the edge areas, thereby improving the user experience.

[0014] In a possible implementation of the first aspect, generating a depth map set corresponding to the image to be processed includes:

[0015] Downsampling the original depth map of the image to be processed N times obtains the N depth maps in the depth map set, wherein a depth threshold range corresponding to each depth map in the N depth maps is determined by a difference between an average foreground depth and an average background depth of the depth map.

[0016] In an embodiment of the present application, the depth value in the depth map can be gradually reduced by downsampling N times, thereby obtaining a depth map with higher precision. Further, targeted processing can be performed on areas with different depth change values ​​in the processed image to improve the stretching deformation caused by areas with larger depth change values.

[0017] In a possible implementation of the first aspect, the value of N is 4, and performing N downsampling processes on the original depth map of the image to be processed to obtain the N depth maps in the depth map set includes:

[0018] Performing weighted averaging processing on the pixels in the original depth map of the processed image in units of s pixels to obtain a first depth map, where s is a positive integer greater than or equal to 4;

[0019] Performing weighted averaging processing on the pixels in the first depth map in units of the s pixels to obtain a second depth map;

[0020] Performing weighted averaging processing on the pixels in the second depth map in units of the s pixels to obtain a third depth map;

[0021] A weighted average process is performed on the pixels in the third depth map in units of the s pixels to obtain a fourth depth map.

[0022] In a possible implementation of the first aspect, the value of N is 4, and performing N downsampling processes on the original depth map of the image to be processed to obtain the N depth maps in the depth map set includes:

[0023] Performing weighted averaging processing on the pixels in the original depth map of the processed image in units of s pixels to obtain a first depth map, where s is a positive integer greater than or equal to 4;

[0024] s 2 Performing weighted averaging processing on the pixel points in the original depth map of the processed image in units of pixels to obtain a second depth map;

[0025] s 3 Performing weighted averaging processing on the pixels in the original depth map of the processed image in units of pixels to obtain a third depth map;

[0026] s 4 The pixel points in the original depth map of the processed image are weighted averaged in units of pixels to obtain a fourth depth map.

[0027] In the embodiment of the present application, the depth values ​​corresponding to the pixels in the depth map obtained by each downsampling are gradually reduced, so the difference in depth values ​​between pixels is also gradually reduced. Therefore, for the foreground and background of the depth map, the difference between the average depth value of the pixels in the foreground and the average depth value of the pixels in the background is also gradually reduced. This shows that the difference between the depth values ​​of the pixels in the foreground and the depth values ​​of the pixels in the background in the depth map obtained by downsampling is not very large, and the depth map gradually shows a smooth trend. Then, the uneven image to be processed is processed based on the depth map showing a smooth trend, which can improve the unevenness of the image to be processed.

[0028] In a possible implementation of the first aspect, the first depth map corresponds to a first depth threshold range, and the first depth threshold range is determined by a difference between an average depth value of pixels in a first foreground and an average depth value of pixels in a first background in the first depth map;

[0029] The second depth map corresponds to a second depth threshold range, and the second depth threshold range is determined by a difference between an average depth of pixels in a second foreground and an average depth of pixels in a second background in the second depth map;

[0030] The third depth map corresponds to a third depth threshold range, where the third depth threshold range is determined by a difference between a depth average of pixels in a third foreground and a depth average of pixels in a third background in the third depth map;

[0031] The fourth depth map corresponds to a fourth depth threshold range, where the fourth depth threshold range is determined by a difference between an average depth of pixels in a fourth foreground and an average depth of pixels in a fourth background in the fourth depth map;

[0032] The maximum value in the first depth threshold range is greater than the maximum value in the second depth threshold range, greater than the maximum value in the third depth threshold range, and greater than the maximum value in the fourth depth threshold range.

[0033] In an embodiment of the present application, a plurality of depth threshold ranges are preset inside the electronic device, and each depth image generated corresponds to a depth threshold range. The larger the difference in the depth map, the uneven transition between the foreground and the background, and the depth map can be used to process a relatively smooth image to be processed, because the change in the depth value in the relatively smooth image to be processed is not very drastic (i.e., the difference is small), so the larger the difference in the depth map, the smaller the maximum value in the depth threshold range corresponding to the depth map. The smaller the difference in the depth map, the smoother the transition between the foreground and the background, and the depth map can be used to process an unsmooth image to be processed, because the change in the depth value in the unsmooth image to be processed is relatively drastic (i.e., the difference is large), so the smaller the difference in the depth map, the larger the maximum value in the depth threshold range corresponding to the depth map.

[0034] In a possible implementation of the first aspect, the processing the target area of ​​the to-be-processed image according to the target depth map in the depth map set to obtain the target image includes:

[0035] Determining a depth change value of a target area of ​​the image to be processed;

[0036] determining a target depth map from the depth map set according to the depth change value;

[0037] A target image is obtained by performing a displacement transformation on the target area of ​​the image to be processed based on the target depth map.

[0038] In the embodiment of the present application, because the depth change values ​​of different areas are processed using the corresponding depth maps during the processing process, the depth change values ​​can be reduced, and the transition imbalance caused by excessive depth change values ​​can be improved, and the stretching deformation phenomenon in the target image can be further improved, thereby improving the image processing effect and improving the user experience.

[0039] In a possible implementation of the first aspect, determining the depth change value of the target area of ​​the image to be processed includes:

[0040] Calculating a depth difference between a target pixel and a preset number of pixel points to obtain the preset number of depth change values, wherein the target pixel point is a pixel point in a target area of ​​the image to be processed;

[0041] The maximum mean value among the preset number of depth change values ​​is used as the depth change value of the target area of ​​the image to be processed.

[0042] In a possible implementation of the first aspect, the preset number of pixels includes eight pixels adjacent to the target pixel.

[0043] In an embodiment of the present application, the depth change value of a certain area in the image to be processed can be determined by the pixels in the area and, for example, 8 pixels around it, which can reflect the degree of depth change in the area.

[0044] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image processing method described in the first aspect or any possible implementation of the first aspect.

[0045] In a third aspect, the present application provides a chip or chip system, comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a circuit, and the at least one processor is configured to execute a computer program or instruction to perform the image processing method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip may be an input / output interface, a pin, or a circuit.

[0046] In one possible implementation, the chip or chip system described above in the embodiments of the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0047] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, the computer executes the image processing method described in the first aspect or any possible implementation of the first aspect.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a communication device, enables the communication device to execute the image processing method described in the first aspect or any possible implementation of the first aspect.

[0049] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of a technical feature, technical solution or beneficial effect in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The following is an introduction to the drawings used in the embodiments of this application.

[0051] Figures 1A to 1E This is a schematic diagram of a lock screen wallpaper interface provided by an embodiment of the present application;

[0052] Figure 2 This is a schematic diagram of switching element A from 2D to 3D provided by an embodiment of the present application;

[0053] Figure 3 1 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application;

[0054] Figure 4 1 is a schematic diagram of a software architecture of an electronic device 100 provided in an embodiment of the present application;

[0055] Figure 5 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0056] Figure 6A is a schematic diagram of a depth map set provided by this application;

[0057] Figure 6B is a schematic diagram of another depth map set provided by this application;

[0058] Figure 7 This is a schematic diagram of processing an image to be processed based on a depth map set provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0060] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0061] First, some terms used in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0062] 1. A depth image, also known as a range image or grayscale image, is an image that uses the distance (depth) from the image collector to each point in the scene as a pixel value. It directly reflects the geometric shape of the visible surface of the scene. In other words, the depth value of a pixel in the depth image of the target scene represents the distance between a point in the target scene and the image collector (usually in millimeters). Generally, the farther a point in the scene is from the image collector, the greater the depth value of the pixel; the closer a point in the scene is to the image collector, the smaller the depth value of the pixel.

[0063] 2. Foreground and background. The foreground refers to the elements closest to the viewer in the image. It guides the viewer's gaze and is the focus of image processing. The background represents objects without significant targets or other distracting elements. It is usually used to distinguish the foreground object from other environmental elements.

[0064] In a depth map, foreground depth typically refers to the depth of field in front of the subject, while background depth refers to the depth of field behind the subject. The depth values ​​of the foreground and background depths in the depth map are also different: pixels in the foreground have smaller depth values ​​(larger pixel values), while pixels in the background have larger depth values ​​(smaller pixel values).

[0065] 3. Pixels and voxels. Pixels are the basic units of an image, the smallest independent unit of information into which an image can be broken down. In two-dimensional (2D) space, pixels have no "thickness"; their most significant characteristic is their two-dimensional nature. For example, in a visible image, each small square is a pixel, and image resolution is typically expressed in pixels per inch (PPI). An image is composed of many pixels, each with its own grayscale value or color value.

[0066] Voxel, short for volume pixel, is the smallest unit of digital data in three-dimensional (3D) space and is widely used in 3D imaging. A voxel can be understood as a generalization of a two-dimensional pixel to three-dimensional space: a set of cubic units distributed at the center of an orthogonal grid. Unlike pixels, voxels are a three-dimensional concept.

[0067] In 2D images, pixels are the basic image elements, while in 3D images, voxels are the basic volume elements. Specifically, a voxel is the extension of a pixel in 3D space.

[0068] Next, the technical problems to be solved by this application are analyzed and proposed by introducing the application scenarios of this application.

[0069] In order to increase the playability of electronic devices, the image displayed by the electronic device (such as a wallpaper image) can change as the user's status changes, wherein the image can be an image displayed in the always on display (AOD) state (which can be simply referred to as an off-screen image), or an image displayed in the lock screen state, or an image displayed on the desktop. For example, in the lock screen state, the lock screen wallpaper displayed by the electronic device is a two-dimensional image. When the electronic device detects that the face rotates / moves, the lock screen wallpaper will also rotate / move and change from a two-dimensional image to a three-dimensional image. For ease of explanation, the following embodiments are mainly described by taking the lock screen scenario (the lock screen wallpaper of the electronic device changes as the user status changes) as an example.

[0070] See Figures 1A to 1E , Figures 1A to 1E This is a schematic diagram of a lock screen wallpaper interface provided by an embodiment of the present application. Figure 1A As shown, when the electronic device 100 is in the lock screen state, the lock screen wallpaper 101 is displayed. If the electronic device detects that the user 200 is looking at the display screen, the lock screen wallpaper 101 displayed is a two-dimensional image, that is, the element A in the foreground of the lock screen wallpaper is flat. Figure 1B As shown, if the electronic device detects that the user 200's head turns to the left (with the user 200 as a reference), the element A in the lock screen wallpaper 101 displayed by the electronic device 100 also turns to the left (the left rotation / movement is based on the user 200 as a reference, and the right rotation is based on the electronic device 100). The displayed lock screen wallpaper 101 is a three-dimensional image, that is, the element A in the foreground of the lock screen wallpaper appears in a three-dimensional state. Figure 1C As shown, if the electronic device detects that the user 200's head turns right (with the user 200 as a reference), the element A in the lock screen wallpaper 101 displayed by the electronic device 100 also turns right (the right rotation / movement is based on the user 200 as a reference, and the left rotation is based on the electronic device 100 as a reference), then the displayed lock screen wallpaper 101 is a three-dimensional image, that is, the element A in the foreground of the lock screen wallpaper 101 appears in a three-dimensional state. Figure 1D As shown, if the electronic device detects that the user 200's head is turned upward (with the user 200 as a reference), the element A in the lock screen wallpaper 101 displayed by the electronic device 100 also rotates upward (the upward rotation / movement is based on the user 200 as a reference, and if the electronic device 100 is used as a reference, it rotates downward), then the displayed lock screen wallpaper 101 is a three-dimensional image, that is, the element A in the foreground of the lock screen wallpaper appears in a three-dimensional state. Figure 1EAs shown, if the electronic device detects that the user 200's head is turned downward (with the user 200 as a reference), the element A in the lock screen wallpaper 101 displayed by the electronic device 100 also rotates downward (the downward rotation / movement is based on the user 200 as a reference, and if the electronic device 100 is used as a reference, it rotates upward), then the displayed lock screen wallpaper 101 is a three-dimensional image, that is, the element A in the foreground of the lock screen wallpaper 101 appears in a three-dimensional state.

[0071] Therefore, when Figure 1A The lock screen wallpaper 101 shown switches to Figure 1B 、 Figure 1C 、 Figure 1D or Figure 1E When any of the lock screen wallpapers 101 shown is displayed, the electronic device needs to switch from displaying a two-dimensional image to displaying a three-dimensional image, that is, to switch element A from a two-dimensional state to a three-dimensional state. It should be noted that the implementation method of switching the lock screen wallpaper 101 from displaying a two-dimensional image to displaying a three-dimensional image is not limited to Figures 1A to 1E In the embodiment shown, the gyroscope sensor of the electronic device can also detect the motion posture of the electronic device and display the three-dimensional image corresponding to the lock screen wallpaper 101 according to the direction corresponding to the motion posture. For example, when the electronic device rotates to the left, the following image is displayed: Figure 1B Lock screen wallpaper 101 shown.

[0072] See Figure 2 , Figure 2 This is a schematic diagram of switching element A from 2D to 3D provided by an embodiment of the present application. Wherein, switching from 2D to 3D is based on a displacement transformation to convert a 2D plane grid into a 3D solid grid, and the plane grid is included in. Figure 2 As shown in the figure, the plane grid is defined by the X-axis and the Y-axis, so the element A in the plane grid has no depth, so the depth of the foreground and background can be considered to be zero. The three-dimensional grid is defined by the X-axis, the Y-axis, and the Z-axis, so the element A in the plane grid has depth, so the depth of the foreground and background are different. The depth map of element A can provide the depth of element A (that is, the length on the Z axis), so during the displacement transformation process, it is necessary to rely on the depth map to perform the displacement transformation of the mesh vertices. Figure 2It can be seen that since the depth value in the foreground in the depth map is smaller (for example, 0.2) and the depth value in the background is larger (for example, 1), there is a sudden change in depth value in the intersection area of ​​the foreground and background in the three-dimensional grid obtained by replacement based on a depth map, such as the case where the difference between the depth value 0.2 and the depth value 1 is large. It can be understood that since the difference between the depth value 0.2 and the depth value 1 is large, this situation can be called a depth value sudden change. The image is composed of individual pixels, and the pixels in the 2D image have no depth value. By assigning the depth value to the corresponding pixel in the 2D image, the conversion from the 2D image to the 3D image can be completed. When a depth value sudden change occurs in a 3D image, the connection between the pixel with a smaller depth value and the pixel with a larger depth value is not smooth enough, which may cause the corresponding area in the 3D image (that is, the area where the depth value sudden change is located) to be stretched and deformed, affecting the user's experience.

[0073] In view of this, the present application provides an image processing method. Taking the replacement of a two-dimensional image to a three-dimensional image as an example, after the electronic device obtains the two-dimensional image that needs to be replaced, it downsamples the depth map corresponding to the two-dimensional image to obtain depth maps corresponding to different levels. Each level of the depth map corresponds to a depth change value. For example, the larger the depth change value (that is, the more drastic the depth change), the larger the corresponding depth map level; the smaller the depth change value (that is, the less drastic the depth change), the smaller the corresponding depth map level. Therefore, in the process of converting a two-dimensional image to a three-dimensional image, in areas where the depth changes dramatically, a depth map of the corresponding level is used for processing. For example, in areas with large depth change values, a depth map with a higher level corresponding to the depth change value is used for replacement transformation, so that the area obtained by the replacement transformation will be smoother; in areas with small depth change values, a depth map with a smaller level corresponding to the depth change value is used for replacement transformation, so that the area obtained by the replacement transformation can retain depth details as much as possible. Different from Figure 2 As shown in the displacement transformation, this application uses depth maps of different levels to correct areas with severe deformation and improve the user experience.

[0074] First, the electronic devices in the embodiments of the present application can be smart screen devices, smart televisions (TVs), mobile phones, tablet computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), wearable devices (such as smart watches and smart bracelets), and other devices with display functions. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic devices.

[0075] For example, taking the electronic device as a mobile phone, Figure 3 1 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. That is, illustratively, Figure 3 The electronic device shown may be a mobile phone.

[0076] like Figure 3 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0077] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0078] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0079] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0080] Processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0081] In one embodiment, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a universal serial bus (USB) interface.

[0082] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0083] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0084] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In one embodiment, electronic device 100 may include N display screens 194, where N is a positive integer greater than 1.

[0085] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0086] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and other factors. It can also optimize parameters such as exposure and color temperature of the captured scene. In one embodiment, the ISP can be located within camera 193.

[0087] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0088] In one embodiment, the camera 193 may include a front camera and a rear camera. The front camera may be located on the side where the display screen 194 is located, for example, at the top or bottom of the display screen 194 , and the rear camera may be located on the back of the display screen 194 .

[0089] In one embodiment, the front camera can be used to capture the user's facial image in real time, and the real-time captured facial image can be sent to the processor 110. The processor 110 can determine whether the user's status has changed based on the user's facial image, such as whether the angle of the user's head has changed, whether the user's face has shifted, whether the user's expression has changed, whether a gesture has been performed, whether the gaze position has changed, whether the appearance features have changed, etc. When the user's status changes, the processor 110 can instruct the display screen 194 to adjust the displayed lock screen wallpaper accordingly. For example, when the angle of the user's head changes, the lock screen wallpaper also rotates accordingly.

[0090] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0091] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0092] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0093] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0094] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0095] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor. For example, music playback, recording, etc. The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In one embodiment, the audio module 170 can be set in the processor 110, or some functional modules of the audio module 170 can be set in the processor 110. The speaker 170A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A. The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be heard by placing the receiver 170B close to the human ear. Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In another embodiment, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In another embodiment, the electronic device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, and directional recording functions, etc. The headphone jack 170D is used to connect wired headphones.

[0096] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In one embodiment, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0097] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In another embodiment, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0098] In one embodiment, the pressure sensor 180A and / or the touch sensor 180K may be provided in the display screen 194 for detecting a touch operation on the display screen 194. When the pressure sensor 180A and / or the touch sensor 180K detects a touch operation (e.g., a sliding operation) on the display screen 194, the touch operation information may be reported to the processor 110. The processor 110 may instruct the display screen 194 to adjust the displayed lock screen wallpaper accordingly based on the touch operation information (e.g., operation type, sliding direction, sliding amplitude, etc.). For example, when the touch operation is a sliding operation, the lock screen wallpaper also rotates / moves accordingly.

[0099] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In one embodiment, the gyroscope sensor 180B can be used to determine the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can also be used for image stabilization, navigation, and somatosensory gaming scenarios. Optionally, the gyroscope sensor 180B can be provided on the circuit board of the electronic device 100.

[0100] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the electronic device's posture, but is not limited to this. It can also be applied to applications such as landscape / portrait screen switching and pedometers. Optionally, accelerometer 180E can be located on the circuit board of electronic device 100.

[0101] In one embodiment, the gyroscope sensor 180B and / or the acceleration sensor 180E can be used to detect whether the electronic device 100 has moved, thereby allowing the processor 110 to determine whether the relative position of the electronic device 100 and the user has changed, such as whether the angle of the user's head relative to the electronic device 100 has changed (also referred to as whether the angle of the electronic device 100 relative to the user has changed), or whether the user's face has shifted relative to the electronic device 100 (also referred to as whether the electronic device 100 has shifted relative to the user). When the relative position of the electronic device 100 and the user changes, the processor 110 can instruct the display screen 194 to adjust the displayed lock screen wallpaper accordingly.

[0102] The air pressure sensor 180C measures air pressure. The magnetic sensor 180D includes a Hall effect sensor, allowing the electronic device 100 to detect the opening and closing of the flip case. The distance sensor 180F measures distance. The fingerprint sensor 180H collects fingerprints. The temperature sensor 180J measures temperature. The ambient light sensor 180L senses ambient light brightness.

[0103] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The electronic device 100 emits infrared light through the light emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100.

[0104] The posture sensor 180M can be used to detect the user status, such as but not limited to the angle of the user's head, the position of the user's face, the user's expression, gestures, the user's line of sight, etc. The detected user status can be used by the processor 110 to instruct the display screen 194 to adjust the displayed lock screen wallpaper.

[0105] This application does not limit the specific type of sensor used to detect the user status.

[0106] Keys 190 include a power button, volume button, and other buttons. Keys 190 can be mechanical or touch-sensitive. Electronic device 100 can receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration prompts. Indicator 192 can be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and more. SIM card interface 195 is used to connect a SIM card.

[0107] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. For example, the software system with a layered architecture can be an Android system, a Harmony operating system (OS), or other software systems. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100.

[0108] Figure 4 Schematic diagram of the software architecture of an electronic device 100 provided in an embodiment of the present application.

[0109] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In one embodiment, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0110] like Figure 4 As shown, the application package may include applications such as camera, gallery, calendar, map, navigation, Bluetooth, music, video, settings, lock screen wallpaper, etc. The lock screen wallpaper in this application can be an independent application or a functional component integrated into other applications. This application does not limit this. The applications in this application can also be replaced by other software such as mini-programs and atomic services.

[0111] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0112] like Figure 4 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0113] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0114] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0115] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0116] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0117] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0118] The notification manager enables applications to display notification information in the status bar. This can be used to convey notification-type messages and can disappear automatically after a short period of time without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders, etc. The notification manager can also be used to display notifications in the form of icons or scrolling text in the top status bar of the system, such as notifications from applications running in the background, or notifications that appear on the screen in the form of dialog windows. For example, a text message can be displayed in the status bar, a notification sound can be emitted, the electronic device 100 can vibrate, an indicator light can flash, etc.

[0119] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0120] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0121] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0122] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0123] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0124] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0125] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0126] A 2D graphics engine is a drawing engine for 2D drawings.

[0127] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0128] The following is an example of the workflow of the software and hardware of the electronic device 100, combined with the display of the lock screen wallpaper scene.

[0129] The display screen 194 of the electronic device 100 is in the lock screen state. When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch single-click operation as an example, the lock screen wallpaper application calls the interface of the application framework layer to start the lock screen wallpaper display, and then starts the display driver by calling the kernel layer, controls the display screen 194 to switch from the off screen state to the lock screen state, and displays the two-dimensional image of the lock screen wallpaper on the display screen 194. When the gyroscope sensor 180B detects a motion gesture, the corresponding hardware interrupt is sent to the kernel layer, and the kernel layer processes the motion gesture into an input event, which is stored in the kernel layer. The lock screen wallpaper application starts the display driver by calling the kernel layer, controls the display screen 194 to change from displaying the two-dimensional image of the lock screen wallpaper to the three-dimensional image corresponding to the motion gesture.

[0130] See Figure 5 , Figure 5 This is a flow chart of an image processing method provided by an embodiment of the present application, which can be applied to Figure 3 The electronic device 100 shown includes but is not limited to the following steps:

[0131] Step S501: Acquire an image to be processed.

[0132] In one possible implementation, the image to be processed includes a two-dimensional image used as a lock screen wallpaper, for example Figure 1A The lock screen wallpaper 101 shown. For example, in the lock screen state, when the electronic device 100 detects that the user's face has shifted, it is necessary to adjust the displayed lock screen wallpaper. For example, the electronic device 100 needs to convert the two-dimensional image corresponding to the lock screen wallpaper into a three-dimensional image corresponding to the displacement of the user's face. It is understandable that in order to save storage space, the electronic device 100 does not store a large number of three-dimensional images. Instead, in the case of a scene where a three-dimensional image needs to be displayed, the two-dimensional image is replaced and transformed into a three-dimensional image. Therefore, the electronic device needs to first obtain the image to be processed, and then process the image to be processed according to steps S502 and S503.

[0133] In one possible implementation, the image to be processed includes an image rendered based on a texture with a discrete effect, that is, an image obtained by mapping a texture with a caustic effect onto a plane. The caustic effect is an optical phenomenon. When light passes through a transparent medium, due to the unevenness of the medium surface, the light refraction does not occur in parallel, but diffuse reflection occurs, and the projected surface forms photon dispersion. Generally speaking, this effect is widely used in rendering to create realistic light and shadow effects, such as scenes such as water ripples and glassware. In one implementation, when the electronic device 100 needs to render an image of a swimming pool, the method adopted is to map a two-dimensional texture with a discrete effect onto the wall of the swimming pool (i.e., a plane) to obtain the swimming pool image. However, due to the presence of the caustic effect, the edge of the swimming pool in the obtained swimming pool image may be deformed and stretched. In order to improve the deformation and stretching problem, the electronic device 100 needs to first obtain the image to be processed, and then process the image to be processed according to steps S502 and S503.

[0134] Step S502: Generate a depth map set corresponding to the image to be processed.

[0135] Among them, the depth map set includes N depth maps, each depth map in the N depth maps corresponds to a depth value range, and the depth value range shows a decreasing trend. For example, the first depth map corresponds to a first depth value range, and the second depth map corresponds to a second depth value range. The second depth value range is smaller than the first depth value range, so the depth maps in the depth map set can be called a downsampling pyramid structure, and N is a positive integer greater than or equal to 2.

[0136] In one possible implementation, the electronic device 100 performs N downsampling processing on the original depth map of the image to be processed, thereby obtaining N depth maps. The original depth map may be a depth map corresponding to the image to be processed pre-stored in the electronic device 100, or a depth map corresponding to the image to be processed generated by the electronic device 100 based on deep learning technology. The present application does not impose any restrictions on the source of the original depth map. The depth value of each pixel in the original depth map in the embodiment of the present application can be used to indicate the distance from the surface of the occluder (i.e., the shadow generating object) in the image to be processed to the viewpoint. The resolution of the original depth map is consistent with the resolution of the image to be processed, that is, the image size is consistent. For example, if the resolution of the image to be processed is 1024*1024, then the resolution of the original depth map is also 1024*1024. 1024*1024 is used to indicate the number of horizontal pixels and the number of vertical pixels.

[0137] It is understood that in image processing, downsampling generally refers to reducing the resolution or size of an image, that is, reducing the number of pixels in the image. The core principle of downsampling is to extract data points from the original data according to a certain ratio. For example, assuming that the size of the image to be processed is L*M, it is downsampled s times to obtain a resolution image of (L / s)(N / s) size, where s should be a common divisor of L and N. Therefore, in an embodiment of the present application, the electronic device downsamples the original depth map of the image to be processed N times by s. Each downsampling can reduce the number of pixels in the original depth map, and the resolution of each depth map finally obtained is less than the resolution of the original depth map. It is understood that since each pixel in the original depth map has a depth value, as the number of pixels decreases, the depth value reflected by the original depth map will also decrease. As the number of downsampling increases, the depth information that can be reflected by the depth map obtained by downsampling becomes more blurred. Among them, each depth map in the depth map set corresponds to a depth value range, and the depth value range is determined by the difference between the foreground depth value and the background depth value of the depth map. The foreground depth value includes the depth value corresponding to the pixel point located in the foreground, and the background depth value includes the depth value corresponding to the pixel point located in the background.

[0138] In one implementation, the electronic device performs weighted averaging on the pixels in the original depth map of the processed image in units of s pixels to obtain a first depth map, where s is a positive integer greater than or equal to 4; performs weighted averaging on the pixels in the first depth map in units of s pixels to obtain a second depth map; performs weighted averaging on the pixels in the second depth map in units of s pixels to obtain a third depth map; and performs weighted averaging on the pixels in the third depth map in units of s pixels to obtain a fourth depth map.

[0139] For example, if N is equal to 4 and s is equal to 4, see Figure 6A , Figure 6A This is a schematic diagram of a depth map set provided by this application. Figure 6AAs shown, the depth map corresponding to level 0 is the original depth map, that is, the depth map without downsampling, and the number of pixels contained in the corresponding pixel map is consistent with the number of pixels of the image to be processed. During the downsampling process, the pixels in the depth map corresponding to level 0 (original depth map) are first weighted averaged in units of 4 pixels to obtain the depth map corresponding to level 1 (i.e., the first depth map). Among them, the depth value of each pixel in the depth map corresponding to level 1 is equal to the weighted average of the depth values ​​of the 4 pixels in the depth map corresponding to level 0. For example, assuming that the depth value range is [0.1], the depth values ​​of these four pixels are 0, 0.5, 0.2, and 0.2, respectively. Then, the depth value of a pixel determined by these four pixels in the depth map corresponding to level 1 = (0+0.5+0.2+0.2) / 4 = 0.225. Then, by analogy, the electronic device performs weighted averaging on the pixels in the depth map corresponding to level 1 (i.e., the first depth map) in units of 4 pixels, thereby obtaining a depth map corresponding to level 2 (i.e., the second depth map). The depth value of each pixel in the depth map corresponding to level 2 is equal to the weighted average of the depth values ​​of the 4 pixels in the depth map corresponding to level 1. The electronic device then performs weighted averaging on the pixels in the depth map corresponding to level 2 (i.e., the second depth map) in units of 4 pixels, thereby obtaining a depth map corresponding to level 3 (i.e., the third depth map); the electronic device then performs weighted averaging on the pixels in the depth map corresponding to level 3 (i.e., the third depth map) in units of 4 pixels, thereby obtaining a depth map corresponding to level 4 (i.e., the fourth depth map). Ultimately, the electronic device can obtain five depth maps, including the original depth map of level 0, the first depth map of level 1, the second depth map of level 2, the third depth map of level 3, and the fourth depth map of level 4.

[0140] In one implementation, weighted averaging is performed on the pixels in the original depth map of the processed image in units of s pixels to obtain a first depth map, where s is a positive integer greater than or equal to 4; 2 The pixel points in the original depth map of the processed image are weighted averaged in units of s to obtain a second depth map; 3 The pixels in the original depth map of the processed image are weighted averaged to obtain a third depth map; 4 The pixel points in the original depth map of the processed image are weighted averaged in units of pixels to obtain a fourth depth map.

[0141] For example, if N is equal to 4 and s is equal to 4, see Figure 6B , Figure 6B This is a schematic diagram of another depth map set provided by this application. Figure 6BAs shown in the figure, the depth map corresponding to level 0 is the original depth map, that is, the depth map without downsampling, and the number of pixels contained in the corresponding pixel map is consistent with the number of pixels of the image to be processed. During the downsampling process, the pixels in the depth map corresponding to level 0 (original depth map) are weighted averaged in units of 4 pixels to obtain the depth map corresponding to level 1 (i.e., the first depth map), wherein the depth value of each pixel in the depth map corresponding to level 1 is equal to the weighted average of the depth values ​​of the 4 pixels in the depth map corresponding to level 0. Then, the electronic device performs weighted averaging on the pixels in units of 4 pixels. 2 = 16 pixels are used as units to perform weighted averaging on the pixels in the depth map corresponding to level 0 (i.e., the original depth map), thereby obtaining a depth map corresponding to level 2 (i.e., the second depth map), wherein the depth value of each pixel in the depth map corresponding to level 2 is equal to the weighted average of the depth values ​​of the 16 pixels in the depth map corresponding to level 0. The electronic device takes every 4 3 = 64 pixels are used as units to perform weighted averaging on the pixels in the depth map corresponding to level 0 (i.e., the original depth map), thereby obtaining a depth map corresponding to level 3 (i.e., the third depth map), wherein the depth value of each pixel in the depth map corresponding to level 3 is equal to the weighted average of the depth values ​​of the 64 pixels in the depth map corresponding to level 0. The electronic device then performs weighted averaging on the pixels in the depth map corresponding to level 0 (i.e., the original depth map) in units of 4 = 64 pixels. 3 = = 256 pixels as a unit, the pixels in the depth map corresponding to level 0 (i.e., the original depth map) are weighted averaged to obtain a depth map corresponding to level 4 (i.e., the fourth depth map), wherein the depth value of each pixel in the depth map corresponding to level 3 is equal to the weighted average of the depth values ​​of 256 pixels in the depth map corresponding to level 0. Ultimately, the electronic device can obtain five depth maps, including the original depth map of level 0, the first depth map of level 1, the second depth map of level 2, the third depth map of level 3, and the fourth depth map of level 4. For other relevant descriptions of "the original depth map of level 0, the first depth map of level 1, the second depth map of level 2, the third depth map of level 3, and the fourth depth map of level 4", please refer to the above Figure 6A , I will not go into details this time.

[0142] For example, assuming that the value of N is 4, that is, the electronic device downsamples the original depth map of the image to be processed four times to obtain four depth maps, including a first depth map, a second depth map, a third depth map, and a fourth depth map. The resolution of the first depth map is smaller than that of the second depth map, which is smaller than that of the third depth map, which is smaller than that of the fourth depth map. That is, as the number of depth maps increases, the depth information reflected by them becomes more blurred. The first depth map corresponds to a first depth threshold range, which is determined by the difference between the average depth of pixels in the first foreground and the average depth of pixels in the first background in the first depth map; the second depth map corresponds to a second depth threshold range, which is determined by the difference between the average depth of pixels in the second foreground and the average depth of pixels in the second background in the second depth map; the third depth map corresponds to a third depth threshold range, which is determined by the difference between the average depth of pixels in the third foreground and the average depth of pixels in the third background in the third depth map; and the fourth depth map corresponds to a fourth depth threshold range, which is determined by the difference between the average depth of pixels in the fourth foreground and the average depth of pixels in the fourth background in the fourth depth map. The maximum value in the first depth threshold range is greater than the maximum value in the second depth threshold range, greater than the maximum value in the third depth threshold range, and greater than the maximum value in the fourth depth threshold range. That is, the greater the difference between the average depths of the foreground and background in the depth map, the greater the threshold value; and the smaller the average difference, the smaller the threshold value.

[0143] It can be understood that during the downsampling process, the depth value corresponding to each pixel in the depth map obtained each time is gradually decreasing, so the difference in depth value between pixels is also gradually decreasing. Therefore, for the foreground and background, the difference between the depth average of the pixels in the foreground and the depth average of the pixels in the background is also gradually decreasing. This shows that the difference between the depth value of the pixels in the foreground and the depth value of the pixels in the background in the depth map obtained by downsampling is not very large, and the depth map gradually presents a smooth trend. Among them, the depth average of the pixels in the foreground is obtained by summing and averaging the depth values ​​of each pixel in the foreground, which can represent the depth information of the foreground; the depth average of the pixels in the background is obtained by summing and averaging the depth values ​​of each pixel in the background, which can represent the depth information of the background.

[0144] In one implementation, the electronic device has multiple depth threshold ranges preset internally, and each depth image generated corresponds to a depth threshold range. In one implementation, the electronic device can determine the corresponding depth threshold range based on the difference between the average depth of pixels in the foreground and the average depth of pixels in the background in the generated depth map. It should be noted that the larger the difference in the depth map, the less smooth the transition between the foreground and the background. In this case, the depth map can be used to process smoother images to be processed. Because the depth values ​​in smoother images to be processed do not change dramatically (i.e., the difference is smaller), the larger the difference in the depth map, the smaller the maximum value in the depth threshold range corresponding to the depth map. The smaller the difference in the depth map, the smoother the transition between the foreground and the background. In this case, the depth map can be used to process non-smooth images to be processed. Because the depth values ​​in non-smooth images to be processed change dramatically (i.e., the difference is larger), the smaller the difference in the depth map, the larger the maximum value in the depth threshold range corresponding to the depth map. Therefore, the transition between the foreground and background in the original depth map of level 0 is not smooth, and can be used to process the image to be processed with a depth value change within 0.1, so it corresponds to the original depth threshold range (for example, greater than or equal to 0 and less than 0.1); the transition between the foreground and background in the first depth map of level 1 is the first smoothness, which can be used to process the image to be processed with a depth value change between 0.1 and 0.2, so it corresponds to the first depth threshold range (for example, greater than or equal to 0.1 and less than 0.2); the transition between the foreground and background in the second depth map of level 2 is the second smoothness, which can be used to process the depth value change between 0.2 and 0.3 , so it corresponds to the second depth threshold range (for example, greater than or equal to 0.2 and less than 0.3); the transition between the foreground and the background in the third depth map of level 3 is the third smoothness, which can be used to process the image to be processed with a depth value change between 0.3 and 0.4, so it corresponds to the third depth threshold range (for example, greater than or equal to 0.3 and less than 0.4); the transition between the foreground and the background in the fourth depth map of level 4 is the fourth smoothness, which can be used to process the image to be processed with a depth value change between 0.4 and 0.5, so it corresponds to the fourth depth threshold range (for example, greater than or equal to 0.4 and less than 0.5).

[0145] Step S503: Processing the target area of ​​the image to be processed according to the target depth map in the depth map set to obtain a target image.

[0146] Specifically, each depth map corresponds to a depth threshold range, so each depth map can reflect the changing state of the depth difference between its corresponding pixels, and thus the changing state of its processing capabilities. For example, a depth threshold range greater than or equal to 0.4 and less than 0.5 indicates that the corresponding depth map can be used to process areas in the image to be processed where the depth change value is between 0.4 and 0.5. Therefore, the depth change value corresponding to the target area is within the depth threshold range corresponding to the template depth map.

[0147] In one possible implementation, the electronic device first determines the depth change value of the target area of ​​the image to be processed, and then determines the target depth map from the depth map set based on the depth change value, and then performs a displacement transformation on the target area of ​​the image to be processed based on the target depth map to obtain the target image. Among them, the size of the target area can be set according to actual needs. For example, the size of the target area is 128*128. This application does not impose any restrictions on the size of the target area. In the scenario of converting a two-dimensional image to a three-dimensional image, the target image finally processed is a three-dimensional image. In the rendering scene, the image finally processed is a rendered image with a caustic effect. Because in the processing process, the depth change values ​​of different areas are processed using their corresponding depth maps, so that the depth change value can be reduced, and then the transition imbalance caused by the excessive depth change value can be improved, and the stretching deformation phenomenon in the target image can be further improved (for example, Figure 2 The stretching deformation phenomenon shown in the figure can be eliminated), thereby improving the image processing effect and enhancing the user experience.

[0148] In one implementation, in order to determine the depth change value of the target area, the electronic device may first calculate the absolute value of the difference in depth value between the target pixel and a preset number of pixel points, thereby obtaining a preset number of absolute values ​​of the difference, and then use the maximum value of the preset number of absolute values ​​of the difference as the depth change value of the target area of ​​the image to be processed. Wherein, the target pixel point is a pixel point in the target area of ​​the image to be processed. Exemplarily, the preset number of pixel points include eight pixel points adjacent to the target pixel point, so the depth value of the target pixel point and the depth values ​​of these eight pixel points are calculated separately to obtain eight depth differences, and the maximum value is selected from these eight depth differences as the depth change value of the target area.

[0149] For example, see Figure 7 , Figure 7 This is a schematic diagram of processing an image to be processed based on a depth map set provided by an embodiment of the present application. Figure 7As shown, the image to be processed includes an image located in the foreground and an image located in the background, and the depth value of the pixel in the foreground image is less than the depth value of the pixel in the background image. In order to determine the depth map corresponding to each area in the image to be processed from the depth map set, the depth change value of each area can be calculated first. Taking the area 702 corresponding to the depth change value of 0.33 as an example, the pixel located at the center of the area can be taken as the target pixel, and the difference between the depth value of the target pixel and the depth values ​​of the eight pixel points around it is calculated. These eight pixel points include the pixels located directly above, directly below, directly to the left, directly to the right, upper left, lower left, upper right and lower right of the target pixel. As shown Figure 7 As shown in the figure, assuming that the depth value of the target pixel is 0.3, the depth values ​​of the eight pixels around it are 0.4, 0.45, 0.63, 0.55, 0.61, 0.4, 0.3 and 0.5 respectively. Therefore, the absolute values ​​of the differences between the depth value of the target pixel and the depth values ​​of these eight pixels are 0.1, 0.15, 0.33, 0.25, 0.31, 0.1, 0 and 0.2 respectively. Therefore, the maximum absolute value of the difference is 0.33, which can be regarded as the depth change value of the area.

[0150] like Figure 7 As shown, regions 701 and 702 are located at the intersection of the foreground and background, and the depth change value in the intersection region may undergo a sudden change. Region 703 is located in the foreground, and its depth change value is unlikely to undergo a sudden change. For region 703 corresponding to a depth change value of 0.05, region 701 corresponding to a depth change value of 0.45, and region 702 corresponding to a depth change value of 0.33, the corresponding depth map can be selected from the depth map set based on the corresponding depth change value. Since the depth change value of 0.05 is within the depth threshold range (0.01], region 703 corresponds to the original depth map of level 0; since the depth change value of 0.45 is within the depth threshold range (0.4.05], region 701 corresponds to the fourth depth map of level 4; since the depth change value of 0.33 is within the depth threshold range (0.3.04], region 702 corresponds to the third depth map of level 3. Next, the electronic device may perform a permutation change on the processed image based on the depth map corresponding to each region. For region 701, the electronic device may smooth the pixels with drastic changes in region 701 according to the fourth depth map, so that the transition of depth values ​​between pixels in region 701 is smoother. Similarly, it can be seen that the electronic device may smooth the pixels with drastic changes in region 702 according to the third depth map, so that the transition of depth values ​​between pixels in region 702 is smoother. Therefore, in the target image obtained after the permutation transformation of the processed image based on the depth maps corresponding to each region, the stretching deformation phenomenon of the intersecting area is optimized.

[0151] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0152] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0153] It should be understood that each step in the above method embodiments provided herein can be implemented by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of this application can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0154] The present application also provides an electronic device, which may include a memory and a processor, wherein the memory may be used to store a computer program, and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0155] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0156] In one possible design, the chip system further includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0157] The chip system can be composed of chips, or can include chips and other discrete devices.

[0158] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0159] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0160] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0161] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device in any of the above embodiments.

[0162] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0163] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.

[0164] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0165] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0166] In short, the above description is only an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of the present invention should be included in the scope of protection of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method includes: Get the image to be processed; Generate a depth map set corresponding to the image to be processed, the depth map set including N depth maps, each depth map in the N depth maps corresponding to a depth threshold range, where N is a positive integer greater than or equal to 2; The target area of ​​the image to be processed is processed according to the target depth map in the depth map set to obtain a target image, wherein the depth change value corresponding to the target area is within the depth threshold range corresponding to the target depth map, and the depth change value is the maximum difference between the depth values ​​of two adjacent pixels in the target area.

2. The method according to claim 1, characterized in that The image to be processed includes a two-dimensional image serving as a lock screen wallpaper, and the target image includes a three-dimensional image serving as the lock screen wallpaper.

3. The method according to claim 1, characterized in that The image to be processed includes a map with a caustic effect, and the target image is an image rendered based on the image to be processed.

4. The method according to any one of claims 1 to 3, characterized in that Generating a depth map set corresponding to the image to be processed includes: Downsampling the original depth map of the image to be processed N times obtains the N depth maps in the depth map set, wherein a depth threshold range corresponding to each depth map in the N depth maps is determined by a difference between an average foreground depth and an average background depth of the depth map.

5. The method according to claim 4, characterized in that The value of N is 4, and performing N downsampling processing on the original depth map of the image to be processed to obtain the N depth maps in the depth map set includes: Performing weighted averaging processing on the pixels in the original depth map of the processed image in units of s pixels to obtain a first depth map, where s is a positive integer greater than or equal to 4; Performing weighted averaging processing on the pixels in the first depth map in units of the s pixels to obtain a second depth map; Performing weighted averaging processing on the pixels in the second depth map in units of the s pixels to obtain a third depth map; A weighted average process is performed on the pixels in the third depth map in units of the s pixels to obtain a fourth depth map.

6. The method according to claim 4, characterized in that The value of N is 4, and performing N downsampling processing on the original depth map of the image to be processed to obtain the N depth maps in the depth map set includes: Performing weighted averaging processing on the pixels in the original depth map of the processed image in units of s pixels to obtain a first depth map, where s is a positive integer greater than or equal to 4; s 2 Performing weighted averaging processing on the pixel points in the original depth map of the processed image in units of pixels to obtain a second depth map; s 3 Performing weighted averaging processing on the pixels in the original depth map of the processed image in units of pixels to obtain a third depth map; s 4 The pixel points in the original depth map of the processed image are weighted averaged in units of pixels to obtain a fourth depth map.

7. The method according to claim 5 or 6, characterized in that The first depth map corresponds to a first depth threshold range, and the first depth threshold range is determined by a difference between an average depth value of pixels located in a first foreground and an average depth value of pixels located in a first background in the first depth map; The second depth map corresponds to a second depth threshold range, and the second depth threshold range is determined by a difference between an average depth of pixels in a second foreground and an average depth of pixels in a second background in the second depth map; The third depth map corresponds to a third depth threshold range, where the third depth threshold range is determined by a difference between a depth average of pixels in a third foreground and a depth average of pixels in a third background in the third depth map; The fourth depth map corresponds to a fourth depth threshold range, where the fourth depth threshold range is determined by a difference between an average depth of pixels in a fourth foreground and an average depth of pixels in a fourth background in the fourth depth map; The maximum value in the first depth threshold range is greater than the maximum value in the second depth threshold range, greater than the maximum value in the third depth threshold range, and greater than the maximum value in the fourth depth threshold range.

8. The method according to any one of claims 1 to 7, characterized in that The step of processing the target area of ​​the image to be processed according to the target depth map in the depth map set to obtain the target image includes: Determining a depth change value of a target area of ​​the image to be processed; determining a target depth map from the depth map set according to the depth change value; A target image is obtained by performing a displacement transformation on the target area of ​​the image to be processed based on the target depth map.

9. The method according to claim 8, characterized in that Determining the depth change value of the target area of ​​the image to be processed includes: Calculating a depth difference between a target pixel and a preset number of pixel points to obtain the preset number of depth change values, wherein the target pixel point is a pixel point in a target area of ​​the image to be processed; The maximum mean value among the preset number of depth change values ​​is used as the depth change value of the target area of ​​the image to be processed.

10. The method according to claim 9, characterized in that The preset number of pixel points includes eight pixel points adjacent to the target pixel point.

11. An electronic device, characterized in that: The electronic device includes: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1-10.

12. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1-10.

13. A computer program product comprising instructions, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 10.

14. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 10.

Images