Sky segmentation method, device, computer equipment and storage medium

Abstract

The present application discloses a method, apparatus, computer device, and storage medium for sky segmentation, relating to the field of image processing. The method comprises: obtaining a first image segmentation probability of an image, the first image segmentation probability being used to indicate the probability that a pixel point on the image belongs to the sky; determining the sky segmentation probability of the sky hemisphere based on the first image segmentation probability of a target pixel point based on a position mapping relationship between the image and the sky hemisphere, wherein the sky hemisphere is a hemisphere model established with the camera that captured the image as the center, the target pixel point being a pixel point with a corresponding spatial point on the sky hemisphere, and the sky segmentation probability being used to indicate the probability that a spatial point on the sky hemisphere belongs to the sky; and determining a second image segmentation probability of the target pixel point based on the sky segmentation probability based on the position mapping relationship. The present application provides a sky segmentation method for determining the sky area in an image by mapping the image to the sky hemisphere, thereby improving the accuracy of sky segmentation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a sky segmentation method, apparatus, computer equipment, and storage medium. Background Art

[0002] Sky segmentation is the process of separating the sky area from the non-sky area in an image. Based on the segmentation results, you can perform sky transformations or add special effects to the sky area, such as changing the sky color or adding shooting star animations to the sky area.

[0003] Related technologies use color thresholding or deep learning network models to perform sky segmentation on the current image. This method calculates the probability that each pixel in the current image belongs to the sky area, obtaining a sky segmentation probability map for the current image, and thus achieving sky segmentation.

[0004] However, when the color features of the sky and ground areas are similar, the ground area may be misidentified as the sky area, resulting in low accuracy of sky segmentation. How to accurately achieve sky segmentation is a problem that needs to be solved. Summary of the Invention

[0005] The present invention provides a method, apparatus, computer device, and storage medium for sky segmentation, which can improve the accuracy of sky segmentation. The technical solution is as follows:

[0006] In one aspect, a sky segmentation method is provided, the method comprising:

[0007] Obtaining a first image segmentation probability of a pixel point on the image, where the first image segmentation probability is used to indicate a probability that the pixel point belongs to the sky;

[0008] Determining, based on a positional mapping relationship between the image and a sky hemisphere, a sky segmentation probability of a spatial point on the sky hemisphere according to the first image segmentation probability of a target pixel, wherein the sky hemisphere is a hemispherical model established with a camera that captured the image as the center, and the target pixel is a pixel point on the sky hemisphere that has a corresponding spatial point;

[0009] Based on the position mapping relationship, a second image segmentation probability of the target pixel point is determined according to the sky segmentation probability of the spatial point on the sky hemisphere, where the second image segmentation probability is used to indicate the probability that the target pixel point belongs to the sky.

[0010] In another aspect, a sky segmentation device is provided, the device comprising:

[0011] an acquisition module, configured to acquire a first image segmentation probability of a pixel point on an image, where the first image segmentation probability is used to indicate a probability that the pixel point on the image belongs to the sky;

[0012] a sky determination module for determining, based on a position mapping relationship between the image and a sky hemisphere and the first image segmentation probability of a target pixel, a sky segmentation probability of a spatial point on the sky hemisphere, wherein the sky hemisphere is a hemispherical model established with the camera that captured the image as the center, and the target pixel is a pixel point on the sky hemisphere that has a corresponding spatial point;

[0013] An image determination module is used to determine a second image segmentation probability of the target pixel point based on the position mapping relationship and the sky segmentation probability of the spatial point on the sky hemisphere, wherein the second image segmentation probability is used to indicate the probability that the target pixel point belongs to the sky.

[0014] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the sky segmentation method as described in any of the above embodiments of the present application.

[0015] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores computer instructions, and the computer instructions are loaded and executed by a processor to implement the sky segmentation method provided in various aspects of the present application.

[0016] In another aspect, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described sky segmentation method.

[0017] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0018] By mapping the image segmentation probability of pixels on an image to the sky hemisphere, pixels belonging to the ground area are not mapped to the sky hemisphere, thus avoiding misclassifying ground areas with similar color to the sky as sky. The sky segmentation result of the image is then determined based on the sky segmentation probability of the spatial points. By incorporating the sky hemisphere model into the sky segmentation process, the accuracy of sky segmentation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 is a schematic diagram of a sky hemisphere provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a camera coordinate system provided by an exemplary embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of an interface for applying a sky segmentation method to video production, provided by an exemplary embodiment of the present application;

[0023] Figure 4 is a structural block diagram of a computer system provided by another exemplary embodiment of the present application;

[0024] Figure 5 is a flowchart of a sky segmentation method provided by an exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of obtaining a first image segmentation probability of an image provided by an exemplary embodiment of the present application;

[0026] Figure 7 is a flowchart of a sky segmentation method provided by an exemplary embodiment of the present application;

[0027] Figure 8 This is a schematic diagram of a process for determining the longitude and latitude of a spatial point based on the coordinates of a pixel point, provided by an exemplary embodiment of the present application;

[0028] Figure 9 This is a schematic diagram of a process for determining the coordinates of a pixel point based on the longitude and latitude of a spatial point, provided by an exemplary embodiment of the present application;

[0029] Figure 10 is a structural block diagram of a sky segmentation device provided by an exemplary embodiment of the present application;

[0030] Figure 11 A structural block diagram of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0032] First, the terms involved in the embodiments of this application are introduced:

[0033] IMU: Short for Inertial Measurement Unit. An IMU generally includes an accelerometer, an angular velocity sensor (gyroscope), and a magnetometer. In this application, the terminal can determine the posture of the camera on the terminal by obtaining data from the IMU component.

[0034] Sky Segmentation: In the image, the sky area and non-sky area are divided to determine the sky area.

[0035] Sky Hemisphere: A hemisphere model with a radius of s, centered on the camera used to capture the image. Typically, s is set to twice the camera's focal length. The location of a point on the sky hemisphere's surface can be uniquely determined by longitude and latitude, similar to how longitude and latitude uniquely identify a point on a globe.

[0036] Indicatively, Figure 1 , using the camera 22 for collecting images as the center of the hemispherical model and twice the focal length of the camera 22 as the radius of the hemispherical model to establish a sky hemisphere 24.

[0037] Optical center: the center of the camera's convex lens.

[0038] The world coordinate system is an absolute coordinate system that is unique and unchanging once selected. This application uses the camera position as the origin, O, with due south as the x-axis, due west as the y-axis, and the vertically downward z-axis as an example to establish the world coordinate system. This application does not restrict the method used to establish the world coordinate system.

[0039] The camera coordinate system is used to indicate the position of the object being photographed relative to the camera. As shown in the figure, this application establishes the camera coordinate system using the optical center (i.e., the pinhole in pinhole imaging) as the origin O and the z-axis, which is perpendicular to the camera's optical center and points toward the object.

[0040] Indicatively, Figure 2 , Figure 2 The pinhole imaging model used in camera photography and the method used in this application to establish the camera coordinate system are shown in Figure 1. Light from source P propagates in a straight line, passes through the camera's optical center O (the pinhole in the pinhole imaging model), and falls on a physical imaging plane at a distance f from the camera lens, resulting in point P'. The camera coordinate system is established with the camera's optical center as the origin O, the plane containing the camera's optical center as the xoy plane, and the z-axis perpendicular to the camera's optical center and pointing toward the subject. Figure 2 A possible way to establish the camera coordinate system is shown.

[0041] like Figure 3 As shown, the user shoots a short video through the terminal. The picture captured by the camera at the initial moment is shown in picture 10. There are elements such as buildings and trees in picture 10, and the blank area in picture 10 is the sky area; in response to the user selecting the special effects function, the terminal determines the sky area in the image through the sky segmentation method and displays the pre-set special effects in the sky area. The picture displayed by the terminal is shown in picture 11; in response to the user rotating the camera to shoot, the terminal will continuously segment the sky according to the shot picture and display the corresponding special effects in the sky area, as shown in picture 12. That is, after the sky area is determined by the sky segmentation method, the sky is replaced with a sky with special effects, and according to the continuous change of the camera posture, the sky with special effects displayed by the terminal will also change continuously accordingly.

[0042] Figure 4 The following is a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device can be a terminal device or a part of a terminal device. The device includes: a bus 101, a processor 102, a memory 103, an IMU component 104, and a camera component 105.

[0043] The processor 102 includes one or more processing cores. The processor 102 executes various functional applications and information processing by running software programs and modules.

[0044] The memory 103 is connected to the processor 102 via the bus 101 .

[0045] The memory 103 may be used to store at least one instruction, and the processor 102 may be used to execute the at least one instruction to implement each step in the following method embodiment.

[0046] In addition, the memory 103 can be implemented by any type of volatile or non-volatile storage device or a combination thereof. The volatile or non-volatile storage device includes but is not limited to: a magnetic disk or an optical disk, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a static random access memory (SRAM), a read-only memory (ROM), a magnetic memory, a flash memory, and a programmable read-only memory (PROM).

[0047] The IMU component 104 may be used to collect IMU data indicating changes in attitude. For example, the IMU data collected by the IMU component 104 in the present application may be used to indicate changes in the attitude of a camera in the terminal.

[0048] The camera component 105 is used to capture images or videos.

[0049] In the related art, the traditional sky segmentation method of setting a color threshold is used to analyze the current image and determine the sky area, thereby replacing the obtained sky area with a pre-set effect. However, when the colors of the sky and the ground are similar, it is easy to cause misjudgment. For example, in an image collected on a rainy day, the color of the sky area is very likely to be similar to the color of the concrete floor, resulting in a decrease in the accuracy of sky segmentation. In this application, a sky hemisphere model centered on the camera is constructed. In the sky segmentation method based on the sky hemisphere model, the pixel points belonging to the ground area on the image will not be mapped to the sky hemisphere, thereby avoiding misjudging the ground area as a sky area with a similar color.

[0050] Figure 5 FIG. 1 shows a flow chart of a sky segmentation method provided by an exemplary embodiment of the present application, which is applied to a terminal. Figure 5 As shown, the method includes:

[0051] Step 220: Obtain a first image segmentation probability of a pixel point on the image;

[0052] The image can be a photo captured by the terminal through the camera component; or any frame of a video captured by the terminal through the camera component.

[0053] The first image segmentation probability is used to indicate the probability that a pixel point on the image belongs to the sky.

[0054] The first image segmentation probability of the pixel points on the image is obtained by the traditional color threshold method; or, by collecting a set of image samples, training a deep learning network model, calling the trained deep learning network model to recognize the image, and obtaining the first image segmentation probability of the pixel points on the image.

[0055] The present application does not limit the method of acquiring an image and the method of acquiring the first image segmentation probability of pixels on the image based on the image.

[0056] For example, Figure 6As shown, the terminal acquires image 32, analyzes image 32 by calling a deep learning network model, and outputs the probability that each pixel on image 32 belongs to the sky, thereby obtaining the first image segmentation probability of each pixel on image 32, as shown in image 34. The numbers on image 34 represent the probability that the pixel belongs to the sky area. For example, when the first image segmentation probability of a pixel is 1, it means that the pixel belongs to the sky area 100%, that is, the pixel definitely belongs to the sky area; when the first image segmentation probability of a pixel is 0.7, it means that there is a 70% probability that the pixel belongs to the sky area; when the first image segmentation probability of a pixel is 0, it means that there is a 0% probability that the pixel belongs to the sky area, that is, the pixel cannot belong to the sky area.

[0057] Step 240: determining the sky segmentation probability of the spatial point on the sky hemisphere according to the first image segmentation probability of the target pixel point based on the position mapping relationship between the image and the sky hemisphere;

[0058] The sky hemisphere is a hemisphere model with a radius of s, centered on the camera used to capture the image. Typically, s is set to twice the camera's focal length f. The location of a point on the sky hemisphere's surface can be uniquely determined by longitude and latitude, similar to how longitude and latitude uniquely identify a point on a globe.

[0059] The target pixel is a pixel that has a corresponding spatial point on the sky hemisphere. A 3D spherical mapping is performed on the pixel on the image. If the spatial point after the 3D spherical mapping falls within the sky hemisphere, the pixel is determined as the target pixel.

[0060] For example, by performing three-dimensional spherical mapping on the pixels on the image, the longitude and latitude of the spatial point corresponding to the pixel are determined; if the latitude of the spatial point belongs to the sky hemisphere, the pixel is determined as the target pixel. For example, if the latitude of the spatial point is greater than 0, the pixel is determined as the target pixel.

[0061] The position mapping relationship is used to indicate a one-to-one mapping relationship between a target pixel point on an image and a spatial point on the sky hemisphere. The process of determining the position of a spatial point on the sky hemisphere using the coordinates of the target pixel point on the image includes: determining the rotational attitude of the camera based on the IMU; determining the position coordinates of the spatial point in the camera coordinate system based on the coordinates of the target pixel point; determining the position coordinates of the spatial point in the world coordinate system based on the rotational attitude of the camera and the position coordinates of the spatial point in the camera coordinate system; and determining the longitude and latitude of the spatial point based on the position coordinates of the spatial point in the world coordinate system.

[0062] The sky segmentation probability is used to indicate the probability that a spatial point on the sky hemisphere belongs to the sky.

[0063] Exemplarily, the first image segmentation probability of the target pixel point is directly determined as the sky segmentation probability of the spatial point corresponding to the target pixel point; or, based on the position mapping relationship between the target pixel point and the spatial point, the first image segmentation probability of the target pixel point and the sky segmentation probability of the spatial point at a historical moment are filtered, and the result obtained after filtering is determined as the sky segmentation probability of the spatial point at the current moment.

[0064] Step 260: Based on the position mapping relationship, determine the second image segmentation probability of the target pixel point according to the sky segmentation probability of the spatial point on the sky hemisphere.

[0065] The target pixel point is a pixel point that has a corresponding spatial point on the sky hemisphere; the remaining pixel points other than the target pixel point do not have a corresponding spatial point on the sky hemisphere, that is, the remaining pixel points are pixel points belonging to the ground area in the image.

[0066] The second image segmentation probability is used to indicate the probability that the target pixel point belongs to the sky.

[0067] Based on the position mapping relationship between the target pixel point on the indication image and the spatial point on the sky hemisphere described in step 240, the sky segmentation probability of the spatial point on the sky hemisphere is mapped to the image, and the second image segmentation probability of the target pixel point in the image is determined; the second image segmentation probability of the remaining pixel points on the image except the target pixel point is determined to be 0.

[0068] Optionally, after determining the sky segmentation probability of the sky hemisphere, the sky segmentation probability cached on the sky hemisphere is updated, that is, the sky segmentation probability cached at a historical moment is replaced by the most recently determined sky segmentation probability.

[0069] Optionally, based on the second image segmentation probability of the pixels in the image, a sky region in the image is determined; and the sky region is replaced with a pre-set sky element, where the sky element is at least one of a video, an animation effect, text, and a picture. For example, the sky region in the image is replaced with a starry sky, or with a sky with fireworks animation, etc.

[0070] Optionally, the method for determining the sky region in the image based on the second image segmentation probability can be based on a pre-set threshold. For example, if the second image segmentation probability of a pixel is greater than 0.7, the pixel is determined to belong to the sky region. This application does not limit the method for determining the sky region based on the second image segmentation probability.

[0071] In summary, the sky segmentation method provided in this embodiment determines the sky region in the image by obtaining a first image segmentation probability for a pixel point on the image and determining the sky segmentation probability for the spatial point based on the positional mapping relationship between the target pixel point and the spatial point. The second segmentation probability for the target pixel point is then determined based on the sky segmentation probability for the spatial point, thereby determining the sky region on the image. By adopting a sky hemisphere model, interference from remaining pixels other than the target pixel point, i.e., pixels belonging to the ground region, is eliminated, preventing the misidentification of ground areas with a similar color to the sky as sky regions, thereby improving the accuracy of sky segmentation.

[0072] In the above embodiment, the sky segmentation process incorporates a sky hemisphere model to distinguish between the sky and the ground. Based on this, the sky segmentation probability can also be filtered using cached data from historical moments within the sky hemisphere. This results in more accurate sky segmentation and smoother edge transitions in the sky region, avoiding jitter in the sky region in the video and further improving sky segmentation accuracy.

[0073] Figure 7 FIG. 1 shows a flow chart of a sky segmentation method provided by an exemplary embodiment of the present application, which is applied to a terminal. Figure 7 As shown, the method includes:

[0074] Step 420: Obtain a first image segmentation probability of a pixel point on the image;

[0075] The method for obtaining the first image segmentation probability of the image is shown in step 220 and will not be described in detail here.

[0076] Step 442: If the spatial point after the three-dimensional spherical mapping of the pixel point on the image belongs to the sky hemisphere, the pixel point is determined as the target pixel point;

[0077] By performing three-dimensional spherical mapping on the pixel points on the image, the longitude and latitude of the spatial point corresponding to the pixel point are determined; when the latitude of the spatial point belongs to the sky hemisphere, the pixel point is determined as the target pixel point.

[0078] The process of performing three-dimensional spherical mapping on the pixel points on the image and determining the longitude and latitude of the spatial point corresponding to the pixel point is as follows: determine the rotation attitude of the camera based on the IMU; determine the position coordinates of the spatial point in the camera coordinate system based on the coordinates of the pixel point; determine the position coordinates of the spatial point in the world coordinate system based on the rotation attitude of the camera and the position coordinates of the spatial point in the camera coordinate system; determine the longitude and latitude of the spatial point based on the position coordinates of the spatial point in the world coordinate system.

[0079] Figure 8Taking a pixel point p on an image as an example, detailed steps for determining the longitude and latitude of a spatial point P corresponding to the pixel point p based on the coordinates of the pixel point p are shown:

[0080] Step 4421: Determine the rotational attitude of the camera based on the IMU;

[0081] Get the IMU attitude R from the phone's IMU imu , IMU attitude is used to indicate the position relationship of IMU relative to the world coordinate system, IMU attitude R imu is a matrix with 3 rows and 3 columns.

[0082] The camera's rotation posture R is calculated by the following formula: c :

[0083] R c =R ic -1 *R imu *R ic

[0084] Among them, R ic It is used to indicate the rotation attitude of the camera coordinate system relative to the IMU coordinate system. Since the position and attitude of the camera and IMU in the terminal are fixed, R ic is a known matrix with 3 rows and 3 columns; the result R calculated by the above formula is c Used to indicate the camera's posture, that is, the camera's current posture relative to the world coordinate system.

[0085] Step 4422: Determine the position coordinates of the spatial point P in the camera coordinate system according to the coordinates of the pixel point p;

[0086] Take any pixel point p on the image as an example, the coordinates of point p on the image are (u p , v p ).

[0087] The three-dimensional coordinate P of the pixel point p on the normalized plane in the camera coordinate system is determined by the following formula sphere_in_unit , where the normalized plane is the plane at unit distance in front of the camera:

[0088]

[0089] Among them, K is the camera's intrinsic matrix (Intrisic Matrix), which is a matrix with 3 rows and 3 columns. K can be expressed as f is the focal length of the camera; u is the horizontal coordinate of the optical center on the image; v is the vertical coordinate of the optical center on the image; the units of f, u, and v are all pixels.

[0090] Note that u and v are the horizontal and vertical coordinates of the center point of the image. For example, if the width of an image is Width pixels and the height is Height pixels, and the image is not distorted, Width = 2*u and Height = 2*v.

[0091] The three-dimensional coordinates P of p on the normalized plane in the camera coordinate system obtained by the above formula are sphere_in_unit For a matrix with 3 rows and 1 column, use P sphere_in_unit_x 、P sphere_in_unit_y 、P sphere_in_unit_z Represent the three elements in the matrix respectively.

[0092] Based on this, the coordinates P of point p on the image in the camera coordinate system are obtained by the following formula: sphere_in_cam :

[0093]

[0094] Where s is the radius of the sky hemisphere.

[0095] Step 4423: Determine the position coordinates of the spatial point P in the world coordinate system based on the rotation posture of the camera and the position coordinates of the spatial point P in the camera coordinate system;

[0096] The position coordinates of the spatial point in the camera coordinate system are converted to the position coordinates in the world coordinate system using the following formula:

[0097] P sphere_in_w =R c *P sphere_in_cam

[0098] Among them, R c is the rotation posture of the camera.

[0099] Step 4424: Determine the longitude and latitude of the spatial point P based on the position coordinates of the spatial point P in the world coordinate system.

[0100] The P obtained in the previous step sphere_in_w For a matrix with 3 rows and 1 column, use P x 、P y 、P z Represent the three elements in the matrix respectively, and schematically obtain the position coordinate matrix of the spatial point P in the world coordinate system as follows:

[0101]

[0102] Calculate the longitude P of the spatial point P longitude and latitude P latitude :

[0103]

[0104] atan2 is the inverse tangent function, which returns the azimuth angle in radians; asin is the inverse sine function, which returns the azimuth angle in radians; and s is the radius of the sky hemisphere.

[0105] Optionally, the longitude P of the spatial point P longitude and latitude P latitude It can be expressed in radians, with a range of [-π, +π]; or, the longitude P of a spatial point P longitude and latitude P latitude It can be expressed in degrees, with the range of angles being [-180°, +180°]. Radians and degrees can be converted to each other.

[0106] After obtaining the longitude and latitude of the spatial point corresponding to the pixel, if the latitude of the spatial point belongs to the sky hemisphere, the pixel is determined as the target pixel. For example, the pixel corresponding to the spatial point with a latitude greater than 0 is determined as the target pixel.

[0107] Step 444: Determine the first image segmentation probability of the target pixel as the first sky segmentation probability of the spatial point corresponding to the target pixel;

[0108] The first sky segmentation probability refers to the probability that a spatial point on the sky hemisphere belongs to the sky, which is obtained by mapping the spatial point based on the corresponding target pixel point on the image.

[0109] Based on the position mapping relationship between the target pixel point and the spatial point determined in step 442 , the first image segmentation probability of the target pixel point is determined as the first sky segmentation probability of the spatial point corresponding to the target pixel point.

[0110] Step 446: Determine a third sky segmentation probability of the spatial point corresponding to the target pixel point by a filtering method based on the first sky segmentation probability and the second sky segmentation probability;

[0111] The second sky segmentation probability is used to indicate the probability that the spatial point belongs to the sky at a historical moment, and the third sky probability is used to indicate the probability that the spatial point belongs to the sky at a current moment.

[0112] The filtering method includes at least one of the following filtering methods: weighted filtering; Kalman filtering; mean filtering; median filtering.

[0113] For example, a weighted filtering method is selected to filter the first sky segmentation probability and the second sky segmentation probability. A filter value a is preset, and the range of a is between 0 and 1. The first sky segmentation probability obtained by step 444 is Pro pic , and the second sky segmentation probability of the historical moment cached on the sky hemisphere is Pro sphereBy Pro sphere and Pro pic Weighted filtering obtains the third sky segmentation probability Pro sphere_filter :

[0114] Pro sphere_filter =a·Pro sphere +(1-a)·Pro pic

[0115] Optionally, the value of the filter value a can be adjusted according to actual conditions. If the sky segmentation result at a historical moment is considered more reliable, the value of a can be increased; if the sky segmentation result at the current moment is considered more reliable, the value of a can be decreased.

[0116] This application does not impose any restrictions on the selection of filtering methods and the setting of filtering values.

[0117] Optionally, the third sky segmentation probability of each spatial point on the sky hemisphere is used to replace the second sky segmentation probability to obtain the sky hemisphere at the current moment for caching; wherein the sky hemisphere at the current moment is used to perform filtering calculation on the third sky segmentation probability at the next moment. sphere_filter Replace the Second Sky Segmentation Probability Pro sphere , get the current sky hemisphere for caching.

[0118] Step 460: Based on the position mapping relationship, determine the second image segmentation probability of the target pixel point according to the third sky segmentation probability of the spatial point on the sky hemisphere.

[0119] The target pixel point is a pixel point that has a corresponding spatial point on the sky hemisphere; the remaining pixel points other than the target pixel point do not have a corresponding spatial point on the sky hemisphere, that is, the remaining pixel points are pixel points belonging to the ground area in the image.

[0120] The second image segmentation probability is used to indicate the probability that the target pixel point belongs to the sky.

[0121] After filtering is completed on the sky hemisphere, the filtered results need to be returned to the image for subsequent operations. Based on the position mapping relationship, the second image segmentation probability of the target pixel is determined according to the sky segmentation probability of the spatial point on the sky hemisphere; the second image segmentation probability of the remaining pixels on the image except the target pixel is set to 0.

[0122] The process of mapping the position of a spatial point on the sky hemisphere to the coordinates of a target pixel point on the image is the opposite of the process in step 442: determining the coordinates of the spatial point in the world coordinate system based on the longitude and latitude of the spatial point on the sky hemisphere; determining the position coordinates of the spatial point in the camera coordinate system based on the position coordinates of the spatial point in the world coordinate system; and obtaining the coordinates of the target pixel point on the image based on the position coordinates of the spatial point in the camera coordinate system.

[0123] Figure 9 The following diagram shows the steps of determining the coordinates of the target pixel corresponding to the spatial point P on the image based on the longitude and latitude of the spatial point P, taking the spatial point P on the sky hemisphere as an example:

[0124] Step 4601: Determine the coordinates of the spatial point P in the world coordinate system based on the longitude and latitude of the spatial point P on the sky hemisphere;

[0125] Based on the longitude P of point P longitude With latitude P latitude The coordinates P of the spatial point P in the world coordinate system are determined by the following formula sphere_in_w :

[0126]

[0127] Step 4602: Determine the position coordinates of the spatial point P in the camera coordinate system based on the position coordinates of the spatial point P in the world coordinate system;

[0128] Based on the position coordinate P of point P in the world coordinate system sphere_in_w The position coordinates of point P in the camera coordinate system are determined by the following formula.

[0129] P sphere_in_cam =R c -1 *P sphere_in_w

[0130] Among them, R c is the rotation posture of the camera.

[0131] P sphere_in_cam For a matrix with 3 rows and 1 column, use P sphere_in_cam_x 、P sphere_in_cam_y 、P sphere_in_cam_z Represent the three elements in the matrix respectively.

[0132] Step 4603: Obtain the coordinates of the pixel point on the image based on the position coordinates of the spatial point P in the camera coordinate system.

[0133] First, the position coordinates P of the space point P in the camera coordinate system sphere_in_cam Project it onto the normalized plane, that is, the plane at a unit distance in front of the camera, and obtain the position coordinates P of the spatial point P on the normalized planesphere_in_unit :

[0134]

[0135] Then based on the position coordinate P of the spatial point P on the normalized plane sphere_in_unit And the camera intrinsic parameter matrix K to obtain the coordinates P of the target pixel point P on the image pic :

[0136] P pic =K*P sphere_in_unit

[0137] After obtaining the position coordinates of the target pixel point corresponding to the spatial point based on the position mapping relationship, the third sky segmentation probability of the spatial point is determined as the second image segmentation probability of the target pixel point.

[0138] Optionally, when mapping the spatial points on the sky hemisphere to the target pixel points on the image, the sky hemisphere can be divided into integers according to longitude and latitude; or, the sky hemisphere can be divided into intervals of 0.1 longitude and latitude, and so on. The smaller the interval of longitude and latitude selected for division, the more spatial points obtained by division, and the more pixels obtained by projection onto the image, then the sky segmentation result on the image will be more accurate, but it will also occupy more memory resources. This application does not limit the method of dividing the sky hemisphere.

[0139] Optionally, the method for determining the sky region in the image based on the second image segmentation probability can be based on a pre-set threshold. For example, if the second image segmentation probability of a pixel is greater than 0.8, the pixel is determined to belong to the sky region. This application does not limit the method for determining the sky region based on the second image segmentation probability.

[0140] Optionally, after determining the second image segmentation probability of the image, a sky region in the image is determined based on the second image segmentation probability of the pixels; and the sky region is replaced with a pre-set sky element, where the sky element is at least one of a video, an animation effect, text, and a picture. For example, the sky region in the image is replaced with a starry sky, or with a sky region depicting a fireworks animation, etc.

[0141] In summary, the method provided in this embodiment is based on the first image segmentation probability of the pixel points on the obtained image, as well as the position mapping relationship between the pixel points and the spatial points. The first image segmentation probability of the target pixel point is mapped to the spatial point on the sky hemisphere to obtain the first sky segmentation probability. The second sky segmentation probability at the historical moment of the spatial point on the sky hemisphere is filtered to obtain the third sky segmentation probability at the current moment, which is then mapped to the image to obtain the sky segmentation result of the image. This sky segmentation method constructs a sky hemisphere model so that pixels belonging to the ground area in the image are not mapped to the sky hemisphere, thereby eliminating the interference of the ground area on the sky segmentation and improving the accuracy of sky segmentation.

[0142] Furthermore, the method provided in this embodiment further improves the accuracy of sky segmentation by fusing historical and current sky segmentation results through filtering on the sky hemisphere. Furthermore, when the sky is replaced with pre-set sky elements based on the sky segmentation results, the resulting sky region is continuous and stable, avoiding jitter and improving the sky segmentation effect.

[0143] In addition, in the method provided in this embodiment, the sky segmentation result is integrated into the sky hemisphere after being obtained. Therefore, when the application needs the sky segmentation result, it can be directly extracted from the cache of the sky hemisphere, and the sky segmentation process can be performed asynchronously in the background, avoiding the occurrence of freezes in the application and ensuring the smoothness of the application.

[0144] The method provided by this application has been well used in applications that require sky segmentation. Taking the production of sky effects in short videos as an example, the method provided by this application can effectively improve the accuracy and efficiency of producing sky effects: First, the method provided by this application determines the rotation posture of the camera relative to the ground based on the data provided by the mobile phone IMU, and thus uses the sky hemisphere model to exclude the ground area from the sky segmentation process, avoiding mistaking the ground area for the sky area; In addition, in the related art, the short video shooting application performs sky segmentation on each frame of video. The inconsistent sky segmentation results in each frame of video will cause the sky area in the video to shake. The method provided by this application combines the historical sky segmentation results and the current sky segmentation results. On the one hand, it makes the sky segmentation result more accurate, and on the other hand, it makes the edge change process of the sky area smoother and more natural; furthermore, the shooting of short videos itself will occupy a large amount of memory in the mobile phone. If the sky segmentation operation is performed on each frame of the image, the memory usage will be too high, resulting in problems such as screen freeze and high terminal power consumption. However, the method provided by the present application caches the sky segmentation probability on the sky hemisphere. Therefore, when the application needs to use the sky segmentation result, there is no need to wait until the sky segmentation process of the current frame is completed. The sky segmentation result on the sky hemisphere can be directly obtained, and the sky segmentation process can be performed asynchronously in the background, thereby improving the smoothness of the video.

[0145] Figure 8 This is a structural block diagram of a sky segmentation device provided by an exemplary embodiment of the present application. Figure 8 As shown, the device includes:

[0146] An acquisition module 720 is configured to acquire a first image segmentation probability of a pixel point on an image, where the first image segmentation probability indicates a probability that the pixel point belongs to the sky;

[0147] A sky determination module 740 is configured to determine, based on a positional mapping relationship between the image and a sky hemisphere and the first image segmentation probability of a target pixel, a sky segmentation probability of a spatial point on the sky hemisphere, wherein the sky hemisphere is a hemispherical model established with the camera that captured the image as the center, and the target pixel is a pixel point on the sky hemisphere that has a corresponding spatial point;

[0148] The image determination module 760 is used to determine the second image segmentation probability of the target pixel point based on the position mapping relationship and the sky segmentation probability of the spatial point on the sky hemisphere, wherein the second image segmentation probability is used to indicate the probability that the target pixel point belongs to the sky.

[0149] In a possible embodiment, the sky determination module 740 is used to determine the pixel point as the target pixel point when the spatial point after three-dimensional spherical mapping of the pixel point on the image belongs to the sky hemisphere; and to determine the sky segmentation probability of the spatial point corresponding to the target pixel point based on the first image segmentation probability of the target pixel point.

[0150] In a possible embodiment, the sky determination module 740 is used to determine the longitude and latitude of the spatial point corresponding to the pixel point by performing three-dimensional spherical mapping on the pixel point on the image; and, when the latitude of the spatial point belongs to the sky hemisphere, determine the pixel point as the target pixel point.

[0151] In a possible embodiment, the sky determination module 740 is configured to determine the first image segmentation probability of the target pixel as the sky segmentation probability of the spatial point corresponding to the target pixel.

[0152] In a possible embodiment, the sky determination module 740 is used to determine the first image segmentation probability of the target pixel point as the first sky segmentation probability of the spatial point corresponding to the target pixel point; and, based on the first sky segmentation probability and the second sky segmentation probability, determine the third sky segmentation probability of the spatial point corresponding to the target pixel point through a filtering method, the second sky probability is used to indicate the probability that the spatial point belongs to the sky at a historical moment, and the third sky probability is used to indicate the probability that the spatial point belongs to the sky at a current moment; wherein the filtering method includes at least one of the following filtering methods: weighted filtering; Kalman filtering; mean filtering; median filtering.

[0153] In one possible embodiment, the sky determination module 740 is used to replace the second sky segmentation probability with the third sky segmentation probability of each spatial point on the sky hemisphere to obtain the sky hemisphere at the current moment for caching; wherein, the sky hemisphere at the current moment is used to perform filtering calculation on the third sky segmentation probability at the next moment.

[0154] In a possible embodiment, the sky determination module 740 is used to determine the rotational posture of the camera based on the inertial measurement unit IMU; determine the position coordinates of the spatial point in the camera coordinate system based on the coordinates of the pixel point; determine the position coordinates of the spatial point in the world coordinate system based on the rotational posture of the camera and the position coordinates of the spatial point in the camera coordinate system; determine the longitude and latitude of the spatial point based on the position coordinates of the spatial point in the world coordinate system.

[0155] In a possible embodiment, the image determination module 760 is configured to determine the second image segmentation probability of remaining pixels on the image except the target pixel as 0.

[0156] In a possible embodiment, the sky determination module 740 is further used to determine the sky area in the image based on the second image segmentation probability of the pixel point; and replace the sky area with a pre-set sky element, where the sky element is at least one of a video, an animation effect, text, and a picture.

[0157] It should be noted that the sky segmentation device provided in the above embodiment is merely an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the sky segmentation device provided in the above embodiment and the sky segmentation method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0158] Figure 9 The following is a block diagram of an electronic device 2000 according to an exemplary embodiment of the present application. The electronic device 2000 may be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. The electronic device 2000 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other similar names.

[0159] Typically, the electronic device 2000 includes a processor 2001 and a memory 2002 .

[0160] The processor 2001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 2001 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). The processor 2001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 2001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 2001 may also include an AI (Artificial Intelligence) processor, which is used to handle computing operations related to machine learning.

[0161] Memory 2002 may include one or more computer-readable storage media, which may be non-transitory. Memory 2002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 2002 is used to store at least one instruction, which is executed by processor 2001 to implement the sky segmentation method provided in the method embodiments of this application.

[0162] In some embodiments, electronic device 2000 may optionally include a peripheral device interface 2003 and at least one peripheral device. Processor 2001, memory 2002, and peripheral device interface 2003 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 2003 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 2004, a display screen 2005, a camera assembly 2006, an audio circuit 2007, and a power supply 2008.

[0163] The peripheral device interface 2003 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 2001 and the memory 2002. In some embodiments, the processor 2001, the memory 2002, and the peripheral device interface 2003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 2001, the memory 2002, and the peripheral device interface 2003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0164] The RF circuit 2004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 2004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 2004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 2004 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 2004 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 2004 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0165] The display screen 2005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 2005 is a touch screen display, the display screen 2005 also has the ability to collect touch signals on the surface of the display screen 2005 or above the surface. The touch signal can be input as a control signal to the processor 2001 for processing. At this time, the display screen 2005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there can be one display screen 2005, which is set on the front panel of the electronic device 2000; in other embodiments, there can be at least two display screens 2005, which are respectively set on different surfaces of the electronic device 2000 or in a folding design; in other embodiments, the display screen 2005 can be a flexible display screen, which is set on the curved surface or folding surface of the electronic device 2000. Even the display screen 2005 can be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 2005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0166] The camera assembly 2006 is used to capture images or videos. Optionally, the camera assembly 2006 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 2006 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0167] The audio circuit 2007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 2001 for processing, or input into the radio frequency circuit 2004 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each disposed in different parts of the electronic device 2000. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 2001 or the radio frequency circuit 2004 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as distance measurement. In some embodiments, the audio circuit 2007 may also include a headphone jack.

[0168] Power supply 2008 is used to power various components in electronic device 2000. Power supply 2008 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 2008 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0169] In some embodiments, the electronic device 2000 further includes one or more sensors 2009 , including but not limited to: an acceleration sensor 2010 , a gyroscope sensor 2011 , a pressure sensor 2012 , an optical sensor 2013 , and a proximity sensor 2014 .

[0170] The accelerometer 2010 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the electronic device 2000. For example, the accelerometer 2010 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 2001 can control the display screen 2005 to display the user interface in a horizontal or vertical view based on the gravity acceleration signal collected by the accelerometer 2010. The accelerometer 2010 can also be used to collect game or user motion data.

[0171] The gyro sensor 2011 can detect the orientation and rotation angle of the electronic device 2000. It can also work with the accelerometer 2010 to collect 3D motions of the user on the electronic device 2000. Based on the data collected by the gyro sensor 2011, the processor 2001 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0172] The pressure sensor 2012 can be set on the side frame of the electronic device 2000 and / or the lower layer of the display screen 2005. When the pressure sensor 2012 is set on the side frame of the electronic device 2000, it can detect the user's grip signal of the electronic device 2000, and the processor 2001 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 2012. When the pressure sensor 2012 is set on the lower layer of the display screen 2005, the processor 2001 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 2005. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0173] Optical sensor 2013 is used to detect ambient light intensity. In one embodiment, processor 2001 can control the display brightness of display screen 2005 based on the ambient light intensity detected by optical sensor 2013. Specifically, when the ambient light intensity is high, the display brightness of display screen 2005 is increased; when the ambient light intensity is low, the display brightness of display screen 2005 is decreased. In another embodiment, processor 2001 can also dynamically adjust the shooting parameters of camera assembly 2006 based on the ambient light intensity detected by optical sensor 2013.

[0174] Proximity sensor 2014, also known as a distance sensor, is typically located on the front panel of electronic device 2000. Proximity sensor 2014 is used to detect the distance between the user and the front of electronic device 2000. In one embodiment, when proximity sensor 2014 detects that the distance between the user and the front of electronic device 2000 is gradually decreasing, processor 2001 controls display screen 2005 to switch from the screen-on state to the screen-off state. When proximity sensor 2014 detects that the distance between the user and the front of electronic device 2000 is gradually increasing, processor 2001 controls display screen 2005 to switch from the screen-off state to the screen-on state.

[0175] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the electronic device 2000, and the electronic device 2000 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0176] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the sky segmentation method provided by the above-mentioned method embodiments.

[0177] Embodiments of the present application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the sky segmentation method described in any of the above embodiments.

[0178] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0179] Those skilled in the art will appreciate that all or part of the steps in the above embodiments may be implemented by hardware or by programs instructing the relevant hardware to perform the steps. The programs may be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. The above are merely optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A sky segmentation method, characterized in that: The method comprises: Obtaining a first image segmentation probability of a pixel point on an image, where the first image segmentation probability is used to indicate a probability that the pixel point belongs to the sky, where the image is any frame of a video captured by the terminal through a camera component; Determining, based on a positional mapping relationship between the image and a sky hemisphere, a sky segmentation probability of a spatial point on the sky hemisphere according to the first image segmentation probability of a target pixel, the sky hemisphere being a hemispherical model established with a camera that captured the image as the center, the target pixel being a pixel having a corresponding spatial point on the sky hemisphere, the sky hemisphere being used to determine corresponding sky segmentation probabilities for images at least two moments in the video; Based on the position mapping relationship, a second image segmentation probability of the target pixel point is determined according to the sky segmentation probability of the spatial point on the sky hemisphere, where the second image segmentation probability is used to indicate the probability that the target pixel point belongs to the sky.

2. The method according to claim 1, characterized in that The determining, based on the position mapping relationship between the image and the sky hemisphere and according to the first image segmentation probability of the target pixel point, the sky segmentation probability of the spatial point on the sky hemisphere includes: If the spatial point after the three-dimensional spherical mapping of the pixel point on the image belongs to the sky hemisphere, determining the pixel point as the target pixel point; The sky segmentation probability of the spatial point corresponding to the target pixel point is determined according to the first image segmentation probability of the target pixel point.

3. The method according to claim 2, characterized in that When the spatial point after the three-dimensional spherical mapping of the pixel point on the image belongs to the sky hemisphere, determining the pixel point as the target pixel point includes: Determine the longitude and latitude of the spatial point corresponding to the pixel point by performing three-dimensional spherical mapping on the pixel point on the image; In a case where the latitude of the spatial point belongs to the sky hemisphere, the pixel point is determined as the target pixel point.

4. The method according to claim 2, characterized in that The determining, based on the first image segmentation probability of the target pixel point, a sky segmentation probability of a spatial point corresponding to the target pixel point comprises: The first image segmentation probability of the target pixel point is determined as the sky segmentation probability of the spatial point corresponding to the target pixel point.

5. The method according to claim 2, characterized in that The determining, based on the first image segmentation probability of the target pixel point, a sky segmentation probability of a spatial point corresponding to the target pixel point comprises: Determining the first image segmentation probability of the target pixel point as the first sky segmentation probability of the spatial point corresponding to the target pixel point; Determining, by filtering, a third sky segmentation probability for the spatial point corresponding to the target pixel based on the first sky segmentation probability and the second sky segmentation probability, wherein the second sky segmentation probability is used to indicate a probability that the spatial point belongs to the sky at a historical moment, and the third sky segmentation probability is used to indicate a probability that the spatial point belongs to the sky at a current moment; The filtering method includes at least one of the following filtering methods: weighted filtering; Kalman filtering; mean filtering; median filtering.

6. The method according to claim 5, characterized in that The method further comprises: Replacing the second sky segmentation probability with the third sky segmentation probability of each spatial point on the sky hemisphere to obtain the sky hemisphere at the current moment for caching; The sky hemisphere at the current moment is used to perform filtering calculation on the third sky segmentation probability at the next moment.

7. The method according to claim 3, characterized in that Determining the longitude and latitude of a spatial point corresponding to the pixel point by performing three-dimensional spherical mapping on the pixel point on the image includes: Determine the rotational attitude of the camera according to an inertial measurement unit (IMU); Determine the position coordinates of the spatial point in the camera coordinate system according to the coordinates of the pixel point; Determining the position coordinates of the spatial point in the world coordinate system according to the rotation posture of the camera and the position coordinates of the spatial point in the camera coordinate system; The longitude and latitude of the spatial point are determined according to the position coordinates of the spatial point in the world coordinate system.

8. The method according to claim 1, characterized in that The method further comprises: The second image segmentation probability of the remaining pixel points on the image except the target pixel point is determined to be zero.

9. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: determining a sky area in the image based on the second image segmentation probability of the pixel point; The sky area is replaced with a preset sky element, where the sky element is at least one of a video, an animation effect, text, and a picture.

10. A sky segmentation device, characterized in that: The device comprises: an acquisition module, configured to acquire a first image segmentation probability of a pixel point on an image, where the first image segmentation probability indicates a probability that the pixel point on the image belongs to the sky, and the image is any frame of a video captured by the terminal through a camera component; A sky determination module is configured to determine, based on a positional mapping relationship between the image and a sky hemisphere and the first image segmentation probability of a target pixel, a sky segmentation probability of a spatial point on the sky hemisphere, wherein the sky hemisphere is a hemispherical model established with a camera that captured the image as the center, the target pixel being a pixel having a corresponding spatial point on the sky hemisphere, and the sky hemisphere is used to determine corresponding sky segmentation probabilities for images at least two moments in the video; An image determination module is used to determine a second image segmentation probability of the target pixel point based on the position mapping relationship and the sky segmentation probability of the spatial point on the sky hemisphere, wherein the second image segmentation probability is used to indicate the probability that the target pixel point belongs to the sky.

11. The device according to claim 10, characterized in that The sky determination module is configured to determine the pixel point as the target pixel point when the spatial point after the three-dimensional spherical mapping of the pixel point on the image belongs to the sky hemisphere; The sky determination module is further configured to determine a sky segmentation probability of a spatial point corresponding to the target pixel point based on the first image segmentation probability of the target pixel point.

12. The device according to claim 11, characterized in that The sky determination module is configured to determine the longitude and latitude of the spatial point corresponding to the pixel point by performing three-dimensional spherical mapping on the pixel point on the image; The sky determination module is further configured to determine the pixel point as the target pixel point when the latitude of the spatial point belongs to the sky hemisphere.

13. The device according to claim 11, characterized in that The sky determination module is configured to determine the first image segmentation probability of the target pixel point as the first sky segmentation probability of the spatial point corresponding to the target pixel point; The sky determination module is further configured to determine, by a filtering method, a third sky segmentation probability of the spatial point corresponding to the target pixel point based on the first sky segmentation probability and the second sky segmentation probability, wherein the second sky segmentation probability is used to indicate a probability that the spatial point belongs to the sky at a historical moment, and the third sky segmentation probability is used to indicate a probability that the spatial point belongs to the sky at a current moment; The filtering method includes at least one of the following filtering methods: weighted filtering; Kalman filtering; mean filtering; median filtering.

14. A computer device, characterized in that: The computer device includes a processor, a memory connected to the processor, and program instructions stored in the memory. The program instructions executed by the processor implement the sky segmentation method according to any one of claims 1 to 9.

15. A computer-readable storage medium storing program instructions, wherein: When the program instructions are executed by a processor, the sky segmentation method according to any one of claims 1 to 9 is implemented.

Images