Image processing method and apparatus, and electronic device

Abstract

Provided in the present disclosure are an image processing method and apparatus, and an electronic device. The method comprises: determining a first image, wherein the first image is an image that is obtained by means of projecting a panoramic image onto a spherical surface, the first image comprises a second image, and the second image comprises an image in the first image that is covered by a field-of-view angle of a user; determining a first angle for horizontal opening of the second image relative to the center of sphere corresponding to the first image and a second angle for vertical opening of the second image relative to the center of sphere; on the basis of the first angle and the second angle, performing equal-angle division on the second image, so as to obtain M*N sub-images, wherein M and N are positive integers that are determined on the basis of a preset resolution; and performing pixel sampling in the M*N sub-images, so as to obtain an image after sampling is performed.

Description

Image processing methods, apparatus and electronic equipment

[0001] This application claims priority to Chinese Patent Application No. 202310736743.4, filed on June 20, 2023, entitled "Image Processing Method, Apparatus and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, and electronic device. Background Technology

[0003] In the field of virtual reality, servers can determine the image viewed by the user in a panoramic image based on the user's field of view and send the image viewed by the user to the virtual reality device.

[0004] Currently, servers can convert spherical panoramic images into planar panoramic images based on equidistant rectangular projection. Within this planar panoramic image, the server identifies the image slice the user is viewing (the planar panoramic image is divided into multiple image slices) and sends this slice to the virtual reality device. However, the uniformity of pixel density in the planar panoramic image obtained from equidistant rectangular projection is poor, resulting in poor uniformity in the server's sampling of the panoramic image based on the viewing angle, thus wasting network resources.

[0005] Summary of the Invention

[0006] This disclosure provides an image processing method, apparatus, and electronic device to solve one or more technical problems in the prior art.

[0007] In a first aspect, this disclosure provides an image processing method, the method comprising: determining a first image, the first image being an image projected onto a sphere from a panoramic image, the first image including a second image, the second image including an image covered by a user's field of view in the first image; determining a first angle of horizontal opening of the second image relative to the center of the sphere corresponding to the first image and a second angle of vertical opening of the second image relative to the center of the sphere; dividing the second image domain into equal angles based on the first angle and the second angle to obtain M*N sub-images, wherein M and N are positive integers determined based on a preset resolution; and performing pixel sampling in the M*N sub-images to obtain a sampled image.

[0008] Secondly, this disclosure provides an image processing apparatus, which includes a first determining module, a second determining module, a segmentation module, and a sampling module, wherein: the first determining module is used to determine a first image, the first image being an image projected onto a sphere from a panoramic image, the first image including a second image, the second image including an image covered by the user's field of view in the first image; the second determining module is used to determine a first angle of horizontal opening of the second image relative to the center of the sphere corresponding to the first image and a second angle of vertical opening of the second image relative to the center of the sphere; the segmentation module is used to divide the second image domain into M*N sub-images at equal angles based on the first angle and the second angle, where M and N are positive integers determined based on a preset resolution; the sampling module is used to perform pixel sampling in the M*N sub-images to obtain a sampled image.

[0009] Thirdly, embodiments of this disclosure provide an electronic device including: a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the image processing methods described in the first aspect above and various possible aspects of the first aspect.

[0010] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image processing methods described in the first aspect and various possible aspects thereof. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this disclosure;

[0013] Figure 2 is a schematic flowchart of an image processing method provided in an embodiment of this disclosure;

[0014] Figure 3 is a schematic diagram of a first image provided in an embodiment of this disclosure;

[0015] Figure 4 is a schematic diagram from a first angle provided in an embodiment of this disclosure;

[0016] Figure 5 is a schematic diagram of a second angle provided in an embodiment of this disclosure;

[0017] Figure 6 is a schematic diagram of a first angle bisector provided in an embodiment of this disclosure;

[0018] Figure 7 is a schematic diagram of determining M*N sub-images according to an embodiment of this disclosure;

[0019] Figure 8 is a schematic diagram of determining the image after sampling according to an embodiment of this disclosure;

[0020] Figure 9 is a schematic diagram of a method for determining a first image provided in an embodiment of this disclosure;

[0021] Figure 10 is a schematic diagram of a process for determining a second image provided in an embodiment of this disclosure;

[0022] Figure 11 is a schematic diagram of a rotated image provided in an embodiment of this disclosure;

[0023] Figure 12 is a schematic diagram of determining an image to be transmitted according to an embodiment of this disclosure;

[0024] Figure 13 is a schematic diagram of an image processing method provided in an embodiment of this disclosure;

[0025] Figure 14 is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure; and,

[0026] Figure 15 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] For ease of understanding, the concepts involved in the embodiments of this disclosure will be explained below.

[0029] Electronic device: A device with wireless transceiver capabilities. Electronic devices can be deployed on land, including indoors or outdoors, handheld, wearable, or vehicle-mounted; they can also be deployed on water (such as on ships). These electronic devices can be mobile phones, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, vehicle-mounted electronic devices, wireless terminals in self-driving vehicles, wireless electronic devices in remote medical care, wireless electronic devices in smart grids, wireless electronic devices in transportation safety, wireless electronic devices in smart cities, wireless electronic devices in smart homes, wearable electronic devices, etc. The electronic devices involved in the embodiments of this disclosure can also be referred to as terminals, user equipment (UE), access electronic devices, vehicle-mounted terminals, industrial control terminals, UE units, UE stations, mobile stations, mobile stations, remote stations, remote electronic devices, mobile devices, UE electronic devices, wireless communication devices, UE agents, or UE devices, etc. Electronic devices can be fixed or mobile.

[0030] The application scenarios of the embodiments of this disclosure will now be described with reference to Figure 1.

[0031] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this disclosure. Referring to Figure 1, it includes a spherical image, a server, and a virtual reality device. The spherical image includes multiple sampling points (circles in the spherical image are sampling points). The server can project the spherical image onto a plane to obtain a planar image. The planar image may include multiple sampling points; the first row of sampling points is located in the Arctic region, the third row in the equatorial region, and the fifth row in the Antarctic region.

[0032] Referring to Figure 1, the server can divide a planar image into 25 image slices. If the user's viewpoint is in front, the server can send image slice 13 to the virtual reality device; if the user's viewpoint is at the top, the server can send image slices 1, 2, 3, 4, and 5 to the virtual reality device. In this way, the server can send image slices within the user's field of view to the virtual reality device based on the user's field of view, reducing the amount of data transmitted.

[0033] It should be noted that Figure 1 is merely an exemplary illustration of the application scenarios of the embodiments of this disclosure, and is not intended to limit the application scenarios of the embodiments of this disclosure.

[0034] In related technologies, a server can determine the image viewed by the user within a panoramic video image based on the user's field of view and send the image viewed by the user to a virtual reality device. For example, as shown in Figure 1, since the server cannot encode images of curved surfaces, it can project the curved surface image onto a plane. If the user's field of view includes image patch 13 in the plane image, the server can send image patch 13 to the virtual reality device. Currently, the server can convert a panoramic image of a curved surface into a panoramic image of a plane based on equidistant rectangular projection. The server can determine the image patch in the panoramic image of the plane viewed by the user and send that image patch to the virtual reality device. However, the uniformity of pixel density in the panoramic image of the plane obtained by equidistant rectangular projection is poor. For example, as shown in Figure 1, the pixel density is higher in the equatorial region and lower in the polar regions of the panoramic image of the plane. If the user's field of view is in the equatorial region, the field of view can cover fewer image patches. However, if the user's field of view is in the polar regions, the field of view will cover all image patches in those regions (i.e., for the same field of view size, the number of image patches covered by the field of view at the poles is greater than the number of image patches covered by the field of view at the equator). Thus, when the server samples images within the field of view, the sampling uniformity is poor, requiring the server to transmit more image patches, leading to a waste of network resources.

[0035] To address the technical problems in related technologies, this disclosure provides an image processing method. An electronic device can acquire a panoramic image corresponding to a first image and a user's viewpoint. Based on the user's viewpoint, the panoramic image is rotated to obtain a rotated image, where the user's viewpoint is at the center of the rotated image. The electronic device can project the rotated image onto a sphere using perspective projection to obtain the first image. The electronic device can determine a first angle of horizontal opening of a second image in the first image relative to the center of the sphere corresponding to the first image, and a second angle of vertical opening of the second image relative to the center of the sphere. Based on the first and second angles, the second image is divided into M*N sub-images at equal angles, resulting in M*N sub-images. M and N are positive integers determined based on a preset resolution. Pixel sampling is performed on the M*N sub-images to obtain a sampled image. In this method, since the electronic device can rotate the panoramic video frame based on the viewpoint, the center of the second image in the first image can be located in the equatorial region of the first image, resulting in a higher pixel density in the second image. Furthermore, since multiple sub-images are obtained by dividing the second image at equal angles, the electronic device can uniformly sample pixels in the second image to obtain the sampled image, thereby improving the accuracy of the sampled image and saving network resources.

[0036] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0037] Figure 2 is a schematic flowchart of an image processing method provided in an embodiment of this disclosure. Referring to Figure 2, the method may include:

[0038] S201. Determine the first image, which includes the second image.

[0039] The execution entity in this embodiment can be an electronic device or an image processing device installed in an electronic device. The image processing device can be implemented in software or a combination of software and hardware; this embodiment does not limit its implementation. Optionally, the electronic device can be any device with on-device computing capabilities, such as a server or computer; this embodiment does not limit its implementation either.

[0040] In some embodiments, the first image can be an image projected onto a sphere. For example, the panoramic image can be a planar image, and the electronic device can project the panoramic image onto the surface of the sphere to obtain a curved first image. If the user's viewing angle is located at the center of the sphere, the curved first image can present a three-dimensional display effect to the user.

[0041] Optionally, the panoramic image can be a planar image obtained based on equirectangular projection (ERP). For example, ERP projection can project the meridians of a sphere into equally spaced vertical lines and the parallels of a sphere into equally spaced horizontal lines. Electronic devices can render a three-dimensional spherical image and convert the spherical image into a planar panoramic image based on ERP projection.

[0042] Optionally, the panoramic image can be a planar image obtained based on cube map projection (CMP). For example, CMP projection can embed a sphere into a cube, projecting the sphere onto the six faces of the cube using perspective projection. Electronic devices can render a three-dimensional spherical image, and based on CMP projection, project this spherical image onto the six faces of the cube, and unfold the six faces of the cube to obtain a planar panoramic image.

[0043] Optionally, the panoramic image can also be an image captured by a panoramic camera device, such as a fisheye camera, which can capture panoramic images. This disclosure does not limit the scope of the panoramic image.

[0044] Optionally, the electronic device can receive panoramic images sent by other devices, or it can obtain panoramic images from a database. This disclosure does not limit this aspect.

[0045] It should be noted that in practical applications, electronic devices can generate panoramic videos. The video frames in a panoramic video are panoramic images, and the electronic device can send each panoramic video frame to a virtual reality device. In this way, the virtual reality device can present the user with a 3D video effect.

[0046] In some embodiments, the first image can be a curved image. For example, the first image is obtained by projecting a planar image onto the surface of a sphere; therefore, the first image can be a curved image. The first image may include a second image, which may include the image covered by the user's field of view (FOV) within the first image. For example, the user's FOV may cover a portion of the first image, and the second image may include that portion. This allows the user to see an effective view even with minor shifts in their viewing angle, improving the user experience.

[0047] It should be noted that the second image can be an image to be transmitted by the electronic device. The electronic device can determine the second image in the first image and project the second image onto a plane to obtain a planar image. The electronic device can encode the planar image and send the encoded planar image to the virtual reality device.

[0048] Optionally, the center of the second image can be the viewpoint of the user's field of view, and the center of the second image can be located in a region within a preset range of the equator of the first image. For example, the region within the preset range of the equator of the first image can be a region within 30 degrees above and below the equator, or it can be a region within 50 degrees above and below the equator. This embodiment of the present disclosure does not limit this.

[0049] For example, the center of the second image can be located within a predetermined range of the equator of the first image, or it can be located directly on the equator of the first image. In this way, since the center of the second image can be the user's viewpoint, and the pixel density is higher within the predetermined range of the equator of the first image, if the center of the second image is within the predetermined range of the equator of the first image, the pixel density within the second image will be higher, thereby improving the uniformity of sampling.

[0050] Optionally, after acquiring the panoramic image, the electronic device can rotate it so that the user's field of view (FOV) is located at the center of the rotated image, and then project the rotated image onto a sphere to obtain the first image. In this way, the user's viewpoint can be located within a preset range around the equator; that is, the center point of the second image is located within a preset range around the equator, and the second image has higher pixel density and higher pixel uniformity. Therefore, the electronic device can accurately obtain the image to be transmitted, saving network resources.

[0051] The first image will now be explained with reference to Figure 3.

[0052] Figure 3 is a schematic diagram of a first image provided in an embodiment of this disclosure. Referring to Figure 3, a first image is included. The first image is a curved surface image of a sphere, and it may include a second image (the images within the black and gray areas of the first image's surface can be the second image within the first image; the images within the black areas can be the images covered by the user's field of view in the first image). The center of the second image can be located at the equator of the sphere corresponding to the first image. In this way, the second image can be directly aligned with the equatorial plane of the sphere, thereby increasing the pixel density of the second image and improving sampling accuracy.

[0053] S202. Determine the first angle of horizontal opening of the second image relative to the center of the sphere corresponding to the first image and the second angle of vertical opening of the second image relative to the center of the sphere.

[0054] In some embodiments, the first angle can be the angle at which the second image opens horizontally relative to the center of the sphere corresponding to the first image. For example, if the second image opens horizontally by 100 degrees relative to the center of the sphere (the center of the sphere corresponding to the first image), then the first angle can be 100 degrees; if the second image opens horizontally by 200 degrees relative to the center of the sphere, then the first angle can be 200 degrees.

[0055] In some embodiments, the second angle can be the vertical opening angle of the second image relative to the center of the sphere corresponding to the first image. For example, if the second image is vertically opened by 50 degrees relative to the center of the sphere corresponding to the first image, the second angle can be 50 degrees; if the second image is vertically opened by 80 degrees relative to the center of the sphere corresponding to the first image, the second angle can be 80 degrees.

[0056] Optionally, the electronic device may determine the first angle of horizontal opening of the second image relative to the center of the sphere corresponding to the first image and the second angle of vertical opening of the second image relative to the center of the sphere based on the following feasible implementation: the angle between the midpoint of the two arcs in the vertical direction of the second image and the center of the sphere is determined as the first angle of horizontal opening, and the angle between the midpoint of the two arcs in the horizontal direction of the second image and the center of the sphere is determined as the second angle of vertical opening.

[0057] Optionally, the second image can be determined based on four arcs: the two vertical arcs can be the left and right arcs of the second image, and the two horizontal arcs can be the upper and lower arcs of the second image.

[0058] Optionally, the electronic device can determine the angle between the lines connecting the midpoints of the left and right arcs to the center of the sphere as the first angle of horizontal opening. For example, since the first image is a curved surface image, the angle between the lines connecting the midpoints of the left and right arcs to the center of the sphere can be the maximum angle of horizontal opening of the second image, and the electronic device can determine this angle as the first angle.

[0059] Optionally, the electronic device can determine the angle between the lines connecting the midpoints of the upper and lower arcs to the center of the sphere as the second angle of vertical opening. For example, since the first image is a curved image, the angle between the lines connecting the midpoints of the upper and right arcs to the center of the sphere can be the maximum angle of vertical opening of the second image, and the electronic device can determine this angle as the second angle. In this way, the electronic device can accurately determine the angle of opening of the second image relative to the center of the sphere, and thus accurately divide the second image into equal angles, improving the accuracy and uniformity of sampling.

[0060] Optionally, since the first image is a curved image, the electronic device can also arbitrarily select a pair of two opposite points on the two vertically opposite arcs of the second image (e.g., a point 10 pixels away from the top left vertex on the left arc and a point 10 pixels away from the top right vertex on the right arc), and determine the angle between the line connecting the pair of opposite points and the center of the sphere as the first angle.

[0061] Optionally, since the first image is a curved image, the electronic device can also arbitrarily select a pair of two opposite points on two horizontally opposite arcs of the second image (e.g., a point 10 pixels away from the upper left vertex on the upper arc and a point 10 pixels away from the lower left vertex on the lower arc), and determine the angle between the line connecting the pair of opposite points and the center of the sphere as the second angle.

[0062] The first angle will now be explained with reference to Figure 4.

[0063] Figure 4 is a schematic diagram of a first angle provided by an embodiment of this disclosure. Referring to Figure 4, it includes: a first image. The first image may include a second image. The center of the second image is located on the equator of the first image. The second image may include a left arc and a right arc. An electronic device (not shown in Figure 4) can determine the angle between the lines connecting the midpoint of the left arc, the midpoint of the right arc, and the center of the sphere as the first angle.

[0064] The second angle will now be explained with reference to Figure 5.

[0065] Figure 5 is a schematic diagram of a second angle provided by an embodiment of this disclosure. Referring to Figure 5, it includes: a first image. The first image may include a second image. The center of the second image is located on the equator of the first image. The second image may include an upper arc and a lower arc. An electronic device (not shown in Figure 5) can determine the angle between the lines connecting the midpoint of the upper arc, the midpoint of the lower arc, and the center of the sphere as the second angle.

[0066] S203. Based on the first angle and the second angle, the second image is divided into M*N sub-images at equal angles.

[0067] Where M and N are positive integers determined based on the preset resolution. For example, if the preset resolution is 800*400, then M is 800 and N is 400; if the preset resolution is 800*1600, then M is 800 and N is 1600.

[0068] Optionally, the preset resolution can be any set resolution, and this embodiment of the present disclosure does not limit it.

[0069] In some embodiments, the electronic device may obtain M*N sub-images based on the following feasible implementation: dividing the first angle into M first bisectors, dividing the second angle into N second bisectors, and dividing the second image into equal angles based on the M first bisectors and N second bisectors to obtain M*N sub-images.

[0070] Optionally, the angle of the first bisector can be the ratio of the first angle to M. For example, if the first angle is 100 degrees and M is 50, the electronic device can divide the first angle into 50 equal first bisectors. For example, if the first angle is 100 degrees and M is 50, the electronic device can determine that each first bisector is 2 degrees, and the sum of the 50 first bisectors is the first angle.

[0071] Optionally, the angle of the second bisector can be the ratio of the second angle to N. For example, if the second angle is 100 degrees and N is 200, the electronic device can divide the second angle into 200 equal second bisectors. Alternatively, if the second angle is 100 degrees and N is 200, the electronic device can determine that each second bisector is 0.5 degrees, and the sum of the 200 second bisectors constitutes the second angle.

[0072] The first angle bisector will now be explained with reference to Figure 6.

[0073] Figure 6 is a schematic diagram of a first bisecting angle provided by an embodiment of this disclosure. For ease of understanding, in the embodiment shown in Figure 6, M can be 3. Referring to Figure 6, it includes: a first image. The first image may include a second image. The center of the second image is located on the equator of the first image. The second image may include a left arc and a right arc. The first angle can be the angle between the lines connecting the midpoint of the left arc, the midpoint of the right arc, and the center of the sphere. Dividing the first angle into three equal parts yields three first bisecting angles. In this way, the electronic device can divide the first angle into multiple first bisecting angles, thereby improving the uniformity of pixels in the sub-images and improving the accuracy of the second image.

[0074] It should be noted that the second bisector angle is similar to the first bisector angle in the embodiment shown in Figure 6, and will not be described again in this disclosure.

[0075] In some embodiments, the electronic device divides the second image into M*N sub-images based on M first bisector angles and N second bisector angles. Specifically, it may involve: determining M+1 vertical arcs associated with the M first bisector angles in the second image, determining N+1 horizontal arcs associated with the N second bisector angles in the second image, and dividing the second image into M*N sub-images based on the M+1 vertical arcs and the N+1 horizontal arcs.

[0076] Optionally, the vertical arc can be an arc passing through the edge of the first bisector angle and the focal point of the first image. For example, the vertical arc can be perpendicular to the equator of the first image. For example, for any first bisector angle (which does not intersect the left and right arcs), and with focal point 1 and focal point 2 between the first bisector angle and the first image, the electronic device can determine one vertical arc as the arc passing through focal point 1 and perpendicular to the equator, and the electronic device can determine another vertical arc as the arc passing through focal point 2 and perpendicular to the equator.

[0077] Optionally, the horizontal arc can be an arc passing through the edge of the second bisector angle and the focal point of the first image. For example, the horizontal arc can be parallel to the equator of the first image. For example, for any second bisector angle (which does not intersect the upper and lower arcs), and with focal point 1 and focal point 2 between the second bisector angle and the first image, the electronic device can determine one horizontal arc as an arc passing through focal point 1 and parallel to the equator, and the electronic device can determine another horizontal arc as an arc passing through focal point 2 and parallel to the equator.

[0078] It should be noted that the M+1 vertical arcs include the left and right arcs of the second image, and the N+1 horizontal arcs include the upper and lower arcs of the second image.

[0079] It should be noted that multiple adjacent sets of arcs in the vertical direction have the same horizontal opening angle relative to the center of the sphere. For example, arc 1 in the vertical direction is adjacent to arc 2 in the vertical direction, and arc 2 in the vertical direction is adjacent to arc 3 in the vertical direction. The angle between the lines connecting the midpoint of arc 1, the midpoint of arc 2 and the center of the sphere is angle A, and the angle between the lines connecting the midpoint of arc 2, the midpoint of arc 3 and the center of the sphere is angle B. Here, angle A and angle B are the same, that is, the second image is divided into equal angles in the vertical direction.

[0080] It should be noted that multiple adjacent sets of horizontal arcs open at the same vertical angle relative to the center of the sphere. For example, horizontal arc 1 is adjacent to horizontal arc 2, and horizontal arc 2 is adjacent to horizontal arc 3. The angle between the lines connecting the midpoints of arc 1 and arc 2 to the center of the sphere is angle A, and the angle between the lines connecting the midpoints of arc 2 and arc 3 to the center of the sphere is angle B. Angles A and B are the same, meaning that the second image is divided into equal angles in the horizontal direction.

[0081] Optionally, the electronic device can divide the second image into M*N grids based on M+1 vertical arcs and N+1 horizontal arcs, where each grid contains a sub-image. For example, the electronic device can divide the second image into two regions vertically based on three vertical arcs (including the left and right arcs), and can divide it into two regions horizontally based on three horizontal arcs (including the upper and lower arcs). That is, the electronic device can obtain four grids based on three vertical and three horizontal arcs, with each grid containing a sub-image.

[0082] The process of determining M*N sub-images will be explained below with reference to Figure 7.

[0083] Figure 7 is a schematic diagram illustrating the determination of M*N sub-images according to an embodiment of this disclosure. For ease of understanding, in the embodiment shown in Figure 7, M is 3 and N is 3. Referring to Figure 7, the system includes: a first image. The first image may include the center of a sphere and a second image, wherein the center of the second image is located on the equator. If M is 3, the electronic device (not shown in Figure 7) can determine four vertical arcs (including the left and right arcs of the second image) in the second image, thereby dividing the second image into three regions at equal angles in the vertical direction.

[0084] Referring to Figure 7, if N is 3, the electronic device can determine four horizontal arcs (including the upper and lower arcs of the second image) in the second image, and then divide the second image into three regions at equal angles in the horizontal direction (the equator is not shown). Based on the three horizontal regions and three vertical regions, the second image can be divided into sub-image 1, sub-image 2, ..., sub-image 9 at equal angles. Since the center of the second image is located on the equator, and the nine sub-images are obtained by dividing the second image at equal angles, the pixel density of the nine sub-images is high, improving the sampling effect.

[0085] S204. Perform pixel sampling on M*N sub-images to obtain the sampled image.

[0086] In some embodiments, the sampled image can be an image projected onto a plane from a second image in a first image. For example, the second image can be an image to be transmitted, and the electronic device can project the second image onto a plane using an isometric projection method to obtain the sampled image. The isometric projection method can be a projection method that projects pixels from multiple sub-images (sub-images are obtained by equally dividing the second image) onto a plane.

[0087] Optionally, the resolution of the sampled image is the same as the preset resolution. For example, if the preset resolution is 800*1600, then based on the preset resolution, 800*1600 sub-images can be obtained from the second image. The electronic device can sample pixels in each sub-image to obtain the sampled image. Therefore, the resolution of the sampled image can be 800*1600.

[0088] Optionally, the electronic device may obtain the sampled image based on the following feasible implementation: for any target location in the sampled image, determine the sub-image corresponding to the target location in the second image, obtain one or more pixels in the sub-image, and perform fusion processing on the one or more pixels to obtain the pixel of the target location in the sampled image.

[0089] Optionally, during the isometric projection of the second image, the electronic device can determine multiple sub-images corresponding to multiple positions in the sampled image in the second image, and then project the pixels in the multiple sub-images to the multiple positions to obtain the sampled image. For example, for position (1,1) in the sampled image, the electronic device can determine the sub-image corresponding to position (1,1) in the second image, and then determine the pixels in the sub-image as the pixels at position (1,1). Repeating the above steps, the sampled image can be obtained.

[0090] Optionally, the electronic device can determine the sub-image corresponding to the target location based on the following formula:

[0091] Among them, F v For the first angle; F h Let M be the width of the sampled image in pixels; N be the height of the sampled image in pixels; m be the coordinates of the target position in the width of the sampled image; n be the coordinates of the target position in the height of the sampled image; and φ be the horizontal opening angle of the sub-image (its position within the first angle). The angle at which the sub-image is vertically opened (its position within the second angle).

[0092] In this way, based on the above formula, the correspondence between multiple positions in the sampled image and multiple sub-images in the second image can be determined, and thus the sampled image can be accurately generated.

[0093] It should be noted that, due to Let φ be the angle at which the second image opens vertically, but this angle cannot be determined in spherical coordinates. Therefore, this angle can be converted to θ in spherical coordinates, so that the position of the sub-image can be accurately located in the second image based on φ and θ.

[0094] In some embodiments, the electronic device may use the following formula to Convert to θ in spherical coordinates:

[0095] The physical meaning of each parameter in the above formula can be referred to the formula for determining the sub-image corresponding to the target position, and will not be elaborated further in this embodiment.

[0096] It should be noted that after the electronic device determines the second image and preset resolution, it can scale the second image, making the projection method of the second image highly adaptable. Furthermore, the electronic device can flexibly determine the resolution of the second image based on network conditions, improving the smoothness of the panoramic video viewing experience for users.

[0097] Optionally, the electronic device can perform a fusion process on one or more pixels to obtain the pixel at the target location in the sampled image. For example, the electronic device can average the pixel values ​​of one or more pixels to obtain the pixel at the target location. For instance, if the sub-image corresponding to the target location includes pixel 1, pixel 2, and pixel 3, the electronic device can average the RGB values ​​of pixel 1, pixel 2, and pixel 3 to obtain a new pixel, and determine the new pixel as the pixel at the target location.

[0098] Optionally, the electronic device may also perform fusion processing on one or more pixels in the sub-image based on any other feasible implementation method, and the embodiments disclosed herein are not limited thereto.

[0099] Optionally, the electronic device can also determine any pixel in the sub-image as the pixel at the target location. For example, if the sub-image corresponding to the target location includes pixels 1, 2, and 3, the electronic device can determine pixel 1 as the pixel at the target location, pixel 2 as the pixel at the target location, and pixel 3 as the pixel at the target location. This embodiment of the present disclosure does not limit this. In this way, the electronic device can project the second image onto the plane to obtain the sampled image. Since the pixel density in the second image is high, the accuracy of the sampled image is high.

[0100] The process of determining the image after sampling will be explained below with reference to Figure 8.

[0101] Figure 8 is a schematic diagram of determining a sampled image according to an embodiment of this disclosure. For ease of understanding, in the embodiment shown in Figure 8, the sampled image may include 9 pixels. Please refer to Figure 8, which includes the corresponding image. The first image may include a second image, and the second image may include 9 sub-images. The sampled image to be generated may include 9 positions for filling pixels. An electronic device (not shown in Figure 8) can project pixels from sub-image 1 to position 1, pixels from sub-image 2 to position 2, ..., pixels from sub-image 9 to position 9. In this way, the electronic device can project the second image onto a planar image to obtain the sampled image. During the projection process, the sampling uniformity is good, and the pixel density within the second image is high; therefore, the accuracy of the sampled image is high.

[0102] This disclosure provides an image processing method. A first image is determined, and a second image is determined by a first horizontal angle relative to the center of a sphere corresponding to the first image and a second vertical angle relative to the center of the sphere. The first angle is divided into M equal first bisectors, and the second angle is divided into N equal second bisectors. Based on the M first bisectors and N second bisectors, the second image is divided into M*N sub-images at equal angles. Pixel sampling is performed on the M*N sub-images to obtain a sampled image. Since the center of the second image is located on the equator of the first image, and the electronic device can divide the second image into multiple sub-images based on equal angle division, the pixel density in the second image is high, and the uniformity of the sub-images is high. This improves the accuracy of sampling in the second image, thereby improving the accuracy of the sampled image, avoiding the transmission of redundant content, and saving network resources.

[0103] Based on the embodiment shown in Figure 2, the method for determining the first image in the above image processing method will be described below with reference to Figure 9.

[0104] Figure 9 is a schematic diagram of a method for determining a first image according to an embodiment of this disclosure. Referring to Figure 9, the method flow includes:

[0105] S901. Obtain the panoramic image corresponding to the first image and the user's viewpoint.

[0106] Optionally, the electronic device can acquire the panoramic image corresponding to the first image based on any feasible implementation method, and the embodiments disclosed herein are not limited in this regard.

[0107] Optionally, the user's viewpoint can be the center point of the area the user is interested in within the panoramic image. For example, the user's viewpoint can be the center point of the user's field of view. If the user is interested in area 1 in the panoramic image, then the user's viewpoint can be the center point of area 1; if the user is interested in area 2 in the panoramic image, then the user's viewpoint can be the center point of area 2. It should be noted that the user's viewpoint can also be any point within the area the user is interested in within the panoramic image, and this embodiment of the present disclosure does not limit this.

[0108] Optionally, the user's viewpoint can be represented based on direction parameters. For example, the direction parameters corresponding to the user's viewpoint can be (α, β, γ), where α can be the horizontal field of view of the user's viewpoint (i.e., the horizontal field of view in the FOV), β can be the vertical field of view of the user's viewpoint (i.e., the vertical field of view in the FOV), and γ can be the diagonal field of view of the user's viewpoint (i.e., the diagonal field of view in the FOV).

[0109] It should be noted that electronic devices can determine the direction parameters corresponding to the viewpoint based on any feasible implementation method, and the embodiments disclosed herein do not limit this.

[0110] Optionally, the electronic device can receive the user's viewpoint transmitted by other devices. For example, the electronic device can receive the user's viewpoint (based on orientation parameters) transmitted by a virtual reality device. Optionally, the electronic device can predict the user's viewpoint. For example, the electronic device can receive the user's viewpoint transmitted by a virtual reality device and, based on that viewpoint, predict the user's viewpoint over a future period of time.

[0111] Optionally, the electronic device may also acquire the user's viewpoint based on any other feasible implementation method, and the embodiments disclosed herein are not limited thereto.

[0112] S902. Based on the user's viewpoint, rotate the panoramic image to obtain a rotated image.

[0113] In some embodiments, the rotated image can be an image obtained by rotating a panoramic image based on the user's viewpoint using an electronic device. For example, the electronic device can determine a rotation matrix based on the user's viewpoint and rotate the panoramic image based on the rotation matrix to obtain the rotated image. For example, the electronic device can process each pixel in the panoramic image based on the rotation matrix to obtain the rotated image.

[0114] Optionally, the user's viewpoint can be at the center of the rotated image. For example, the electronic device can rotate the panoramic image based on the rotation matrix corresponding to the user's viewpoint to obtain a rotated image, where the user's viewpoint can be located at the center of the rotated image. For instance, for a panoramic image obtained by shooting an object from the front, if the user's viewpoint is located at the top of the panoramic image, the electronic device can rotate the panoramic image to obtain a rotated image, where the rotated image can be a panoramic image taken from the top of the object, and the center of the panoramic image can be the user's viewpoint.

[0115] It should be noted that the user's viewpoint can be a point within the central region of the rotated image, or any point in the rotated image; this disclosure does not limit this.

[0116] Optionally, the electronic device can determine a rotation matrix based on the user's viewpoint and rotate the panoramic image based on the rotation matrix to obtain a rotated image. Optionally, the electronic device can determine the rotation matrix based on the following formula:

[0117] Among them, R v α is the rotation matrix; α can be the horizontal field of view of the user's viewpoint (i.e., the horizontal field of view in the FOV), β can be the vertical field of view of the user's viewpoint (i.e., the vertical field of view in the FOV), and γ can be the diagonal field of view of the user's viewpoint (i.e., the diagonal field of view in the FOV).

[0118] Optionally, the electronic device can process each pixel in the panoramic video based on a rotation matrix to obtain a rotated image. Since the rotated image is obtained based on a rotation matrix determined from the user's viewpoint, the user's viewpoint can be located at the center of the rotated image.

[0119] Optionally, the electronic device rotates the panoramic image based on the user's viewpoint. After obtaining the rotated image, a second image can be determined from the rotated image. Specifically, the electronic device can determine the second image from the rotated image by taking the user's viewpoint as the center, determining the image covered by the user's field of view in the rotated image, and adding images of a preset range to each edge of the image covered by the user's field of view to obtain the second image.

[0120] Optionally, since the user's viewpoint is located at the center of the rotating image, the electronic device uses this viewpoint as the center to determine the image covered by the user's field of view in the rotating image. The user's field of view can be any viewing angle range, and this embodiment of the disclosure does not limit this. For example, if the user's field of view is 90 degrees * 100 degrees, the electronic device can determine a 90-degree * 100-degree region in the rotating image using the viewpoint as the center, thereby obtaining the image covered by the user's field of view.

[0121] Optionally, to prevent users from missing new images when they slightly turn their heads, the electronic device can determine a second image based on the image covered by the user's field of view. For example, the electronic device can add images of a preset range to each edge of the image covered by the user's field of view to obtain a second image. For instance, the electronic device can add images of a 20-degree range to the upper edge, lower edge, left edge, and right edge of the target area of ​​the image covered by the field of view to obtain a second image. In this way, even when the user slightly turns their head away from the image covered by the current field of view, they can still see the relevant image, improving the user experience.

[0122] It should be noted that the size of the image added to each edge of the image covered by the electronic device in the field of view can be the same or different, and this disclosure does not limit this.

[0123] The process of determining the second image will now be explained with reference to Figure 10.

[0124] Figure 10 is a schematic diagram of a process for determining a second image according to an embodiment of this disclosure. Referring to Figure 10, a rotated image is included. The viewpoint is located at the center of the rotated image. Based on the user's field of view, the image covered by the user's field of view is determined in the rotated image, centered on the viewpoint, wherein the image covered by the user's field of view is located in the central region of the rotated image. Images within a preset range are added to the top, bottom, left, and right edges of the image covered by the user's field of view, thereby obtaining the second image. In this way, the second image can include the image covered by the user's field of view, allowing the user to see the corresponding image even with slight head rotation, improving the user experience and saving network resources.

[0125] The rotated image will now be explained with reference to Figure 11.

[0126] Figure 11 is a schematic diagram of a rotated image provided by an embodiment of this disclosure. Referring to Figure 11, it includes a panoramic image. An electronic device (not shown in Figure 11) can determine that a second image is located to the left of the panoramic image. The electronic device can determine a rotation matrix based on the orientation parameters of the viewpoint in the second image, and process each pixel in the panoramic image based on the rotation matrix to obtain the rotated image. The second image is located at the center of the rotated image, and its center point is also located at the center of the rotated image. Thus, after rotating the panoramic image, the rotated image is projected onto the surface of a three-dimensional sphere. The center point of the second image can be located at the equator of the three-dimensional sphere, meaning the user's viewpoint is located at the equator of the three-dimensional sphere, thereby increasing the pixel density of the second image.

[0127] Optionally, after the electronic device determines the second image, since the maximum range of the second image can be 180 degrees * 180 degrees, the electronic device can project other images besides the second image onto the plane using an equal-angle projection method, and then stitch them together with the sampled images corresponding to the second image. In this way, when the user's viewpoint shifts away from the second image, the corresponding image can still be seen, improving the user experience.

[0128] Optionally, when projecting other images besides the second image of the first image onto the plane at equal angles, the number of sampling points can be reduced, thereby reducing the image resolution of the other images after they are projected onto the plane and saving network transmission resources.

[0129] Optionally, the electronic device can determine a first arc based on the top-left and bottom-right vertices of the second image, and a second arc based on the bottom-left and top-right vertices of the second image. Then, based on the first and second arcs, the image in the first image, excluding the second image, is divided into four images to be projected. The four images to be projected are then projected onto a plane using an equal-angle projection method to obtain four planar images. These four planar images are then stitched together onto the sampled image corresponding to the second image. The server can then send the sampled image, which consists of the stitched four planar images, to the virtual reality device.

[0130] For example, an electronic device can obtain four images to be projected based on a first arc and a second arc. These four images are located above, below, to the left, and to the right of the second image, respectively. The electronic device can generate images of the upper, lower, left, and right regions of the second image using equal-angle projection. It then stitches the image of the upper region onto the top of the corresponding sampled image of the second image, the image of the lower region onto the bottom of the corresponding sampled image, the image of the left region onto the left side of the corresponding sampled image, and the image of the right region onto the right side of the corresponding sampled image, thus obtaining the image to be transmitted.

[0131] The process of determining the image to be transmitted will be explained below with reference to Figure 12.

[0132] Figure 12 is a schematic diagram of determining an image to be transmitted according to an embodiment of this disclosure. Referring to Figure 12, the image to be transmitted may include a target image, image 1, image 2, image 3, and image 4. Image 1 is the image above the second image in the first image, projected onto a plane using an isometric projection. Image 2 is the image to the right of the second image in the first image, projected onto a plane using an isometric projection. Image 3 is the image below the second image domain in the first image, projected onto a plane using an isometric projection. Image 4 is the image to the left of the second image in the first image, projected onto a plane using an isometric projection.

[0133] S903. Based on perspective projection, the rotated image is projected onto the sphere to obtain the first image.

[0134] Optionally, the electronic device can project the rotating image onto a sphere using perspective projection to obtain a first image. For example, the electronic device can project the rotating image onto the surface of a sphere using perspective projection to obtain the first image. In this way, after the rotating image is projected onto the sphere using perspective projection, the shape of the image covered by the user's field of view is similar to the shape of the second image, thus ensuring that the user can see an effective image even when moving their head, improving the user experience. For example, in the embodiment shown in Figure 10, by projecting the rotating image in Figure 10 onto the sphere using perspective projection, the shape of the image covered by the user's field of view is similar to the shape of the image within a preset range, thereby improving the user experience.

[0135] This disclosure provides a method for determining a first image. The method involves acquiring a panoramic image corresponding to the first image and a user's viewpoint. Based on the user's viewpoint, the panoramic image is rotated to obtain a rotated image. Then, using perspective projection, the rotated image is projected onto a sphere to obtain the first image. In this way, the center of the second image within the first image can be located at the equator of the first image, and the second image has a higher pixel density, thereby improving the accuracy of determining the sampled image and saving network resources.

[0136] Based on any of the above embodiments, the process of the above image processing method will be described below with reference to FIG13.

[0137] Figure 13 is a schematic diagram of an image processing method provided in an embodiment of this disclosure. Referring to Figure 13, it includes: a panoramic image, a server, and a virtual reality device. The panoramic image includes a second image, the center of which is located at the center of the panoramic image. The server can project the panoramic image onto the surface of a sphere based on perspective projection to obtain a first image. The first image may include the second image, the center of which is located at the equator of the first image.

[0138] Referring to Figure 13, the server can divide the second image in the first image into 9 sub-images based on an equal-angle division method (the division process of the second image can be referred to the above embodiments, and will not be repeated here). The server then projects the pixels of the 9 sub-images onto 9 positions in the sampled image to be generated, thus obtaining the sampled image. The sampled image may include 9 pixels (exemplary illustration), where the first pixel is determined based on pixels within sub-image 1 of the second image, the second pixel is determined based on pixels within sub-image 2 of the second image, and so on, until the 9th pixel is determined based on pixels within sub-image 9 of the second image.

[0139] Referring to Figure 13, after the server determines the sampled image, it can send the sampled image to the virtual reality device. The virtual reality device can then play the sampled image, presenting a 3D effect to the user and improving the user experience. In this way, the center of the second image in the first image can be located on the equator of the first image. The second image has a higher pixel density, and since multiple sub-images are obtained by dividing the second image at equal angles, the electronic device can uniformly sample the pixels in the second image to obtain the sampled image, thereby improving the accuracy of the sampled image and saving network resources.

[0140] This disclosure provides an image processing method, apparatus, and electronic device. The electronic device can determine a first image, wherein the first image is an image projected onto a sphere as a panoramic image. The first image may include a second image, which includes the image covered by the user's field of view in the first image. The method determines a first angle of horizontal opening of the second image relative to the center of the sphere corresponding to the first image and a second angle of vertical opening of the second image relative to the center of the sphere. Based on the first and second angles, the second image is divided into M*N sub-images at equal angles, where M and N are positive integers determined based on a preset resolution. Pixel sampling is performed on the M*N sub-images to obtain a sampled image. In the above method, since the multiple sub-images are obtained by dividing the second image at equal angles, the electronic device achieves good uniformity in sampling the second image, resulting in high accuracy of the sampled image and thus saving network resources.

[0141] Figure 14 is a schematic diagram of an image processing device provided in an embodiment of this disclosure. Referring to Figure 14, the image processing device 140 includes a first determining module 141, a second determining module 142, a dividing module 143, and a sampling module 144, wherein: the first determining module 141 is used to determine a first image, which is an image projected onto a sphere from a panoramic image; the first image includes a second image, which includes the image covered by the user's field of view in the first image; the second determining module 142 is used to determine a first angle of horizontal opening of the second image relative to the center of the sphere corresponding to the first image and a second angle of vertical opening of the second image relative to the center of the sphere; the dividing module 143 is used to divide the second image domain into M*N sub-images at equal angles based on the first angle and the second angle, where M and N are positive integers determined based on a preset resolution; the sampling module 144 is used to perform pixel sampling in the M*N sub-images to obtain a sampled image.

[0142] According to one or more embodiments of this disclosure, the division module 143 is specifically used to: divide the first angle into M first bisectors, divide the second angle into N second bisectors; and divide the second image into M*N sub-images based on the M first bisectors and the N second bisectors.

[0143] According to one or more embodiments of this disclosure, the division module 143 is specifically used to: determine M+1 vertical arcs associated with the M first bisectors in the second image; determine N+1 horizontal arcs associated with the N second bisectors in the second image; and divide the second image into M*N sub-images based on the M+1 vertical arcs and the N+1 horizontal arcs.

[0144] According to one or more embodiments of this disclosure, the sampling module 144 is specifically configured to: for any target location in the sampled image; determine the sub-image corresponding to the target location in the second image; acquire one or more pixels in the sub-image, and perform fusion processing on the one or more pixels to obtain the pixel of the target location in the sampled image.

[0145] According to one or more embodiments of this disclosure, the second determining module 142 is specifically used to: determine the angle between the midpoint of the two arcs in the vertical direction of the second image and the center of the sphere as the first angle of horizontal opening; and determine the angle between the midpoint of the two arcs in the horizontal direction of the second image and the center of the sphere as the second angle of vertical opening.

[0146] According to one or more embodiments of this disclosure, the first determining module 141 is specifically configured to: acquire a panoramic image corresponding to the first image and a user's viewpoint; rotate the panoramic image based on the user's viewpoint to obtain a rotated image, wherein the user's viewpoint is at the center of the rotated image; and project the rotated image onto a sphere based on the perspective projection method to obtain the first image.

[0147] According to one or more embodiments of this disclosure, the first determining module 141 is specifically configured to: determine the image covered by the user's field of view in the rotating image with the user's viewpoint as the center; and add images of a preset range to each edge of the image covered by the user's field of view to obtain the second image.

[0148] The image processing apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0149] Figure 15 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Referring to Figure 15, it shows a schematic diagram of the structure of an electronic device 1500 suitable for implementing an embodiment of this disclosure. The electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 15 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this disclosure.

[0150] As shown in Figure 15, the electronic device 1500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage device 1508 into a random access memory (RAM) 1503. The RAM 1503 also stores various programs and data required for the operation of the electronic device 1500. The processing unit 1501, ROM 1502, and RAM 1503 are interconnected via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0151] Typically, the following devices can be connected to I / O interface 1505: input devices 1506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1509. Communication device 1509 allows electronic device 1500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 15 shows electronic device 1500 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0152] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1509, or installed from a storage device 1508, or installed from a ROM 1502. When the computer program is executed by the processing device 1501, it performs the functions defined in the methods of embodiments of this disclosure.

[0153] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0154] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0155] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0156] This disclosure provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the image processing methods that may be involved in the above embodiments.

[0157] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the image processing methods described in the above embodiments.

[0158] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0161] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0162] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0164] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0165] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0166] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message. As an optional but non-limiting implementation, the prompt message can be sent to the user in the form of a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0167] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0168] It is understood that the data involved in this technical solution (including but not limited to the data itself, its acquisition, or its use) shall comply with the requirements of relevant laws, regulations, and provisions. Data may include information, parameters, and messages, such as flow control instructions.

[0169] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0170] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0171] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method, comprising: A first image is determined, which is an image projected onto a sphere as a panoramic image. The first image includes a second image, which includes the image covered by the user's field of view in the first image. Determine a first angle at which the second image is horizontally opened relative to the center of the sphere corresponding to the first image and a second angle at which the second image is vertically opened relative to the center of the sphere; Based on the first angle and the second angle, the second image domain is divided into M*N sub-images at equal angles, where M and N are positive integers determined based on a preset resolution. Pixel sampling is performed on the M*N sub-images to obtain the sampled image.

2. The method according to claim 1, wherein, The second image is divided into M*N sub-images based on the first angle and the second angle, including: The first angle is divided into M equal first bisectors, and the second angle is divided into N equal second bisectors. Based on the M first bisectors and the N second bisectors, the second image is divided into M*N sub-images at equal angles.

3. The method according to claim 2, wherein, The second image is divided into M*N sub-images based on the M first bisectors and the N second bisectors, including: Determine the M+1 vertical arcs associated with the M first bisecting angles in the second image; Determine the N second bisecting angles and associate them with N+1 horizontal arcs in the second image; Based on the M+1 vertical arcs and the N+1 horizontal arcs, the second image is divided into M*N sub-images.

4. The method according to any one of claims 1-3, wherein, The step of performing pixel sampling in the M*N sub-images of the second image to obtain the sampled image includes: For any target location in the sampled image; Determine the sub-image corresponding to the target location in the second image; One or more pixels in the sub-image are obtained, and the one or more pixels are fused to obtain the pixel at the target location in the sampled image.

5. The method according to any one of claims 1-3, wherein, Determining the first angle of horizontal opening of the second image relative to the center of the sphere corresponding to the first image and the second angle of vertical opening of the second image relative to the center of the sphere includes: The angle between the midpoint of the two vertical arcs in the second image and the line connecting the center of the sphere is determined as the first angle of the horizontal opening. The angle between the midpoint of the two horizontal arcs in the second image and the line connecting the center of the sphere is determined as the second angle of vertical opening.

6. The method according to any one of claims 1-3, wherein, Determining the first image includes: Obtain the panoramic image corresponding to the first image and the user's viewpoint; Based on the user's viewpoint, the panoramic image is rotated to obtain a rotated image, wherein the user's viewpoint is at the center of the rotated image; The first image is obtained by projecting the rotated image onto a sphere using perspective projection.

7. The method according to claim 6, wherein, After rotating the panoramic image based on the user's viewpoint to obtain a rotated image, the method further includes: Centered on the user's viewpoint, determine the image covered by the user's field of view in the rotated image; The second image is obtained by adding images of a preset range to each edge of the image covered by the user's field of view.

8. An image processing apparatus, comprising a first determining module, a second determining module, a segmentation module, and a sampling module, wherein: The first determining module is used to determine a first image, the first image being an image projected onto a sphere, the first image including a second image, the second image including an image covered by the user's field of view in the first image; The second determining module is used to determine a first angle at which the second image is horizontally opened relative to the center of the sphere corresponding to the first image and a second angle at which the second image is vertically opened relative to the center of the sphere; The partitioning module is used to partition the second image domain at equal angles based on the first angle and the second angle to obtain M*N sub-images, where M and N are positive integers determined based on a preset resolution. The sampling module is used to perform pixel sampling in the M*N sub-images to obtain the sampled image.

9. An electronic device, comprising: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the image processing method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the image processing method as described in any one of claims 1-7.

Images