Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment, and the method comprises the steps: determining a first image, the first image is an image obtained by projecting a panoramic image to a spherical surface, the first image comprises a second image, and the second image comprises an image covered by a field angle of a user in the first image; determining a first angle of horizontal opening of the second image relative to a sphere center corresponding to the first image and a second angle of vertical opening of the second image relative to the sphere center; based on the first angle and the second angle, performing equal-angle division on the second image domain to obtain M * N sub-images, M and N being positive integers determined based on a preset resolution; and carrying out pixel sampling in the M * N sub-images to obtain a sampled image.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular, to an image processing method, device and electronic device. Background Art

[0002] In the field of virtual reality, the server can determine the image viewed by the user in the panoramic image based on the user's viewing angle, and send the image viewed by the user to the virtual reality device.

[0003] Currently, the server can convert the spherical panoramic image into a flat panoramic image based on equirectangular projection, determine the image slices viewed by the user in the flat panoramic image (the flat panoramic image will be divided into multiple image slices), and send the image slices viewed by the user to the virtual reality device. However, the uniformity of the pixel density of the flat panoramic image obtained by equirectangular projection is poor, and the uniformity of the sampling of the panoramic image by the server based on the viewing angle range is poor, resulting in a waste of network resources. Summary of the invention

[0004] The present disclosure provides an image processing method, device and electronic device, which are used to solve one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides an image processing method, the method comprising:

[0006] Determine a first image, where the first image is an image obtained by projecting the panoramic image onto a spherical surface, the first image includes a second image, and the second image includes an image covered by the user's field of view in the first image;

[0007] Determine a first horizontal angle of the second image relative to a sphere center corresponding to the first image and a second vertical angle of the second image relative to the sphere center;

[0008] Based on the first angle and the second angle, the second image domain is divided into equal angles to obtain M*N sub-images, where M and N are positive integers determined based on a preset resolution;

[0009] Pixel sampling is performed in the M*N sub-images to obtain a sampled image.

[0010] In a second aspect, the present disclosure provides an image processing device, the image processing device comprising a first determination module, a second determination module, a division module and a sampling module, wherein:

[0011] The first determination module is used to determine a first image, where the first image is an image obtained by projecting a panoramic image onto a spherical surface, the first image includes a second image, and the second image includes an image covered by the user's field of view in the first image;

[0012] The second determination module is used to determine a first horizontal angle of the second image relative to a sphere center corresponding to the first image and a second vertical angle of the second image relative to the sphere center;

[0013] The division module is used to divide the second image domain into equal angles based on the first angle and the second angle to obtain M*N sub-images, where M and N are positive integers determined based on a preset resolution;

[0014] The sampling module is used to perform pixel sampling in the M*N sub-images to obtain a sampled image.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the image processing method as described in the first aspect and various possible aspects of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0019] The present disclosure provides an image processing method, device and electronic device, the electronic device can determine a first image, wherein the first image is an image of a panoramic image projected onto a spherical surface, the first image can include a second image, the second image includes an image covered by the user's field of view in the first image, determine a first angle of horizontal opening of the second image relative to the center of the sphere corresponding to the first image and a second angle of vertical opening of the second image relative to the center of the sphere, divide the second image into equal angles based on the first angle and the second angle, obtain M*N sub-images, wherein M and N are positive integers determined based on a preset resolution, and perform pixel sampling in the M*N sub-images to obtain a sampled image. In the above method, since the multiple sub-images are images obtained based on the equal angle division of the second image, the electronic device has better uniformity in sampling the second image, and the accuracy of the sampled image is higher, thereby saving network resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0022] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of a first image provided by an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of a first angle provided for an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of a second angle provided for an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of a first bisecting angle provided in an embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of determining M*N sub-images provided in an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram of determining an image after sampling provided by an embodiment of the present disclosure;

[0029] Fig. 9 A schematic diagram of a method for determining a first image provided by an embodiment of the present disclosure;

[0030] Fig.10 A schematic diagram of a process for determining a second image provided by an embodiment of the present disclosure;

[0031] Fig.11 A schematic diagram of a rotating image provided by an embodiment of the present disclosure;

[0032] Fig.12 A schematic diagram of determining an image to be transmitted provided by an embodiment of the present disclosure;

[0033] Fig.13 A schematic diagram of an image processing method provided by an embodiment of the present disclosure;

[0034] Fig.14A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure; and

[0035] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0037] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0038] Electronic device: a device with wireless transceiver function. Electronic devices can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted; they can also be deployed on the water (such as ships, etc.). The electronic device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a vehicle-mounted electronic device, a wireless terminal in self-driving, a wireless electronic device in remote medical, a wireless electronic device in smart grid, a wireless electronic device in transportation safety, a wireless electronic device in smart city, a wireless electronic device in smart home, a wearable electronic device, etc. The electronic device involved in the embodiments of the present disclosure can also be called a terminal, a user equipment (UE), an access electronic device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote electronic device, a mobile device, a UE electronic device, a wireless communication device, a UE agent or a UE device, etc. Electronic devices may also be fixed or mobile.

[0039] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0040] Figure 1A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 , including a spherical image, a server and a virtual reality device, wherein the spherical image includes a plurality of sampling points (the circles in the spherical image are sampling points), and the server can project the spherical image onto a plane to obtain a plane image. The plane image may include a plurality of sampling points, the first row of sampling points of the plane image is located in the Arctic region, the third row of sampling points is located in the equatorial region, and the fifth row of sampling points is located in the Antarctic region.

[0041] See also Figure 1 , the server can divide the plane image into 25 image slices. If the user's viewpoint is in the front, the server can send image slice 13 to the virtual reality device. If the user's viewpoint is at the top, the server can send image slices 1, 2, 3, 4, and 5 to the virtual reality device. In this way, the server can send image slices within the field of view to the virtual reality device based on the user's field of view, thereby reducing the amount of data transmitted.

[0042] It should be noted that Figure 1 The application scenarios of the embodiments of the present disclosure are merely illustrative and are not intended to limit the application scenarios of the embodiments of the present disclosure.

[0043] In the related art, the server can determine the image viewed by the user in the panoramic video based on the user's field of view, and send the image viewed by the user to the virtual reality device. Figure 1 As shown, since the server cannot encode the image of the curved surface, the server can project the image of the curved surface onto a plane. If the viewing angle of the user includes the image slice 13 in the plane image, the server can send the image slice 13 to the virtual reality device. Currently, the server can convert the panoramic image of the curved surface into a panoramic image of the plane based on equidistant rectangular projection. The server can determine the image slice in the panoramic image of the plane viewed by the user and send the image slice to the virtual reality device. However, the uniformity of the pixel density of the panoramic image of the plane obtained by equidistant rectangular projection is poor. For example, Figure 1 As shown, the pixel density of the equatorial part of the panoramic image of the plane is relatively large, and the pixel density of the polar part is relatively small. If the user's field of view is in the equatorial part, the field of view can cover fewer image slices, but if the user's field of view is in the polar region, the field of view will cover all the image slices in the polar region (that is, for the same size of field of view, the number of image slices covered by the field of view at the poles is greater than the number of image slices covered by the field of view at the equator). In this way, when the server samples the images within the field of view, the sampling uniformity is poor, and the server needs to transmit more image slices, which leads to a waste of network resources.

[0044] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides an image processing method, where an electronic device can obtain a panoramic image corresponding to a first image and a user's viewpoint, and rotate the panoramic image based on the user's viewpoint to obtain a rotated image, wherein the user's viewpoint is at the center of the rotated image. The electronic device can project the rotated image onto a spherical surface based on a perspective projection to obtain a first image, and the electronic device can determine a first horizontal opening angle of a second image in the first image relative to the center of a sphere corresponding to the first image and a second vertical opening angle of the second image relative to the center of the sphere, and divide the second image into equal angles based on the first angle and the second angle to obtain M*N sub-images, wherein M and N are positive integers determined based on a preset resolution, and pixel sampling is performed in the M*N sub-images to obtain the sampled image. In the above method, since the electronic device can rotate the panoramic video frame based on the viewpoint, the center of the second image in the first image can be located at the equatorial part of the first image, and the pixel density of the second image is relatively high. Moreover, since the multiple sub-images are images obtained based on dividing the second image at equal angles, the electronic device can uniformly sample the pixels in the second image to obtain the sampled image, thereby improving the accuracy of the sampled image and saving network resources.

[0045] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0046] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0047] S201: Determine a first image, where the first image includes a second image.

[0048] The execution subject in the embodiments of the present disclosure may be an electronic device, or an image processing device provided in the electronic device. The image processing device may be implemented based on software, or the image processing device may be implemented based on a combination of software and hardware, which is not limited in the embodiments of the present disclosure. Optionally, the electronic device may be any device with terminal computing capabilities, for example, the electronic device may be a server, a computer, or other devices, which is not limited in the embodiments of the present disclosure.

[0049] The first image may be an image obtained by projecting the panoramic image onto a spherical surface. For example, the panoramic image may be a flat image, and the electronic device may project the panoramic image onto the spherical surface of a sphere, thereby obtaining a first image of a curved surface. If the user's viewing angle is located at the center of the sphere, the first image of the curved surface may present a three-dimensional display effect to the user.

[0050] Optionally, the panoramic image may be a planar image obtained based on equirectangular projection (ERP). For example, ERP projection may project the longitude of a sphere into equally spaced vertical lines and the latitude of a sphere into equally spaced horizontal lines, and the electronic device may render a spherical image with a three-dimensional effect, and may convert the spherical image into a planar panoramic image based on ERP projection.

[0051] Optionally, the panoramic image may be a planar image obtained based on a cube map projection (CMP). For example, the CMP projection may embed a spherical surface into a cube, and project the spherical surface onto the six faces of the cube based on a perspective projection method. The electronic device may render a spherical image with a three-dimensional effect, and may project the spherical image onto the six faces of the cube based on the CMP projection, and unfold the six faces of the cube to obtain a planar panoramic image.

[0052] Optionally, the panoramic image may also be an image captured by a panoramic camera device. For example, a fisheye camera may capture a panoramic image, which is not limited in the embodiments of the present disclosure.

[0053] Optionally, the electronic device may receive the panoramic image sent by other devices, and the electronic device may also obtain the panoramic image from a database, which is not limited in the embodiments of the present disclosure.

[0054] It should be noted that in actual application, the electronic device can generate a panoramic video, wherein the video frames in the panoramic video are panoramic images, and the electronic device can send each panoramic video frame in the panoramic video to a virtual reality device, so that the virtual reality device can present a three-dimensional video effect to the user.

[0055] The first image may be an image of a curved surface. For example, the first image is an image obtained by projecting a plane image onto the surface of a sphere, and therefore, the first image may be an image of a curved surface. The first image may include a second image, and the second image may include an image covered by the user's field of view in the first image. For example, the user's field of view (FOV) may cover a portion of the image in the first image, and the second image may include the portion of the image, so that when the user's perspective moves slightly, an effective picture can also be seen, thereby improving the user experience.

[0056] It should be noted that the second image can be an image to be transmitted by the electronic device. The electronic device can determine the second image in the first image and project the second image onto a plane to obtain a plane image. The electronic device can encode the plane image and send the encoded plane image to the virtual reality device.

[0057] Optionally, the center of the second image may be the viewpoint of the user's field of view, and the center of the second image may be located in an area within a preset range of the equator of the first image. For example, the area within the preset range of the equator of the first image may be an area within 30 degrees above and below the equator, and the area within the preset range of the equator of the first image may also be an area within 50 degrees above and below the equator, which is not limited in the embodiments of the present disclosure.

[0058] For example, the center of the second image can be located in an area within a preset range of the equator of the first image, and the center of the second image can also be located on the equator of the first image. In this way, since the center of the second image can be the user's viewpoint, and the pixel density in the area within the preset range of the equator of the first image is higher, therefore, if the center of the second image is in the area within the preset range of the equator of the first image, the pixel density in the second image is higher, thereby improving the uniformity of sampling.

[0059] Optionally, after the electronic device acquires the panoramic image, it can rotate the panoramic image so that the user's FOV viewpoint is located at the center of the rotated image, and the rotated image is projected onto a spherical surface, thereby obtaining a first image. In this way, the user's viewpoint can be located in an area within a preset range of the equator, that is, the center point of the second image is located in an area within a preset range of the equator. The second image has a higher pixel density and a higher pixel uniformity. Therefore, the electronic device can accurately obtain the image to be transmitted and save network resources.

[0060] Next, combine Figure 3 , the first image is described.

[0061] Figure 3 A schematic diagram of a first image provided by an embodiment of the present disclosure. Figure 3 , including a first image. The first image is a curved surface image of a sphere, and the first image may include a second image (the image in the black area and the image in the gray area on the surface of the first image may be the second image in the first image, and the image in the black area may be the image covered by the user's field of view in the first image). The center of the second image may be located at the equator of the sphere corresponding to the first image, so that the second image may face the equatorial plane of the sphere, thereby increasing the pixel density of the second image and improving the sampling accuracy.

[0062] S202, determining a first horizontal angle of the second image relative to the sphere center corresponding to the first image and a second vertical angle of the second image relative to the sphere center.

[0063] The first angle may be the horizontally opened angle of the second image relative to the center of the sphere corresponding to the first image. For example, if the second image is horizontally opened 100 degrees relative to the center of the sphere (the center of the sphere corresponding to the first image), the first angle may be 100 degrees, and if the second image is horizontally opened 200 degrees relative to the center of the sphere, the first angle may be 200 degrees.

[0064] The second angle may be an angle of the second image vertically opened relative to the sphere center corresponding to the first image. For example, if the second image is vertically opened 50 degrees relative to the sphere center corresponding to the first image, the second angle may be 50 degrees; if the second image is vertically opened 80 degrees relative to the sphere center corresponding to the first image, the second angle may be 80 degrees.

[0065] Optionally, the electronic device may determine a first horizontal opening angle of the second image relative to the center of a sphere corresponding to the first image and a second vertical opening angle of the second image relative to the center of the sphere based on the following feasible implementation method: determining the angle between the line connecting the midpoints of two arcs in the vertical direction of the second image and the center of the sphere as the first horizontal opening angle, and determining the angle between the line connecting the midpoints of two arcs in the horizontal direction of the second image and the center of the sphere as the second vertical opening angle.

[0066] Optionally, the second image may be determined based on four arcs, where two arcs in the vertical direction may be a left arc and a right arc of the second image, and two arcs in the horizontal direction may be an upper arc and a lower arc of the second image.

[0067] Optionally, the electronic device may determine the angle between the midpoint of the left arc, the midpoint of the right arc and the line connecting the center of the sphere as the first angle of horizontal opening. For example, since the first image is a curved surface image, the angle between the midpoint of the left arc, the midpoint of the right arc and the line connecting the center of the sphere may be the maximum angle of the second image in the horizontal direction, and the electronic device may determine the angle as the first angle.

[0068] Optionally, the electronic device may determine the angle between the midpoint of the upper arc, the midpoint of the lower arc and the line connecting the center of the sphere as the second vertically opened angle. For example, since the first image is a curved image, the angle between the midpoint of the upper arc, the midpoint of the right arc and the line connecting the center of the sphere may be the maximum angle of the second image opened in the vertical direction, and the electronic device may determine the angle as the second angle. In this way, the electronic device may accurately determine the angle of the second image opened relative to the center of the sphere, and then accurately divide the second image into equal angles, thereby improving the accuracy and uniformity of sampling.

[0069] Optionally, since the first image is a curved image, the electronic device can also arbitrarily select a group of two relative points on two vertically opposite arcs of the second image (e.g., a point 10 pixels away from the upper left vertex on the left arc, and a point 10 pixels away from the upper right vertex on the right arc), and determine the angle between the line between the group of two relative points and the center of the sphere as the first angle.

[0070] Optionally, since the first image is a curved image, the electronic device can also arbitrarily select a group of two relative points on two horizontal relative arcs of the second image (e.g., a point 10 pixels away from the upper left vertex on the upper arc, and a point 10 pixels away from the lower left vertex on the lower arc), and determine the angle between the group of two relative points and the line connecting the center of the sphere as the second angle.

[0071] Next, combine Figure 4 , the first angle is explained.

[0072] Figure 4 A schematic diagram of a first angle provided by an embodiment of the present disclosure. Figure 4 , including: a first image. The first image may include a second image. The center of the second image is located on the equator of the first image. The second image may include a left arc and a right arc. Electronic device ( Figure 4 (Not shown) The included angle between the midpoint of the left arc, the midpoint of the right arc and the line connecting the center of the ball can be determined as the first angle.

[0073] Next, combine Figure 5 , the second angle is explained.

[0074] Figure 5 A schematic diagram of a second angle provided by an embodiment of the present disclosure. Figure 5 , including: a first image. The first image may include a second image. The center of the second image is located on the equator of the first image. The second image may include an upper arc and a lower arc. Electronic device ( Figure 5 (Not shown) The included angle between the midpoint of the upper arc, the midpoint of the lower arc and the line connecting the center of the ball can be determined as the second angle.

[0075] S203 , dividing the second image into equal-angle segments based on the first angle and the second angle to obtain M*N sub-images.

[0076] Wherein, M and N are positive integers determined based on the preset resolution. For example, if the preset resolution is 800*400, M is 800 and N is 400; if the preset resolution is 800*1600, M is 800 and N is 1600.

[0077] Optionally, the preset resolution may be any set resolution, which is not limited in the embodiments of the present disclosure.

[0078] Among them, the electronic device can obtain M*N sub-images based on the following feasible implementation method: divide the first angle evenly into M first bisecting angles, divide the second angle evenly into N second bisecting angles, and divide the second image at equal angles based on the M first bisecting angles and the N second bisecting angles to obtain M*N sub-images.

[0079] Optionally, the angle of the first bisecting angle may be a ratio of the first angle to M. For example, if the first angle is 100 degrees and M is 50, the electronic device may evenly divide the first angle into 50 first bisecting angles. For example, if the first angle is 100 degrees and M is 50, the electronic device may determine that each first bisecting angle is 2 degrees, and the sum of the 50 first bisecting angles may be the first angle.

[0080] Optionally, the angle of the second bisecting angle may be a ratio of the second angle to N. For example, if the second angle is 100 degrees and N is 200, the electronic device may evenly divide the second angle into 200 second bisecting angles. For example, if the second angle is 100 degrees and N is 200, the electronic device may determine that each second bisecting angle is 0.5 degrees, and the sum of the 200 second bisecting angles may be the second angle.

[0081] Next, combine Figure 6 , the first bisector angle is explained.

[0082] Figure 6 A schematic diagram of a first bisecting angle provided in an embodiment of the present disclosure. Figure 6 In the embodiment shown, M can be 3, see Figure 6 , including: a first image. The first image may include a second image. The center of the second image is located on the equator of the first image. The second image may include a left arc and a right arc. The first angle may be the angle between the midpoint of the left arc, the midpoint of the right arc and the center of the sphere. Dividing the first angle into three equal parts, three first bisecting angles may be obtained. In this way, the electronic device may divide the first angle into multiple first bisecting angles, thereby improving the uniformity of pixels in the sub-image and improving the accuracy of the second image.

[0083] It should be noted that the second bisector angle is Figure 6 The first bisecting angle in the illustrated embodiment is similar, and the embodiments of the present disclosure will not be described in detail herein.

[0084] Among them, the electronic device divides the second image at equal angles based on M first bisecting angles and N second bisecting angles to obtain M*N sub-images, which can be specifically: determining M+1 vertical arcs associated with the M first bisecting angles in the second image, determining N+1 horizontal arcs associated with the N second bisecting angles in the second image, and dividing the second image into M*N sub-images based on the M+1 vertical arcs and the N+1 horizontal arcs.

[0085] Optionally, the arc in the vertical direction may be an arc in the vertical direction passing through the edge of the first bisector and the focus of the first image. For example, the arc in the vertical direction may be perpendicular to the equator of the first image. For example, for any first bisector (the first bisector does not intersect the left arc and the right arc), there is focus 1 and focus 2 between the first bisector and the first image, the electronic device may determine that one arc in the vertical direction is an arc passing through focus 1 and perpendicular to the equator, and the electronic device may determine that another arc in the vertical direction is an arc passing through focus 2 and perpendicular to the equator.

[0086] Optionally, the arc in the horizontal direction may be an arc in the horizontal direction passing through the edge of the second bisector and the focus of the first image. For example, the arc in the horizontal direction may be parallel to the equator of the first image. For example, for any second bisector (the second bisector does not intersect with the upper arc and the lower arc), there is focus 1 and focus 2 between the second bisector and the first image, the electronic device may determine that one arc in the horizontal direction is an arc passing through focus 1 and parallel to the equator, and the electronic device may determine that another arc in the horizontal direction is an arc passing through focus 2 and parallel to the equator.

[0087] It should be noted that the M+1 arcs in the vertical direction include the left arc and the right arc of the second image, and the N+1 arcs in the horizontal direction include the upper arc and the lower arc of the second image.

[0088] It should be noted that the multiple groups of adjacent arcs in the vertical direction have the same angle relative to the horizontal opening of the center of the sphere. For example, arc 1 in the vertical direction is adjacent to arc 2 in the vertical direction, arc 2 in the vertical direction is adjacent to arc 3 in the vertical direction, the angle between the midpoint of arc 1, the midpoint of arc 2 and the center of the sphere is angle A, and the angle between the midpoint of arc 2, the midpoint of arc 3 and the center of the sphere is angle B, wherein angle A is the same as angle B, that is, the second image is divided into equal angles in the vertical direction.

[0089] It should be noted that the multiple groups of adjacent arcs in the horizontal direction have the same vertical angle relative to the center of the sphere. For example, arc 1 in the horizontal direction is adjacent to arc 2 in the horizontal direction, arc 2 in the horizontal direction is adjacent to arc 3 in the horizontal direction, the angle between the midpoint of arc 1, the midpoint of arc 2 and the center of the sphere is angle A, and the angle between the midpoint of arc 2, the midpoint of arc 3 and the center of the sphere is angle B, wherein angle A is the same as angle B, that is, the second image is divided into equal angles in the horizontal direction.

[0090] Optionally, the electronic device may divide the second image into M*N grids based on M+1 arcs in the vertical direction and N+1 arcs in the horizontal direction, wherein the image in each grid may be a sub-image. For example, the electronic device may divide the second image into two areas in the vertical direction based on three arcs in the vertical direction (including a left arc and a right arc), and the electronic device may divide the second image into two areas in the horizontal direction based on three arcs in the horizontal direction (including an upper arc and a lower arc), that is, the electronic device may obtain four grids based on three arcs in the vertical direction and three arcs in the horizontal direction, and the image in each grid may be a sub-image.

[0091] Next, combine Figure 7 , the process of determining M*N sub-images is explained.

[0092] Figure 7 A schematic diagram of determining M*N sub-images provided in an embodiment of the present disclosure. Figure 7 In the embodiment shown, M is 3 and N is 3, see Figure 7 , including: a first image. The first image may include the center of the sphere and a second image, wherein the center of the second image is located on the equator. If M is 3, then the electronic device ( Figure 7 (not shown) four vertical arcs (including the left arc and the right arc of the second image) can be determined in the second image, and then the second image can be divided into three areas with equal angles in the vertical direction.

[0093] See also Figure 7 , if N is 3, the electronic device can determine 4 horizontal arcs in the second image (including the upper arc and the lower arc of the second image), and then divide the second image into 3 areas at equal angles in the horizontal direction (the equator is not shown). Based on the 3 areas in the horizontal direction and the 3 areas in the vertical direction, the second image can be divided into sub-image 1, sub-image 2, ..., sub-image 9 at equal angles. In this way, since the center of the second image is located on the equator, and the 9 sub-images are obtained based on the equal-angle division of the second image, the pixel density of the 9 sub-images is relatively high, which improves the sampling effect.

[0094] S204 , performing pixel sampling in the M*N sub-images to obtain a sampled image.

[0095] The sampled image may be an image obtained by projecting the second image in the first image onto a plane. For example, the second image may be an image to be transmitted, and the electronic device may project the second image onto a plane based on an equiangular projection method to obtain the sampled image, wherein the equiangular projection may be a projection method of projecting pixels in a plurality of sub-images (the sub-images are obtained by dividing the second image at equal angles) onto a plane.

[0096] Optionally, the resolution of the sampled image is the same as the preset resolution. For example, if the preset resolution is 800*1600, 800*1600 sub-images can be obtained in the second image based on the preset resolution, and the electronic device can sample pixels in each sub-image to obtain the sampled image. Therefore, the resolution of the sampled image can be 800*1600.

[0097] Optionally, the electronic device can obtain the sampled image based on the following feasible implementation method: for any target position in the sampled image, determine the sub-image corresponding to the target position in the second image, obtain one or more pixels in the sub-image, and fuse the one or more pixels to obtain the pixel of the target position in the sampled image.

[0098] Optionally, in the process of performing equi-angle projection on the second image, the electronic device may determine multiple sub-images corresponding to multiple positions in the sampled image in the second image, and then project the pixels in the multiple sub-images to multiple positions to obtain the sampled image. For example, for position (1, 1) in the sampled image, the electronic device may determine the sub-image corresponding to position (1, 1) in the second image, and then determine the pixels in the sub-image as the pixels at position (1, 1), and repeat the above steps to obtain the sampled image.

[0099] Optionally, the electronic device may determine the sub-image corresponding to the target position based on the following formula:

[0100]

[0101] Among them, F v is the first angle; F h is the second angle, M is the number of pixels in the width of the image after sampling; N is the number of pixels in the height of the image after sampling; m is the coordinate of the target position in the width of the image after sampling; n is the coordinate of the target position in the height of the image after sampling; φ is the horizontal opening angle of the sub-image (position in the first angle), is the vertically opened angle of the sub-image (position in the second angle).

[0102] In this way, based on the above formula, the correspondence between multiple positions in the sampled image and multiple sub-images in the second image can be determined, and then the sampled image can be accurately generated.

[0103] It should be noted that due to is the vertical opening angle of the second image, and the position of this angle cannot be determined in the spherical coordinate system. Therefore, this angle can be converted into θ in the spherical coordinate system. In this way, the position of the sub-image can be accurately located in the second image based on φ and θ.

[0104] Among them, the electronic device can be based on the following formula Convert to θ in spherical coordinates:

[0105]

[0106] Among them, the physical meaning of each parameter in the above formula can refer to the formula for determining the sub-image corresponding to the target position, and the embodiment of the present disclosure will not elaborate on this.

[0107] It should be noted that after the electronic device determines the second image and the preset resolution, it can achieve the effect of scaling the second image, so that the projection method of the second image has higher adaptability. In addition, the electronic device can flexibly determine the resolution of the second image based on network conditions, thereby improving the smoothness of users viewing panoramic videos.

[0108] Optionally, the electronic device may perform fusion processing on one or more pixels to obtain the pixel at the target position in the sampled image. For example, the electronic device may perform average processing on the pixel values ​​of one or more pixels to obtain the pixel at the target position. For example, if the sub-image corresponding to the target position includes pixel 1, pixel 2, and pixel 3, the electronic device may average the RGB value of pixel 1, the RGB value of pixel 2, and the RGB value of pixel 3 to obtain a new pixel, and determine the new pixel as the pixel at the target position.

[0109] Optionally, the electronic device may also perform fusion processing on one or more pixels in the sub-image based on any other feasible implementation manner, which is not limited in the embodiments of the present disclosure.

[0110] Optionally, the electronic device may also determine any pixel in the sub-image as the pixel of the target position. For example, if the sub-image corresponding to the target position includes pixel 1, pixel 2, and pixel 3, the electronic device may determine pixel 1 as the pixel of the target position, the electronic device may determine pixel 2 as the pixel of the target position, and the electronic device may determine pixel 3 as the pixel of the target position. The embodiments of the present disclosure are not limited to this. In this way, the electronic device can project the second image into the plane to obtain the sampled image. Since the pixel density in the second image is high, the accuracy of the sampled image is high.

[0111] Next, combine Figure 8 , the process of determining the image after sampling is explained.

[0112] Figure 8 A schematic diagram of determining an image after sampling provided by an embodiment of the present disclosure. Figure 8 In the embodiment shown, the sampled image may include 9 pixels, see Figure 8 , including the corresponding image. The first image may include the second image, the second image may include 9 sub-images, and the image to be generated after sampling may include 9 filled pixel positions. Electronic device ( Figure 8 (not shown) can project pixels in sub-image 1 to position 1, pixels in sub-image 2 to position 2, ..., pixels in sub-image 9 to position 9. In this way, the electronic device can project the second image into the plane image to obtain a sampled image. During the projection process, the sampling uniformity is good, and the pixel density in the second image is high. Therefore, the accuracy of the sampled image is high.

[0113] The disclosed embodiment provides an image processing method, which determines a first image, determines a first horizontal angle of a second image relative to the center of a sphere corresponding to the first image, and a second vertical angle of the second image relative to the center of a sphere, divides the first angle into M first bisecting angles on average, divides the second angle into N second bisecting angles on average, divides the second image into M*N sub-images based on the M first bisecting angles and the N second bisecting angles at equal angles, and obtains M*N sub-images, and performs pixel sampling in the M*N sub-images to obtain a sampled image. Since the center of the second image is located on the equator of the first image, and the electronic device can divide the second image into multiple sub-images based on equal angle division, the pixel density in the second image is relatively high, and the uniformity of the sub-images is relatively high, which improves the accuracy of sampling in the second image, thereby improving the accuracy of the sampled image, avoiding the transmission of redundant content, and saving network resources.

[0114] exist Figure 2Based on the embodiment shown below, combined with Fig. 9 , a method for determining the first image in the above-mentioned image processing method is described.

[0115] Fig. 9 A schematic diagram of a method for determining a first image provided by an embodiment of the present disclosure. Fig. 9 The method flow includes:

[0116] S901: Acquire a panoramic image corresponding to a first image and a user's viewpoint.

[0117] Optionally, the electronic device may acquire the panoramic image corresponding to the first image based on any feasible implementation method, which is not limited in the embodiments of the present disclosure.

[0118] Optionally, the user's viewpoint may be the center point of the area that the user is paying attention to in the panoramic image. For example, the user's viewpoint may be the center point of the user's field of view. If the user is paying attention to area 1 in the panoramic image, the user's viewpoint may be the center point of area 1. If the user is paying attention to area 2 in the panoramic image, the user's viewpoint may be the center point of area 2. It should be noted that the user's viewpoint may also be any point in the area that the user is paying attention to in the panoramic image, and the embodiments of the present disclosure are not limited to this.

[0119] Optionally, the user's viewpoint may be represented based on a direction parameter. For example, the direction parameter corresponding to the user's viewpoint may be (α, β, γ), where α may be the horizontal field of view of the user's viewpoint (i.e., the horizontal field of view in the FOV), β may be the vertical field of view of the user's viewpoint (i.e., the vertical field of view in the FOV), and γ may be the diagonal field of view of the user's viewpoint (i.e., the diagonal field of view in the FOV).

[0120] It should be noted that the electronic device can determine the direction parameter corresponding to the viewpoint based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0121] Optionally, the electronic device may receive the user's viewpoint sent by other devices. For example, the electronic device may receive the user's viewpoint sent by a virtual reality device (represented based on a direction parameter). Optionally, the electronic device may predict the user's viewpoint. For example, the electronic device may receive the user's viewpoint sent by a virtual reality device, and based on the viewpoint, predict the user's viewpoint within a future period of time.

[0122] Optionally, the electronic device may also obtain the user's viewpoint based on any other feasible implementation manner, which is not limited in the embodiments of the present disclosure.

[0123] S902: Rotate the panoramic image based on the user's viewpoint to obtain a rotated image.

[0124] The rotated image may be an image obtained by rotating the panoramic image by the electronic device based on the user's viewpoint. For example, the electronic device may determine a rotation matrix based on the user's viewpoint, and rotate the panoramic image based on the rotation matrix to obtain a rotated image. For example, the electronic device may process each pixel in the panoramic image based on the rotation matrix to obtain a rotated image.

[0125] Optionally, the user's viewpoint may be at the center of the rotated image. For example, after the electronic device rotates the panoramic image based on the rotation matrix corresponding to the user's viewpoint, a rotated image may be obtained, and the user's viewpoint may be located at the center of the rotated image. For example, for a panoramic image obtained by photographing an object from the front, if the user's viewpoint is located on the upper side of the panoramic image, the electronic device may rotate the panoramic image to obtain a rotated image, wherein the rotated image may be a panoramic image photographed from the upper side of the object, and the center of the panoramic image may be the user's viewpoint.

[0126] It should be noted that the user's viewpoint may also be a point in the central area of ​​the rotating image, or may be any point of the rotating image, which is not limited in the embodiments of the present disclosure.

[0127] Optionally, the electronic device may determine a rotation matrix based on the user's viewpoint, and rotate the panoramic image based on the rotation matrix to obtain a rotated image. Optionally, the electronic device may determine the rotation matrix based on the following formula:

[0128]

[0129] Among them, R v is a rotation matrix; α may be the horizontal field of view angle of the user's viewpoint (i.e., the horizontal field of view angle in FOV), β may be the vertical field of view angle of the user's viewpoint (i.e., the vertical field of view angle in FOV), and γ may be the diagonal field of view angle of the user's viewpoint (i.e., the diagonal field of view angle in FOV).

[0130] Optionally, the electronic device may process each pixel in the panoramic video based on the rotation matrix to obtain a rotated image. Since the rotated image is obtained based on the rotation matrix determined based on the user's viewpoint, the user's viewpoint may be located at the center of the rotated image.

[0131] Optionally, the electronic device rotates the panoramic image based on the user's viewpoint, and after obtaining the rotated image, it can also determine the second image in the rotated image. The method for the electronic device to determine the second image in the rotated image can specifically be: with the user's viewpoint as the center, determine the image covered by the user's field of view in the rotated image, and add images of a preset range at each edge of the image covered by the user's field of view to obtain the second image.

[0132] Optionally, since the user's viewpoint is located at the center of the rotating image, the electronic device determines the image covered by the user's field of view in the rotating image with the viewpoint as the center, wherein the user's field of view can be in any viewing angle range, which is not limited in the embodiments of the present disclosure. For example, if the user's field of view is 90 degrees * 100 degrees, the electronic device can determine a 90 degree * 100 degree area in the rotating image with the viewpoint as the center, thereby obtaining the image covered by the user's field of view.

[0133] Optionally, in order to avoid the user being unable to see the new image when turning his head slightly, the electronic device can determine the second image based on the image covered by the user's field of view. For example, the electronic device can add images of a preset range to each edge of the image covered by the user's field of view, and thus obtain the second image. For example, the electronic device can add an image of a 20-degree range to the upper edge of the image covered by the field of view, add an image of a 20-degree range to the lower edge of the image covered by the field of view, add an image of a 20-degree range to the left edge of the image covered by the field of view, and add an image of a 20-degree range to the right edge of the target area, and thus obtain the second image. In this way, when the user turns his head slightly, although he leaves the image covered by the current field of view, he can still see related images, thereby improving the user experience.

[0134] It should be noted that the sizes of the images added by the electronic device at the edges of the image covered by the field of view may be the same or different, and this is not limited in the embodiments of the present disclosure.

[0135] Next, combine Fig.10 , the process of determining the second image is described.

[0136] Fig.10 A schematic diagram of a process for determining a second image provided by an embodiment of the present disclosure. Fig.10 , including a rotating image. Wherein, the viewpoint is located at the center of the rotating image. Based on the user's field of view, the image covered by the user's field of view is determined in the rotating image with the viewpoint as the center, wherein the image covered by the user's field of view is located in the central area of ​​the rotating image. An image in a preset range is added to the upper edge, lower edge, left edge and right edge of the image covered by the user's field of view, thereby obtaining a second image. In this way, the second image can include the image covered by the user's field of view, and the user can see the corresponding picture by slightly rotating his head, thereby improving the user experience and saving network resources.

[0137] Next, combine Fig.11 , the rotation of the image is explained.

[0138] Fig.11 A schematic diagram of a rotating image provided by an embodiment of the present disclosure. Fig.11, including: panoramic images. Among them, electronic equipment ( Fig.11 (not shown) can determine that the second image is located on the left side of the panoramic image. The electronic device can determine the rotation matrix based on the direction parameters of the viewpoint in the second image, and based on the rotation matrix, process each pixel in the panoramic image to obtain a rotated image. Among them, the second image is located in the center of the rotated image, and the center point of the second image is located at the center of the rotated image. In this way, after the panoramic image is rotated, the rotated image is projected onto the surface of the three-dimensional sphere, and the center point of the second image can be located at the equator of the three-dimensional sphere, that is, the user's viewpoint is located at the equator of the three-dimensional sphere, thereby increasing the pixel density of the second image.

[0139] Optionally, after the electronic device determines the second image, since the maximum range of the second image can be 180 degrees * 180 degrees, the electronic device can project other images other than the second image into a plane based on an equiangular projection method, and splice them with the corresponding samples of the second image. In this way, when the user's perspective switches to the second image, the corresponding picture can also be seen, thereby improving the user experience.

[0140] Optionally, when performing equi-angle projection on other images other than the second image of the first image, the number of sampling points may be reduced, thereby reducing the image resolution of other images after being projected onto the plane, thereby saving network transmission resources.

[0141] Optionally, the electronic device can determine a first arc based on the upper left vertex and the lower right vertex of the second image, and determine a second arc based on the lower left vertex and the upper right vertex of the second image, and then divide the image in the first image except the second image into four images to be projected based on the first arc and the second arc, and project the four images to be projected onto a plane based on an equi-angle projection method to obtain four plane images, and splice the four plane images onto the sampled image corresponding to the second image, and the server can send the sampled image of the four plane images to the virtual reality device.

[0142] For example, based on the first arc and the second arc, the electronic device can obtain four images to be projected, wherein the four images to be projected are respectively located on the upper side, the lower side, the left side and the right side of the second image. The electronic device can generate an image of the area above the second image, an image of the area below the second image, an image of the area on the left side of the second image and an image of the area on the right side of the second image based on an equi-angle projection method, and splice the image of the area above the second image to the upper side of the image after sampling corresponding to the second image, splice the image of the area below the second image to the lower side of the image after sampling corresponding to the second image, splice the image of the area on the left side of the second image to the left side of the image after sampling corresponding to the second image, and splice the image of the area on the right side of the second image to the right side of the image after sampling corresponding to the second image, to obtain the image to be transmitted.

[0143] Next, combine Fig.12 , a process of determining an image to be transmitted is described.

[0144] Fig.12 A schematic diagram of determining an image to be transmitted provided by an embodiment of the present disclosure. Fig.12 , including images to be transmitted. The images to be transmitted may include a target image, image 1, image 2, image 3, and image 4. Image 1 is an image on the upper side of the second image in the first image, and is projected onto a plane based on an equiangular projection method; image 2 is an image on the right side of the second image in the first image, and is projected onto a plane based on an equiangular projection method; image 3 is an image on the lower side of the second image in the first image, and is projected onto a plane based on an equiangular projection method; image 4 is an image on the left side of the second image in the first image, and is projected onto a plane based on an equiangular projection method.

[0145] S903: Project the rotated image onto a spherical surface based on perspective projection to obtain a first image.

[0146] Optionally, the electronic device may project the rotating image onto a spherical surface based on perspective projection to obtain the first image. For example, the electronic device may project the rotating image onto the surface of a sphere based on perspective projection to obtain the first image. In this way, after the rotating image is projected onto the spherical surface based on perspective projection, the shape of the image covered by the user's field of view is similar to the shape of the second image, thereby ensuring that the user can see the effective picture even when moving his head, thereby improving the user experience. For example, Fig.10 In the embodiment shown, based on the perspective projection method, Fig.10 The rotating image in the image is projected onto a spherical surface, and the shape of the image covered by the user's field of view is similar to the shape of the image in a preset range, thereby improving the user experience.

[0147] The disclosed embodiment provides a method for determining a first image, obtaining a panoramic image corresponding to the first image and a user's viewpoint, rotating the panoramic image based on the user's viewpoint to obtain a rotated image, and projecting the rotated image onto a spherical surface based on a perspective projection method to obtain the first image. In this way, the center of the second image in the first image can be located at the equator of the first image, and the pixel density of the second image is relatively large, thereby improving the accuracy of determining the image after sampling and saving network resources.

[0148] Based on any of the above embodiments, Fig.13 , the process of the above-mentioned image processing method is explained.

[0149] Fig.13 A schematic diagram of an image processing method provided by an embodiment of the present disclosure. Fig.13 , including: a panoramic image, a server and a virtual reality device. The panoramic image includes a second image, and the center of the second image is located at the center of the panoramic image. The server can project the panoramic image onto the surface of a sphere based on perspective projection to obtain a first image. The first image can include the second image, and the center of the second image is located at the equator of the first image.

[0150] See also Fig.13 , the server can divide the second image in the first image into 9 sub-images based on an equal-angle division method (the division process of the second image can refer to the above embodiment, and the embodiment of the present disclosure will not be repeated here), and project the pixels of the 9 sub-images to 9 positions in the image after sampling to be generated, so as to obtain the image after sampling. Among them, the image after sampling may include 9 pixels (exemplary illustration), the first pixel is determined based on the pixel in sub-image 1 of the second image, the second pixel is determined based on the pixel in sub-image 2 of the second image, ..., the ninth pixel is determined based on the pixel in sub-image 9 of the second image.

[0151] See also Fig.13 After the server determines the sampled image, it can send the sampled image to the virtual reality device, and the virtual reality device can play the sampled image to present a three-dimensional effect to the user and improve the user experience. In this way, the center of the second image in the first image can be located on the equator in the first image, and the pixel density of the second image is relatively high. In addition, since the multiple sub-images are graphics obtained by dividing the second image at equal angles, the electronic device can uniformly sample the pixels in the second image to obtain the sampled image, thereby improving the accuracy of the sampled image and saving network resources.

[0152] Fig.14This is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. Fig.14 The image processing device 140 includes a first determining module 141, a second determining module 142, a dividing module 143 and a sampling module 144, wherein:

[0153] The first determination module 141 is used to determine a first image, where the first image is an image obtained by projecting a panoramic image onto a spherical surface, the first image includes a second image, and the second image includes an image covered by the user's field of view in the first image;

[0154] The second determination module 142 is used to determine a first horizontal angle of the second image relative to the sphere center corresponding to the first image and a second vertical angle of the second image relative to the sphere center;

[0155] The division module 143 is used to divide the second image domain into equal angles based on the first angle and the second angle to obtain M*N sub-images, where M and N are positive integers determined based on a preset resolution;

[0156] The sampling module 144 is used to perform pixel sampling in the M*N sub-images to obtain a sampled image.

[0157] According to one or more embodiments of the present disclosure, the division module 143 is specifically used to:

[0158] Divide the first angle into M first bisected angles on average, and divide the second angle into N second bisected angles on average;

[0159] Based on the M first bisecting angles and the N second bisecting angles, the second image is divided into equal angles to obtain M*N sub-images.

[0160] According to one or more embodiments of the present disclosure, the division module 143 is specifically used to:

[0161] Determine M+1 vertical arcs associated with the M first bisecting angles in the second image;

[0162] Determine N+1 arcs in the horizontal direction associated with the N second bisecting angles in the second image;

[0163] Based on the M+1 vertical arcs and the N+1 horizontal arcs, the second image is divided into M*N sub-images.

[0164] According to one or more embodiments of the present disclosure, the sampling module 144 is specifically used for:

[0165] For any target position in the image after sampling;

[0166] Determine a sub-image corresponding to the target position in the second image;

[0167] One or more pixels in the sub-image are acquired, and a fusion process is performed on the one or more pixels to obtain the pixel at the target position in the sampled image.

[0168] According to one or more embodiments of the present disclosure, the second determining module 142 is specifically configured to:

[0169] Determine the included angle between the midpoints of the two arcs in the vertical direction of the second image and the line connecting the center of the sphere as the first horizontal opening angle;

[0170] The included angle between the midpoints of the two arcs in the horizontal direction of the second image and the line connecting the center of the sphere is determined as the second vertically opened angle.

[0171] According to one or more embodiments of the present disclosure, the first determining module 141 is specifically configured to:

[0172] Acquire a panoramic image corresponding to the first image and a user's viewpoint;

[0173] Rotating the panoramic image based on the user's viewpoint to obtain a rotated image, wherein the user's viewpoint is at the center of the rotated image;

[0174] Based on the perspective projection method, the rotated image is projected onto a spherical surface to obtain the first image.

[0175] According to one or more embodiments of the present disclosure, the first determining module 141 is specifically configured to:

[0176] Taking the user's viewpoint as the center, determining an image covered by the user's field of view in the rotated image;

[0177] An image in a preset range is added to each edge of the image covered by the user's field of view to obtain the second image.

[0178] The image processing device provided in the embodiment of the present disclosure may be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0179] Fig.15 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.15, which shows a schematic diagram of the structure of an electronic device 1500 suitable for implementing the embodiment of the present disclosure. The electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.15 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0180] like Fig.15 As shown, the electronic device 1500 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage device 1508 to a random access memory (RAM) 1503. Various programs and data required for the operation of the electronic device 1500 are also stored in the RAM 1503. The processing device 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0181] Typically, the following devices may be connected to the I / O interface 1505: input devices 1506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1509. The communication devices 1509 may allow the electronic device 1500 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.15 The electronic device 1500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0182] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1509, or installed from a storage device 1508, or installed from a ROM 1502. When the computer program is executed by the processing device 1501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0183] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0184] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0185] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0186] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the image processing method that may be involved in various embodiments as described above is implemented.

[0187] An embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the image processing method that may be involved in various embodiments as described above is implemented.

[0188] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0189] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0191] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0192] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0193] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0194] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0195] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0196] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server or storage medium that performs the operation of the technical solution of the present disclosure based on the prompt message. As an optional but non-limiting implementation method, in response to receiving an active request from a user, the method of sending a prompt message to the user can be, for example, a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0197] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0198] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws and regulations. The data may include information, parameters and messages, such as flow switching indication information.

[0199] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0200] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0201] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Determine a first image, where the first image is an image obtained by projecting the panoramic image onto a spherical surface, the first image includes a second image, and the second image includes an image covered by the user's field of view in the first image; Determine a first horizontal angle of the second image relative to a sphere center corresponding to the first image and a second vertical angle of the second image relative to the sphere center; Based on the first angle and the second angle, the second image domain is divided into equal angles to obtain M*N sub-images, where M and N are positive integers determined based on a preset resolution; Pixel sampling is performed in the M*N sub-images to obtain a sampled image.

2. The method according to claim 1, characterized in that The step of dividing the second image at equal angles based on the first angle and the second angle to obtain M*N sub-images includes: Divide the first angle equally into M first bisected angles, and divide the second angle equally into N second bisected angles; Based on the M first bisecting angles and the N second bisecting angles, the second image is divided into equal angles to obtain M*N sub-images.

3. The method according to claim 2, characterized in that The step of dividing the second image at equal angles based on the M first bisecting angles and the N second bisecting angles to obtain M*N sub-images includes: Determine M+1 vertical arcs associated with the M first bisecting angles in the second image; Determine N+1 arcs in the horizontal direction associated with the N second bisecting angles in the second image; Based on the M+1 vertical arcs and the N+1 horizontal arcs, the second image is divided into M*N sub-images.

4. The method according to any one of claims 1 to 3, characterized in that: The step of sampling pixels in the M*N sub-images of the second image to obtain the sampled image includes: For any target position in the image after sampling; Determine a sub-image corresponding to the target position in the second image; One or more pixels in the sub-image are acquired, and a fusion process is performed on the one or more pixels to obtain the pixel at the target position in the sampled image.

5. The method according to any one of claims 1 to 3, characterized in that: The determining of a first horizontal angle of the second image relative to a sphere center corresponding to the first image and a second vertical angle of the second image relative to the sphere center includes: Determine the included angle between the midpoints of the two arcs in the vertical direction of the second image and the line connecting the center of the sphere as the first horizontal opening angle; The included angle between the midpoints of the two arcs of the second image in the horizontal direction and the line connecting the center of the sphere is determined as the second vertically opened angle.

6. The method according to any one of claims 1 to 3, characterized in that: The determining of the first image comprises: Acquire a panoramic image corresponding to the first image and a user's viewpoint; Based on the viewpoint of the user, the panoramic image is rotated to obtain a rotated image, wherein the viewpoint of the user is at the center of the rotated image; Based on a perspective projection method, the rotated image is projected onto a spherical surface to obtain the first image.

7. The method according to claim 6, characterized in that After rotating the panoramic image based on the user's viewpoint to obtain a rotated image, the method further includes: Taking the user's viewpoint as the center, determining an image covered by the user's field of view in the rotated image; An image in a preset range is added to each edge of the image covered by the user's field of view to obtain the second image.

8. An image processing device, characterized in that: It includes a first determination module, a second determination module, a division module and a sampling module, wherein: The first determination module is used to determine a first image, where the first image is an image obtained by projecting a panoramic image onto a spherical surface, the first image includes a second image, and the second image includes an image covered by the user's field of view in the first image; The second determination module is used to determine a first horizontal angle of the second image relative to a sphere center corresponding to the first image and a second vertical angle of the second image relative to the sphere center; The division module is used to divide the second image domain into equal angles based on the first angle and the second angle to obtain M*N sub-images, where M and N are positive integers determined based on a preset resolution; The sampling module is used to perform pixel sampling in the M*N sub-images to obtain a sampled image.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the image processing method according to any one of claims 1 to 7 is implemented.

Images