Panoramic image processing method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides a panoramic image processing method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a viewpoint coordinate corresponding to a panoramic image to be displayed, and determining a view coverage area in the panoramic image according to the viewpoint coordinate; a preset angle is obtained, a sampling area in the panoramic image is generated according to the preset angle and the view coverage area, and the similarity between the shape of the sampling area and the shape of the view coverage area meets a preset threshold value; and acquiring pixels in the sampling area according to a target resolution to obtain a target image corresponding to the panoramic image. According to the technical scheme, the sampling area is determined based on the viewpoint coordinates, the sampling area is sampled according to the target resolution, the target image meeting the watching expectation can be obtained, the target image replaces the panoramic image for transmission, and bandwidth consumption is reduced.

Description

Technical Field

[0001] This disclosure relates to computer technology, and more particularly to a method, apparatus, device, medium, and program product for processing panoramic images. Background Technology

[0002] Currently, the development of immersive multimedia technology has driven the popularization of panoramic images. For example, users can watch panoramic videos through head-mounted devices, including movies, short videos, documentaries, or games. Since the data volume of panoramic videos is far greater than that of traditional videos, directly transmitting the entire panoramic video would result in a significant waste of bandwidth. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, medium, and program product for processing panoramic images, which can reduce bandwidth consumption.

[0004] In a first aspect, embodiments of this disclosure provide a method for processing panoramic images, including:

[0005] Obtain the viewpoint coordinates corresponding to the panoramic image to be displayed, and determine the field of view coverage area in the panoramic image based on the viewpoint coordinates, wherein the viewpoint coordinates represent the position of the gaze point in the panoramic image;

[0006] A preset angle is obtained, and a sampling region in the panoramic image is generated based on the preset angle and the field of view coverage area. The similarity between the shape of the sampling region and the shape of the field of view coverage area satisfies a preset threshold.

[0007] The pixels in the sampling area are obtained according to the target resolution to obtain the target image corresponding to the panoramic image, wherein the target resolution is the panoramic image resolution in the client.

[0008] Secondly, embodiments of this disclosure also provide a panoramic image processing apparatus, the apparatus comprising:

[0009] The coordinate acquisition module is used to acquire the viewpoint coordinates corresponding to the panoramic image to be displayed, and to determine the field of view coverage area in the panoramic image based on the viewpoint coordinates, wherein the viewpoint coordinates represent the position of the gaze point in the panoramic image.

[0010] The region generation module is used to obtain a preset angle and generate a sampling region in the panoramic image based on the preset angle and the field of view coverage area. The similarity between the shape of the sampling region and the shape of the field of view coverage area satisfies a preset threshold.

[0011] The image generation module is used to obtain pixels in the sampling area according to the target resolution to obtain the target image corresponding to the panoramic image, wherein the target resolution is the panoramic image resolution in the client.

[0012] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0013] One or more processors;

[0014] Storage device for storing one or more programs.

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic image processing method as described in any embodiment of this disclosure.

[0016] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the panoramic image processing method as described in any embodiment of this disclosure.

[0017] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the panoramic image processing method as described in any embodiment of this disclosure.

[0018] This disclosure provides a method for processing panoramic images. Based on the viewpoint coordinates of the panoramic image to be displayed, the method determines the field of view coverage area in the panoramic image. Then, based on the field of view coverage area and a preset angle, it generates a sampling area with a similar shape but larger area, avoiding artifacts in the target image due to insufficient coverage of the sampling area. Next, pixels in the sampling area are obtained according to the resolution of the panoramic image on the client side. The obtained pixels are then used to construct the target image, reducing data transmission volume. This disclosure, by determining the sampling area based on viewpoint coordinates and sampling the area according to the target resolution, can obtain a target image that meets the user's viewing expectations. This target image is then transmitted instead of the panoramic image, reducing bandwidth consumption. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A schematic flowchart illustrating a panoramic image processing method provided in an embodiment of this disclosure;

[0021] Figure 2 This is a schematic diagram of a sampling area provided in an embodiment of the present disclosure;

[0022] Figure 3 A schematic flowchart illustrating another panoramic image processing method provided in this embodiment of the present disclosure;

[0023] Figure 4 This is a schematic flowchart illustrating yet another panoramic image processing method provided in an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a panoramic image processing device provided in an embodiment of this disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0032] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0033] Figure 1 This is a flowchart illustrating a panoramic image processing method provided in an embodiment of the present disclosure. This embodiment is applicable to scenarios involving the transmission of panoramic images, such as the efficient transmission of panoramic video or panoramic images. The method can be executed by a panoramic image processing device, which can be implemented in software and / or hardware, optionally through an electronic device, such as a mobile terminal, PC, or server.

[0034] like Figure 1 As shown, the method includes:

[0035] S110. Obtain the viewpoint coordinates corresponding to the panoramic image to be displayed, and determine the field of view coverage area in the panoramic image based on the viewpoint coordinates, wherein the viewpoint coordinates represent the position of the gaze point in the panoramic image.

[0036] A panoramic image can be a spherical image obtained by capturing the real world in 360 degrees using professional camera equipment. For example, panoramic images include video frames from panoramic videos or panoramic photographs. Panoramic images can be displayed using panoramic image playback devices; the panoramic image to be displayed can be selected as the panoramic image to be shown on the device.

[0037] When creating panoramic video, multiple cameras at different angles are typically used to record real-world visual scenes. Images captured by multiple cameras at the same time are stitched together, using techniques such as feature point matching or feature point fusion to reconstruct the real-world visual field. The stitched images are then projected onto a sphere to obtain panoramic video frames. Combining these panoramic video frames according to their timestamps yields a 360-degree video, also known as spherical panoramic video. After projection and mapping, the panoramic video frames are encapsulated into a two-dimensional planar data frame before video encoding, resulting in an encoded video stream for transmission.

[0038] Since viewpoint coordinates represent the position of the gaze point in a panoramic image, viewpoint coordinates can be represented as (yaw, pitch, roll). Here, yaw represents the angle of the head-mounted device's left-right (corresponding to longitude) offset, pitch represents the angle of the head-mounted device's up-down (latitude) offset, and roll represents the tilt angle of the head-mounted device.

[0039] The field of view coverage area refers to the area of ​​the image that can be observed using the viewpoint coordinates as the observation reference. For example, when looking at the center of a panoramic image, the field of view includes the vertical field of view (θ1, θ2) and the horizontal field of view (Φ1, Φ2). Therefore, the field of view coverage area can be defined as the area extending upwards by θ1 degrees, downwards by θ2 degrees, to the left by Φ1 degrees, and to the right by Φ2 degrees, starting from the center of the panoramic image. The endpoints of these movements are defined by the latitude and longitude lines on the panoramic image.

[0040] For example, determining the field of view coverage area in the panoramic image based on the viewpoint coordinates includes: obtaining the field of view angle corresponding to the client, wherein the field of view angle represents the field of view angle range of the client device corresponding to the client. The field of view coverage area in the panoramic image is determined based on the viewpoint coordinates and the field of view angle. The client is used to display the target image corresponding to the panoramic image.

[0041] Specifically, starting from the gaze point position represented by the viewpoint coordinates, the first endpoint is obtained by moving upwards by θ1 degrees on the panoramic image (which can be visualized as a sphere). The second endpoint is obtained by moving downwards by θ2 degrees from the starting point. The third endpoint is obtained by moving left by Φ1 degrees from the starting point, and the fourth endpoint is obtained by moving right by Φ2 degrees from the starting point. Then, combining the latitude lines corresponding to the first and second endpoints on the panoramic image, and the longitude lines corresponding to the third and fourth endpoints, the field of view coverage area is determined. It should be noted that on the panoramic image, vertical movement can be considered as movement towards the North or South Pole, and horizontal movement can be considered as movement towards the west or east. The closed area enclosed by the latitude and longitude lines corresponding to each endpoint on the panoramic image is considered the field of view coverage area. It should be noted that (θ1, θ2) and (Φ1, Φ2) are time angles representing the angular range of the client device's field of view. The angular range of the client device's field of view limits the range of the panoramic image that the user can view.

[0042] S120. Obtain a preset angle, and generate a sampling region in the panoramic image based on the preset angle and the field of view coverage area. The similarity between the shape of the sampling region and the shape of the field of view coverage area satisfies a preset threshold.

[0043] The preset threshold represents the similarity threshold. The similarity threshold can be set according to the application scenario.

[0044] A preset angle is used to expand the area covered by the field of view. The preset angle includes a preset pitch angle and a preset yaw angle. To determine a suitable preset angle, a head motion trajectory dataset can be constructed, and candidate angles required for the head motion speed (i.e., the viewpoint motion speed) under a set delay can be calculated based on the head motion trajectory dataset. The impact of candidate angles at different head motion speeds on the viewport can be tested to obtain the preset angle. This method can effectively reduce visual artifacts and maintain good pixel density.

[0045] In this embodiment, the preset angle represents the angle at which the field of view expands outwards. If the preset angle is too large, it will reduce the pixel density within the viewport, affecting image quality. If the preset angle is too small, the sampling area will not completely cover the viewport, causing the user to see artifacts. Here, the viewport refers to the panoramic image area that can be displayed on the display screen of the client device.

[0046] Since panoramic images can be presented in the form of spherical images, they can be seen as image textures pasted on the surface of a sphere. Therefore, a panoramic image can be seen as the surface of a sphere with image textures pasted on it.

[0047] The sampling area represents a spherical region obtained by extending the field of view coverage area outwards from a preset angle. The sampling area can cover the user's viewport and its surrounding area.

[0048] In this embodiment of the disclosure, generating a sampling region in the panoramic image based on the preset angle and the field of view coverage area includes:

[0049] The latitude coverage angle of the field of view is increased by the preset pitch angle to obtain the target latitude coverage angle. The longitude coverage angle of the field of view is increased by the preset yaw angle to obtain the target longitude coverage angle. Based on the target latitude and longitude coverage angles, the field of view is projected to obtain the sampling area in the panoramic image. The area between the field of view and the sampling area is called the margin. The similarity between the shape of the sampling area and the shape of the field of view is determined, and the similarity exceeds a preset similarity threshold.

[0050] Among them, projection processing can be perspective projection processing. Perspective projection processing is a central projection method used to form a projection area covering the field of view on the sphere of the panoramic image. In this method, the difference in latitude between points of the same longitude and the boundary of the field of view in the panoramic image is equal to the pitch angle, and the difference in longitude between points of the same latitude is equal to the yaw angle.

[0051] For example, the preset angles include a pitch angle of 4 degrees and a yaw angle of 8 degrees. The field of view coverage area is represented as the panoramic image region covered by longitudes from 45 degrees east to 45 degrees west and latitudes from 35 degrees north to 35 degrees south. After projecting the field of view coverage area based on the preset angles, the target longitude coverage angle is from 53 degrees east to 53 degrees west, and the target latitude coverage angle is from 39 degrees north to 39 degrees south. Then, within the projection area formed by the meridians of 53 degrees east and 53 degrees west, and the parallels of 39 degrees north and 39 degrees south in the panoramic image, perspective projection is performed on the field of view coverage area to obtain a sampling area with the same shape but a larger area as the field of view coverage area. Figure 2 This is a schematic diagram of a sampling area provided in an embodiment of this disclosure. Figure 2 As shown, a field of view coverage area 210 and a sampling area 220 are presented on the panoramic image 200, wherein the area between the field of view coverage area 210 and the sampling area 220 represents the margin 230. The similarity of the shapes of the field of view coverage area 210 and the sampling area 220 satisfies a preset threshold.

[0052] It's important to note that the preset angle plays a crucial role in balancing transmission efficiency and user experience. Choosing a suitable preset angle better accommodates user head movements. To determine the appropriate preset angle, a quantitative analysis can be performed on a dataset consisting of head motion trajectories observed during panoramic video viewing. Head motion speed is calculated based on these trajectories, and candidate angles representing slow, moderate, and fast head movements are selected based on the distribution of these speeds. Then, the impact of different candidate angles on the viewport is compared to obtain performance scores for each angle. The preset angle is then selected from the candidate angles based on these performance scores, effectively reducing visual artifacts in the viewport while maintaining good pixel density.

[0053] S130. Obtain the pixels in the sampling area according to the target resolution to obtain the target image corresponding to the panoramic image, wherein the target resolution is the panoramic image resolution in the client.

[0054] Since traditional encoding techniques cannot encode panoramic images, they need to be projected into a two-dimensional image before encoding and transmission. The target image represents the two-dimensional planar image of the area viewed by the user in the panoramic image after projection and encoding. Specifically, the target image is a two-dimensional planar image suitable for display on the client side.

[0055] The sampling representation describes the operation of acquiring pixels from a sampling region in a panoramic image according to a set sampling method. To achieve uniform sampling and ensure the uniformity of pixel distribution in the target image, the sampling method can be set to equal-angle sampling. This disclosure does not specifically limit the sampling method; random sampling or layered sampling can also be used to sample the sampling region.

[0056] For example, based on the target resolution, the number of pixels in the width direction and the number of pixels in the height direction of the target image are determined. Based on the number of pixels in the width direction and the number of pixels in the height direction, pixels in the sampling area are sampled to obtain the target image corresponding to the panoramic image, wherein the angle difference between any two adjacent pixels in the width direction and the angle difference between any two adjacent pixels in the height direction of the target image are equal.

[0057] Assume the coverage angle of the sampling area is (F) w F h Let M*N be the number of pixels in the width and height directions of the target image. Then, the correspondence between pixels in the sampling area of ​​the panoramic image and sampling points (m,n) in the target image is expressed as: based on a coverage angle of (F... w F h The system uses a first matrix to determine the width and height, a second matrix to determine the number of pixels in the width and height directions, and a third matrix to determine the sampling point (m,n). Matrix multiplication of the first, second, and third matrices is then performed to obtain the coordinates [φ, θ', 1] of the pixels within the sampling area of ​​the panoramic image. Here, φ represents the longitude of the sampled pixel in the panoramic image, and θ' represents the angle between the sampled pixel and the equator in the panoramic image.

[0058] Further, sampling the pixels in the sampling region based on the number of pixels in the width direction and the number of pixels in the height direction to obtain the target image corresponding to the panoramic image includes: determining the width sampling angle and the height sampling angle of the sampling region based on the number of pixels in the width direction, the number of pixels in the height direction, and the resolution of the sampling region; and acquiring the pixels in the sampling region based on the width sampling angle and the height sampling angle to obtain the target image corresponding to the panoramic image.

[0059] For example, suppose the sampling area represents the spherical region covered by 60 degrees east longitude, 60 degrees west longitude, 40 degrees north latitude, and 40 degrees south latitude in the panoramic image, and the target image has a resolution of 100*50. Using equal-angle sampling, one pixel is sampled every 1.2 degrees along the longitude direction and one pixel is sampled every 1.6 degrees along the latitude direction in the panoramic image to obtain the target image.

[0060] Optionally, pixels in the sampling area can be obtained based on the image size or pixel density of the client device, and this disclosure does not specifically limit this. The image size can be represented by the width and height of the image, i.e., the width and height of the canvas carrying the image information.

[0061] The technical solution of this disclosure determines the field of view coverage area in the panoramic image based on the viewpoint coordinates corresponding to the panoramic image to be displayed. Then, it generates a sampling area with a similar shape but larger area to the field of view coverage area based on the field of view coverage area and a preset angle, avoiding artifacts in the target image due to insufficient coverage of the sampling area. Then, it obtains pixels in the sampling area according to the resolution of the panoramic image on the client, and constructs the target image based on the obtained pixels, thereby reducing data transmission volume. This disclosure, by determining the sampling area based on viewpoint coordinates and sampling the sampling area according to the target resolution, can obtain a target image that meets the user's viewing expectations. This target image is then transmitted instead of the panoramic image, reducing bandwidth consumption.

[0062] Figure 3 This is a flowchart illustrating another panoramic image processing method provided by an embodiment of the present disclosure. Based on the above embodiments, this embodiment adds a panoramic image rotation step that limits the field of view coverage area to not satisfy a preset position condition.

[0063] like Figure 3 As shown, the method includes:

[0064] S310. Obtain the viewpoint coordinates corresponding to the panoramic image to be displayed, and determine the field of view coverage area in the panoramic image based on the viewpoint coordinates, wherein the viewpoint coordinates represent the position of the gaze point in the panoramic image.

[0065] S320. If the position of the field of view coverage area does not meet the preset position conditions, then a rotation matrix is ​​determined according to the viewpoint coordinates, wherein the preset position conditions indicate that the center coordinates of the field of view coverage area coincide with the center coordinates of the panoramic image of the panoramic image.

[0066] Because the central region (or equatorial region) of a panoramic image has a high and uniform pixel density, the preset position condition can be set so that the center coordinates of the field of view coverage area coincide with the center coordinates of the panoramic image. This design increases the pixel density of the rotated field of view coverage area, thereby presenting better image quality. Uniform pixel distribution can avoid image distortion.

[0067] For example, based on the positional relationship between the central region of the panoramic image and the field of view coverage area, it is determined whether the position of the field of view coverage area meets a preset positional condition. For instance, if the field of view coverage area coincides with the central region, it is determined that the position of the field of view coverage area meets the preset positional condition. Alternatively, when the field of view coverage area is located within the central region, it is determined whether the position of the field of view coverage area meets the preset positional condition. It is understood that there may be many positional relationships that meet the preset positional condition, and this disclosure does not specifically limit them.

[0068] The central region of the panoramic image can be understood as the area that expands outwards from the center of the panoramic image (the intersection of the 0-degree longitude and the 0-degree latitude) according to the field of view.

[0069] It should be noted that if the location of the field of view coverage area determined by the viewpoint coordinates meets the preset location conditions, that is, if the user's current viewing direction is the equator, then there is no need to rotate the panoramic image.

[0070] The rotation matrix is ​​used to rotate the panoramic image. The rotation matrix can be determined based on the current viewpoint coordinates. For example, it can be determined based on the sine and cosine values ​​of yaw, pitch, and roll.

[0071] S330. The panoramic image is rotated using the rotation matrix so that the center coordinates of the field of view coverage area in the rotated panoramic image coincide with the center coordinates of the panoramic image.

[0072] For example, the positions of pixels in the panoramic image are updated according to the rotation matrix to obtain a rotated panoramic image. For instance, the rotation matrix is ​​multiplied by the pixels in the panoramic image to update the pixel positions, resulting in a rotated panoramic image. In the rotated panoramic image, the field of view coverage area is located within the central region of the panoramic image. In this case, the field of view coverage area in the rotated panoramic image can be represented as a region on a sphere extending outwards from the intersection of the 0° longitude and 0° latitude lines, based on the field of view angle.

[0073] S340. Obtain a preset angle, and generate a sampling region in the panoramic image based on the preset angle and the field of view coverage area, wherein the similarity between the shape of the sampling region and the shape of the field of view coverage area satisfies a preset threshold.

[0074] S350. Obtain the pixels in the sampling area according to the target resolution to obtain the target image corresponding to the panoramic image, wherein the target resolution is the panoramic image resolution in the client.

[0075] Figure 4 This is a schematic flowchart illustrating yet another panoramic image processing method provided in an embodiment of this disclosure. Figure 4As shown, the method includes: determining the field-of-view coverage area 410 in the panoramic image 400 based on the viewpoint coordinates sent by the client. Figure 4 It is known that the current field of view coverage area 410 is not in the central region of the panoramic image 400, and the shape of the field of view coverage area 410 is irregular. The panoramic image 400 is rotated according to the viewpoint coordinates to align the equator with the user's current viewing direction, resulting in a second panoramic image 420. This places the field of view coverage area 410 in the central region of the second panoramic image 420, where it has a more regular shape and can be approximated as a rectangle. A spherical region with the same shape as the field of view coverage area 410 but a larger area, i.e., the sampling region 430, is generated using perspective projection. The area between the sampling region 430 and the field of view coverage area 410 is called the margin. A preset angle ensures that when the viewpoint moves slightly, the new field of view coverage area still falls within the sampling region 430; therefore, all pixels in the sampling region 430 are valid. Even if the field of view coverage area 410 moves to the edge of the sampling region 430, since the sampling region 430 has the same shape as the field of view coverage area 410, it can provide enough pixels to avoid visual artifacts. Starting from the center of sampling region 430, sampling is performed uniformly outwards from the four edges of sampling region 430 using an equal-angle sampling method. Since sampling region 430 has a uniform pixel density and a large number of pixels, equal-angle sampling of sampling region 430 yields the target image 440 (where x represents pixels), which can improve projection quality and mitigate artifacts or image distortion. Assume the coverage angle of the sampling region (F...) w F h The number of pixels in the width and height directions of the target image 440 is represented as M*N. The correspondence between pixels in the panoramic image and sampling points (m,n) is as follows:

[0076]

[0077] Then, a target image 440 with a width of M and a height of N is formed based on each sampling point. After encoding the target image 440, it is transmitted to the head-mounted device 450. Since the target image is obtained by sampling only the sampling area and then encoded and transmitted, the transmission efficiency of panoramic video can be improved and bandwidth consumption can be reduced.

[0078] The technical solution of this disclosure generates a rotation matrix using viewpoint coordinates, and rotates the panoramic image according to the rotation matrix to ensure that the position of the field of view coverage area of ​​the rotated panoramic image meets preset position conditions. Then, a sampling area with the same shape but larger area as the field of view coverage area is generated based on a preset angle and the field of view coverage area to avoid artifacts in the target image due to insufficient coverage. Next, the sampling area is generated according to the preset angle and the field of view coverage area, and the pixels in the sampling area are obtained according to the target resolution to obtain the target image. This disclosure ensures that the field of view coverage area is located in the central region of the panoramic image, where the pixel density is high, avoiding image distortion caused by low pixel density, thereby reducing the probability of image distortion in the target image and improving image quality.

[0079] Figure 5 This is a schematic diagram of a panoramic image processing device provided in an embodiment of the present disclosure. The device can be implemented in the form of software and / or hardware, and optionally, it can be implemented by an electronic device, such as a mobile terminal, a PC, or a server.

[0080] like Figure 5 As shown, the device includes: a coordinate acquisition module 510, a region generation module 520, and an image generation module 530.

[0081] The coordinate acquisition module 510 is used to acquire the viewpoint coordinates corresponding to the panoramic image to be displayed, and to determine the field of view coverage area in the panoramic image based on the viewpoint coordinates, wherein the viewpoint coordinates represent the position of the gaze point in the panoramic image.

[0082] The region generation module 520 is used to obtain a preset angle and generate a sampling region in the panoramic image based on the preset angle and the field of view coverage area. The similarity between the shape of the sampling region and the shape of the field of view coverage area satisfies a preset threshold.

[0083] The image generation module 530 is used to obtain pixels in the sampling area according to the target resolution to obtain the target image corresponding to the panoramic image, wherein the target resolution is the panoramic image resolution in the client.

[0084] Optionally, it also includes:

[0085] An image rotation module is used to determine a rotation matrix based on the viewpoint coordinates if the position of the field of view coverage area does not meet a preset position condition. The preset position condition indicates that the center coordinates of the field of view coverage area coincide with the center coordinates of the panoramic image. The panoramic image is rotated using the rotation matrix so that the center coordinates of the field of view coverage area in the rotated panoramic image coincide with the center coordinates of the panoramic image.

[0086] Further, rotating the panoramic image using the rotation matrix includes:

[0087] The positions of pixels in the panoramic image are updated according to the rotation matrix to obtain the rotated panoramic image.

[0088] Optionally, the coordinate acquisition module 510 is specifically used for:

[0089] Obtain the field of view angle corresponding to the client, wherein the field of view angle represents the field of view angle range of the client device corresponding to the client;

[0090] The field of view coverage area in the panoramic image is determined based on the viewpoint coordinates and the field of view angle.

[0091] Optionally, the preset angle is used to expand the area of ​​the field of view coverage area, and the preset angle includes a preset pitch angle and a preset yaw angle;

[0092] The region generation module 520 is specifically used for:

[0093] The target latitude coverage angle is obtained by increasing the latitude coverage angle of the field of view coverage area according to the preset pitch angle.

[0094] The target longitude coverage angle is obtained by increasing the longitude coverage angle of the field of view coverage area according to the preset yaw angle.

[0095] Based on the target latitude coverage angle and the target longitude coverage angle, the field of view coverage area is projected to obtain the sampling area in the panoramic image.

[0096] Optionally, the image generation module 530 is specifically used for:

[0097] Based on the target resolution, determine the number of pixels in the width direction and the number of pixels in the height direction of the target image;

[0098] Based on the number of pixels in the width direction and the number of pixels in the height direction, the pixels in the sampling area are sampled to obtain the target image corresponding to the panoramic image. In the target image, the angle difference between any two adjacent pixels in the width direction is equal, and the angle difference between any two adjacent pixels in the height direction is equal.

[0099] Further, the step of sampling pixels in the sampling region based on the number of pixels in the width direction and the number of pixels in the height direction to obtain the target image corresponding to the panoramic image includes:

[0100] The width sampling angle and height sampling angle of the sampling area are determined based on the number of pixels in the width direction, the number of pixels in the height direction, and the resolution of the sampling area.

[0101] Based on the width sampling angle and the height sampling angle, the pixels in the sampling area are obtained to obtain the target image corresponding to the panoramic image.

[0102] The panoramic image processing apparatus provided in this disclosure can execute the panoramic image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0103] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0104] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 6 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 6 The diagram below shows the structure of the terminal device or server 600. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0105] like Figure 6 As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An edit / output (I / O) interface 605 is also connected to bus 604.

[0106] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0107] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0108] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0109] The electronic device provided in this embodiment and the panoramic image processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0110] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the panoramic image processing method provided in the above embodiments.

[0111] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0112] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0113] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0114] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0115] Obtain the viewpoint coordinates corresponding to the panoramic image to be displayed, and determine the field of view coverage area in the panoramic image based on the viewpoint coordinates, wherein the viewpoint coordinates represent the position of the gaze point in the panoramic image;

[0116] A preset angle is obtained, and a sampling region in the panoramic image is generated based on the preset angle and the field of view coverage area. The similarity between the shape of the sampling region and the shape of the field of view coverage area satisfies a preset threshold.

[0117] The pixels in the sampling area are obtained according to the target resolution to obtain the target image corresponding to the panoramic image, wherein the target resolution is the panoramic image resolution in the client.

[0118] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0121] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0122] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0123] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0124] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0125] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for processing panoramic images, characterized in that, include: Obtain the viewpoint coordinates corresponding to the panoramic image to be displayed, and determine the field of view coverage area in the panoramic image based on the viewpoint coordinates, wherein the viewpoint coordinates represent the position of the gaze point in the panoramic image; A preset angle is obtained, and a sampling region in the panoramic image is generated based on the preset angle and the field of view coverage area. The similarity between the shape of the sampling region and the shape of the field of view coverage area satisfies a preset threshold. The pixels in the sampling area are obtained according to the target resolution to obtain the target image corresponding to the panoramic image, wherein the target resolution is the panoramic image resolution in the client.

2. The method according to claim 1, characterized in that, Also includes: If the position of the field of view coverage area does not meet the preset position conditions, then a rotation matrix is ​​determined according to the viewpoint coordinates, wherein the preset position conditions indicate that the center coordinates of the field of view coverage area coincide with the center coordinates of the panoramic image of the panoramic image. The panoramic image is rotated using the rotation matrix so that the center coordinates of the field of view coverage area in the rotated panoramic image coincide with the center coordinates of the panoramic image.

3. The method according to claim 2, characterized in that, The step of rotating the panoramic image using the rotation matrix includes: The positions of pixels in the panoramic image are updated according to the rotation matrix to obtain the rotated panoramic image.

4. The method according to claim 1, characterized in that, Determining the field of view coverage area in the panoramic image based on the viewpoint coordinates includes: Obtain the field of view angle corresponding to the client, wherein the field of view angle represents the field of view angle range of the client device corresponding to the client; The field of view coverage area in the panoramic image is determined based on the viewpoint coordinates and the field of view angle.

5. The method according to claim 1, characterized in that, in, The preset angle is used to expand the area of ​​the field of view coverage area, and the preset angle includes a preset pitch angle and a preset yaw angle; The step of generating the sampling region in the panoramic image based on the preset angle and the field of view coverage area includes: The target latitude coverage angle is obtained by increasing the latitude coverage angle of the field of view coverage area according to the preset pitch angle. The target longitude coverage angle is obtained by increasing the longitude coverage angle of the field of view coverage area according to the preset yaw angle. Based on the target latitude coverage angle and the target longitude coverage angle, the field of view coverage area is projected to obtain the sampling area in the panoramic image.

6. The method according to claim 1, characterized in that, The step of obtaining pixels in the sampling region according to the target resolution to obtain the target image corresponding to the panoramic image includes: Based on the target resolution, determine the number of pixels in the width direction and the number of pixels in the height direction of the target image; Based on the number of pixels in the width direction and the number of pixels in the height direction, the pixels in the sampling area are sampled to obtain the target image corresponding to the panoramic image. In the target image, the angle difference between any two adjacent pixels in the width direction is equal, and the angle difference between any two adjacent pixels in the height direction is equal.

7. The method according to claim 6, characterized in that, The step of sampling pixels in the sampling region based on the number of pixels in the width direction and the number of pixels in the height direction to obtain the target image corresponding to the panoramic image includes: The width sampling angle and height sampling angle of the sampling area are determined based on the number of pixels in the width direction, the number of pixels in the height direction, and the resolution of the sampling area. Based on the width sampling angle and the height sampling angle, the pixels in the sampling area are obtained to obtain the target image corresponding to the panoramic image.

8. A panoramic image processing apparatus, characterized in that, include: The coordinate acquisition module is used to acquire the viewpoint coordinates corresponding to the panoramic image to be displayed, and to determine the field of view coverage area in the panoramic image based on the viewpoint coordinates, wherein the viewpoint coordinates represent the position of the gaze point in the panoramic image. The region generation module is used to obtain a preset angle and generate a sampling region in the panoramic image based on the preset angle and the field of view coverage area. The similarity between the shape of the sampling region and the shape of the field of view coverage area satisfies a preset threshold. The image generation module is used to obtain pixels in the sampling area according to the target resolution to obtain the target image corresponding to the panoramic image, wherein the target resolution is the panoramic image resolution in the client.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic image processing method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the panoramic image processing method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the panoramic image processing method as described in any one of claims 1-7.

Images