Image processing method and device, electronic equipment and storage medium

Abstract

This application provides an image processing method, apparatus, electronic device, and storage medium. The method includes acquiring a planar image, wherein the planar image is determined by projection of a first spherical image, the first spherical image being an image obtained by capturing the content to be photographed from 360 degrees; inputting the planar image into a trained saliency model; generating a second spherical image corresponding to the planar image based on the planar image through the saliency model, and determining the coordinates of the spherical pixels corresponding to the second spherical image; determining the coordinates of the planar pixels corresponding to the planar image based on the second spherical image and the coordinates of the spherical pixels through the saliency model; and outputting a target image with a salient region corresponding to the planar image based on the planar pixel coordinates through the saliency model. This solves the technical problem in the prior art that the field of view prediction of the planar image obtained by projecting a spherical image is inaccurate, and achieves the purpose of accurately determining the salient region of the planar image.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the commercialization of 5G and the rapid development of new multimedia technologies, virtual reality videos (such as panoramic videos and 360-degree videos) have become increasingly popular in recent years. Unlike traditional videos, virtual reality videos allow users to view 360-degree video content, thus requiring significant bandwidth for transmission. However, users typically focus on areas of interest when watching virtual reality videos. To reduce bandwidth consumption, the areas of interest can be transmitted at high resolution, while other areas are transmitted at lower resolution. Therefore, accurately identifying the areas of interest for the user is crucial.

[0003] In existing technologies, virtual reality videos are generally projected from spherical images into planar images before transmission. However, the projected planar images usually suffer from image distortion and pixel warping, and the image distortion becomes increasingly severe from the equatorial plane corresponding to the spherical image to the North and South Poles. Traditional models cannot accurately learn the features of planar images, which leads to inaccurate determination of salient regions by traditional models. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an image processing method, apparatus, electronic device and storage medium to overcome all or part of the deficiencies in the prior art.

[0005] To achieve the above objectives, this application provides an image processing method, comprising: acquiring a planar image, wherein the planar image is determined by projection of a first spherical image, the first spherical image being an image obtained by capturing the content to be captured from 360 degrees; inputting the planar image into a trained saliency model; generating a second spherical image corresponding to the planar image based on the planar image through the saliency model, and determining the coordinates of spherical pixels corresponding to the second spherical image; determining the coordinates of planar pixels corresponding to the planar image through the saliency model based on the second spherical image and the coordinates of the spherical pixels; and outputting a target image with a salient region corresponding to the planar image through the saliency model based on the coordinates of the planar pixels.

[0006] Optionally, determining the planar pixel coordinates corresponding to the planar image based on the second spherical image and the spherical pixel coordinates includes: determining any one of a plurality of tangent points corresponding to the second spherical image, and determining a tangent plane of a preset size centered on the tangent point based on the tangent point; projecting the spherical pixel coordinates onto the tangent plane based on the spherical pixel coordinates to determine the projected coordinates of the spherical pixel coordinates in the tangent plane; and determining the planar pixel coordinates based on the projected coordinates.

[0007] Optionally, the step of projecting the spherical pixel coordinates onto the tangent plane based on the spherical pixel coordinates to determine the projected coordinates of the spherical pixel coordinates in the tangent plane includes: establishing a coordinate system corresponding to the tangent plane with the tangent point as the center; dividing the tangent plane into regions based on the coordinate system corresponding to the tangent plane and calculating the unit coordinates of each region; determining the region corresponding to the spherical pixel coordinates in the tangent plane; and calculating the projected coordinates based on the spherical pixel coordinates and the unit coordinates of the region corresponding to the spherical pixel coordinates.

[0008] Optionally, determining the coordinates of the planar pixel point based on the projected coordinates includes: determining the coordinates of the planar pixel point in the tangent plane using the following formula:

[0009]

[0010] Among them, Γ x (φ,θ) are the x-coordinates of the planar pixel points in the tangent plane, Γ y (φ, θ) represents the ordinate of the planar pixel in the tangent plane, θ represents the abscissa of the spherical pixel, and φ represents the ordinate of the spherical pixel. γ φ is the abscissa in the projected coordinate system. γ The vertical coordinate is the ordinate in the projected coordinate system.

[0011] Optionally, the loss function used to train the saliency model is determined by the following formula: ι=L S-MSE (S,Q)+L CC (S,Q)+L KL (S,Q), where ι is the loss function, L S-MSE (S,Q) represents the weights, and L... CC (S,Q) represents a linear correlation, L KL (S,Q) represents the difference relationship, where S is the target image and Q is the labeled sample image.

[0012] Optionally, the linear correlation is determined by the following formula: L CC (S,Q)=1-CC(S,Q), Among them, L CC (S,Q) represents the linear correlation, CC(S,Q) is the linear correlation coefficient, Cov(S,Q) is the covariance, σ(S) is the standard deviation of the target image, σ(Q) is the standard deviation of the labeled sample image, S is the target image, and Q is the labeled sample image; the difference relationship is determined by the following formula: L KL (S,Q)=KL(S,Q), Among them, L KL (S,Q) represents the difference relationship, KL(S,Q) represents the difference between the target image and the labeled sample image under the condition of information loss, S is the target image, Q is the labeled sample image, ε is the regularization constant, n is the total number of initial plane pixels, and i is the current pixel.

[0013] Optionally, the saliency model is a convolutional neural network model. The step of outputting a target image with a salient region corresponding to the planar image through the saliency model based on the planar pixel coordinates includes: in response to determining that the planar image has a corresponding preset calibration image, inputting the preset calibration image into the saliency model; extracting a first feature corresponding to the planar pixel coordinates and a second feature corresponding to the preset calibration image using the convolutional layers of the saliency model based on the planar pixel coordinates and the preset calibration image; and outputting the target image through the saliency model based on the first feature and the second feature.

[0014] Based on the same inventive concept, this application also provides an image processing apparatus, comprising: an acquisition module configured to acquire a planar image, wherein the planar image is determined by projection of a first spherical image, the first spherical image being an image obtained by capturing the content to be captured from 360 degrees; a first determination module configured to input the planar image into a trained saliency model, generate a second spherical image corresponding to the planar image based on the planar image through the saliency model, and determine the coordinates of spherical pixels corresponding to the second spherical image; a second determination module configured to determine the coordinates of planar pixels corresponding to the planar image based on the second spherical image and the coordinates of the spherical pixels through the saliency model; and an output determination module configured to output a target image with a salient region corresponding to the planar image based on the planar pixel coordinates through the saliency model.

[0015] Based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0016] Based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the method described above.

[0017] As can be seen from the above description, the image processing method, apparatus, electronic device, and storage medium provided in this application include: acquiring a planar image, wherein the planar image is determined by projection of a first spherical image, the first spherical image being an image obtained by capturing the content to be photographed from 360 degrees; inputting the planar image into a trained saliency model; generating a second spherical image corresponding to the planar image based on the planar image through the saliency model, and determining the coordinates of the spherical pixels corresponding to the second spherical image; representing the second spherical image with specific numerical values ​​in the form of coordinates to establish an accurate association with the planar image subsequently; determining the coordinates of the planar pixels corresponding to the planar image based on the second spherical image and the coordinates of the spherical pixels through the saliency model, making the association between the planar image and the second spherical image more accurate; and outputting a target image with a salient region corresponding to the planar image through the saliency model based on the planar pixel coordinates. The saliency model can accurately extract the features of the planar image, thereby achieving the purpose of accurately determining the salient region of the planar image through the saliency model. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of the image processing method according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the structure of the image processing apparatus according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0023] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0024] As described in the background section, with the commercialization of 5G and the rapid development of new multimedia technologies, virtual reality videos (such as panoramic videos and 360-degree videos) have become increasingly popular in recent years. Virtual reality videos allow users to view 360-degree video content; therefore, the bandwidth consumed by virtual reality videos differs from that of traditional videos. For example, transmitting a 4K panoramic video to a client and allowing the user to view it from all angles requires a data rate of 400Mb / s, while traditional 4K video streaming only requires a data rate of 25Mb / s. When watching virtual reality videos, users typically focus on areas of interest. Furthermore, due to the limited field of view of head-mounted devices, users may only see 20% to 30% of the full video content, with the remaining areas being wasted. Therefore, accurately identifying the areas of interest for the user is crucial.

[0025] Virtual reality videos are typically projected from a spherical image into a planar image before transmission. However, the projected planar image usually suffers from image distortion and pixel warping, and the image distortion becomes increasingly severe from the equatorial plane corresponding to the spherical image to the North and South Poles. Traditional models cannot accurately learn the features of the planar image, which leads to inaccurate determination of salient regions by traditional models.

[0026] In view of this, embodiments of this application propose an image processing method, referring to... Figure 1 This includes the following steps:

[0027] Step 101: Obtain a planar image, wherein the planar image is determined by projection of a first spherical image, and the first spherical image is an image obtained by taking a 360-degree picture of the content to be photographed.

[0028] In this step, to improve the user's viewing experience, the first spherical image is typically projected into a planar image, and this projection step is usually completed during the image capture stage. However, the planar image obtained by projecting the first spherical image often suffers from image distortion and pixel warping. The planar image, determined by the projection of the first spherical image, can be obtained from various sources. For example, it can be obtained from panoramic video, or it can be determined from a panoramic image captured by the user.

[0029] Step 102: Input the planar image into the trained saliency model, generate a second spherical image corresponding to the planar image through the saliency model based on the planar image, and determine the coordinates of the spherical pixel points corresponding to the second spherical image.

[0030] In this step, traditional saliency models extract features from the input image to determine the corresponding salient regions, which are the areas the user's gaze focuses on. However, due to image distortion and pixel warping in planar images, traditional saliency models cannot accurately extract features from planar images. To address this issue, this embodiment uses a spherical convolutional neural network (SCNN) model. This model divides the input image into multiple regions and extracts features from each region using different weights obtained through model training. The trained SCNN model achieves high accuracy in feature extraction from spherical images. Therefore, it's necessary to establish a connection between the planar image and the spherical image that the SCNN model can process. A second spherical image corresponding to the planar image is generated using the saliency model. The parameters of the second spherical image can be set according to user requirements; for example, based on user needs, the radius of the second spherical image can be set to half the length of the planar image. The coordinates of the second spherical image also need to be determined; for example, the coordinates of the spherical pixels corresponding to the second spherical image can be determined by establishing a coordinate system. The second spherical image is represented by specific numerical values ​​in the form of coordinates so that it can be accurately correlated with the planar image later.

[0031] It should be noted that the saliency model used in this embodiment can be a spherical convolutional neural network model with the following structure. The spherical convolutional neural network model consists of an encoder and a decoder. The encoder, as the backbone feature extraction network, obtains feature layers one after another through a shrinking path. The encoder includes four spherical convolutional layers and four ReLU activation layers, with three spherical pooling operations between the spherical convolutional layers and the ReLU activation layers. As a feature enhancement network, the decoder uses the initially effective feature layers obtained by the backbone feature extraction network to perform feature fusion through an expansion path to obtain the final enhanced features. The decoder includes three spherical convolutional layers, followed by corresponding ReLU activation function layers, with three upsampling layers between the spherical convolutional layers and the ReLU activation layers.

[0032] Step 103: Based on the second spherical image and the coordinates of the spherical pixels, determine the coordinates of the planar pixels corresponding to the planar image using the saliency model.

[0033] In this step, the coordinates of the planar pixels corresponding to the planar image are determined using the second spherical image that can be processed by the spherical convolutional neural network model and the coordinates of the corresponding spherical pixels in the second spherical image. A numerical correlation is established between the second spherical image and the planar image, and this correlation is quantified to make the relationship between the two images more accurate.

[0034] Step 104: Based on the coordinates of the planar pixels, output the target image with a salient region corresponding to the planar image through the saliency model.

[0035] In this step, features of the planar image are extracted using a saliency model, which is a spherical convolutional neural network (SNN) model. The SNN model can identify the equatorial plane and the North and South Poles corresponding to the spherical image. Therefore, after establishing a correlation between the spherical image and the planar image, the SNN model can also identify the area from the equatorial plane corresponding to the first spherical image to the North and South Poles in the planar image. Since image distortion in the planar image increases significantly from the equatorial plane corresponding to the first spherical image to the North and South Poles, the SNN model assigns relatively small weights to coordinates near the poles of the first spherical image within the planar image, and relatively large weights to coordinates near the equatorial plane area of ​​the first spherical image within the planar image. The saliency model highlights the coordinates of undistorted areas in the planar image. The saliency model accurately extracts features from the planar image, thereby achieving the goal of accurately determining the salient regions of the planar image. Subsequently, the salient regions of the planar image can be transmitted at high resolution, while the remaining regions can be transmitted at lower resolution, improving the user experience and reducing communication bandwidth consumption.

[0036] The above scheme obtains a planar image, which is determined by projecting a first spherical image. The first spherical image is obtained by capturing the content to be photographed from 360 degrees. The planar image is input into a trained saliency model. Based on the planar image, the saliency model generates a second spherical image corresponding to the planar image and determines the coordinates of the spherical pixels corresponding to the second spherical image. The second spherical image is represented by specific numerical values ​​in coordinate form to establish an accurate association with the planar image later. Based on the second spherical image and the spherical pixel coordinates, the saliency model determines the coordinates of the planar pixels corresponding to the planar image, making the association between the planar image and the second spherical image more accurate. Based on the planar pixel coordinates, the saliency model outputs a target image with a salient region corresponding to the planar image. The saliency model can accurately extract features from the planar image, thereby achieving the goal of accurately determining the salient region of the planar image through the saliency model.

[0037] In some embodiments, determining the planar pixel coordinates corresponding to the planar image based on the second spherical image and the spherical pixel coordinates includes: determining any one of a plurality of tangent points corresponding to the second spherical image, and determining a tangent plane of a preset size centered on the tangent point based on the tangent point; projecting the spherical pixel coordinates onto the tangent plane based on the spherical pixel coordinates to determine the projected coordinates of the spherical pixel coordinates in the tangent plane; and determining the planar pixel coordinates based on the projected coordinates.

[0038] In this embodiment, to establish a relationship between the coordinates of spherical pixels and planar pixels, other planes can be used to display the coordinates of spherical pixels and planar pixels. Since the second spherical image has a tangent point, a tangent plane can be established using the tangent point of the second spherical image. The preset size can be determined according to actual needs. For example, to quickly determine the size of the tangent plane, the size of the planar image can be determined as the size of the tangent plane. The spherical pixels are projected onto the tangent plane to obtain the projected coordinates, and the relationship between the tangent plane and the spherical pixels is established first. Then, the coordinates of the planar pixels are determined through the projected coordinates, establishing a relationship between the planar pixel coordinates and the projected coordinates. Since the trained spherical convolutional neural network model has high accuracy when extracting features from the spherical image, after establishing the numerical relationship between the planar image and the second spherical image, the feature extraction of the planar pixels by the trained spherical convolutional neural network model also has high accuracy.

[0039] In some embodiments, the step of projecting the spherical pixel coordinates onto the tangent plane based on the spherical pixel coordinates to determine the projected coordinates of the spherical pixel coordinates in the tangent plane includes: establishing a coordinate system corresponding to the tangent plane with the tangent point as the center; dividing the tangent plane into regions based on the coordinate system corresponding to the tangent plane and calculating the unit coordinates of each region; determining the region corresponding to the spherical pixel coordinates in the tangent plane; and calculating the projected coordinates based on the spherical pixel coordinates and the unit coordinates of the region corresponding to the spherical pixel coordinates.

[0040] In this embodiment, since it is necessary to establish a connection between the second spherical image and the planar image using other planes, the projection coordinates of the spherical pixel coordinates of the second spherical image in the tangent plane are first determined. A coordinate system corresponding to the tangent plane is established with the tangent point as the center. To improve the efficiency of determining the projection coordinates, the tangent plane can be divided into regions. For example, for ease of description, it can be considered as dividing the horizontal axis with the center as the dividing line, with the horizontal axis to the left of the center being the first horizontal axis and the horizontal axis to the right of the center being the second horizontal axis; it can also be considered as dividing the vertical axis with the center as the dividing line, with the vertical axis above the center being the first vertical axis and the vertical axis below the center being the second vertical axis. The center of the tangent plane is the center of the circle containing the spherical pixel coordinates. The tangent plane is divided into eight regions, where the first region is the first horizontal axis, and the unit coordinate of the first region is p. γ(-1,0) =(-tanΔ θ The second region is the second horizontal axis, and the unit coordinate of the second region is p. γ(1,0) =(tanΔ θ The third region is defined by the first vertical axis, and the unit coordinate of the third region is p. γ(0,1)= (0, tanΔ φ The fourth region is the second vertical axis, and the unit coordinate of the fourth region is p. γ(0,-1) = (0, -tanΔ) φ The fifth region is the area enclosed by the first horizontal axis and the first vertical axis, and the unit coordinate of the fifth region is p. γ(-1,+1) =(-tanΔ θ ,+secΔ θ tanΔ φ The sixth region is the area enclosed by the second horizontal axis and the first vertical axis, and the unit coordinate of the sixth region is p. γ(1,1) =(tanΔ θ ,secΔ θ tanΔ φ The seventh region is the area enclosed by the second horizontal axis and the second vertical axis, and the unit coordinate of the seventh region is p. γ(+1,-1) =(+tanΔ) θ ,-secΔ θ tanΔ φ The eighth region is the area enclosed by the second vertical axis and the second horizontal axis, and the unit coordinate of the eighth region is p. γ(-1,-1) =(-tanΔ θ ,-secΔ θ tanΔ φ ); where Δ θ and Δ φ This is the preset step size.

[0041] The positive and negative signs of the spherical pixel coordinates are matched with the positive and negative signs of the unit coordinates in each region. If the positive and negative signs of the spherical pixel coordinates are the same as those of the unit coordinates in one region, that region is defined as the region corresponding to the spherical pixel coordinates in the tangent plane. For example, if the x-coordinate of the spherical pixel is positive and the y-coordinate is negative, a unit coordinate with a positive x-coordinate and a negative y-coordinate is searched among all regions, thus determining that the spherical pixel coordinates are in the seventh region in the tangent plane. Based on the spherical pixel coordinates and the unit coordinates of the region corresponding to the spherical pixel coordinates, the projected coordinates are calculated. For example, when the spherical pixel coordinates are (2, -2), the unsigned value is multiplied by the unit coordinates of the seventh region to determine the projected coordinates, which are (+2tanΔ). θ -2secΔ θ tanΔ φ The relationship between the second spherical image and the tangent plane is numericalized in the form of coordinates, thereby allowing for a precise determination of this relationship.

[0042] In some embodiments, determining the planar pixel coordinates based on the projected coordinates includes: determining the planar pixel coordinates in the tangent plane using the following formula:

[0043]

[0044] Among them, Γ x (φ,θ) are the x-coordinates of the planar pixel points in the tangent plane, Γ y (φ, θ) represents the ordinate of the planar pixel in the tangent plane, θ represents the abscissa of the spherical pixel, and φ represents the ordinate of the spherical pixel. γ φ is the abscissa in the projected coordinate system. γ The vertical coordinate is the ordinate in the projected coordinate system.

[0045] In this embodiment, the planar image needs to be associated with the second spherical image using a tangent plane. Therefore, based on the spherical pixel coordinates and projection coordinates corresponding to the second spherical image, the planar pixel coordinates corresponding to the planar image are determined in the tangent plane. The planar pixel coordinates can be determined using a formula. By using the tangent plane, the relationship between the second spherical image and the planar image is numericalized in coordinate form, thereby accurately determining the relationship between the second spherical image and the planar image. This allows the subsequent saliency model to accurately extract the features of the planar pixels.

[0046] In some embodiments, the loss function used to train the saliency model is determined by the following formula: ι=L S-MSE (S,Q)+L CC (S,Q)+L KL (S,Q), where ι is the loss function, L S-MSE (S,Q) represents the weights, and L... CC (S,Q) represents a linear correlation, L KL (S,Q) represents the difference relationship, where S is the target image and Q is the labeled sample image.

[0047] In this embodiment, the saliency model is a spherical convolutional neural network (SNN). This model divides the input image into multiple regions and learns the features of each region by setting different weights. Therefore, these weights are incorporated into the loss function calculation, guiding the training direction of the saliency model based on the weights corresponding to each region. To further train the saliency model, linear correlation and difference relationships are also introduced to guide its training. The linear correlation coefficient measures the linear correlation between the target image and the labeled sample image; a larger coefficient indicates greater similarity. The difference relationship measures the difference between the target image and the labeled sample image under information loss; a smaller difference value indicates less difference between the two images. These loss functions make the training direction of the saliency model more accurate.

[0048] In some embodiments, the linear correlation is determined by the following formula: L CC (S,Q)=1-CC(S,Q), Among them, L CC (S,Q) represents the linear correlation, CC(S,Q) is the linear correlation coefficient, Cov(S,Q) is the covariance, σ(S) is the standard deviation of the target image, σ(Q) is the standard deviation of the labeled sample image, S is the target image, and Q is the labeled sample image; the difference relationship is determined by the following formula: L KL (S,Q)=KL(S,Q), Among them, L KL (S,Q) represents the difference relationship, KL(S,Q) represents the difference between the target image and the labeled sample image under the condition of information loss, S is the target image, Q is the labeled sample image, ε is the regularization constant, n is the total number of initial plane pixels, and i is the current pixel.

[0049] In this embodiment, the linear correlation and the difference relationship need to be determined based on the formula, and the two abstract correlation relationships are quantified to make the determination of the two relationships more accurate, thereby making the training direction of the saliency model more accurate.

[0050] In some embodiments, the saliency model is a convolutional neural network model, and the step of outputting a target image with a salient region corresponding to the planar image through the saliency model based on the planar pixel coordinates includes: in response to determining that the planar image has a corresponding preset calibration image, inputting the preset calibration image into the saliency model; extracting a first feature corresponding to the planar pixel coordinates and a second feature corresponding to the preset calibration image using the convolutional layers of the saliency model based on the planar pixel coordinates and the preset calibration image; and outputting the target image through the saliency model based on the first feature and the second feature.

[0051] In this embodiment, considering that a user might continuously view a set of related planar images, the related planar images are marked. For example, "related" can mean that the planar images share the same source as other planar images; for instance, related planar images may originate from the same panoramic video. Before inputting the planar image into the saliency model, it is detected whether other planar images with the same markings as the input planar image have corresponding target images output by the saliency model. In response to determining the existence of such target images, one of these target images can be input together with the input planar image into the saliency model. This target image is a preset calibration image, which corrects the output result of the saliency model for the currently input planar image, enabling the saliency model to output the target image efficiently and accurately. To further improve the accuracy of the target image output, the target image can be the target image output by the saliency model at the moment before the input planar image is input into the saliency model. It should be noted that the saliency model in this embodiment can also be embedded into other models according to actual needs. For example, in order to further improve the user's viewing experience, the saliency model can be embedded into the FoV model. The FoV model can accurately represent the image that the human eye or machine vision system can see, and can also analyze the image.

[0052] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0053] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0054] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides an image processing apparatus.

[0055] refer to Figure 2 The image processing apparatus includes:

[0056] The acquisition module 10 is configured to acquire a planar image, wherein the planar image is determined by projection of a first spherical image, and the first spherical image is an image obtained by capturing the content to be captured in 360 degrees.

[0057] The first determining module 20 is configured to input the planar image into a trained saliency model, generate a second spherical image corresponding to the planar image based on the planar image through the saliency model, and determine the coordinates of the spherical pixel points corresponding to the second spherical image.

[0058] The second determining module 30 is configured to determine the coordinates of the planar pixels corresponding to the planar image based on the second spherical image and the coordinates of the spherical pixels, using the saliency model.

[0059] The output module 40 is configured to output a target image with a salient region corresponding to the planar image based on the coordinates of the planar pixel points and through the salientity model.

[0060] Using the aforementioned device, a planar image is acquired, wherein the planar image is determined by projecting a first spherical image, which is an image obtained by capturing the content to be photographed from 360 degrees. The planar image is input into a trained saliency model. Based on the planar image, a second spherical image corresponding to the planar image is generated through the saliency model, and the coordinates of the spherical pixels corresponding to the second spherical image are determined. The second spherical image is represented by specific numerical values ​​in coordinate form to establish an accurate association with the planar image subsequently. Based on the second spherical image and the spherical pixel coordinates, the coordinates of the planar pixels corresponding to the planar image are determined through the saliency model, making the association between the planar image and the second spherical image more accurate. Based on the planar pixel coordinates, the target image with a salient region corresponding to the planar image is output through the saliency model. The saliency model can accurately extract features from the planar image, thereby achieving the purpose of accurately determining the salient region of the planar image through the saliency model.

[0061] In some embodiments, the second determining module 30 is further configured to determine any one of a plurality of tangent points corresponding to the second spherical image, and based on the tangent point, determine a tangent plane of a preset size centered on the tangent point; based on the spherical pixel coordinates, project the spherical pixel coordinates onto the tangent plane to determine the projected coordinates of the spherical pixel coordinates in the tangent plane; and based on the projected coordinates, determine the planar pixel coordinates.

[0062] In some embodiments, the second determining module 30 is further configured to establish a coordinate system corresponding to the tangent plane with the tangent point as the center, divide the tangent plane into regions based on the coordinate system corresponding to the tangent plane, and calculate the unit coordinates of each region; determine the region corresponding to the spherical pixel coordinates in the tangent plane; and calculate the projected coordinates based on the spherical pixel coordinates and the unit coordinates of the region corresponding to the spherical pixel coordinates.

[0063] In some embodiments, the second determining module 30 is further configured to determine the planar pixel coordinates based on the projection coordinates, including: determining the planar pixel coordinates in the tangent plane using the following formula: , Among them, Γ x (φ,θ) are the x-coordinates of the planar pixel points in the tangent plane, Γ y (φ, θ) represents the ordinate of the planar pixel in the tangent plane, θ represents the abscissa of the spherical pixel, and φ represents the ordinate of the spherical pixel. γ φ is the abscissa in the projected coordinate system. γ The vertical coordinate is the ordinate in the projected coordinate system.

[0064] In some embodiments, a third determining module is further configured to determine the loss function for training the saliency model by the following formula: ι=L S-MSE (S,Q)+L CC (S,Q)+L KL (S,Q), where ι is the loss function, L S-MSE (S,Q) represents the weights, and L... CC (S,Q) represents a linear correlation, L KL (S,Q) represents the difference relationship, where S is the target image and Q is the labeled sample image.

[0065] In some embodiments, the third determining module is further configured to determine the linear correlation using the following formula: L CC (S,Q)=1-CC(S,Q), Among them, L CC (S,Q) represents the linear correlation, CC(S,Q) is the linear correlation coefficient, Cov(S,Q) is the covariance, σ(S) is the standard deviation of the target image, σ(Q) is the standard deviation of the labeled sample image, S is the target image, and Q is the labeled sample image; the difference relationship is determined by the following formula: L KL (S,Q)=KL(S,Q), Among them, L KL (S,Q) represents the difference relationship, KL(S,Q) represents the difference between the target image and the labeled sample image under the condition of information loss, S is the target image, Q is the labeled sample image, ε is the regularization constant, n is the total number of initial plane pixels, and i is the current pixel.

[0066] In some embodiments, the output module 40 is further configured such that the saliency model is a convolutional neural network model, and in response to determining that the planar image has a corresponding preset calibration image, the preset calibration image is input into the saliency model; based on the planar pixel coordinates and the preset calibration image, the convolutional layer of the saliency model is used to extract the first feature corresponding to the planar pixel coordinates and the second feature corresponding to the preset calibration image, respectively; based on the first feature and the second feature, the target image is output through the saliency model.

[0067] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0068] The apparatus of the above embodiments is used to implement the corresponding image processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0069] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image processing method described in any of the above embodiments.

[0070] Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0071] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0072] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0073] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0074] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0075] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0076] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0077] The electronic devices described above are used to implement the corresponding image processing methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0078] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the image processing method as described in any of the above embodiments.

[0079] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0080] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the image processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0081] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0082] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0083] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0084] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: A planar image is acquired, wherein the planar image is determined by projection of a first spherical image, the first spherical image being an image obtained by capturing the content to be captured from 360 degrees; The planar image is input into a trained saliency model, which generates a second spherical image corresponding to the planar image and determines the coordinates of the spherical pixels corresponding to the second spherical image. The saliency model is a spherical convolutional neural network model, which is used to divide the input image into multiple regions. By using different weights obtained through model training in each region, features of each region are extracted. The weights of the polar regions corresponding to the image are less than the weights of the equatorial plane regions corresponding to the image. Based on the second spherical image and the coordinates of the spherical pixels, the coordinates of the planar pixels corresponding to the planar image are determined by the saliency model; Based on the coordinates of the planar pixels, the target image with a salient region corresponding to the planar image is output through the saliency model; Based on the second spherical image and the coordinates of the spherical pixels, the coordinates of the planar pixels corresponding to the planar image are determined, including: Determine any one of the multiple tangent points corresponding to the second spherical image, and based on the tangent point, determine a tangent plane of a preset size centered on the tangent point; Based on the coordinates of the spherical pixel, the coordinates of the spherical pixel are projected onto the tangent plane to determine the projected coordinates of the spherical pixel in the tangent plane; Based on the projected coordinates, determine the coordinates of the planar pixel point; Determining the coordinates of the planar pixel point based on the projected coordinates includes: The coordinates of the planar pixels in the tangent plane are determined by the following formula: ， ， in, Let x be the x-coordinate of the planar pixel point in the tangent plane. The ordinate of the planar pixel in the tangent plane. The x-coordinate of the pixel on the sphere. The ordinate of the pixel on the sphere. The x-coordinate in the projected coordinate system. The ordinate in the projected coordinate system; The loss function used to train the saliency model is determined by the following formula: ， in, Let the loss function be... As weight, Indicates a linear correlation. Indicates a difference relationship. The target image, The image shows the labeled sample image.

2. The method according to claim 1, characterized in that, The step of projecting the spherical pixel coordinates onto the tangent plane based on the spherical pixel coordinates to determine the projected coordinates of the spherical pixel coordinates in the tangent plane includes: With the tangent point as the center, establish a coordinate system corresponding to the tangent plane. Based on the coordinate system corresponding to the tangent plane, divide the tangent plane into regions and calculate the unit coordinates of each region. Determine the region in the tangent plane corresponding to the coordinates of the spherical pixel point; The projected coordinates are calculated based on the coordinates of the spherical pixel and the unit coordinates of the region corresponding to the spherical pixel.

3. The method according to claim 1, characterized in that, The linear correlation is determined by the following formula: ， ， in, The linear correlation is... The linear correlation coefficient, For covariance, The standard deviation of the target image is . The standard deviation of the labeled sample images is _____. The target image, The labeled sample image; The difference relationship is determined by the following formula: ， ， in, The difference relationship is described above. The difference between the target image and the labeled sample image under the condition of information loss. The target image, The labeled sample image, Here is the regularization constant. This represents the initial total number of planar pixels. This refers to the current pixel.

4. The method according to claim 1, characterized in that, The step of outputting the target image with a salient region corresponding to the planar image based on the coordinates of the planar pixels and through the salientity model includes: In response to determining that the planar image has a corresponding preset calibration image, the preset calibration image is input into the saliency model; Based on the planar pixel coordinates and the preset calibration image, the convolutional layer of the saliency model is used to extract the first feature corresponding to the planar pixel coordinates and the second feature corresponding to the preset calibration image, respectively. Based on the first feature and the second feature, the target image is output through the saliency model.

5. An image processing apparatus, characterized in that, include: The acquisition module is configured to acquire a planar image, wherein the planar image is determined by projection of a first spherical image, and the first spherical image is an image obtained by capturing the content to be captured in 360 degrees; The first determining module is configured to input the planar image into a trained saliency model, generate a second spherical image corresponding to the planar image based on the planar image through the saliency model, and determine the coordinates of the spherical pixels corresponding to the second spherical image. The saliency model is a spherical convolutional neural network model, which is used to divide the input image into multiple regions. By using different weights obtained through model training in each region, features of each region are extracted respectively. The weights of the polar regions corresponding to the image are less than the weights of the equatorial plane regions corresponding to the image. The second determining module is configured to determine the coordinates of the planar pixels corresponding to the planar image based on the second spherical image and the coordinates of the spherical pixels, using the saliency model. The output module is configured to output a target image with a salient region corresponding to the planar image based on the coordinates of the planar pixels and through the salientity model. The second determining module is further configured to: determine any one of a plurality of cutting points corresponding to the second spherical image; and based on the cutting point, determine a cutting plane of a preset size centered on the cutting point; project the spherical pixel coordinates onto the cutting plane based on the spherical pixel coordinates to determine the projected coordinates of the spherical pixel coordinates in the cutting plane; and determine the planar pixel coordinates based on the projected coordinates. The second determining module is further configured to determine the coordinates of planar pixels in the tangent plane using the following formula: ， ， in, Let x be the x-coordinate of the planar pixel point in the tangent plane. The ordinate of the planar pixel in the tangent plane. The x-coordinate of the pixel on the sphere. The ordinate of the pixel on the sphere. The x-coordinate in the projected coordinate system. The ordinate in the projected coordinate system; The third determining module is configured to determine the loss function for training the saliency model using the following formula: ， in, Let the loss function be... As weight, Indicates a linear correlation. Indicates a difference relationship. The target image, The image shows the labeled sample image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 4.

Images