A multi-channel image processing system for browser side

Abstract

The application relates to the technical field of image processing, and discloses a multi-channel image processing system for a browser end, which comprises a multi-channel image preprocessing module, a multi-channel path decision module, a three-dimensional reconstruction module, a two-dimensional super-resolution reconstruction module and an interactive presentation module. Multi-channel image data is acquired through a browser API and color space standardization processing is performed to obtain a scene data packet. Depth semantic analysis is performed on the data packet to obtain a path identifier. Path decision is performed based on the path identifier. A three-dimensional path performs three-dimensional imaging reconstruction on the scene data packet to obtain a three-dimensional scene object. A two-dimensional path performs super-resolution reconstruction on the scene data packet to obtain a super-resolution stylized image. Finally, mixed fusion display is performed to obtain a fusion image. The application can improve the multi-channel image processing efficiency for the browser end.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multi-channel image processing system for browser-based applications. Background Technology

[0002] With the trend of image processing technology migrating to the browser, existing solutions generally lack intelligent processing path selection mechanisms for multi-channel data. Traditional systems either force the adoption of a 3D reconstruction process or only perform 2D image enhancement, failing to dynamically adapt the processing mode according to the actual characteristics of the image data. Due to the lack of correlation analysis between depth and color channel edge information and quantitative evaluation of depth data richness, serious resource waste occurs during processing, and problems such as distorted 3D reconstruction results or poor 2D enhancement effects frequently occur, making it impossible to balance processing efficiency and output quality.

[0003] Meanwhile, existing browser-based image processing systems struggle to achieve efficient fusion and presentation of multi-channel data. On one hand, 3D reconstruction algorithms often rely on complex computing resources, resulting in high response latency and poor compatibility when running in a browser environment. On the other hand, 2D super-resolution technologies focus primarily on improving sharpness, lacking organic integration with 3D information, and stylization processing is disconnected from resolution enhancement. This prevents the system from fully utilizing the complementary advantages of multi-channel data, leading to output results that either lack a sense of 3D spatial hierarchy or have monotonous 2D visual effects, failing to meet the diverse needs of interactive displays on browsers. Therefore, balancing processing efficiency and output quality while satisfying the diverse needs of interactive displays on browsers has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a multi-channel image processing system for browser-based applications to address the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides a multi-channel image processing system for browser-based applications, characterized in that the system includes a multi-channel image preprocessing module, a multi-channel path decision module, a 3D reconstruction module, a 2D super-resolution reconstruction module, and an interactive presentation module, wherein:

[0006] The multi-channel image preprocessing module is used to obtain multi-channel image data of the same scene through the browser API, and to perform color space normalization processing on the multi-channel image data to obtain scene data packets;

[0007] The multi-channel path decision module is used to perform deep semantic analysis on the scene data packet to obtain a path identifier. When the path identifier is three-dimensional dominant, the three-dimensional reconstruction module is executed. When the semantic analysis result is two-dimensional dominant, the two-dimensional super-resolution reconstruction module is executed.

[0008] The 3D reconstruction module is used to perform 3D imaging reconstruction on the depth channel and camera intrinsic parameters in the scene data package to obtain a 3D scene object;

[0009] The two-dimensional super-resolution reconstruction module is used to perform super-resolution reconstruction on the color channels in the scene data package to obtain a super-resolution stylized image.

[0010] The interactive presentation module is used to blend and display the 3D scene object and the super-resolution stylized image to obtain a blended image.

[0011] In a preferred embodiment, the step of obtaining multi-channel image data of the same scene via a browser API and performing color space normalization processing on the multi-channel image data to obtain a scene data package includes:

[0012] The browser API is used to obtain multi-channel image data of the same scene, wherein the multi-channel image data includes depth channel, color channel and camera intrinsic parameters;

[0013] Invalid values ​​are filled into the depth channel;

[0014] Perform color space normalization on the color channels;

[0015] The processed depth channel, color channel, and camera intrinsic parameters are uniformly encapsulated into a scene data package.

[0016] In a preferred embodiment, the step of performing deep semantic analysis on scene data packets to obtain path identifiers includes:

[0017] Extract the salient edge map of the luminance channel in the color channel, and simultaneously calculate the effective depth transition edge map of the depth channel;

[0018] By analyzing the spatial overlap and gradient direction consistency between the salient edge map of the brightness channel and the effective depth transition edge map, a depth-edge consistency metric is obtained.

[0019] The information entropy of the depth channel is calculated to evaluate the information richness of the depth image in the depth channel. Based on the depth-edge consistency metric and the information entropy, the three-dimensional tendency is calculated to make path decisions and obtain a scene three-dimensional tendency score. If the three-dimensional tendency score is greater than or equal to a preset three-dimensional dominance threshold, the corresponding path is identified as three-dimensional dominant; otherwise, the path is identified as two-dimensional dominant.

[0020] In a preferred embodiment, the step of analyzing the spatial overlap and gradient direction consistency between the salient edge map of the luminance channel and the effective depth transition edge map to obtain a depth-edge consistency metric includes:

[0021] Based on the salient edge map of the brightness channel and the effective depth transition edge map, the number of overlapping pixels in spatial location is calculated to obtain the spatial overlap value.

[0022] Based on the salient edge map of the brightness channel and the effective depth transition edge map, the absolute angle difference of their gradient directions is calculated at the overlapping pixel positions to obtain the gradient direction consistency value.

[0023] Based on the spatial overlap value and the gradient direction consistency value, a weighted linear combination is performed to obtain a depth-edge consistency metric.

[0024] In a preferred embodiment, the step of performing 3D imaging reconstruction on the depth channel and camera intrinsic parameters in the scene data packet to obtain a 3D scene object includes:

[0025] Based on the depth channel and camera intrinsic parameters in the scene data package, a sparse 3D point cloud is generated by back projection calculation.

[0026] Each point in the sparse 3D point cloud is colored using color channels, and the point cloud normal vector is calculated by fusing the normal map channels.

[0027] The Poisson surface reconstruction algorithm is called to reconstruct the sparse 3D point cloud into a triangular mesh model;

[0028] Lightweight lighting baking and texture mapping are performed on the triangular mesh model based on the color channels to generate a 3D scene object.

[0029] In a preferred embodiment, the mathematical expression of the Poisson surface reconstruction algorithm is as follows:

[0030]

[0031] In the formula, V(x) is the vector field value at any point x in space, x is the coordinate of any point in three-dimensional space, N is the total number of points in the point cloud, i is the point cloud index, and n i Let P be the unit normal vector at that point, F be the selected smoothing kernel function, and P be the unit normal vector at that point. i Let i be the three-dimensional coordinates of the i-th input point. Here, χ is the Laplace operator, and x is the indicator function. Let L be the divergence operator, L be the discrete Laplacian operator matrix, and b be a column vector consisting of the divergence values ​​calculated on all voxels.

[0032] In a preferred embodiment, the step of performing super-resolution reconstruction on the color channels in the scene data packet to obtain a super-resolution stylized image includes:

[0033] Based on the color channels in the scene data packet, hierarchical features are extracted through a visual self-attention mechanism to obtain deep features;

[0034] Based on the adaptive channel attention mechanism, the deep features are subjected to a nonlinear transformation to obtain attention features;

[0035] Based on preset style transfer weights, the attention features are reconstructed at the pixel level to obtain a super-resolution stylized image.

[0036] In a preferred embodiment, the mathematical expression for the adaptive channel attention mechanism is as follows:

[0037]

[0038] In the formula, z c Let F be the global average pooling value for the c-th channel, where C is the total number of channels, c is the channel index, and F is the average pooling value for the c-th channel. c This represents a two-dimensional feature map of the depth feature on the c-th channel, where H and W are the height and width of the feature map, respectively, (i,j) are the spatial coordinates, i is the height index, j is the width index, and σ c Let be the standard deviation of the c-th channel eigenvalue, and T be an adjustable temperature parameter. c Let σ be the normalized attention weight for the c-th channel, k be the temporary channel index used in the calculation of the normalized attention weight, and σ be the normalized attention weight. k Let be the standard deviation of the eigenvalues ​​of the k-th channel. This indicates channel-by-channel multiplication. The attention feature of the c-th channel after applying channel attention weights. In a preferred embodiment, the step of blending and fusing the 3D scene object and the super-resolution stylized image to obtain a fused image includes:

[0039] Based on the path identifier, determine the main output image data and the auxiliary image data;

[0040] The main output image data and the auxiliary image data are fused at the pixel level to generate a fused image.

[0041] In a preferred embodiment, determining the main output image data and auxiliary image data based on the path identifier includes:

[0042] When the path identifier is 3D dominant, the 2D projection image of the 3D scene object is the main output image data, and the super-resolution stylized image is the auxiliary image data;

[0043] When the path identifier is two-dimensional dominant, the super-resolution stylized image is the main output image data, and the two-dimensional projection image of the three-dimensional scene object is the auxiliary image data.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. This invention significantly improves the intelligence and efficiency of multi-channel image processing on the browser side by constructing an adaptive path decision mechanism based on depth-edge consistency measurement and information entropy. This mechanism extracts the salient edge map of the luminance channel and the effective depth transition edge map of the depth channel in the color channel, accurately analyzes their spatial overlap and gradient direction consistency, and combines the information entropy of the depth channel to evaluate data richness, constructing a three-dimensional tendency scoring system to achieve dynamic adaptation of the processing path. When the three-dimensional tendency score reaches a preset threshold, the three-dimensional reconstruction process is automatically activated; otherwise, two-dimensional super-resolution reconstruction is performed. This avoids the resource waste and effect imbalance caused by traditional fixed path processing, ensuring that the optimal path can be used to complete the processing under different data feature scenarios, greatly improving processing efficiency and the relevance of output results. At the same time, this decision mechanism achieves the objectivity and accuracy of path selection through quantitative indicators, effectively avoiding the subjective errors caused by human intervention, and providing a reliable pre-judgment basis for subsequent processing.

[0046] 2. The lightweight 3D reconstruction and stylized super-resolution fusion presentation system for multi-channel data built in this invention fully explores and efficiently integrates the value of multi-dimensional data. In terms of 3D reconstruction, sparse 3D point clouds are generated through back projection calculation, and the point cloud attributes are optimized by fusing color channels and normal map channels. With the help of Poisson surface reconstruction algorithm and lightweight lighting baking and texture mapping technology, high-precision 3D scene objects are generated while ensuring smooth operation on the browser side. This solves the pain points of traditional 3D reconstruction algorithms in the browser environment, such as computational complexity and poor compatibility. In terms of 2D processing, the system achieves accurate extraction and optimization of depth features through visual self-attention mechanism and adaptive channel attention mechanism. Combined with pixel-level reconstruction with preset style transfer weights, the super-resolution image has both clarity and artistic stylization effect. The interactive presentation module determines the main and auxiliary data through path identification and adopts a pixel-level weighted fusion strategy to organically combine the 2D projection of the 3D scene object with the super-resolution stylized image. This preserves the sense of hierarchy in 3D space while taking into account the delicate texture and stylized expression of 2D images, significantly improving the overall presentation effect of image processing on the browser side and meeting diverse interactive display needs. Attached Figure Description

[0047] Figure 1 This is a system architecture diagram of a browser-oriented multi-channel image processing system provided in one embodiment of the present invention.

[0048] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments belong to some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “said” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0051] Depending on the context, the word "if" or "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0052] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0053] In practice, the server-side equipment deployed in a browser-oriented multi-channel image processing system may consist of one or more devices. This browser-oriented multi-channel image processing system can be implemented as a business instance, a virtual machine, or hardware devices. For example, it can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, it can be understood as software deployed on a cloud node, providing a browser-oriented multi-channel image processing system to various user terminals. Alternatively, it can be implemented as a virtual machine deployed on one or more devices in a cloud node, with application software installed to manage each user terminal. Or, it can also be implemented as a server composed of numerous identical or different types of hardware devices, with one or more hardware devices configured to provide a browser-oriented multi-channel image processing system to various user terminals.

[0054] In terms of implementation, a browser-oriented multi-channel image processing system and a user client are mutually compatible. That is, if a browser-oriented multi-channel image processing system is implemented as an application installed on a cloud service platform, then the user client is implemented as a client that establishes a communication connection with the application; or if a browser-oriented multi-channel image processing system is implemented as a website, then the user client is implemented as a webpage; or if a browser-oriented multi-channel image processing system is implemented as a cloud service platform, then the user client is implemented as a mini-program in an instant messaging application.

[0055] like Figure 1 The diagram shown is a system architecture diagram of a multi-channel image processing system for browsers provided in an embodiment of the present invention.

[0056] The browser-oriented multi-channel image processing system 100 described in this invention can be located on a cloud server. In terms of implementation, it can function as one or more service devices, or as an application installed in the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed into a website. Depending on the implemented functions, the browser-oriented multi-channel image processing system 100 may include a multi-channel image preprocessing module 101, a multi-channel path decision module 102, a 3D reconstruction module 103, a 2D super-resolution reconstruction module 104, and an interactive presentation module 105. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0057] In this embodiment of the invention, in a browser-oriented multi-channel image processing system, each of the above modules can be implemented independently and can call other modules. Here, "calling" can be understood as a module connecting to multiple modules of another type and providing corresponding services to those connected modules. The browser-oriented multi-channel image processing system provided by this embodiment of the invention allows for adjustment of the applicability of the system architecture without modifying the program code, through adding modules and directly calling them. This enables cluster-based horizontal expansion, achieving the goal of quickly and flexibly expanding a browser-oriented multi-channel image processing system. In practical applications, the above modules can be set in the same device or different devices, or they can be set in a virtual device, such as a service instance in a cloud server.

[0058] The following describes, with reference to specific embodiments, the various components and specific workflows of a browser-oriented multi-channel image processing system:

[0059] The multi-channel image preprocessing module is used to obtain multi-channel image data of the same scene through the browser API, and to perform color space normalization processing on the multi-channel image data to obtain scene data packets;

[0060] In this embodiment of the invention, the step of obtaining multi-channel image data of the same scene through a browser API and performing color space normalization processing on the multi-channel image data to obtain a scene data package includes:

[0061] The browser API is used to obtain multi-channel image data of the same scene, wherein the multi-channel image data includes depth channel, color channel and camera intrinsic parameters;

[0062] Invalid values ​​are filled into the depth channel;

[0063] Perform color space normalization on the color channels;

[0064] The processed depth channel, color channel, and camera intrinsic parameters are uniformly encapsulated into a scene data package.

[0065] It should be noted that by obtaining multi-channel image data of the same scene through the browser API, the browser's standard interface is used to synchronously call the available color camera and depth sensor on the device to request a media stream that contains both video and depth tracks. From this stream, a time-aligned color image and a depth image representing the same physical scene are captured. At the same time, the camera's intrinsic parameter matrix is ​​read from the sensor metadata or preset configuration.

[0066] It should be noted that invalid value filling refers to the process of repairing and suppressing data loss and random errors in the depth image caused by sensor limitations in the depth channel. Invalid value filling involves traversing the depth map and replacing pixels marked as invalid with the statistical median of the depth values ​​of the surrounding valid pixels.

[0067] It should be noted that color space normalization refers to the process of converting color channels from their original color space to the CIELAB color space in order to effectively separate the brightness and color information of an image. Specifically, this conversion process maps pixel values ​​from device-related spaces such as RGB to the three components of the LAB space through standardized mathematical transformation: the L component represents lightness, and the A and B components together represent chroma.

[0068] Furthermore, after conversion, the L channel is normalized. First, the original brightness value of each pixel is taken and the average brightness value of all pixels in the entire image is subtracted. The difference obtained in the previous step is divided by the standard deviation of the brightness values ​​of the entire image. Then, the result of the division is multiplied by a preset target standard deviation value. Finally, the product is added to a preset target average value.

[0069] Furthermore, the pre-set target standard deviation is 0.2, and the pre-set target average is 0.5.

[0070] The multi-channel path decision module is used to perform deep semantic analysis on the scene data packet to obtain a path identifier. When the path identifier is three-dimensional dominant, the three-dimensional reconstruction module is executed. When the semantic analysis result is two-dimensional dominant, the two-dimensional super-resolution reconstruction module is executed.

[0071] In this embodiment of the invention, the step of performing deep semantic analysis on scene data packets to obtain path identifiers includes:

[0072] Extract the salient edge map of the luminance channel in the color channel, and simultaneously calculate the effective depth transition edge map of the depth channel;

[0073] By analyzing the spatial overlap and gradient direction consistency between the salient edge map of the brightness channel and the effective depth transition edge map, a depth-edge consistency metric is obtained.

[0074] The information entropy of the depth channel is calculated to evaluate the information richness of the depth image in the depth channel. Based on the depth-edge consistency metric and the information entropy, the three-dimensional tendency is calculated to make path decisions and obtain a scene three-dimensional tendency score. If the three-dimensional tendency score is greater than or equal to a preset three-dimensional dominance threshold, the corresponding path is identified as three-dimensional dominant; otherwise, the path is identified as two-dimensional dominant.

[0075] It should be noted that the method for extracting the salient edge map of the luminance channel in the color channel is as follows: Calculate the absolute value of the luminance difference between each pixel in the image and its right-hand neighbor, and the absolute value of the luminance difference between each pixel and its lower neighbor. Take the larger of these two absolute values ​​as the edge response intensity of that pixel. After traversing the entire image, an edge intensity map is obtained. Finally, apply a preset global threshold to the intensity map and mark pixels with an intensity greater than this threshold as edge points to generate a binarized salient edge map. The preset global threshold is the value at the 95th percentile after sorting the intensity values ​​of all pixels in the image from smallest to largest.

[0076] It should be noted that the effective depth jump edge map of the depth channel is calculated by calculating the absolute value of the depth difference between each pixel and its right-hand neighbor and the absolute value of the depth difference between each pixel and its bottom neighbor. The larger of the two is taken as the local depth jump variable of that point. After traversing, a depth jump intensity map is obtained. Then, a preset effective jump threshold is applied, and pixels with jump variables greater than this threshold are marked as effective depth jump points, generating a binarized effective depth jump edge map. The preset effective jump threshold is the depth value difference corresponding to an actual spatial distance of 10 centimeters.

[0077] It should be noted that the information entropy of the depth channel is calculated by treating all pixel depth values ​​in the depth map as a random variable, counting the frequency of each depth value, and dividing by the total number of pixels to obtain the probability. Then, according to the information entropy formula Summing through all possible depth values ​​or intervals yields the resulting entropy value Hs.

[0078] It should be noted that calculating the three-dimensional tendency for path decision-making specifically refers to the process of weighted linear fusion of the depth-edge consistency metric and the depth map information entropy value after appropriate normalization to [0,1]. Specifically, the normalized depth-edge consistency metric and the depth map information entropy value are multiplied by their respective weight coefficients and then added together to obtain the scene's three-dimensional tendency score, where both weight coefficients are 1 / 2.

[0079] Furthermore, since the depth-edge consistency metric value is already between 0 and 1, it is directly considered as normalized, while the depth map information entropy value is divided by the theoretical maximum value to achieve normalization.

[0080] It should be noted that the preset value of the three-dimensional dominant threshold is 0.5.

[0081] In this embodiment of the invention, the step of analyzing the spatial overlap and gradient direction consistency between the salient edge map of the brightness channel and the effective depth transition edge map to obtain a depth-edge consistency metric includes:

[0082] Based on the salient edge map of the brightness channel and the effective depth transition edge map, the number of overlapping pixels in spatial location is calculated to obtain the spatial overlap value.

[0083] Based on the salient edge map of the brightness channel and the effective depth transition edge map, the absolute angle difference of their gradient directions is calculated at the overlapping pixel positions to obtain the gradient direction consistency value.

[0084] Based on the spatial overlap value and the gradient direction consistency value, a weighted linear combination is performed to obtain a depth-edge consistency metric.

[0085] It should be noted that the specific process of calculating the spatial overlap value is as follows: traverse every pixel position in the image, and determine the position as an overlap point if and only if the salient edge map is marked as an edge point and the effective depth transition edge map is also marked as an edge point. Count the total number of overlap points in the entire image, and then divide the total number of overlap points by the total number of edge points in the salient edge map. The quotient obtained is the spatial overlap value.

[0086] It should be noted that calculating the gradient direction consistency value first requires obtaining the spatial gradient direction maps of the salient edge map and the effective depth jump edge map, respectively. Then, for each overlapping point, the gradient direction angle of the brightness edge and the gradient direction angle of the depth jump edge at that point are extracted, and the absolute difference between these two angles is calculated. Finally, the number of overlapping points whose absolute difference is less than a preset angle tolerance threshold is counted, and this number is divided by the total number of overlapping points. The obtained ratio is the gradient direction consistency value, where the preset angle tolerance threshold is 30 degrees.

[0087] Furthermore, the gradient direction map is calculated by first calculating the difference between each pixel and its right-side neighboring pixel as an approximation of the gradient in the horizontal direction; then calculating the difference between each pixel and its lower-side neighboring pixel as an approximation of the gradient in the vertical direction, and combining them to obtain the gradient direction map.

[0088] Furthermore, the approximate values ​​of the horizontal and vertical gradients of the gradient direction angle basis are extracted using the following formula: The gradient direction angle was calculated. .

[0089] It should be noted that the process of performing a weighted linear combination to calculate the depth-edge consistency metric is as follows: multiply the spatial overlap value by the first weight coefficient to obtain the contribution of the spatially overlapping part; then, multiply the gradient direction consistency value by the second weight coefficient to obtain the contribution of the direction-consistent part; finally, add the two contributions to obtain the final depth-edge consistency metric value.

[0090] It should be noted that the depth-edge consistency metric is a scalar value that combines spatial overlap and gradient direction consistency, ranging from [0,1]. The higher the value, the more reliable and complete the 3D structural information extracted from the current multi-channel data.

[0091] The 3D reconstruction module is used to perform 3D imaging reconstruction on the depth channel and camera intrinsic parameters in the scene data package to obtain a 3D scene object;

[0092] In this embodiment of the invention, the step of performing three-dimensional imaging reconstruction on the depth channel and camera intrinsic parameters in the scene data packet to obtain a three-dimensional scene object includes:

[0093] Based on the depth channel and camera intrinsic parameters in the scene data package, a sparse 3D point cloud is generated by back projection calculation.

[0094] Each point in the sparse 3D point cloud is colored using color channels, and the point cloud normal vector is calculated by fusing the normal map channels.

[0095] The Poisson surface reconstruction algorithm is called to reconstruct the sparse 3D point cloud into a triangular mesh model;

[0096] Lightweight lighting baking and texture mapping are performed on the triangular mesh model based on the color channels to generate a 3D scene object.

[0097] It should be noted that the core computational process for generating sparse 3D point clouds through back projection is as follows: Based on the coordinates of each effective pixel in the depth image and its corresponding depth value, as well as the camera's intrinsic parameter matrix, the coordinates of the 3D point corresponding to that pixel are calculated using the following formula:

[0098]

[0099] In the formula, x, y, and z are the coordinates of points f in three-dimensional space. x ,f y These are the camera's focal length parameters, C x and C y The coordinates are the principal point coordinates. u and v are the coordinates of the effective pixels in the depth image, and ω is the depth value corresponding to the coordinates of the effective pixels.

[0100] Perform this calculation on all valid pixels in the depth map, and the resulting set of three-dimensional points is the sparse three-dimensional point cloud.

[0101] It should be noted that each point in the sparse 3D point cloud is colored using color channels. Based on the coordinates of the effective pixels from which it originates, the color value at that location is directly extracted from the corresponding color channel image and assigned to the 3D point as its color attribute.

[0102] The normal vector calculation of point cloud by fusing normal map channels is achieved by calculating the normal vector of each point in the point cloud. First, based on the depth image, the initial surface normal vector of each pixel is estimated by calculating the depth difference between each pixel and its neighboring pixels. Then, for each point in the point cloud, its corresponding pixel normal initial vector is used as the normal vector of the 3D point, and normalization is performed by dividing by the theoretical maximum value of the vector to obtain the final unit normal vector.

[0103] It should be noted that lightweight lighting baking uses a basic Lambertian lighting model to color the texture. Specifically, it sets one or more virtual light source directions for the scene, calculates the cosine of the angle between the normal vector of the mesh vertex and the light source direction as the lighting intensity coefficient at that vertex, and multiplies this coefficient by the texture color value corresponding to that vertex to obtain the final color.

[0104] Texture mapping projects the vertices of the triangular mesh model reconstructed from the Poisson surface onto a two-dimensional texture coordinate space, establishing a mapping relationship between the vertices and the pixels of the texture image.

[0105] In this embodiment of the invention, the mathematical expression of the Poisson surface reconstruction algorithm is as follows:

[0106]

[0107] In the formula, V(x) is the vector field value at any point x in space, x is the coordinate of any point in three-dimensional space, N is the total number of points in the point cloud, i is the point cloud index, and n i Let P be the unit normal vector at that point, F be the selected smoothing kernel function, and P be the normal vector at that point. i Let i be the three-dimensional coordinates of the i-th input point. Here, χ is the Laplace operator, and x is the indicator function. Let L be the divergence operator, L be the discrete Laplacian operator matrix, and b be a column vector consisting of the divergence values ​​calculated on all voxels.

[0108] It should be noted that the selected smoothing kernel function expression is as follows:

[0109]

[0110] In the formula, F is the selected smoothing kernel function, and r = xp i This represents the distance from spatial point x to the i-th point p in the point cloud. i The vector, ||r||, is the Euclidean distance between the two points. The Gaussian kernel width parameter is used to control the smoothing range.

[0111] It should be noted that the Discrete Laplace operator matrix is ​​calculated by dividing the space containing the point cloud into a dense cubic grid. Then, at the center point of each grid, the Laplace value at that point is approximated using the values ​​of its neighboring grid centers through the finite difference method. Specifically, it finds its six most direct neighbors in space: the center points of the small cubes located directly to its left, right, front, back, top, and bottom. The function values ​​at these six direct neighbor points are added together to obtain a sum. Then, the sum is subtracted by six times the function value of the center point itself, and the result is considered as the continuous Laplace value at that center point. Combining these approximate relationships at all grid points forms a large system of linear equations, and the coefficient matrix of this system is the Discrete Laplace operator matrix.

[0112] The two-dimensional super-resolution reconstruction module is used to perform super-resolution reconstruction on the color channels in the scene data package to obtain a super-resolution stylized image.

[0113] In this embodiment of the invention, the step of performing super-resolution reconstruction on the color channels in the scene data packet to obtain a super-resolution stylized image includes:

[0114] Based on the color channels in the scene data packet, hierarchical features are extracted through a visual self-attention mechanism to obtain deep features;

[0115] Based on the adaptive channel attention mechanism, the deep features are subjected to a nonlinear transformation to obtain attention features;

[0116] Based on preset style transfer weights, the attention features are reconstructed at the pixel level to obtain a super-resolution stylized image.

[0117] It should be noted that the visual self-attention mechanism refers to: dividing the color channel into a sequence of image patches of fixed size, performing linear projection on each image patch to obtain a patch embedding vector, and adding a learnable positional encoding to preserve spatial information; then, inputting the embedding sequence into a multi-layer Transformer encoder, each layer of which contains a multi-head self-attention module and a feedforward neural network module, calculating the global dependencies between patches through self-attention, extracting local and global features of the image layer by layer, and finally outputting a high-dimensional feature sequence, which is then reshaped into a two-dimensional feature map form according to the spatial location of the image patches to obtain the depth features.

[0118] It should be noted that the pixel-level reconstruction process first upsamples the spatial size of the attention features to half of the target resolution through a first transposed convolutional layer and then fuses the features. Subsequently, the features are further upsampled and refined through subsequent transposed convolutional layers. Finally, a convolutional layer maps the high-dimensional features back to the RGB color space, and a stylized activation function based on preset style transfer weights is applied to adjust the output pixel values ​​to obtain the super-resolution stylized image.

[0119] Furthermore, the function expression of the stylized activation function with pre-set style transfer weights is designed as follows:

[0120]

[0121] In the formula, StyleActivate(θ) is the stylization activation function, γ is the style scaling factor, Tanh is the hyperbolic tangent function, and β is the style bias factor. These are stylized original pixel feature values;

[0122] The style scaling factor is directly derived from the normalized value of the global brightness standard deviation, while the style bias factor is the normalized value of the average brightness of all pixels in the channel.

[0123] It should be noted that super-resolution stylized images are the result of color channels undergoing super-resolution reconstruction and stylization processing. They represent a high-quality image that has been enhanced in spatial resolution and given a specific artistic visual effect.

[0124] In this embodiment of the invention, the mathematical expression of the adaptive channel attention mechanism is as follows:

[0125]

[0126] In the formula, z c Let F be the global average pooling value for the c-th channel, where C is the total number of channels, c is the channel index, and F is the average pooling value for the c-th channel. c This represents a two-dimensional feature map of the depth feature on the c-th channel, where H and W are the height and width of the feature map, respectively, (i,j) are the spatial coordinates, i is the height index, j is the width index, and σ c Let be the standard deviation of the c-th channel eigenvalue, and T be an adjustable temperature parameter, s c Let σ be the normalized attention weight for the c-th channel, k be the temporary channel index used in the calculation of the normalized attention weight, and σ be the normalized attention weight. k Let be the standard deviation of the eigenvalues ​​of the k-th channel. This indicates channel-by-channel multiplication. This represents the attention feature of the c-th channel after applying channel attention weights.

[0127] It should be noted that the adjustable temperature parameter is set to a range of [0.1, 10], and the depth feature is at the [0.1, 10]th [0.1, 10]. The two-dimensional feature map on each channel refers to the two-dimensional matrix [H, W] corresponding to the c-th channel.

[0128] The interactive presentation module is used to blend and display the 3D scene object and the super-resolution stylized image to obtain a blended image.

[0129] In this embodiment of the invention, the step of mixing and fusing the three-dimensional scene object and the super-resolution stylized image to obtain a fused image includes:

[0130] Based on the path identifier, determine the main output image data and the auxiliary image data;

[0131] The main output image data and the auxiliary image data are then fused at the pixel level to generate a fused image.

[0132] It should be noted that the pixel-level weighted fusion process is performed separately on the three color channels (R, G, B) at the same pixel coordinate position (i, j) in the two images. For each channel, the fused pixel value I... fusea (i,j,channel) is calculated using the following weighted summation formula:

[0133]

[0134] In the formula, I fusea( i, j, channel) are the merged pixel values, I main( i,j,channel) are the main output image data, I aux (i,j,channel) represents auxiliary image data, ω represents the dominant weight, (1-ω) represents the auxiliary weight, i represents the height index, i represents the width index, and channel represents the color channel.

[0135] Furthermore, the dominant weight is set to 0.7, and the color channel values ​​are 0 for red, 1 for green, and 2 for blue.

[0136] In this embodiment of the invention, determining the main output image data and auxiliary image data based on the path identifier includes:

[0137] When the path identifier is 3D dominant, the 2D projection image of the 3D scene object is the main output image data, and the super-resolution stylized image is the auxiliary image data;

[0138] When the path identifier is two-dimensional dominant, the super-resolution stylized image is the main output image data, and the two-dimensional projection image of the three-dimensional scene object is the auxiliary image data.

[0139] It's important to note that using a 2D projected image of a 3D scene object for subsequent pixel-level weighted fusion is crucial to the design. The 3D scene object itself is a geometric model and cannot be directly processed at the pixel level with a 2D super-resolution stylized image. Rendering it as a 2D projected image is a necessary step to convert it into a data format isomorphic to a 2D image.

[0140] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0141] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-channel image processing system for browser-based applications, characterized in that, The system includes a multi-channel image preprocessing module, a multi-channel path decision module, a 3D reconstruction module, a 2D super-resolution reconstruction module, and an interactive presentation module, wherein: The multi-channel image preprocessing module is used to obtain multi-channel image data of the same scene through the browser API, and to perform color space normalization processing on the multi-channel image data to obtain scene data packets; The multi-channel path decision module is used to perform deep semantic analysis on the scene data packet to obtain a path identifier. When the path identifier is three-dimensional dominant, the three-dimensional reconstruction module is executed; when the path identifier is two-dimensional dominant, the two-dimensional super-resolution reconstruction module is executed. The 3D reconstruction module is used to perform 3D imaging reconstruction on the depth channel and camera intrinsic parameters in the scene data package to obtain a 3D scene object; The two-dimensional super-resolution reconstruction module is used to perform super-resolution reconstruction on the color channels in the scene data package to obtain a super-resolution stylized image. The method for performing deep semantic analysis on scene data packets to obtain path identifiers includes: Extract the salient edge map of the luminance channel in the color channel, and simultaneously calculate the effective depth transition edge map of the depth channel; By analyzing the spatial overlap and gradient direction consistency between the salient edge map of the brightness channel and the effective depth transition edge map, a depth-edge consistency metric is obtained. The information entropy of the depth channel is calculated to evaluate the information richness of the depth image in the depth channel. Based on the depth-edge consistency metric and the information entropy, the three-dimensional tendency is calculated to make path decisions and obtain a scene three-dimensional tendency score. If the three-dimensional tendency score is greater than or equal to a preset three-dimensional dominance threshold, the corresponding path identifier is three-dimensional dominant; otherwise, the path identifier is two-dimensional dominant. The interactive presentation module is used to blend and display the 3D scene objects and the super-resolution stylized image, including: Based on the path identifier, determine the main output image data and the auxiliary image data; The main output image data and the auxiliary image data are pixel-level weighted and fused to generate a fused image; The step of determining the main output image data and auxiliary image data based on the path identifier includes: When the path identifier is 3D dominant, the 2D projection image of the 3D scene object is the main output image data, and the super-resolution stylized image is the auxiliary image data; When the path identifier is two-dimensional dominant, the super-resolution stylized image is the main output image data, and the two-dimensional projection image of the three-dimensional scene object is the auxiliary image data.

2. The multi-channel image processing system for browser-side applications as described in claim 1, characterized in that, The method for obtaining multi-channel image data of the same scene via a browser API, and performing color space normalization processing on the multi-channel image data to obtain a scene data package, includes: Obtain multi-channel image data of the same scene through the browser API, wherein the multi-channel image data includes depth channel, color channel and camera intrinsic parameters; Invalid values ​​are filled into the depth channel; Perform color space normalization on the color channels; The processed depth channel, color channel, and camera intrinsic parameters are uniformly encapsulated into a scene data package.

3. The multi-channel image processing system for browser-side applications as described in claim 2, characterized in that, The analysis of the spatial overlap and gradient direction consistency between the salient edge map of the brightness channel and the effective depth transition edge map yields a depth-edge consistency metric, including: Based on the salient edge map of the brightness channel and the effective depth transition edge map, the number of overlapping pixels between the two in spatial location is calculated to obtain the spatial overlap value. Based on the salient edge map of the brightness channel and the effective depth transition edge map, the absolute angle difference of their gradient directions is calculated at the overlapping pixel positions to obtain the gradient direction consistency value. Based on the spatial overlap value and the gradient direction consistency value, a weighted linear combination is performed to obtain a depth-edge consistency metric.

4. The multi-channel image processing system for browser-side applications as described in claim 1, characterized in that, The method for performing 3D imaging reconstruction on the depth channel and camera intrinsic parameters in the scene data packet to obtain a 3D scene object includes: Based on the depth channel and camera intrinsic parameters in the scene data package, a sparse 3D point cloud is generated by back projection calculation. Each point in the sparse 3D point cloud is colored using color channels, and the point cloud normal vector is calculated by fusing the normal map channels. The Poisson surface reconstruction algorithm is called to reconstruct the sparse 3D point cloud into a triangular mesh model; Lightweight lighting baking and texture mapping are performed on the triangular mesh model based on the color channels to generate a 3D scene object.

5. A multi-channel image processing system for browser-side applications as described in claim 4, characterized in that, The mathematical expression for the Poisson surface reconstruction algorithm is as follows: ； In the formula, V(x) is any point in space. The vector field value at point, where x is the coordinate of any point in 3D space, N is the total number of points in the point cloud, i is the point cloud index, and n is the vector field value. i Let P be the unit normal vector at that point, F be the selected smoothing kernel function, and P be the normal vector at that point. i Let i be the three-dimensional coordinates of the i-th input point. Here, χ is the Laplace operator, and x is the indicator function. Let L be the divergence operator, L be the discrete Laplacian operator matrix, and b be a column vector consisting of the divergence values ​​calculated on all voxels.

6. The multi-channel image processing system for browser-side applications as described in claim 1, characterized in that, The method for performing super-resolution reconstruction of the color channels in the scene data packet to obtain a super-resolution stylized image includes: Based on the color channels in the scene data packet, hierarchical features are extracted through a visual self-attention mechanism to obtain deep features; Based on the adaptive channel attention mechanism, the deep features are subjected to a nonlinear transformation to obtain attention features; Based on preset style transfer weights, the attention features are reconstructed at the pixel level to obtain a super-resolution stylized image.

7. A multi-channel image processing system for browser-side applications as described in claim 6, characterized in that, The mathematical expression for the adaptive channel attention mechanism is as follows: ； In the formula, z c Let F be the global average pooling value for the c-th channel, where C is the total number of channels, c is the channel index, and F is the average pooling value for the c-th channel. c This represents a two-dimensional feature map of the depth feature on the c-th channel, where H and W are the height and width of the feature map, respectively, (i,j) are the spatial coordinates, i is the height index, j is the width index, and σ c Let be the standard deviation of the c-th channel eigenvalue, and T be an adjustable temperature parameter, s c Let σ be the normalized attention weight for the c-th channel, k be the temporary channel index used in the calculation of the normalized attention weight, and σ be the normalized attention weight. k Let be the standard deviation of the eigenvalues ​​of the k-th channel. This indicates channel-by-channel multiplication. This represents the attention feature of the c-th channel after applying channel attention weights.

Images