Image processing method, related device and storage medium

Abstract

The embodiments of the present application relate to the field of image processing, and provide an image processing method, related device and storage medium, the method comprising: for a rendering model to be processed, obtaining the first visual space position information and depth information of the first model vertex in the rendering model to be processed; performing surface subdivision processing on the first model vertex to obtain the second model vertex and the second visual space position information; obtaining the offset of the first model vertex relative to the reference viewpoint, and reversely offset the second model vertex based on the offset to obtain the third visual space position information of the offset second model vertex; correcting the occlusion relationship of the third visual space position information through the depth information to obtain the target space position information. The embodiments of the present application reversely offset each vertex according to the pupil distance to ensure the consistency of the viewpoint position, realize seamless splicing between multiple channels, improve the quality and display effect of image rendering, and help adapt to the display requirements of ultra-large viewing areas.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and more specifically to an image processing method, related devices and storage media. Background Art

[0002] Stereoscopic rendering is a computer graphics technology used to create three-dimensional visual effects with a sense of depth. It simulates the difference in binocular vision of the human eye by generating parallax images seen by the left and right eyes respectively, thereby presenting a stereoscopic image on the display device. This technology is commonly used in virtual reality (VR), augmented reality (AR), mixed reality (XR), and 3D movies and games, which can provide users with a more realistic and immersive visual experience.

[0003] In the related art, when drawing the left and right eye images, the virtual camera is moved to the left or right by a certain distance according to the pupil distance so that the left and right eye images have a certain viewing distance difference, and then the left and right eye images are transmitted to the left eye channel and right eye channel of the stereo display device respectively, so as to produce a more realistic stereoscopic sense for the brain. Since the instantaneous display field of ordinary VR, AR, XR and other display devices is relatively small, this method is only applicable to scenes with a single field of view and a small field of view (such as less than 120 degrees).

[0004] Therefore, it is urgent to propose a new technical solution to solve at least one of the technical problems existing in the related technologies, namely, a single field of view and a small field of view angle, so as to realize full-view stereoscopic rendering. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, a related device and a storage medium to achieve seamless stitching between multiple channels, improve the quality of image rendering and display effects, and adapt to the display requirements of ultra-large viewing areas.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0007] For the rendering model to be processed, obtaining first view space position information and depth information of a first model vertex in the rendering model to be processed;

[0008] Performing surface subdivision processing on the first model vertices to obtain second model vertices and corresponding second visual space position information;

[0009] Obtaining an offset of a first model vertex relative to a reference viewpoint, and performing a reverse offset on a second model vertex based on the offset to obtain third view space position information of the offset second model vertex;

[0010] The occlusion relationship of the third visual space position information is corrected by using the depth information to obtain the target space position information of the to-be-processed rendering model.

[0011] In a second aspect, an embodiment of the present application provides an image processing device having a function of implementing the image processing method provided in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0012] In one embodiment, the image processing device comprises:

[0013] An acquisition module is configured to acquire, for a to-be-processed rendering model, first visual space position information and depth information of a first model vertex in the to-be-processed rendering model;

[0014] A tessellation module is configured to perform tessellation processing on the first model vertices to obtain second model vertices and corresponding second visual space position information;

[0015] An offset module is configured to obtain an offset of a first model vertex relative to a reference viewpoint, and reversely offset a second model vertex based on the offset to obtain third view space position information of the offset second model vertex;

[0016] The correction module is configured to correct the occlusion relationship of the third visual space position information according to the depth information to obtain the target space position information of the rendering model to be processed.

[0017] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions, and when the instructions are executed on a computer, the computer executes the image processing method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image processing method described in the first aspect when executing the computer program.

[0019] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor coupled to a transceiver, and is used to execute the technical solution provided in the first aspect of the embodiment of the present application. In a possible design, the chip can also be a dedicated hardware structure for implementing the technical solution provided in the first aspect above, for example, the processing involving the neural network model can be implemented by a dedicated neural network processor or a graphics processor.

[0020] In a sixth aspect, an embodiment of the present application provides a chip system, which includes a processor for implementing the functions involved in the above-mentioned first aspect, for example, generating or processing the information involved in the image processing method provided in the above-mentioned first aspect.

[0021] In a possible design, the chip system also includes a memory, which is connected to the processor through a circuit structure. The memory is used to store program instructions and data required by the terminal. The chip system can be composed of chips, or it can include chips and other discrete devices. Further optionally, the chip also includes a communication interface, and the processor is connected to the communication interface. The communication interface is used to receive data and / or information to be processed, the processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing results through the communication interface. The communication interface can be an input and output interface.

[0022] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image processing method provided in the first aspect.

[0023] Compared with the prior art, in the embodiment of the present application, for the rendering model to be processed, the first visual space position information and depth information of the first model vertex in the rendering model to be processed are obtained; the first model vertex is subjected to surface subdivision processing to obtain the second model vertex and the corresponding second visual space position information; the offset of the first model vertex relative to the reference viewpoint is obtained, and the second model vertex is reversely offset based on the offset to obtain the third visual space position information of the offset second model vertex; the occlusion relationship of the third visual space position information is corrected by the depth information to obtain the target space position information of the rendering model to be processed. The embodiment of the present application ensures the consistency of the viewpoint position of all channels when drawing the left eye or right eye picture by reversely offsetting each vertex according to the pupil distance, thereby realizing seamless splicing between multiple channels. And the deformation problem after vertex offset is solved by surface subdivision, which effectively solves the problems existing in the current multi-channel seamless splicing 3D display, improves the quality and display effect of image rendering, and helps to adapt to the display requirements of ultra-large viewing areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By reading the detailed description of the embodiments of the present application with reference to the accompanying drawings, the objectives, features and advantages of the embodiments of the present application will become easy to understand. Among them:

[0025] Figure 1 A schematic diagram of the principle of a projection method provided for related technologies;

[0026] Figure 2 A schematic diagram of a flow chart of an image processing method according to an embodiment of the present application;

[0027] Figure 3 A schematic diagram of an original flow chart of the information acquisition step of an embodiment of the present application;

[0028] Figures 4 to 7 A schematic diagram of the principle of the image processing method of an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of a structure of a computing device according to an embodiment of the present application;

[0031] Fig.10 A schematic diagram of the structure of a terminal device in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of the structure of a server in an embodiment of the present application.

[0033] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0034] The terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects (for example, the first full-field stereoscopic rendering and the second full-field stereoscopic rendering are respectively represented as different full-field stereoscopic renderings, and the others are similar), and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules that appear in the embodiments of the present application is only a logical division, and there may be other division methods when implemented in actual applications, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling between modules, and the communication connection may be electrical or other similar forms, which are not limited in the embodiments of the present application. Moreover, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0035] The embodiments of the present application also provide an image processing method, related devices and storage media, which can be applied to full-field stereoscopic rendering scenes, especially full-field stereoscopic rendering systems executed in real time. The full-field stereoscopic rendering system may include an image processing device, which can be deployed in an integrated manner or separately. The image processing device is at least used to perform image processing based on the model to be rendered to obtain the target spatial position information of the rendering model to be processed. Further optionally, it also includes an image recognition device for rendering and displaying an optimized model image according to the target spatial position information of the rendering model to be processed. Among them, the image processing device may be an application for performing image processing on the rendering model to be processed, or a server or terminal device on which an application for performing image processing on the rendering model to be processed is installed.

[0036] The solution provided in the embodiment of the present application involves technologies such as stereoscopic rendering, which is specifically described by the following embodiments:

[0037] Among them, stereo rendering is a computer graphics technology used to create three-dimensional visual effects with a sense of depth. It simulates the binocular vision difference of the human eye by generating parallax images seen by the left and right eyes respectively, thereby presenting a stereoscopic image on the display device. This technology is commonly used in virtual reality (VR), augmented reality (AR), mixed reality (XR), and 3D movies and games, which can provide users with a more realistic and immersive visual experience.

[0038] In the related art, when drawing the left and right eye images, the virtual camera is moved to the left or right by a certain distance according to the pupil distance so that the left and right eye images have a certain viewing distance difference, and then the left and right eye images are transmitted to the left eye channel and right eye channel of the stereo display device respectively, so as to produce a more realistic stereoscopic sense for the brain. Since the instantaneous display field of ordinary VR, AR, XR and other display devices is relatively small, this method is only applicable to scenes with a single field of view and a small field of view (such as less than 120 degrees).

[0039] In another related technology, a multi-channel rendering technology is used to seamlessly stitch multiple captured images into a continuous image. Figure 1As shown, a grid is constructed based on a video image, and the grid vertices are translated to the left and right by a certain distance according to the pupil distance, and then the translated points are re-projected onto the spherical projection surface as the vertex positions of the left and right eye images. However, due to the lack of depth information in video images, this method is only applicable to video images and cannot be applied to stereo rendering scenes. It is understandable that since video images in related technologies do not contain depth information, the object projection orientation is mainly used as the left and right eye reference points for calculation. Objects at a distance of 1 meter and objects at a distance of 100 meters in the same orientation cannot be distinguished in this scenario, and the same offset can only be used when calculating the left and right eye reference points. However, the actual situation is that the difference in spherical projection angles of the left and right eyes of objects at a distance of 1 meter should be relatively large, and the difference in projection angles of objects at a distance of 100 meters should be relatively small. The deviation value will increase as the distance to the viewpoint increases. In addition, since this method cannot distinguish the depth of vertices in the same orientation, it is easy to translate objects at different distances by the same distance, resulting in large visual errors, and it is difficult to apply to stereo rendering scenes.

[0040] Therefore, it is urgent to propose a new technical solution to solve at least one of the technical problems existing in the related technologies, namely, single field of view, small field of view angle, and large visual error, so as to realize full-view stereoscopic rendering.

[0041] Compared with the prior art, in an embodiment of the present application, for a rendering model to be processed, first visual space position information and depth information of a first model vertex in the rendering model to be processed are obtained; surface subdivision processing is performed on the first model vertex to obtain a second model vertex and corresponding second visual space position information; the offset of the first model vertex relative to the reference viewpoint is obtained, and the second model vertex is reversely offset based on the offset to obtain the third visual space position information of the offset second model vertex; the occlusion relationship of the third visual space position information is corrected by the depth information to obtain the target space position information of the rendering model to be processed.

[0042] Specifically, the embodiment of the present application increases the density of vertices by performing surface subdivision processing on the model vertices, reduces the image deformation problem caused by vertex offset, and improves the rendering quality of 3D images. The embodiment of the present application obtains the offset of the first model vertex relative to the reference viewpoint, and reversely offsets the second model vertex based on the offset, thereby ensuring the consistency of the viewpoint position of all channels when drawing the left eye or right eye picture, and realizing seamless splicing between multiple channels. The embodiment of the present application corrects the occlusion relationship of the third visual space position information by using depth information to obtain the target space position information of the rendering model to be processed, thereby ensuring the accuracy of the spatial projection relationship of the images between different channels under a super-large field of view, and avoiding the visual discomfort caused by the error of the spatial projection relationship. It can be seen that the embodiment of the present application can ensure that the parallax image caused by the pupil distance is correctly generated in the 3D display mode of multi-channel seamless splicing, thereby improving the overall effect of 3D display and the visual experience of the audience. This method is particularly suitable for annular or spherical display screens with super-large fields of view, and can provide good 3D image effects for display requirements under a wide range of viewing angles. Furthermore, compared to the traditional method of directly increasing the number of rendering channels or significantly improving the performance of the graphics card, the embodiments of the present application reduce the dependence on hardware resources by optimizing the rendering algorithm, which helps to reduce the cost and power consumption of the 3D display system.

[0043] Regarding the multi-channel splicing scene, in the embodiment of the present application, the visual space position information and depth information can be processed and transmitted as independent data. This means that this information can be exchanged and fused between different channels, thereby supporting the integrated processing of multiple channels. In the embodiment of the present application, the position of the vertex is adjusted by using the offset and depth information, which makes the visual space position information of the model vertex flexible according to the needs of different channels. This flexibility is one of the key factors supporting multi-channel splicing. Depth information ensures that the correction of the occlusion relationship is accurate and consistent to a certain extent. In different channels, the reasonable use of depth information can ensure that the occlusion relationship under different viewing angles can be correctly processed, thereby supporting the viewpoint consistency of multi-channel splicing. The surface subdivision calculation shader in the embodiment of the present application can be applied to multiple channels at the same time, subdividing and adjusting different model vertices, and finally obtaining the respective target space position information. This process is unified in multiple channels, which helps to splice different information streams. When performing multi-channel splicing, different image data streams can be processed in parallel, and each independently generates target space position information. These independent data streams can be effectively integrated in the final splicing stage. Therefore, the embodiment of the present application supports multi-channel stitching technology by utilizing the independence and adjustability of the visual space position information and depth information, as well as the processing ability of the surface subdivision calculation shader for different vertices. The image data in different channels can be processed independently and then finally integrated to ensure the flexibility and accuracy of the stitching process.

[0044] It should be noted that the computing device involved in the embodiments of the present application may be a server and / or a terminal device.

[0045] The server involved in the embodiments of the present application can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0046] The terminal device involved in the embodiments of the present application may be a device that provides voice and / or data connectivity to a user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. For example, a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device, which exchanges voice and / or data with a wireless access network. For example, a personal communication service (English full name: Personal Communication Service, English abbreviation: PCS) phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (Wireless Local Loop, English abbreviation: WLL) station, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA) and other devices.

[0047] Reference Figure 2 , Figure 2 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. The method comprises steps 201 to 204:

[0048] Step 201: For a rendering model to be processed, first view space position information and depth information of a first model vertex in the rendering model to be processed are obtained.

[0049] In the embodiments of the present application, the model to be rendered refers to a three-dimensional model that needs to go through a rendering process to generate a final image in computer graphics. In practical applications, these models may include various shapes and structures in a virtual scene to represent objects, scenes, and / or characters. Before rendering, the model to be rendered may be just a mathematical description or a data set, which includes basic elements such as vertices, edges, and faces that constitute the model.

[0050] Model vertices are the basic units that make up a 3D model. In 3D graphics, vertices are points that define the shape of a model. They are usually corners or intersections of a model. Vertices contain the spatial position information of the model (such as x, y, z coordinates), and may also include other attributes such as color, texture coordinates, and normal vectors. By connecting these vertices, edges and faces of the model are formed, and then a complete 3D shape is formed.

[0051] Simply put, the model to be rendered is a set of three-dimensional data that needs to be drawn or rendered, and the model vertices are the basic building blocks in these data sets, defining the geometric shape of the model.

[0052] In the embodiments of the present application, the model space position information refers to the coordinate information of each vertex in the three-dimensional model, and these coordinates are defined in the local coordinate system of the model itself. Specifically, the model space (also called object space or local space) is a relative coordinate system, which is centered on the origin of the model and is used to describe the geometric shape and structure inside the model.

[0053] Depth information refers to the distance between any object point in the three-dimensional space of the real scene and the observation point (such as a camera). This information can be represented by a depth image, in which the value of each pixel corresponds to the distance of a point in the scene relative to the observation point. Depth information describes the spatial distance of each point on the surface of an object in a three-dimensional scene relative to the observation point (such as a camera). In short, it reflects the distance relationship of objects in three-dimensional space. The method of obtaining depth information in the embodiments of the present application is described below and will not be expanded here.

[0054] As an optional embodiment, in step 201, the first view space position information and depth information of the first model vertex in the to-be-processed rendering model are obtained, see Figure 3 As shown, it can be implemented as:

[0055] 301, obtaining model space position information of a first model vertex in the model space;

[0056] 302, projecting the model space position information into the visual space to obtain corresponding first visual space position information;

[0057] 303. Calculate depth information of vertices of the first model based on the first view space position information.

[0058] The above steps are described below with reference to specific examples.

[0059] In 301, the model space position information of the first model vertex in the model space is obtained. Specifically, the core of step 301 is to extract the position information of the vertex in the model space from the data set of the three-dimensional model, which is a basic step in the rendering pipeline and lays the foundation for subsequent projection and depth calculation. In step 301, the spatial position information of each vertex needs to be extracted from the data structure of the three-dimensional model. The position of the vertex is usually expressed in three-dimensional coordinates (x, y, z). These coordinate values ​​describe the position of the vertex relative to the origin of the model. The position in the model space is usually stored in the form of floating point numbers and may be in the vertex buffer (Vertex Buffer Object, VBO) or similar data structure of the model. When obtaining vertex position information, it is usually necessary to access these data structures and read the corresponding coordinate values.

[0060] For example, suppose there is a simple three-dimensional cube model, and each vertex has a three-dimensional coordinate in the model space. For example, a vertex of the cube may have coordinates (1.0, 2.0, 3.0) in the model space. In step 301, these coordinate values ​​are extracted from the model data and recorded. The result of step 301 is to obtain the exact position of each vertex in the model space. These position data will be used in subsequent transformation steps, such as converting it to world space or view space for further rendering processing.

[0061] Further optionally, it is assumed that the first view space position information includes at least: the projection space coordinates and / or camera space coordinates of the first model vertex. In an embodiment of the present application, the first view space position information refers to the position of the vertex in the view space. For example, the view space can be a coordinate system in which the camera or observation point is located at the origin, and the z-axis of the view space coordinate system usually points in the direction of the field of view. The coordinates in this space are prepared for subsequent projection calculations. Of course, the view space can also be other coordinate systems, which are only examples here.

[0062] Camera space coordinates refer to the position of the vertex in the camera coordinate system. The vertex coordinates in model space are transformed to camera space through the view matrix transformation. The view matrix describes the position and orientation of the camera, thereby transforming the vertex position in model space to the camera coordinate system.

[0063] Projection space coordinates are the positions of vertices in the projection coordinate system. Projection space coordinates are obtained by further transforming the camera space coordinates of the vertex through a projection matrix (usually a perspective projection or an orthographic projection matrix). The projection matrix defines how to map three-dimensional coordinates to two-dimensional screen coordinates, a process that involves depth testing and clipping operations.

[0064] Based on the above assumptions, in 302, the model space position information of the first model vertex is projected into the visual space to obtain the corresponding visual space position information, which can be implemented as follows: the model space position information of the first model vertex is input into the vertex shader; the model space position information is projected into the projection space and / or the camera space respectively through the view matrix transformation to obtain the projection space coordinates and / or camera space coordinates corresponding to the first model vertex.

[0065] It can be understood that the vertex shader is a component in the graphics rendering pipeline that is responsible for processing and transforming the vertices of the three-dimensional model. It is a programming stage in modern graphics processing units (GPUs) that is mainly used to calculate and modify vertices during the rendering process. The role of the vertex shader is to transform the vertex coordinates of the model from model space to world space, then to camera space, and finally to screen space. This process involves a series of matrix transformation operations (such as model transformation, view transformation, and projection transformation) so that the model can be displayed correctly on the screen. The vertex shader can also calculate other attributes of the vertex, such as lighting information, texture coordinates, and color. These attributes will be used in subsequent rendering stages to generate the final image effect. During the vertex shader processing, the attributes of the vertex will be passed to the fragment shader. During this process, the vertex shader can set some variables so that these attributes can be interpolated to the fragment level in the rasterization stage to generate more delicate rendering effects. The final output of the vertex shader is the transformed and calculated vertex data, including the new position coordinates and other vertex attributes. These output data will be passed to the next stage of the graphics pipeline, the rasterization stage. Vertex shaders are usually defined by developers using high-level shading languages ​​(such as GLSL, HLSL, etc.), which provide a high degree of flexibility and control, allowing the graphics rendering process to be customized and optimized as needed.

[0066] The view matrix transformation involved in the above steps is used to transform the coordinates of objects in the three-dimensional space from the model space (or world space) to the camera space (view space). This process is to simulate the effect of looking at the scene from the perspective of the camera (observation point), so that the position and direction of the object in the camera's field of view are correctly converted and rendered.

[0067] The view matrix transforms the world space coordinates of the object into the camera space coordinates, making the camera the origin of the new coordinate system. In the camera space, the z-axis of the view coordinate system points to the viewing direction of the camera, and the x-axis and y-axis define the horizontal and vertical directions of the camera. Through the view matrix transformation, the coordinates of the three-dimensional object are adjusted to the position under the camera's perspective. This can correctly simulate and render the distance and direction of objects in the scene. Further optionally, the view matrix change can be to first calculate the camera's view vector (the direction from the camera to the target point), the right vector (the orthogonal direction of the view vector), and the up vector (defining the up direction of the camera). Then, these vectors are used to construct a rotation matrix and a translation matrix. The rotation matrix rotates the coordinates of the object into the camera's coordinate system. The translation matrix translates the coordinate system to the camera's position, so that the camera position becomes the origin of the coordinate system. Finally, the rotation matrix and the translation matrix are combined to obtain the view matrix V. In actual calculations, this usually involves matrix multiplication operations to form the final view matrix.

[0068] In this way, the view matrix transforms the coordinates of the object from world space to camera space, so that the position of the object is adjusted relative to the camera position. The transformed coordinates can be used in the clipping and rasterization process to determine which parts of the scene are visible in the camera's field of view and which parts are obscured or beyond the field of view. After this transformation, the position of the object is adjusted relative to the camera's perspective, and the way the object's position is presented in the view will conform to the camera's viewing angle.

[0069] In an optional example, the calculation process of the projection space coordinates and / or the camera space coordinates of the first model vertex is expressed as the following formula:

[0070]

[0071]

[0072]

[0073] in, is the model space coordinate of the first model vertex in the model space coordinate system, is the transformation matrix from the model space coordinate system to the world space coordinate system, is the transformation matrix from the world space coordinate system to the camera space coordinate system, is the transformation matrix from the camera space coordinate system to the projection space coordinate system, is the projection space coordinate of the first model vertex, is the camera space coordinate of the first model vertex.

[0074] The role of these formulas is to gradually transform the coordinates of the vertices from the model space to the final projection space coordinates through a series of matrix transformations. This process ensures the correct rendering of the 3D model on the screen. Each matrix transformation is dealing with a different coordinate system transformation, ensuring that the position of the vertex is correctly mapped in different coordinate systems so that it can be finally displayed on the screen. Specifically, the model transformation matrix determines the position of the model in the world. The view transformation matrix determines the position and orientation of the camera. The projection transformation matrix determines how to map the 3D coordinates to the 2D screen coordinates. The combination of these three matrices ensures the coordinate transformation in the entire rendering pipeline from the 3D model to the final screen image.

[0075] Continuing with the above assumption, in 303, the depth information of the first model vertex is calculated based on the first visual space position information, which can be implemented as follows:

[0076] First, the z-axis coordinate component and the w-axis coordinate component of the first model vertex in the projection space are extracted from the first visual space position information. is the z-axis coordinate component of the first model vertex, is the w-axis coordinate component of the first model vertex. Exemplarily, in the projection space, the coordinates of the vertex include three-dimensional coordinates and two homogeneous coordinate components z and w, which are used to represent depth information and perform normalization operations.

[0077] Then, the ratio of the z-axis coordinate component to the w-axis coordinate component of the first model vertex is used as the depth information. The process of obtaining the depth information is as follows: .

[0078] It can be understood that by Divide by the w-axis coordinate component , which can convert the coordinates in the projection space into the depth value in the standardized device coordinates. The depth information here is It is a normalized depth value that represents the distance of the first model vertex relative to the camera.

[0079] In the embodiment of the present application, the depth information provides the relative depth of the vertex. By calculating the z-axis coordinate component divided by the w-axis coordinate component, the depth value of the projection space is calculated and normalized to the range of [0, 1] for comparison with other vertices. This process ensures the correctness and consistency of the depth value and is suitable for the calculation of the depth buffer (Z-buffer).

[0080] In the projection space, the depth information may vary due to different views, and it can be converted to a depth value in the normalized device coordinate system by dividing by . This normalization process maps the depth information to a suitable range in the screen space, which is convenient for further image processing and rendering.

[0081] The depth information calculated here can be used for depth testing to determine which pixels should be rendered and which pixels are occluded. The depth test determines which objects are in front by comparing the depth values ​​of different pixels, thereby achieving occlusion culling and realistic image rendering effects. In the rendering pipeline, the depth buffer is used to store the depth value of each pixel. Calculating and correctly applying depth information can effectively avoid object occlusion and overlap problems, ensuring that the final rendered image has an accurate three-dimensional effect. By calculating depth information and using depth testing, objects that are not in the field of view can be rendered less, thereby improving rendering performance and efficiency.

[0082] In the above steps, the process of calculating depth information based on visual space information is mainly to divide the z-axis coordinate component in the projection space by the w-axis coordinate component to calculate the standardized depth value. This process ensures the accuracy of the depth information, so that the final image can correctly display the front and back relationship of the object, and supports depth testing and other depth-related image processing technologies.

[0083] Step 202: Perform surface subdivision processing on the first model vertices to obtain second model vertices and corresponding second view space position information.

[0084] In the embodiment of the present application, tessellation is used to add details and smoothness to the surface of a three-dimensional model. Tessellation, or tessellation processing technology, is a technology that automatically inserts new vertices between the vertices of a model during graphics processing. These newly inserted vertices can increase the details of the model and make the surface look smoother and more delicate. Tessellation technology subdivides the surface by inserting additional vertices between the vertices of the original model. These new vertices are not added at will, but are calculated according to certain algorithms and rules to ensure that the subdivided surface maintains continuity and smoothness. Through tessellation processing, a second model that is more refined than the original model can be obtained. The second model has more vertices and a more complex geometric structure, and can more accurately represent the shape and details of the object. At the same time, the second visual space position information corresponding to the second model will also be updated accordingly to reflect the accurate position of the subdivided model in the three-dimensional space. Surface tessellation is widely used in movies, games, product design and other fields to improve the visual effects and realism of three-dimensional models. For example, in film and game production, tessellation can be used to create characters and scenes with rich details and realistic textures; in product design, tessellation can help designers control the shape and appearance of the product more finely.

[0085] In summary, tessellation is a powerful graphics processing technology that can significantly improve the visual effect and realism of a three-dimensional scene by increasing the vertices and details of the model. In the embodiment of the present application, the tessellation processing of the vertices of the first model is to obtain a more refined second model and the corresponding second visual space position information.

[0086] Specifically, in step 202, the first model vertices are subjected to surface subdivision processing to obtain the second model vertices and the corresponding second view space position information, which can be implemented as follows:

[0087] The first model vertex is input into the tessellation control shader, and the edge subdivision level of the triangle patch where the first model vertex is located is calculated; based on the edge subdivision level, the first model vertex is tessellated through the tessellation calculation shader to obtain the second model vertex and the second visual space position information corresponding to the second model vertex.

[0088] In an embodiment of the present application, the tessellation control shader (TCS) is an important component of the graphics rendering pipeline, which is specifically used to control the tessellation process. It plays a key role in the rendering pipeline at the tessellation stage. The tessellation control shader is responsible for determining the subdivision level of the triangle patch (or other primitive) where each input vertex is located. These levels determine how to subdivide based on the original model and how many sub-patches the subdivided patch will be divided into. The tessellation control shader generates subdivision scheduling data, which is used for actual vertex generation and calculation in the tessellation evaluation shader (TES).

[0089] During the tessellation process, the vertex data of the initial model is input into the tessellation control shader. The control shader receives this vertex data and determines the areas that need to be tessellated. The control shader calculates the tessellation level for each triangle patch based on the input tessellation strategy (which may include view distance, geometric complexity, etc.). For example, the tessellation level can be dynamically adjusted based on the distance from the camera to ensure that there are more details in the areas closer to the camera. The control shader generates the tessellation level information and passes it to the tessellation compute shader (TES), which will use this information to actually generate the tessellated vertices and patches. In this way, the tessellation control shader can enhance the details of the model by determining the tessellation level, making the model surface smoother and more delicate. For complex models, adjusting the tessellation level can add more vertices in certain areas of the model to improve details. By controlling the tessellation level during the tessellation stage, you can balance the rendering quality and performance requirements. The tessellation control shader can dynamically adjust the tessellation level based on factors such as viewpoint and distance, thereby improving visual effects while optimizing rendering performance. The tessellation control shader allows developers to flexibly set the tessellation strategy according to their needs, which is of great significance for different application scenarios (such as games, movie special effects, etc.). It provides a smooth transition between different levels of detail and adapts to different display and performance requirements.

[0090] The tessellation control shader is an important stage in the graphics rendering pipeline, used to control and calculate the model tessellation process. It determines how to subdivide the model's patches, thus affecting the model details and visual effects of the final rendering. By adjusting the subdivision level, the tessellation control shader can improve the model details while optimizing rendering performance and achieving high-quality visual effects.

[0091] In an embodiment of the present application, the strip edge subdivision level is used to indicate the range of the number of surface subdivision processing times corresponding to each triangle edge in the triangle patch. The strip edge subdivision level is used to control the subdivision degree of the triangle patch. It specifically indicates how many times each triangle edge needs to be subdivided. The strip edge subdivision level is an integer value, which indicates how many smaller segments each triangle edge is divided into during the subdivision process. For example, a strip edge subdivision level of 2 means that each edge is divided into 2 segments, resulting in 3 vertices (including the two endpoints of the original edge). The subdivision level determines the number of times each edge is subdivided. For example, if the strip edge subdivision level of the triangle is 3, then each edge will be divided into 3 segments, forming 4 new vertices. In this way, the original triangle will be subdivided into more small triangles, thereby increasing the details and smoothness of the model.

[0092] It can be explained that the Tessellation Control Shader (TCS) receives vertex data for each triangle and calculates the tessellation level for each edge based on the input tessellation strategy and requirements. These calculations can be based on a variety of factors, such as view distance, detail requirements, model complexity, etc. The Tessellation Control Shader generates the tessellation level information for each edge and passes it to the Tessellation Compute Shader (TES). The TES uses this level information to actually perform the tessellation operation, generating new vertices and patches. In this way, by setting the tessellation level of the edge, you can precisely control the level of detail of the model. The higher the tessellation level, the more each edge is divided, and the higher the detail and smoothness of the model. This is very important for creating complex geometry and improving visual effects. Dynamic adjustment of the tessellation level of the edge can help balance performance and graphics quality during rendering. For example, at a distance from the camera, a lower tessellation level can be used to reduce the amount of calculations; at a distance from the camera, a higher tessellation level can be used to obtain higher details. Precisely controlling the number of tessellation processes for each edge helps improve rendering effects and the realism of the model. The subdivided model can present smoother surfaces and finer details, improving the overall visual experience. The adjustment of the strip subdivision level allows for flexibility in different application scenarios. For example, in games, the subdivision level can be adjusted according to the real-time performance of the game; in film production, a higher subdivision level may be used to obtain the best visual effects.

[0093] In summary, the strip edge subdivision level is used to indicate the range of times each edge in a triangle patch is processed during the subdivision process. It is a key parameter for controlling the details and smoothness of the model. By adjusting the strip edge subdivision level, you can accurately control the degree of subdivision of the model, thereby balancing rendering performance and graphics quality. The surface subdivision control shader uses this level information to generate subdivided vertices and patches, so that the final model can achieve higher details and better visual effects.

[0094] Optionally, the number of tessellation processes corresponding to each triangle edge in the triangle patch is associated with the angle formed by connecting the two end vertices of the triangle patch and the viewpoint. In the triangle patch, the angle is formed by connecting the viewpoint and the two vertices in the triangle patch. This angle can help determine the degree of attention of the viewpoint to the triangle patch. Figure 4 As shown, the angle formed by the line connecting the two end vertices of the triangle patch and the viewpoint can be Figure 4 The angle Angle shown in .

[0095] Furthermore, the greater the angle, the more times the surface is subdivided. When the angle is large, the viewpoint is farther from the triangle, which means that the patch is smaller in the view. In order to ensure that the details of distant objects are not too rough, it is usually necessary to increase the number of subdivisions to ensure the smoothness and integrity of the details. When the angle is small, the viewpoint is closer to the triangle patch, and the patch occupies a larger space in the view. At this time, the need for subdivision is lower because the details are sufficient to be displayed, and the number of subdivisions can be appropriately reduced to optimize computational efficiency.

[0096] It can be found that by adjusting the number of subdivision processes according to the angle, the details can be increased when the viewpoint is far away from the triangle, thereby improving the visual quality of distant objects. When the viewpoint is close to the triangle, the details are sufficient, so reducing the number of subdivision processes helps optimize performance. Reducing the number of subdivisions in areas close to the viewpoint can significantly reduce the amount of calculation and improve rendering efficiency. This is particularly important for real-time rendering (such as games) and helps maintain a smooth frame rate. By reasonably adjusting the number of subdivision processes, a smooth transition from far to near can be achieved, avoiding the visual effects affected by insufficient details while ensuring that the details are rich enough. This method allows the degree of subdivision to be flexibly adjusted according to different application requirements. For specific viewing areas that require high details, a higher subdivision level can be used; for other areas, a lower subdivision level is used.

[0097] For example, the process of obtaining the number of tessellation processing times corresponding to each triangle edge is expressed as the following formula:

[0098]

[0099] in, Indicates the number of tessellation processes corresponding to the triangle edges in the current triangle patch. Indicates the maximum number of surface subdivisions, Indicates the angle formed by the line connecting the two end vertices and the viewpoint in the current triangle patch. Indicates a preset constant. For example, It can be set to 1. The preset constant is used to adjust the sensitivity of this relationship to ensure that the subdivision process meets the requirements. Furthermore, the maximum number of subdivisions is determined based on the graphics card settings, such as Can be read from the GPU.

[0100] The above formula allows the number of subdivisions to be adjusted dynamically according to the angle between the viewpoint and the triangle, so that the subdivision process can provide appropriate details and smoothness at different viewing angles. Through the above formula, the degree of subdivision can be dynamically adjusted considering the angle between the viewpoint and the vertex to improve the efficiency and effect of rendering. By associating the number of subdivisions of each triangle edge in the triangle patch with the angle formed by the line connecting the viewpoint and the vertex, the degree of subdivision can be dynamically adjusted according to the viewpoint position. This method not only improves the detail performance of the model, but also optimizes the rendering performance, making it rich in details and more computationally efficient, and is particularly suitable for real-time graphics applications that require efficient rendering.

[0101] For example, see Figure 5 As shown, the angle Angle can be subdivided. The subdivision angle here can be set to 1°. For further information, see Figure 6 , after the above surface subdivision process, we can get Figure 6 Multiple triangles are shown.

[0102] Step 203: Obtain the offset of the first model vertex relative to the reference viewpoint, and reversely offset the second model vertex based on the offset to obtain the third view space position information of the offset second model vertex.

[0103] In the embodiment of the present application, the reference viewpoint involved in step 203 refers to a reference viewpoint used to calculate the position and offset of the model vertex. The reference viewpoint is a predefined reference point in three-dimensional space, which is used to calculate the view space position and offset of the model vertex. It can be a fixed viewpoint or reference position, which is compared with other viewpoints of the model (such as the current viewpoint or other points in the world space coordinate system). The reference viewpoint is used to calculate the offset of the vertex relative to this reference point. This offset is usually a vector from the reference viewpoint to the model vertex, which is used to determine the change in the position of the vertex relative to the reference viewpoint. Through the reference viewpoint, the vertex can be converted from one view space coordinate system to another view space coordinate system. This is very important for the transformation and view adjustment of the model, especially when performing coordinate conversion between different perspectives.

[0104] The reference viewpoint is a reference point used to calculate the position and offset of the model vertices. In step 203, the reference viewpoint is used to obtain the offset of the first model vertex relative to the viewpoint, and the second model vertex is reversely offset based on the offset to obtain its position in the new view space. This process is a key part of the model view transformation and position adjustment, which helps to achieve accurate rendering effects and view conversion.

[0105] In the above step 203, obtaining the offset of the first model vertex relative to the reference viewpoint can be implemented as follows: determining the type of image currently being processed. If the type of image currently being processed is a left-eye image, half of the interpupillary distance between the two eyes is used as the absolute offset, and the negative value of the absolute offset is taken as the first offset corresponding to the left-eye image. If the type of image currently being processed is a right-eye image, half of the interpupillary distance between the two eyes is used as the absolute offset, and the positive value of the absolute offset is taken as the second offset corresponding to the right-eye image.

[0106] In an embodiment of the present application, the interpupillary distance (IPD) refers to the distance between the two eyes. In stereoscopic vision, this distance is used to simulate the parallax of the two eyes, thereby creating a sense of depth. For example, half of the interpupillary distance is used as the absolute offset because the parallax between the two eyes is based on this distance. Exemplarily, the interpupillary distance can be set to a static value, for example, one is selected from the statistical values ​​of the human interpupillary distance range, that is, between 55mm and 75mm. Of course, the interpupillary distance can also be dynamically adjusted according to the user's height, gender, body shape, face shape, etc.

[0107] In the embodiment of the present application, the image type is a left eye image or a right eye image. In stereoscopic vision and stereoscopic image processing, the left eye image and the right eye image refer to two view images, which correspond to the perspectives of the left eye and the right eye, respectively. These two views are the core parts of realizing stereoscopic vision, and create a three-dimensional sense of depth by simulating the parallax of the human eye. The left eye image is an image taken or rendered from the perspective of the left eye, simulating the scene seen by the left eye. In a stereoscopic vision system, the left eye image usually has a left perspective offset relative to a reference viewpoint (for example, a center point or a virtual camera position). This offset is usually equal to half the interpupillary distance between the two eyes (negative value). The right eye image is not repeated. The left eye image and the right eye image are used together to generate a stereoscopic visual effect, so that the viewer can feel the depth and stereoscopic sense.

[0108] For the left eye image, the offset is based on the reference viewpoint moving to the left. Therefore, we use half of the interpupillary distance between the two eyes as the absolute offset and take a negative value. For the right eye image, the offset is based on the reference viewpoint moving to the right. Therefore, we use half of the interpupillary distance between the two eyes as the absolute offset and take a positive value.

[0109] In step 203, the image viewing angle is adjusted based on the offset of the interpupillary distance between the two eyes to achieve a stereoscopic visual effect. For the left eye image and the right eye image, negative and positive offsets are used respectively to simulate the parallax between the two eyes, thereby visually generating a sense of depth. This method is crucial in stereoscopic image rendering and helps create a realistic three-dimensional visual experience. For example, see Figure 7As shown, assuming that the interpupillary distance between both eyes is PD, based on this, the absolute offset can be set to PD / 2.

[0110] In the above step 203, the second model vertex is reversely offset based on the offset to obtain the third view space position information of the second model vertex after the offset, which can be implemented as follows:

[0111] In the tessellation calculation shader, the sum of the offset of the corresponding image type and the x-axis coordinate component in the second view space position information of the second model vertex is used as the third view space position information of the second model vertex.

[0112] Here, it should be noted that the tessellation compute shader is a stage in the graphics pipeline that is used to subdivide the surface of a 3D model to make it smoother and more detailed. The process of tessellation usually involves subdividing a coarse polygonal mesh into more small polygons, thereby improving the geometric accuracy and visual effect of the model. In modern graphics APIs (such as DirectX or OpenGL), the tessellation compute shader usually includes three main stages: the tessellation control shader decides how to subdivide the base mesh into finer meshes; the tessellation evaluation shader calculates the new position of each vertex in the subdivided mesh; the compute shader is used for further computing tasks, such as data processing or additional tessellation operations.

[0113] In the graphics pipeline, the tessellation compute shader is located in the tessellation stage of the rendering pipeline, which is used to further process vertices to generate higher resolution tessellation surfaces. This stage is usually immediately after the vertex shader to create the final tessellation mesh data for use in subsequent rendering stages.

[0114] Exemplarily, in the above steps, for the left eye image or the right eye image, according to different viewpoint offsets (such as half of the interpupillary distance between the two eyes), these offsets are applied in the tessellation calculation shader. In the tessellation calculation shader, the offset corresponding to the image type is added to the x-axis coordinate component of the second model vertex in the second view space position. This calculation is to adjust the position of the second model vertex to make it conform to the viewpoint offset of the target view.

[0115] Here, the tessellation compute shader is a stage in the graphics rendering pipeline that is used to further subdivide the model mesh to make it smoother and more detailed. It operates on the mesh during the tessellation control and evaluation stages to generate higher resolution tessellation surfaces.

[0116] Furthermore, after obtaining the third visual space position information of the second model vertex after offset, the original projection matrix is ​​used to transform the offset second model vertex to the projection position on the screen. Specifically, it is expressed as: .in, is the projection space coordinate of the second model vertex, is the third view space position coordinate after the second model vertex is offset, It is the transformation matrix from the camera space coordinate system to the projection space coordinate system.

[0117] Step 204: Correct the occlusion relationship of the third view space position information by using the depth information to obtain the target space position information of the to-be-processed rendering model.

[0118] Specifically, in step 204, in the tessellation calculation shader, the product of the third view space position information of the second model vertex and the depth information is used as the target space position information of the to-be-processed rendering model.

[0119] As mentioned above, depth information refers to the distance value of each pixel in the scene relative to the viewpoint, indicating the front-back relationship of the point in the view. Depth information is used to determine which objects or points are blocked by other objects, thereby determining which parts should be rendered. In step 204, by applying the depth information to the third view space position, the visibility of the model can be corrected to ensure that only the foreground objects are displayed in the final rendering result, while the blocked parts are hidden.

[0120] In some implementations, continuing with the setting in the above example, in step 204, the depth information may be used Correct the offset vertex depth to eliminate the occlusion relationship changes caused by vertex offset. This process can be expressed as: ;in, is the z-axis coordinate component of the second model vertex after offset, is the w-axis coordinate component of the second model vertex after offset; is the original depth information.

[0121] In an embodiment of the present application, for a rendering model to be processed, the first visual space position information and depth information of the first model vertex in the rendering model to be processed are obtained; the first model vertex is subjected to surface subdivision processing to obtain the second model vertex and the corresponding second visual space position information; the offset of the first model vertex relative to the reference viewpoint is obtained, and the second model vertex is reversely offset based on the offset to obtain the third visual space position information of the second model vertex after the offset; the occlusion relationship of the third visual space position information is corrected by the depth information to obtain the target space position information of the rendering model to be processed. In an embodiment of the present application, by reversely offsetting each vertex according to the pupil distance, the consistency of the viewpoint position of all channels when drawing the left eye or right eye picture is ensured, and seamless splicing between multiple channels is achieved. And the deformation problem after vertex offset is solved by surface subdivision, which effectively solves the problems existing in the current multi-channel seamless splicing 3D display, improves the quality and display effect of image rendering, and helps to adapt to the display requirements of ultra-large viewing areas.

[0122] An image processing method in an embodiment of the present application is described above. An image processing device (such as a server) that executes the above image processing method is introduced below.

[0123] See also Figure 8 ,like Figure 8 The image processing device in the embodiment of the present application can realize the above Figure 2 The steps of the image processing method executed in the corresponding embodiment. The functions implemented by the image processing device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. The image processing device may include an input acquisition module, a surface subdivision module, an offset module, and a correction module. The functional implementation of each module can refer to Figure 2 The operations performed in the corresponding embodiments are not described in detail here.

[0124] Specifically, the acquisition module is configured to acquire, for the to-be-processed rendering model, first visual space position information and depth information of a first model vertex in the to-be-processed rendering model;

[0125] A tessellation module is configured to perform tessellation processing on the first model vertices to obtain second model vertices and corresponding second visual space position information;

[0126] An offset module is configured to obtain an offset of a first model vertex relative to a reference viewpoint, and reversely offset a second model vertex based on the offset to obtain third view space position information of the offset second model vertex;

[0127] The correction module is configured to correct the occlusion relationship of the third visual space position information according to the depth information to obtain the target space position information of the rendering model to be processed.

[0128] In some implementations, the acquisition module, which acquires the first visual space position information and depth information of the first model vertex in the to-be-processed rendering model, is configured as follows:

[0129] Obtaining model space position information of the first model vertex in the model space;

[0130] Projecting the model space position information into the visual space to obtain corresponding first visual space position information;

[0131] Depth information of vertices of the first model is calculated based on the first view space position information.

[0132] In some embodiments, the first visual space position information includes at least: projection space coordinates and / or camera space coordinates of the first model vertex; an acquisition module projects the model space position information into the visual space to obtain corresponding visual space position information, and is configured as follows:

[0133] Inputting model space position information of the first model vertex into the vertex shader;

[0134] The model space position information is projected into the projection space and / or the camera space respectively through the view matrix transformation to obtain the projection space coordinates and / or the camera space coordinates corresponding to the first model vertex.

[0135] In some embodiments, the calculation process of the projection space coordinates and / or camera space coordinates of the first model vertex is expressed as the following formula:

[0136]

[0137]

[0138]

[0139] in, is the model space coordinate of the first model vertex in the model space coordinate system, is the transformation matrix from the model space coordinate system to the world space coordinate system, is the transformation matrix from the world space coordinate system to the camera space coordinate system, is the transformation matrix from the camera space coordinate system to the projection space coordinate system, is the projection space coordinate of the first model vertex, is the camera space coordinate of the first model vertex.

[0140] In some implementations, the acquisition module calculates the depth information of the first model vertex based on the first view space position information, and is configured to:

[0141] Extract the z-axis coordinate component and the w-axis coordinate component of the first model vertex in the projection space from the first visual space position information; wherein, is the z-axis coordinate component of the first model vertex, is the w-axis coordinate component of the first model vertex;

[0142] The ratio of the z-axis coordinate component to the w-axis coordinate component of the first model vertex is used as the depth information; wherein the process of obtaining the depth information is expressed as follows: .

[0143] In some implementations, the tessellation module performs tessellation processing on the first model vertices to obtain the second model vertices and the corresponding second visual space position information, and is configured as follows:

[0144] Input the first model vertex into the tessellation control shader, and calculate the strip edge tessellation level of the triangle patch where the first model vertex is located; the strip edge tessellation level is used to indicate the range of the number of tessellation processing times corresponding to each triangle edge in the triangle patch;

[0145] Based on the edge subdivision level, the first model vertices are subdivided by a surface subdivision calculation shader to obtain second model vertices and second visual space position information corresponding to the second model vertices.

[0146] In some embodiments, the number of tessellation processing times corresponding to each triangle edge in a triangular patch is associated with the angle formed by connecting the two end vertices of the triangular patch and the viewpoint; the larger the degree of the angle, the more tessellation processing times.

[0147] In some implementations, the process of obtaining the number of tessellation processing times corresponding to each triangle edge is expressed as the following formula:

[0148]

[0149] in, Indicates the number of tessellation processes corresponding to the triangle edges in the current triangle patch. Indicates the maximum number of tessellation times, which is determined based on the graphics card settings. Indicates the angle formed by the line connecting the two end vertices and the viewpoint in the current triangle patch. Indicates a preset constant.

[0150] In some embodiments, the offset module, which obtains the offset of the first model vertex relative to the reference viewpoint, is configured to:

[0151] Determine the type of image currently being processed; wherein the image type is a left-eye image or a right-eye image;

[0152] If the image type currently being processed is a left-eye image, half of the interpupillary distance between the two eyes is used as the absolute offset, and the negative value of the absolute offset is taken as the first offset corresponding to the left-eye image;

[0153] If the image type currently being processed is a right-eye image, half of the interpupillary distance between the two eyes is used as the absolute offset, and the positive value of the absolute offset is taken as the second offset corresponding to the right-eye image;

[0154] The offset module performs reverse offset on the second model vertex based on the offset to obtain the third visual space position information of the second model vertex after offset, and is configured as follows:

[0155] In the tessellation calculation shader, the sum of the offset of the corresponding image type and the x-axis coordinate component in the second view space position information of the second model vertex is used as the third view space position information of the second model vertex.

[0156] In some implementations, the correction module performs occlusion relationship correction on the third visual space position information through the depth information to obtain the target space position information of the to-be-processed rendering model, and is configured as follows:

[0157] In the surface subdivision calculation shader, the product of the third view space position information of the second model vertex and the depth information is used as the target space position information of the to-be-processed rendering model.

[0158] In an embodiment of the present application, the image processing device ensures the consistency of the viewpoint position by reversely offsetting each vertex according to the pupil distance, achieves seamless stitching between multiple channels, improves the quality of image rendering and display effects, and helps to adapt to the display requirements of ultra-large viewing areas.

[0159] The image processing device in the embodiment of the present application is described above from the perspective of modular functional entities. The image processing device in the embodiment of the present application is described below from the perspective of hardware processing.

[0160] It should be noted that Figure 8 The devices shown can all have Fig. 9 The structure shown, when Figure 8 The image processing device shown has the following Fig. 9 When the structure shown is Fig. 9 The processor in the device can realize the same or similar functions of each module provided by the device embodiment corresponding to the device, Fig. 9 The memory in the storage device stores a computer program that needs to be called when the processor executes the above-mentioned image processing method.

[0161] For example, the memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0162] An embodiment of the present application further relates to a processor, which includes multiple storage units for calling a computer program or computer instruction stored in the memory so that the processor executes the method described in any of the above embodiments.

[0163] For example, in an embodiment of the present application, the processor is an integrated circuit chip with signal processing capabilities. For example, the processor can be an FPGA, a general-purpose processor, a DSP, an ASIC or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, a SoC, a CPU, a network processor (network processor, NP), a microcontroller (micro controller unit, MCU), a PLD or other integrated chip, which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. In a possible implementation, the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a program code, and when the program code is run on the computer, the computer executes the above method embodiment.

[0164] The present application also provides a terminal device, such as Fig.10 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc., taking the mobile phone as an example:

[0165] Fig.10 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.10 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1010, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Fig.10 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0166] Combine the following Fig.10 A detailed introduction to the various components of the mobile phone:

[0167] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0168] The memory 1010 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1010. The memory 1010 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1010 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0169] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0170] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Fig.10 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0171] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0172] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to, for example, another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1010 for further processing.

[0173] Wi-Fi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the Wi-Fi module 1070. It provides users with wireless broadband Internet access. Fig.10 A Wi-Fi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0174] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1010 and calling data stored in the memory 1010, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; optionally, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0175] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 1080 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.

[0176] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0177] In the embodiment of the present application, the processor 1080 included in the mobile phone also has the function of controlling and executing the above image processing method flow executed by the image processing device.

[0178] The present application also provides a server. Fig.11 , Fig.11 1 is a schematic diagram of a server structure provided by an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1122 (for example, one or more processors) and memory 1132, and one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 may be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1122 may be configured to communicate with the storage medium 1130 to execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0179] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0180] The steps performed by the server in the above embodiment can be based on the Fig.11 For example, the steps performed by the data processing device or the image processing device in the above-mentioned embodiment can be based on the structure of the server 1100. Fig.11For example, the central processor 1122 performs the following operations by calling the instructions in the memory 1132: for the to-be-processed rendering model, obtain the first visual space position information and depth information of the first model vertex in the to-be-processed rendering model; perform surface subdivision processing on the first model vertex to obtain the second model vertex and the second visual space position information; obtain the offset of the first model vertex relative to the reference viewpoint, and reversely offset the second model vertex based on the offset to obtain the third visual space position information of the offset second model vertex; perform occlusion relationship correction on the third visual space position information by using the depth information to obtain the target space position information.

[0181] Full-field stereo rendering can also be performed through the input and output interface 1158.

[0182] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0183] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0184] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0185] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0187] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0188] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0189] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: For the rendering model to be processed, obtaining first view space position information and depth information of a first model vertex in the rendering model to be processed; Performing surface subdivision processing on the first model vertices to obtain second model vertices and corresponding second visual space position information; Obtaining an offset of a first model vertex relative to a reference viewpoint, and performing a reverse offset on a second model vertex based on the offset to obtain third view space position information of the offset second model vertex; The occlusion relationship of the third visual space position information is corrected by using the depth information to obtain the target space position information of the rendering model to be processed; The step of obtaining the first visual space position information and depth information of the first model vertex in the to-be-processed rendering model includes: Obtaining model space position information of the first model vertex in the model space; Projecting the model space position information into the visual space to obtain corresponding first visual space position information; Calculating depth information of vertices of the first model based on the first visual space position information; Wherein, the first visual space position information includes at least: the projection space coordinates and / or camera space coordinates of the first model vertex; The projecting the model space position information into the visual space to obtain the corresponding visual space position information includes: Inputting model space position information of the first model vertex into the vertex shader; Projecting the model space position information to the projection space and / or the camera space respectively through view matrix transformation to obtain the projection space coordinates and / or the camera space coordinates corresponding to the first model vertex; The calculation process of the projection space coordinates and / or camera space coordinates of the first model vertex is expressed as the following formula: in, is the model space coordinate of the first model vertex in the model space coordinate system, is the transformation matrix from the model space coordinate system to the world space coordinate system, is the transformation matrix from the world space coordinate system to the camera space coordinate system, is the transformation matrix from the camera space coordinate system to the projection space coordinate system, is the projection space coordinate of the first model vertex, is the camera space coordinate of the first model vertex; The step of calculating the depth information of the first model vertices based on the first visual space position information includes: Extract the z-axis coordinate component and the w-axis coordinate component of the first model vertex in the projection space from the first visual space position information; wherein, is the z-axis coordinate component of the first model vertex, is the w-axis coordinate component of the first model vertex; The ratio of the z-axis coordinate component to the w-axis coordinate component of the first model vertex is used as the depth information; wherein the process of obtaining the depth information is expressed as follows: .

2. The image processing method according to claim 1, characterized in that: The step of performing surface subdivision processing on the first model vertices to obtain the second model vertices and corresponding second visual space position information includes: Input the first model vertex into the tessellation control shader, and calculate the strip edge tessellation level of the triangle patch where the first model vertex is located; the strip edge tessellation level is used to indicate the range of the number of tessellation processing times corresponding to each triangle edge in the triangle patch; Based on the edge subdivision level, the first model vertices are subdivided by a surface subdivision calculation shader to obtain second model vertices and second visual space position information corresponding to the second model vertices.

3. The image processing method according to claim 2, characterized in that: The number of tessellation processes corresponding to each triangle edge in the triangle patch is associated with the angle formed by the line connecting the two end vertices of the triangle patch and the viewpoint; The larger the angle is, the more times the surface subdivision process is performed.

4. The image processing method according to claim 3, characterized in that: The process of obtaining the number of surface subdivision processing times corresponding to each triangle edge is expressed as the following formula: in, Indicates the number of tessellation processes corresponding to the triangle edges in the current triangle patch. Indicates the maximum number of tessellation times, which is determined based on the graphics card settings. Indicates the angle formed by the line connecting the two end vertices and the viewpoint in the current triangle patch. Indicates a preset constant.

5. The image processing method according to claim 1, characterized in that: The step of obtaining the offset of the first model vertex relative to the reference viewpoint includes: Determine the type of image currently being processed; wherein the image type is a left-eye image or a right-eye image; If the image type currently being processed is a left-eye image, half of the interpupillary distance between the two eyes is used as the absolute offset, and the negative value of the absolute offset is taken as the first offset corresponding to the left-eye image; If the image type currently being processed is a right-eye image, half of the interpupillary distance between the two eyes is used as the absolute offset, and the positive value of the absolute offset is taken as the second offset corresponding to the right-eye image; The step of reversely offsetting the second model vertex based on the offset to obtain the third view space position information of the offset second model vertex includes: In the tessellation calculation shader, the sum of the offset of the corresponding image type and the x-axis coordinate component in the second view space position information of the second model vertex is used as the third view space position information of the second model vertex.

6. The image processing method according to claim 1, characterized in that: The step of correcting the occlusion relationship of the third visual space position information by using the depth information to obtain the target space position information of the to-be-processed rendering model includes: In the surface subdivision calculation shader, the product of the third view space position information of the second model vertex and the depth information is used as the target space position information of the to-be-processed rendering model.

7. An image processing device, characterized in that: The device includes: An acquisition module is configured to acquire, for a to-be-processed rendering model, first visual space position information and depth information of a first model vertex in the to-be-processed rendering model; A tessellation module is configured to perform tessellation processing on the first model vertices to obtain second model vertices and corresponding second visual space position information; An offset module is configured to obtain an offset of a first model vertex relative to a reference viewpoint, and reversely offset a second model vertex based on the offset to obtain third view space position information of the offset second model vertex; A correction module is configured to correct the occlusion relationship of the third visual space position information by using the depth information to obtain the target space position information of the rendering model to be processed; The acquisition module, which acquires the first visual space position information and depth information of the first model vertex in the to-be-processed rendering model, is configured as follows: Obtaining model space position information of the first model vertex in the model space; Projecting the model space position information into the visual space to obtain corresponding first visual space position information; Calculating depth information of vertices of the first model based on the first visual space position information; Wherein, the first visual space position information includes at least: the projection space coordinates and / or camera space coordinates of the first model vertex; The acquisition module projects the model space position information into the visual space to obtain the corresponding visual space position information, and is configured as follows: Inputting model space position information of the first model vertex into the vertex shader; Projecting the model space position information to the projection space and / or the camera space respectively through view matrix transformation to obtain the projection space coordinates and / or the camera space coordinates corresponding to the first model vertex; The calculation process of the projection space coordinates and / or camera space coordinates of the first model vertex is expressed as the following formula: in, is the model space coordinate of the first model vertex in the model space coordinate system, is the transformation matrix from the model space coordinate system to the world space coordinate system, is the transformation matrix from the world space coordinate system to the camera space coordinate system, is the transformation matrix from the camera space coordinate system to the projection space coordinate system, is the projection space coordinate of the first model vertex, is the camera space coordinate of the first model vertex; The acquisition module calculates the depth information of the first model vertex based on the first visual space position information, and is configured as follows: Extract the z-axis coordinate component and the w-axis coordinate component of the first model vertex in the projection space from the first visual space position information; wherein, is the z-axis coordinate component of the first model vertex, is the w-axis coordinate component of the first model vertex; The ratio of the z-axis coordinate component to the w-axis coordinate component of the first model vertex is used as the depth information; wherein the process of obtaining the depth information is expressed as follows: .

8. A computing device, characterized in that It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the image processing method according to any one of claims 1 to 6.

10. A chip, characterized in that: The chip includes a processor coupled to the transceiver, and is used to execute the image processing method according to any one of claims 1 to 6.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images