Image processing method and device and computing equipment

Abstract

The invention discloses an image processing method. The method comprises the following steps: receiving a plurality of pieces of grid data of a component; transmitting the plurality of grid data to a GPU; clustering the plurality of pieces of grid data to obtain a plurality of pieces of grid clustering data; transmitting the plurality of grid clustering data to a GPU (Graphics Processing Unit); and the GPU is used for rendering the plurality of pieces of grid data and rendering the plurality of pieces of grid clustering data after receiving the plurality of pieces of grid clustering data. In the embodiment of the invention, since the grid data needs a certain time to be processed into the grid clustering data, the CPU can directly transmit a plurality of pieces of grid data to the GPU for rendering after receiving the grid data of one component, the GPU is prevented from being in an idle state in the process of generating the grid clustering data by the CPU, and the processing efficiency of the CPU is improved. Therefore, the screen cannot timely display the rendered image. After the GPU receives the grid clustering data, the grid data can be deleted, and then the grid clustering data is rendered, so that a patch after one component is dispersed can be added to a GPU video memory at one time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method, device and computing equipment. BACKGROUND

[0002] In the modern industrial design process, computer aided design (CAD) and computer aided engineering (CAE) are indispensable design tools. When engineers and designers use CAD or CAE to design a physical model, non-uniform rational B-spline (NURBS) can be used to accurately describe various complex geometric shapes. When CAD or CAE renders the physical model, NURBS will discretize the continuous curve into mesh patches. If CAD or CAE designs a high-precision and high-complexity physical model, NURBS will discretize hundreds of millions of mesh patches, resulting in very high computational load for the processor to render. SUMMARY

[0003] To solve the above problems, an image processing method is provided in the embodiments of the present application. After receiving mesh data of a new component, the mesh data is directly transmitted to a GPU for rendering before clustering the mesh data, so as to avoid that the screen cannot timely display the rendered image. In addition, the present application also provides an image processing device and a computing equipment corresponding to the image processing method.

[0004] To this end, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, the embodiments of the present application provide an image processing method, comprising: receiving a plurality of mesh data; the plurality of mesh data of the component is data discretized from a three-dimensional model of the component; transmitting the plurality of mesh data to a GPU; clustering the plurality of mesh data to obtain a plurality of mesh clustering data; transmitting the plurality of mesh clustering data to the GPU; the GPU is used for rendering the plurality of mesh data, and rendering the plurality of mesh clustering data after receiving the plurality of mesh clustering data.

[0006] In this embodiment, since it takes a certain time to process the mesh data into mesh cluster data, the method can preferentially transmit the plurality of mesh data directly to the GPU for rendering after receiving the mesh data of a component. Then, the method clusters the plurality of mesh data into mesh cluster data, and transmits the mesh cluster data to the GPU for rendering. In the process of generating the mesh cluster data by the CPU, the GPU is allowed to render the received mesh data, so that the GPU is not in an idle state, and the screen cannot display the rendered image in time.

[0007] In an embodiment, the transmitting the plurality of mesh data to the GPU specifically comprises: detecting whether the number of the plurality of mesh data is greater than a set number; and in a case where the number of the plurality of mesh data is greater than the set number, transmitting the plurality of mesh data to the GPU.

[0008] In this embodiment, if the number of mesh data is relatively small, the method takes a relatively short time to generate mesh cluster data, and all mesh data can be directly processed into mesh cluster data to improve the speed of data transmission to the GPU.

[0009] In an embodiment, the transmitting the plurality of mesh cluster data to the GPU specifically comprises: detecting whether all the plurality of mesh data are clustered into the plurality of mesh cluster data within a set time; and in a case where all the plurality of mesh data are clustered into the plurality of mesh cluster data within the set time, transmitting the plurality of mesh cluster data to the GPU.

[0010] In this embodiment, if it is detected within the set time that all mesh data have not been processed into mesh cluster data, the method can transmit the processed mesh cluster data to the GPU driver, so that the GPU driver calls a rendering pipeline for rendering to improve the rendering efficiency.

[0011] In an embodiment, the method further comprises: in a case where all the plurality of mesh data are not clustered into the plurality of mesh cluster data within the set time, transmitting the mesh cluster data that have completed clustering to a GPU driver; and the GPU driver is configured to call a rendering pipeline to render the mesh cluster data that have completed clustering.

[0012] In this embodiment, if it is detected within the set time that all mesh data have been processed into mesh cluster data, the method can transmit the processed mesh cluster data to the GPU, so that the GPU starts a drawing call component for rendering to improve the rendering efficiency.

[0013] In an embodiment, the method further comprises: receiving the deleted mesh data and / or mesh cluster data sent by the GPU driver; the deleted mesh data refers to the mesh data and / or mesh cluster data after the GPU driver performs the view frustum culling and mesh occlusion culling on the received mesh data and / or mesh cluster data; the deleted mesh cluster data refers to the mesh cluster data after the GPU driver performs the view frustum culling and mesh occlusion culling on the received mesh cluster data; and sending the deleted mesh data and / or mesh cluster data to the GPU.

[0014] In this embodiment, the GPU driver performs the view frustum culling and mesh occlusion culling on the received mesh data and / or mesh cluster data after receiving the mesh data and / or mesh cluster data, and the deleted mesh data and / or mesh cluster data can be forwarded to the GPU for rendering, which can save the workload of the GPU rendering.

[0015] In an embodiment, before the plurality of mesh data is transmitted to the GPU, the method further comprises: identifying the surface of the component to which each mesh data in the plurality of mesh data belongs; the surface of the component comprises at least one layer; and marking the same layer ID for the mesh data corresponding to each layer of the component; and the GPU is configured to uniformly render the mesh data and / or mesh cluster data corresponding to the same layer ID.

[0016] In this embodiment, the method can identify the layer of the surface of the component, mark the same layer ID for the corresponding mesh data of the same layer, and enable the GPU to uniformly render the mesh data or mesh cluster data of the same layer ID so that the color of the rendering of one layer is the same.

[0017] In an embodiment, the method further comprises: sending the depth information and layer ID of each mesh data in the plurality of mesh data and the depth information and layer ID of each mesh cluster data in the plurality of mesh cluster data to the GPU; and the GPU is configured to determine the relationship between each layer of the component according to the depth information and layer ID.

[0018] In this embodiment, the method can transmit the depth information and layer ID of each mesh data and mesh cluster data to the GPU, so that the GPU can determine the occlusion relationship between each component and each layer of the component according to the depth information and layer ID during the rendering process, thereby improving the correctness of the calculated occlusion relationship.

[0019] In a second aspect, an image processing apparatus is provided in the embodiments of the present application, comprising: a first processing unit configured to receive a plurality of mesh data of an assembly; the plurality of mesh data being discrete data of a three-dimensional model of the assembly; a second processing unit configured to transmit the plurality of mesh data to a graphics processing unit (GPU); a third processing unit configured to cluster the plurality of mesh data to obtain a plurality of mesh cluster data; and a fourth processing unit configured to transmit the plurality of mesh cluster data to the GPU; the GPU being configured to render the plurality of mesh data, and render the plurality of mesh cluster data after receiving the plurality of mesh cluster data.

[0020] In an embodiment, the second processing unit is specifically configured to detect whether the number of the plurality of mesh data is greater than a set number; and the second processing unit is specifically configured to transmit the plurality of mesh data to the GPU when the number of the plurality of mesh data is greater than the set number.

[0021] In an embodiment, the fourth processing unit is specifically configured to detect whether the plurality of mesh data are all clustered into the plurality of mesh cluster data within a set time; and the fourth processing unit is specifically configured to transmit the plurality of mesh cluster data to the GPU when the plurality of mesh data are all clustered into the plurality of mesh cluster data within the set time.

[0022] In an embodiment, the fourth processing unit is further configured to transmit the mesh cluster data that have been clustered to a GPU driver when the plurality of mesh data are not all clustered into the plurality of mesh cluster data within the set time; and the GPU driver is configured to call a rendering pipeline to render the mesh cluster data that have been clustered.

[0023] In an embodiment, the fourth processing unit is further configured to receive deleted mesh data and / or mesh cluster data sent by the GPU driver; the deleted mesh data refers to mesh data and / or mesh cluster data after the GPU driver performs view frustum culling and mesh occlusion culling on the received mesh data and / or mesh cluster data; the deleted mesh cluster data refers to mesh cluster data after the GPU driver performs view frustum culling and mesh occlusion culling on the received mesh cluster data; and the fourth processing unit is further configured to send the deleted mesh data and / or mesh cluster data to the GPU.

[0024] In an implementation, the first processing unit, before the plurality of mesh data is transmitted to the GPU, is further configured to identify surfaces of the components to which each of the plurality of mesh data belongs; the surface of the component includes at least one layer; the first processing unit is further configured to mark the same layer ID for the mesh data corresponding to each layer of the component; and the GPU is configured to uniformly render the mesh data and / or mesh cluster data corresponding to the same layer ID.

[0025] In an implementation, the fourth processing unit is further configured to send, to the GPU, the depth information and the layer ID of each of the plurality of mesh data and the depth information and the layer ID of each of the plurality of mesh cluster data; and the GPU is configured to determine the relationship between each layer of the component according to the depth information and the layer ID.

[0026] In a third aspect, an embodiment of the present application provides a computing device, including: at least one memory; at least one processor configured to execute instructions stored in the memory to cause the computing device to perform the embodiments of the first aspect.

[0027] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device, the computing device performs the embodiments of the first aspect.

[0028] In a fifth aspect, an embodiment of the present application provides a computer program product including instructions, characterized in that the computer program product stores instructions, when the instructions are executed by a computing device, the computing device implements the embodiments of the first aspect.

[0029] In a sixth aspect, an embodiment of the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the embodiments of the first aspect.

[0030] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the embodiments of the first aspect.

[0031] In an eighth aspect, the embodiments of the present application provide a computer program product containing instructions, characterized in that the computer program product stores instructions, which, when executed by a computing device cluster, cause the computing device cluster to implement various possible implementations of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0032] The drawings needed to be used in the description of the embodiments or the prior art are briefly introduced as follows.

[0033] Figure 1 An architecture schematic diagram of an image processing system provided in the embodiments of the present application is shown in FIG. 1.

[0034] Figure 2 A process schematic diagram of processing grid data by a CPU provided in the embodiments of the present application is shown in FIG. 2.

[0035] Figure 3 A flow schematic diagram of processing grid data by a CPU provided in the embodiments of the present application is shown in FIG. 3.

[0036] Figure 4 A schematic diagram of determining occlusion relationship according to depth and layer ID by a GPU provided in the embodiments of the present application is shown in FIG. 4.

[0037] Figure 5 A flow schematic diagram of removing grid cluster data in an occlusion position by a GPU 120 provided in the embodiments of the present application is shown in FIG. 5.

[0038] Figure 6 A structure schematic diagram of a GPU driver provided in the embodiments of the present application is shown in FIG. 6.

[0039] Figure 7 A flow schematic diagram of an image processing method provided in the embodiments of the present application is shown in FIG. 7.

[0040] Figure 8 A structure schematic diagram of an image processing apparatus provided in the embodiments of the present application is shown in FIG. 8.

[0041] Figure 9 A structure schematic diagram of a computing device provided in the embodiments of the present application is shown in FIG. 9.

[0042] Figure 10 An architecture schematic diagram of a computing device cluster provided in the embodiments of the present application is shown in FIG. 10.

[0043] Figure 11 An architecture schematic diagram of another computing device cluster provided in the embodiments of the present application is shown in FIG. 11. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0045] The term "and / or", used in the present document, is used to describe the association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In the present document, the symbol " / " represents an or relationship of associated objects, for example, A / B represents A or B.

[0046] The terms "first" and "second" and the like in the description and claims of the present document are used to distinguish different objects, and are not used to describe the specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, and are not used to describe the specific order of the response messages.

[0047] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design schemes. Rather, the use of the words "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0048] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0049] Before introducing the technical solutions protected by the present application, several professional terms related to the technical solutions protected by the present application are explained in advance, which are as follows:

[0050] CAD is a tool that uses computers to assist in designing and drawing various graphics. CAD software can be used to design buildings, mechanical parts, electronic components, and various products. CAD software can make designers complete design work faster and more accurately, and facilitate modification and adjustment of design.

[0051] CAE is a process that uses computer software to assist engineers in engineering design, analysis, and optimization. Through CAE software, engineers can perform various simulations such as structural analysis, fluid mechanics analysis, thermal analysis, etc. to verify and optimize design schemes, reduce the cost and time of experimental testing, and improve the accuracy and efficiency of engineering design.

[0052] NURBS, is a mathematical representation widely used in computer graphics and CAD for representing smooth curves and surfaces. NURBS is generally composed of control points, knots, and weights. Control points define the key points of the curve shape. The position of the control points affects the shape of the curve, determining the path and degree of curvature of the curve. Knots define the influence range of control points on the curve. Knots determine the parameterization of the curve, controlling the interpolation and connection rules between control points. The weight is associated with the value of each control point, used to adjust the degree of influence of the control point on the shape of the curve. The weight value can make the curve be more strongly controlled in some parts, so as to achieve more flexible shape adjustment.

[0053] Industrial visualization is the application of advanced visual technology to the industrial field to display and present data, design, simulation results and other information in engineering and manufacturing processes. Engineers and designers can better understand and analyze complex industrial systems, processes and equipment through industrial visualization technology, so as to better optimize design, production and operation processes.

[0054] Render pipeline refers to the process of converting three-dimensional scene description into final images in computer graphics. The rendering pipeline includes geometry processing stage, rasterization stage, pixel processing stage, and output synthesis stage. Among them, the geometry processing stage is responsible for processing and transforming input geometric data (such as vertices, polygons, etc.), and performs viewpoint transformation and projection to convert geometric shapes in the scene to vertices in screen space. The rasterization stage is used to convert geometric primitives (such as triangles) into pixel points on the screen and determine the color value of each pixel. The pixel processing stage is used for shading, texture mapping, lighting calculation and other operations on each pixel to calculate the final pixel color value. The output synthesis stage is used to combine all processed pixels into the final image, which can include blending, transparency processing and other operations.

[0055] Mesh is a basic structure of a three-dimensional model composed of a series of vertices, edges connecting these vertices, and faces formed. Mesh is usually used to represent and describe the surface geometry and topology of three-dimensional objects. Mesh data refers to the data corresponding to one or more meshes.

[0056] Meshlet is a rendering technique used in graphics rendering to optimize the graphics pipeline for performance. Meshlet refers to the division of a mesh into smaller, compact sub-meshes. Sub-meshes are composed of adjacent vertices and triangles to facilitate more efficient processing and optimization during rendering. Meshlet data refers to the data corresponding to one or more meshlets.

[0057] A draw call is a command in graphics programming that instructs a graphics processing unit (GPU) to perform a rendering operation. When a developer needs to display a three-dimensional object or scene on a screen, a draw call is sent to the GPU to instruct the GPU how to draw a geometry (e.g., triangle, line, etc.) and apply corresponding textures, lighting, and materials, etc. to finally render an image.

[0058] GPU-driven is a pipeline that offloads as much work as possible from the central processing unit (CPU) to the GPU, such as view frustum culling. Unlike traditional rendering pipelines, GPU-driven takes advantage of the powerful parallel computing power of the GPU, saves communication costs between the CPU and the GPU, and frees up the CPU to do more.

[0059] Level of detail (LOD) is a technique used in the fields of programming and computer graphics, which gradually reduces the number of polygons of an object or model as it moves away from the observer, thereby improving rendering efficiency. Since the number of polygons decreases as the model moves away, in theory, the observer will not perceive any decline in the rendering quality of the model.

[0060] Frustum / occlusion / HiZ culling is an operation performed in the early and middle stages of the rendering process, which aims to remove invisible models or parts in advance, thereby reducing the load of subsequent processes. Among them, frustum culling refers to discarding parts of the scene that are not within the camera's visible range. Occlusion culling refers to discarding content that is completely behind invisible objects. Depth culling refers to using the z-buffer for quick culling.

[0061] A shader is a program or function in computer graphics that defines the rendering effects of graphics. Shaders are usually executed on GPUs to handle various stages in the graphics rendering pipeline, such as the vertex shading stage, fragment shading stage, etc., to achieve realistic graphics effects. Shaders can be divided into vertex shaders, geometry shaders, and fragment shaders. Vertex shaders are used to process the position, normal, texture coordinates, and other information of input vertices, usually used to perform spatial transformation and vertex attribute calculation to determine the position of vertices on the screen. Geometry shaders are used to process the geometric information of geometry, such as creating new geometric shapes or performing geometric transformations. Fragment shaders are used to calculate the final color value of each pixel, usually performing lighting calculations, texture sampling, and other calculations that affect the final color of the pixel.

[0062] Extended reality (XR) is a term that covers virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. XR aims to describe a series of technologies and applications that provide users with more immersive and interactive experiences through the interaction between digital and real-world.

[0063] Geographic information system (GIS) is a comprehensive discipline that combines geography and cartography, and has been widely applied in different fields such as map navigation, disaster relief, and environmental science. It is a computer system used to input, store, query, analyze, and display geographic data.

[0064] Metaverse refers to all 3D virtual network worlds entered through virtual reality glasses, extended reality glasses, mobile phones, personal computers, and electronic game consoles.

[0065] Hierarchical Z-buffering (HZB) is an algorithm for implementing occlusion culling in image space, which can select appropriate hierarchical depth by bounding volume projection information to reduce the number of depth tests.

[0066] Taking CAD as an example, in the CAD drawing process, it is generally completed by CPU and GPU together. The CPU is responsible for performing the main computing tasks of the entire drawing software, such as processing geometric calculations, computing and converting of graphic data, responding and managing of software functions, etc. The CPU plays a role similar to the "brain" in CAD drawing, controlling and managing the entire computing process. The GPU is responsible for the graphic rendering and display tasks in CAD drawing, processing a large amount of graphic data, and converting the graphic data into pixels displayed on the screen. The GPU plays a role in accelerating and optimizing the rendering process in CAD drawing, and can quickly process complex graphic calculations and rendering tasks, making the drawing process more smooth and efficient.

[0067] In the related art, when engineers and designers use CAD drawing, frequent modification, addition and deletion of components are inevitable, which will trigger operations such as surface discretization, scene tree update and drawing call submission, thereby increasing the number of facets. The increase in the number of facets will increase the data interaction between the CPU and the GPU, causing communication bottlenecks between the CPU and the GPU. In addition, a super large CAD / CAE model produces hundreds of millions of facets after discretization. Due to the limitation of GPU memory, the discretized facets cannot be added to the GPU memory at once, so how to organize and load the super large scale facet CAD / CAE model is also a problem to be solved.

[0068] In order to solve the defects existing in the related art, the embodiment of the present application provides an image processing method, which comprises: the CPU receives a plurality of mesh data of a component, and transmits the plurality of mesh data to the GPU; the CPU clusters the plurality of mesh data to obtain a plurality of mesh clustering data; the CPU transmits the plurality of mesh clustering data to the GPU; and the GPU stops rendering the mesh data and renders the mesh clustering data. In the embodiment of the present application, since it takes a certain time to process the mesh data into mesh clustering data, the CPU can preferentially transmit the mesh data directly to the GPU for rendering after receiving the mesh data of a component, and then process the mesh data into mesh clustering data and transmit it to the GPU for rendering. The CPU generates mesh clustering data in the process, and lets the GPU render the received mesh data, which can avoid the GPU being in an idle state during this process, causing the screen to not display the rendered image in time. After receiving the mesh clustering data, the GPU can delete the mesh data and then render the mesh clustering data, so that the facets of a component after discretization can be added to the GPU memory at once.

[0069] In another solution, after the CPU receives multiple grid data of a component, it transmits a set number of grid data from the multiple grid data to the GPU; the CPU clusters the grid data other than the set number of grid data from the multiple grid data to obtain multiple grid cluster data; the CPU transmits the multiple grid cluster data to the GPU. After the GPU completes the rendering operation of the set number of grid data, it can seamlessly render the grid cluster data, avoiding the GPU being idle during this process, resulting in the screen being unable to display the rendered image in a timely manner. Compared with the above technical solution, this technical solution can reduce the amount of data transmitted between the CPU and the GPU, and reduce the workload of the GPU rendering.

[0070] The set number may be a parameter set by the user or determined based on the computing power of the CPU and GPU. Optionally, the time it takes the GPU to render the set number of mesh data is greater than or equal to the sum of the time it takes the CPU to cluster the mesh data other than the set number of mesh data from the plurality of mesh data and the time it takes the CPU to send the plurality of mesh cluster data to the GPU.

[0071] The CPU can pre-process mesh data in real time to cluster it into meshes, accelerating data transfer between the CPU and GPU. The CPU can uniformly render mesh data in different layers using custom depth settings, reducing rendering pipeline complexity and improving rendering efficiency. When submitting rendering data to the CPU, the GPU-driven pipeline can be organically integrated with the deferred and forward pipelines, reordering the shaders required by the deferred and forward pipelines to support the submission of the entire scene with a single draw call.

[0072] Figure 1 FIG. 1 is a structural diagram of an image processing system provided in an embodiment of the present application. Figure 1 As shown, the image processing system 100 may include a CPU 110, a GPU 120, a memory 130, and a bus 140. The CPU 110, GPU 120, and memory 130 may establish a communication connection via the bus 140 to enable data transmission between them. The image processing system 100 may be deployed and run on one or more computing devices, which may be servers, computers, desktop computers, portable computers, and the like. It should be understood that this application does not limit the number of CPUs 110, GPUs 120, and memory 130 in the image processing system 100.

[0073] The image processing system 100 can be inserted into existing industrial graphics CAD, CAE, game, XR, GIS, metaverse, etc. software as a plug-in. The interface of the plug-in of the present application can be embedded in the adaptation layer of the industrial graphics engine, so that the industrial graphics software can call the interface to realize the functions that the rendering system 100 protected by the present application can realize.

[0074] The CPU 110 is used to be responsible for the management of overall computing tasks and software functions.

[0075] The GPU 120 is used to be responsible for the graphics rendering and display tasks.

[0076] The memory 130 can include a volatile memory, such as a random access memory (RAM). The memory 130 can include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard disk drive (HDD), or a solid state drive (SSD).

[0077] The memory 130 stores executable program codes, so that the CPU 110 and the GPU 120 execute the executable program codes to realize the functions of the respective modules, thereby realizing the image rendering method. That is, the memory 130 stores instructions for executing the image rendering method.

[0078] The bus 140 can be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 1 Only one line is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus 140 can include a path for transmitting information between the components of the image processing system 100 (e.g., the CPU 110, the GPU 120, and the memory 130, etc.).

[0079] When the industrial graphics software appears modification, addition and deletion components, a large amount of mesh data will be generated. In the embodiment of the application, after the CPU 110 receives the input mesh data, the CPU 110 can process the mesh data, process the mesh data into mesh clustering data, and then transmit the mesh clustering data to the GPU 120 to accelerate the data transmission speed between the CPU 110 and the GPU 120. Exemplarily, first, the CPU 110 can read and load the original mesh data, including vertex data, face data and other information of the object. Then, the CPU 110 can perform optimization processing on the original mesh data, such as vertex reduction, face merging and the like, to reduce the complexity and size of the mesh data, thereby improving the subsequent rendering efficiency. Secondly, the CPU 110 can reorganize and distribute the processed mesh data according to certain rules and algorithms to generate mesh clustering data. The mesh clustering data can be a set of closely related vertices and triangular fragments, which are designed to better utilize the parallel processing and cache performance of the GPU 120, thereby improving the efficiency of graphics rendering. Finally, the CPU 110 can transmit the generated mesh clustering data to the GPU 120 for further graphics rendering operations such as geometry processing, lighting calculation and the like.

[0080] In the related art, the data processing process between the CPU and the GPU is executed in a serial manner. That is, after the CPU receives a mesh data, the CPU can process the received mesh data into mesh clustering data. In this process, the GPU is in an idle state, waiting to receive the mesh clustering data, which causes the screen to be unable to display the rendered image in time.

[0081] In the embodiment of the application, the data processing process between the CPU 110 and the GPU 120 is executed in a parallel manner. That is, after the CPU 110 receives multiple mesh data, the CPU 110 can directly transmit the multiple mesh data to the GPU 120 for rendering. At the same time, the CPU 110 can cluster the multiple mesh data to obtain multiple mesh clustering data, and then transmit the multiple mesh clustering data to the GPU 120.

[0082] As Figure 2As shown, the CPU 110 can obtain surfaces of each component of the designed model in the industrial visualization and assign the same layer identification (ID) to the mesh data corresponding to each layer of each surface. Generally, a CAD designed model itself can be accompanied by surface layers and inner layers that have consistent or very close positional information, which are necessary information for correct rendering. For example, taking a car model as an example, in the process of designing the car model by CAD, the car surface can have multiple surface layer components, such as a car surface stain, car paint, and car metal surface. Although these surface layer components are very close to each other or even have the same positional information, the CAD still needs to follow the occlusion relationship of the car surface stain (layer 0) blocking the car paint (layer 1) and the car paint blocking the car metal surface (layer 2) for rendering.

[0083] When an engineer or designer wants to float a certain surface layer component to the surface for observation, the layer information can be used to determine and remove all components with a layer number lower than the layer number of the surface layer component to be observed. For example, still taking the car model as an example, when an engineer or designer wants to float the car metal surface, all components with a layer number lower than the car metal surface, i.e., the car surface stain (layer 0) and the car paint (layer 1), can be removed, so that the car metal surface is floated to the surface.

[0084] In the process of adding the layer ID to each layer by the CPU 110, the layer ID can be defined as follows: the larger the layer ID, the lower the priority, and the smaller the layer ID, the higher the priority. The size of the layer ID can be marked in the order from inside to outside in position. That is, the layer ID of the surface layer or inner layer on the inside is marked with a larger value, and the layer ID of the surface layer or inner layer on the outside is marked with a smaller value. The size of the layer ID can be marked in the order from deep to shallow in color. That is, the layer ID of the surface layer or inner layer with a deep color is marked with a larger value, and the layer ID of the surface layer or inner layer with a shallow color is marked with a smaller value. For example, still taking the car model as an example, the car surface can have three components: a car surface stain, car paint, and a car metal surface. Since the car paint is attached to the car metal surface and the car surface stain is attached to the car paint, the CPU 110 can define the layer ID of the car surface stain as layer 0, the layer ID of the car paint as layer 1, and the layer ID of the car metal surface as layer 2.

[0085] GPU 120 receives the plurality of mesh data, groups the plurality of mesh data, and groups the mesh data at the same component level as a group. GPU 120 can render the mesh data of the same group according to the depth information, and render the mesh data of different groups according to the priority order of the layers, to realize the function of layer priority in the industrial visualization scene. In addition, GPU 120 can select "custom depth" as an algorithm to process the fusion calculation of layer occlusion and space occlusion based on this function.

[0086] The mesh data is composed of triangle vertex information. The mesh clustering data is clustered based on this, and the process is time-consuming. In the embodiment of the application, two rendering pipelines can be arranged between CPU 110 and GPU 120, which are mesh data pipeline and mesh clustering data pipeline. When CPU 110 determines that the currently received mesh data is the mesh data of a new component, it can transmit the received mesh data to GPU 120 through the mesh data pipeline. At the same time, CPU 110 can use a clustering algorithm to cluster the received mesh data to obtain mesh clustering data. CPU 110 can use an LOD algorithm to simplify the mesh clustering data into different triangle data amount detail versions, for example, LOD 0 is not simplified, LOD 1 is generated by removing 1 / 2 of the triangle patches while retaining the shape as much as possible, LOD 2 is generated by removing 1 / 4 of the triangle patches while retaining the shape as much as possible, and so on. Finally, CPU 110 transmits the mesh clustering data to GPU 120 through the mesh clustering data pipeline.

[0087] Exemplarily, an industrial graphics software adds a rabbit component as an example. As shown in Figure 3 Assuming that there are 1280 triangle patches in the rabbit mesh, there are 1280x3=3840 point information, and 3840x3=11520 float coordinates. In the rendering process, the maximum number of triangle patches (the number is selected according to the thread architecture of the hardware, generally 128) is grouped and clustered, and the level, material and parent network of each type of triangle patch obtained are consistent, and all the triangle patches in each type form a whole continuous mesh. The process of CPU 110 transmitting the received mesh data to GPU 120 is as follows:

[0088] Step S301, CPU 110 receives mesh data of a component.

[0089] Step S302, the CPU 110 detects whether the number of component vertices in the mesh data is less than a set number. In one case, the CPU 110 determines that the number of component vertices in the mesh data is less than the set number, and executes step S303. In another case, the CPU 110 determines that the number of component vertices in the mesh data is greater than or equal to the set number, and executes step S304.

[0090] Further regarding step S302, if the number of component vertices in the mesh data is relatively small, the CPU 110 consumes less time to generate the mesh clustering data in real time, and can directly cluster the mesh data. If the number of component vertices in the mesh data is relatively large, the CPU 110 can invoke a background multithreading to cluster the mesh data, so as to avoid long waiting time.

[0091] Step S303, the CPU 110 processes the received mesh data into mesh clustering data.

[0092] Step S304, the CPU 110 invokes a background multithreading to process the received mesh data into mesh clustering data.

[0093] Step S305, the CPU 110 detects whether all the received mesh data is processed into mesh clustering data. In one case, the CPU 110 determines that all the mesh data is processed into mesh clustering data, and executes step S306. In another case, the CPU 110 determines that not all the mesh data is processed into mesh clustering data, and executes step S307.

[0094] Further regarding step S305, when the CPU 110 invokes a background multithreading to cluster the mesh data, some threads cluster the mesh data at a relatively fast speed, and some threads cluster the mesh data at a relatively slow speed, so there is a case that not all the mesh data is processed into mesh clustering data.

[0095] Step S306, the CPU 110 transmits the mesh data to the GPU driver of the GPU 120, so that the GPU driver invokes a rendering pipeline to render.

[0096] Step S307, the CPU 110 transmits the mesh data to the GPU 120, so that the GPU 120 starts a drawing invocation component to render.

[0097] In the embodiments of the present application, when engineers and designers modify, add, and delete components, generating a large number of facets, they transmit the mesh data of a component to CPU 110. Because it takes time for CPU 110 to process the mesh data into mesh clusters, CPU 110 can directly transmit the received mesh data to GPU 120 to prevent the screen from displaying the rendered image in a timely manner. After CPU 110 clusters the mesh data into mesh clusters, it transmits them to GPU 120. GPU 120 switches the rendering pipeline to render based on the mesh cluster data.

[0098] After receiving the mesh data or mesh cluster data, the GPU 120 can render the mesh data or mesh cluster data of different layers to delete the mesh data or mesh cluster data in the occluded position. In the embodiment of the present application, during the rendering process, the GPU 120 can determine the occlusion relationship of objects in the same layer based on the depth information, and determine the occlusion relationship of objects between different layers in order from high priority layer to low priority layer based on the layer ID. Compared with the related art that determines the object occlusion relationship only based on the depth information, the GPU 120 of the present application determines the physical occlusion relationship based on the depth information and the layer ID, which can improve the accuracy of the calculated occlusion relationship.

[0099] For example, Figure 4 As shown, CPU 110 receives mesh data corresponding to components in two layers, namely layer 0 and layer 1. The depth of the object in layer 0 is 0.5. The depth of the object in layer 1 is 0.8. In layer 1, GPU 120 can determine based on the depth information that the depth of the square, 0.5, is less than the depth of the circle, 0.8 (0.5<0.8), so the square occludes the circle. GPU 120 determines based on the depth information and the layer ID that the sum of the square's depth and the layer ID, 1.5, is greater than the sum of the circle's depth and the layer ID, 0.8 (1.5>0.8), so the circle occludes the square.

[0100] In an embodiment of the present application, the GPU 120 can customize the depth information, which includes the original depth information and the ID of the layer. The GPU 120 can use floating-point numbers with different bit numbers to represent the customized depth information. A floating-point number is a data type used to represent real numbers. According to the definition of the Institute of Electronics Engineers (IEEE), a floating-point number is composed of a sign, an exponent, and a fraction. The sign part indicates whether the floating-point number is positive or negative. The exponent part is used to indicate the order of magnitude (i.e., power) of the floating-point number, usually in the form of optional counting method. The fraction part, also known as significant digits, represents the main part of the floating-point number and is used to store the actual numerical value. Compared with integers, floating-point numbers can better represent non-integers, very large or very small numbers.

[0101] When the CPU 110 and the GPU 120 perform data transmission, the template of the data needs to be specified in advance. Assuming that the CPU 110 transmits a 64-bit data, it needs to be determined whether the data is a 64-bit floating-point number or two 32-bit floating-point numbers. Therefore, the CPU 110 needs to transmit information such as the position, size, and how to obtain the depth information and the ID of the layer from the floating-point number to the GPU 120, so that the GPU 120 can accurately obtain the depth information and the ID of the layer from the local template.

[0102] In an embodiment, taking a 32-bit floating-point number as an example. When the GPU 120 embeds the depth information and the ID of the layer into the template, the GPU 120 can place the ID of the layer in the integer part of the floating-point number and the depth (the depth value is between 0 and 1) in the decimal part of the floating-point number.

[0103] In another embodiment, taking a 64-bit floating-point number as an example. When the GPU 120 embeds the depth information and the ID of the layer into the template, the GPU 120 can place the ID of the layer in the 32-bit from left to right in the 64-bit floating-point number and the depth value in the 32-bit from right to left in the 64-bit floating-point number.

[0104] Exemplarily, taking the mesh cluster data as an example. As shown in Figure 5 The process of the GPU 120 to remove the mesh cluster data in the occluded position is specifically as follows:

[0105] In step S501, the GPU 120 clears the floating-point number record depth information and the ID of the layer in the local cache template.

[0106] In step S502, the GPU 120 renders the mesh cluster data of each layer and writes the depth information and the ID of the layer in the form of floating-point number into the template.

[0107] In step S503, the GPU 120 merges the depth information in the layer and the layer ID in the stencil after rendering is completed, to generate custom depth information.

[0108] In step S504, the GPU 120 generates an HZB according to the custom depth information.

[0109] In step S505, the GPU 120 performs occlusion deletion on the mesh clustering data according to the HZB.

[0110] In the embodiment of the present application, the GPU 120 determines the physical occlusion relationship according to the depth information and the ID of the layer, and then performs occlusion deletion on the mesh clustering data in the occluded position, so as to realize the deletion of objects in different layers in the same pipeline.

[0111] The GPU driver inside the GPU 120 can transfer the calculation task performed by the CPU 110 to the GPU 120, so that the GPU 120 performs the calculation task of the CPU 110. In the embodiment of the present application, as shown in FIG. 1, the GPU driver inside the GPU 120 can transfer the calculation task performed by the CPU 110 to the GPU 120, so that the GPU 120 performs the calculation task of the CPU 110. Figure 6As shown, the GPU driver 600 can divide into a pre-depth pipeline node 601, an HZB multi-level building pipeline node 602, a meshlet clear pipeline node 603, a frustum culling pipeline node 604, an occlusion culling for mesh pipeline node 605, a data streaming pipeline node 606, a meshlet creation pipeline node 607, a meshlet culling pipeline node 608, an index compaction for object pipeline node 609, a build triangles pipeline node 610, a draw command generation pipeline node 611, and a draw call classification pipeline node 612 according to the execution function. The pre-depth pipeline node 601, the HZB multi-level building pipeline node 602, the meshlet clear pipeline node 603, the frustum culling pipeline node 604, the occlusion culling for mesh pipeline node 605, the data streaming pipeline node 606, the meshlet creation pipeline node 607, the meshlet culling pipeline node 608, the index compaction for object pipeline node 609, the build triangles pipeline node 610, the draw command generation pipeline node 611, and the draw call classification pipeline node 612 can be implemented by software, or can be implemented by hardware, or can be implemented by software and hardware in combination.

[0112] The pre-depth pipeline node 601 can convert the depth information of the last frame to the viewpoint of the current frame to obtain converted depth information, and generate the depth information of the current frame according to the converted depth information, so as to improve the efficiency and performance of the graphics rendering. The depth information refers to the custom depth information, which includes the original depth information and the ID of the layer.

[0113] The HZB multi-level building pipeline node 602 can use a compute shader to generate the HZB according to the depth information of the current frame, so as to improve the efficiency of the depth test.

[0114] The meshlet clear pipeline node 603 can initialize a global count data structure for managing meshlet data according to the HZB, so as to optimize the rendering process and improve the efficiency and performance of the rendering.

[0115] The view frustum culling pipeline node 604 can project a view frustum into the scene according to the initialized data structure, and then check whether each object intersects with the view frustum. If the view frustum culling pipeline node 604 detects that an object is completely outside the view frustum, the object is culled to obtain mesh data or mesh cluster data corresponding to the object after view frustum culling. The view frustum culling pipeline node 604 culls the mesh data or mesh cluster data corresponding to the object outside the view frustum, which can reduce unnecessary rendering calculations, save GPU 120 resources, and improve rendering performance.

[0116] The mesh occlusion culling pipeline node 605 can determine which objects are occluded by other objects by checking the depth information of the objects in the HZB, and cull the occluded objects to obtain mesh data or mesh cluster data corresponding to the objects after view frustum culling and mesh occlusion. The mesh occlusion culling pipeline node 605 culls the mesh data or mesh cluster data corresponding to the occluded objects, which can reduce unnecessary rendering calculations and improve rendering performance.

[0117] The data streaming pipeline node 606 can send the mesh data or mesh cluster data corresponding to the objects after view frustum culling and mesh occlusion to the CPU 110, and stream the mesh data or mesh cluster data corresponding to the objects after view frustum culling and mesh occlusion to the GPU 120, which can save memory and allow dynamic loading of data when needed to adapt to different scene requirements. In the embodiments of the present application, the data used by the data streaming pipeline node 606 to calculate the mesh view frustum and occlusion is the bounding sphere data (each mesh has one), which includes the center and radius information. The CPU 110 obtains from the operation results of the GPU 120 which meshes are not culled, and these meshes need to upload full data to the GPU 120 for rendering calculation by the GPU 120 using the full data.

[0118] The mesh cluster creation pipeline node 607 can package the collected mesh data or mesh cluster data into a continuous cache after collecting the mesh data or mesh cluster data corresponding to the objects after view frustum culling and mesh occlusion, so that the cached data can be quickly accessed and processed during rendering, thereby improving rendering efficiency and performance.

[0119] When a computer stores data, a pointer is used to point to the corresponding storage position. If the stored data is not continuous, the pointers are pointer1, pointer2, pointer3, etc., which are independent of each other. If the stored data is continuous, the pointers are pointer1, pointer1+singleDataSize*1, pointer1+singleDataSize*2, etc. Therefore, continuous caching is very beneficial to data access.

[0120] The mesh cluster occlusion pipeline node 608 can traverse the data stored in the continuous buffer and use the HZB to calculate whether each mesh data or mesh cluster data is visible in the current view. If the mesh cluster occlusion pipeline node 608 determines that a mesh data or mesh cluster data is off-screen or occluded by other objects, it can cull the mesh data or mesh cluster data, thereby reducing unnecessary rendering work.

[0121] The object index compression pipeline node 609 can sort the culled mesh data or mesh cluster data according to the vertex indexes of the mesh data or mesh cluster data, and store the sorted culled mesh data or mesh cluster data in another continuous buffer. The object index compression pipeline node 609 analyzes and reorganizes the vertex indexes of the mesh data or mesh cluster data to eliminate redundant and unnecessary indexes and use as few indexes as possible to represent the same geometric shape.

[0122] The triangle pipeline node 610 may construct triangle indexes based on vertices in a continuous buffer, thereby providing a basis for subsequent rendering and graphics processing.

[0123] The drawing command generation pipeline node 611 may generate a drawing call for each object. A drawing call is an operation in which the CPU 110 calls a graphics application programming interface to instruct the GPU 120 to perform a graphics drawing operation for each object.

[0124] The draw call classification pipeline node 612 can classify different types of draw calls for optimization and processing within the graphics rendering pipeline. For example, draw calls can be divided into different groups, such as opaque draw calls, transparent draw calls, and shadow draw calls, to enable different optimization strategies to be applied during the rendering process. Because draw calls can be bottlenecks and cause performance losses, the draw call classification pipeline node 612 manages, categorizes, and merges them, helping the next draw call render a significant portion of the entire scene.

[0125] In the embodiment of the present application, since the draw calls of different objects may execute different shaders and the states of the rendering pipeline may also be different when the draw calls are executed, the GPU 120 needs to classify related objects that execute the same shader and are in the same rendering pipeline state into one category, and put the corresponding draw calls into continuous video memory for management, so that one draw call submission can render the entire scene.

[0126] After the forward pipeline receives the draw call, it invokes a corresponding number of shaders for rendering. The shaders invoked by the forward pipeline can include vertex shaders, geometry shaders, and pixel shaders. Vertex shaders are responsible for processing and transforming input vertex data, such as transforming the position, color, texture coordinates, etc. of vertices, in order to convert the input vertex data into a form suitable for rendering on the screen. Geometry shaders can operate on and transform input geometry primitives (such as points, lines, triangles, etc.), such as creating new geometry primitives, modifying the properties of geometry primitives, etc. Geometry shaders further process the geometry data output by vertex shaders to generate the final geometry primitives. Pixel shaders can process and compute input pixel data, such as computing and operating on the color, depth, transparency, etc. of pixels, to determine the color and properties of the final pixels displayed on the screen.

[0127] After the deferred pipeline receives the draw call, it invokes a visibility buffer and a corresponding number of shaders for rendering. The visibility buffer is a data structure used to store visibility information for each pixel in the scene. The visibility buffer is commonly used in graphics rendering processes to determine which pixels need to be rendered and which pixels should be occluded or culled. Shaders are used to process pixel data. In the deferred rendering pipeline, shaders determine the visible pixels in the scene and then perform shading processing on these pixels. The deferred pipeline renders by separating visibility and shading, which helps to improve rendering efficiency, especially when dealing with complex scenes and large amounts of geometry, effectively optimizing rendering performance.

[0128] The implementation process of the technical solutions in the protection of Figure 2 will be introduced in the form of a flow.

[0129] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application. As shown in Figure 7 , the method is executed by the CPU 110 described above, and the specific implementation process is as follows:

[0130] Step S701, receiving a plurality of mesh data of components.

[0131] The plurality of mesh data received by the CPU 110 is generally generated when industrial graphics software modifies, adds, and deletes components. Mesh data is data discretized from a three-dimensional model of a component.

[0132] Step S702, transmitting the plurality of mesh data to the GPU.

[0133] In order to accelerate the output transmission speed between the CPU 110 and the GPU 120, the CPU 110 can process the received mesh data into mesh cluster data, and then transmit the mesh cluster data to the GPU 120. Since it takes a certain time for the CPU 110 to process the mesh data into the mesh cluster data, the GPU 120 is in an idle state during the time period, resulting in that the screen cannot display the rendered image in time. In the embodiment of the present application, after receiving the mesh data of a new component, the CPU 110 can directly send multiple mesh data to the GPU 120 for rendering, so as to avoid that the screen cannot display the rendered image in time.

[0134] In the embodiment of the present application, the CPU 110 can detect whether the number of the multiple mesh data is greater than a set number. In a case where the CPU 110 determines that the number of the mesh data is greater than the set number, the CPU 110 performs the step S702. That is, the CPU 110 transmits multiple mesh data in the multiple mesh data to the GPU 120. In a case where the CPU 110 determines that the number of the mesh data is less than or equal to the set number, the CPU 110 can not need to directly transmit part of the mesh data to the GPU 120, but locally process all the mesh data into mesh cluster data, so as to improve the output data speed between the CPU 110 and the GPU 120.

[0135] In the embodiment of the present application, in the industrial visualization, the CPU 110 can obtain the surfaces of each component of the designed model, and assign the same layer ID to the mesh data corresponding to each layer of each surface. In the process of adding the layer ID to each layer by the CPU 110, the CPU 110 can define that the larger the layer ID is, the lower the priority is, and the smaller the layer ID is, the higher the priority is. The size of the layer ID can be marked in the order from inside to outside according to the position. That is, the layer ID value of the surface layer or the inner layer located on the inside is relatively large, and the layer ID value of the surface layer or the inner layer located on the outside is relatively small. The size of the layer ID can be marked in the order from deep to shallow according to the color. That is, the layer ID value of the surface layer or the inner layer with deep color is relatively large, and the layer ID value of the surface layer or the inner layer with shallow color is relatively small.

[0136] In step S703, the multiple mesh data are clustered to obtain multiple mesh cluster data.

[0137] The process of the CPU 110 processing the mesh data into mesh cluster data can be as follows: first, the CPU 110 can read and load the original mesh data, including vertex data, face data, and other information of the object. Then, the CPU 110 can perform optimization processing on the original mesh data, such as vertex reduction, face merging, and other operations, to reduce the complexity and size of the mesh data, thereby improving the rendering efficiency. Second, the CPU 110 can reorganize and distribute the processed mesh data according to certain rules and algorithms to generate mesh cluster data. The mesh cluster data can be a set of closely related vertices and triangle fragments, designed to better utilize the parallel processing and cache performance of the GPU 120, thereby improving the efficiency of graphics rendering. Finally, the CPU 110 can transmit the generated mesh cluster data to the GPU 120 for further graphics rendering operations such as geometry processing, lighting calculation, etc.

[0138] Step S704, transmitting the plurality of mesh cluster data to the GPU.

[0139] During the process of generating mesh cluster data, the CPU 110 can detect whether all the mesh data is clustered into a plurality of mesh cluster data within a set time. In one case, the CPU 110 detects that not all mesh data is processed into mesh cluster data within the set time, and can transmit the processed mesh cluster data to the GPU driver to call the rendering pipeline for rendering to improve rendering efficiency. In another case, the CPU 110 detects that all mesh data has been processed into mesh cluster data within the set time, and can transmit the processed mesh cluster data to the GPU 120 to start a draw call component for rendering to improve rendering efficiency.

[0140] When the CPU 110 sends mesh data and mesh cluster data to the GPU, it can also send the depth information and layer ID of the mesh data, as well as the depth information and layer ID of the mesh cluster data, to the GPU 120. The GPU is used to determine the relationship between the layers of the components according to the depth information and layer ID. During the rendering process, the GPU 120 can determine the occlusion relationship between the components and the layers of the components according to the depth information and layer ID to improve the correctness of the calculated occlusion relationship.

[0141] After receiving the mesh data and / or mesh cluster data, the GPU driver performs frustum culling and mesh occlusion culling on the mesh data and / or mesh cluster data. The deleted mesh data and / or mesh cluster data can be forwarded to the GPU 120 for rendering, which can save the workload of the GPU 120 for rendering.

[0142] In the embodiment of the present application, since it takes a certain time to process the mesh data into mesh cluster data, the CPU 110 can directly transmit a plurality of mesh data to the GPU 120 for rendering after receiving the mesh data of one component. Then, the CPU 110 clusters the plurality of mesh data into mesh cluster data and transmits the mesh cluster data to the GPU 120 for rendering. In the process of generating the mesh cluster data, the GPU 120 renders the received mesh data, which can avoid the GPU 120 being idle in this process, so that the screen cannot display the rendered image in time. After receiving the mesh cluster data, the GPU 120 can delete the mesh data and then render the mesh cluster data, so that the patches of one component after discretization can be added to the GPU 120 memory at one time.

[0143] Figure 8 A structural schematic diagram of an image processing device provided in the embodiment of the present application is shown in FIG. 8. As shown in FIG. 8, the image processing device 800 can be divided into a first processing unit 810, a second processing unit 820, a third processing unit 830 and a fourth processing unit 840 according to the execution function. The image processing device 700 specifically implements the following process: Figure 8

[0144] The first processing unit 810 is configured to receive a plurality of mesh data of a component. The plurality of mesh data is the data discretized from a three-dimensional model of the component. The second processing unit 820 is configured to transmit the plurality of mesh data to a GPU. The third processing unit 830 is configured to cluster the plurality of mesh data to obtain a plurality of mesh cluster data. The fourth processing unit 840 is configured to transmit the plurality of mesh cluster data to the GPU. The GPU is configured to render the plurality of mesh data and render the plurality of mesh cluster data after receiving the plurality of mesh cluster data.

[0145] In an embodiment, the second processing unit 820 is specifically configured to detect whether the number of the plurality of mesh data is greater than a set number. The second processing unit 820 is specifically configured to transmit the plurality of mesh data to the GPU when the number of the plurality of mesh data is greater than the set number.

[0146] In an embodiment, the fourth processing unit 840 is specifically configured to detect whether the plurality of mesh data is all clustered into the plurality of mesh cluster data within a set time. The fourth processing unit 840 is specifically configured to transmit the plurality of mesh cluster data to the GPU when the plurality of mesh data is all clustered into the plurality of mesh cluster data within the set time.

[0147] ​In one embodiment, the fourth processing unit 840 is further configured to transmit the clustered mesh cluster data to a GPU driver if all mesh data are not clustered into mesh cluster data within a set time. The GPU driver is configured to invoke a rendering pipeline to render the clustered mesh cluster data.

[0148] In one embodiment, the fourth processing unit 840 is further configured to receive deleted mesh data and / or mesh cluster data sent by the GPU driver. The deleted mesh data refers to the mesh data and / or mesh cluster data after the GPU driver performs frustum culling and mesh occlusion culling on the received mesh data. The deleted mesh cluster data refers to the mesh cluster data after the GPU driver performs frustum culling and mesh occlusion culling on the received mesh cluster data. The fourth processing unit 840 is further configured to send the deleted mesh data and / or mesh cluster data to the GPU.

[0149] In one embodiment, before transmitting the plurality of mesh data to a graphics processor (GPU), the first processing unit 810 is further configured to identify the surface of the component to which each mesh data in the plurality of mesh data belongs. The surface of the component includes at least one layer. The first processing unit 810 is further configured to label the mesh data corresponding to each layer of the component with the same layer ID. The GPU is configured to uniformly render the mesh data and / or mesh cluster data corresponding to the same layer ID.

[0150] In one embodiment, the fourth processing unit 840 is further configured to send depth information and layer IDs of each mesh data in the plurality of mesh data, and depth information and layer IDs of each mesh cluster data in the plurality of mesh cluster data, to the GPU. The GPU is configured to determine the relationship between the various layers of the component based on the depth information and layer IDs.

[0151] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. Figure 9 As shown, computing device 900 includes a bus 910, a processor 920, a memory 930, and a communication interface 940. Processor 920, memory 930, and communication interface 940 communicate with each other via bus 910. Computing device 900 may be a server, a computer, a portable notebook, a cabinet, etc. It should be understood that this application does not limit the number of processors and memories in computing device 900.

[0152] The bus 910 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one line is used in the figure to represent a bus, but this does not mean that there is only one bus or only one type of bus. The bus 910 can include paths for transmitting information between various components (e.g., the processor 920, the memory 930, the communication interface 940) of the computing device 900.

[0153] The processor 920 can be the CPU 110 and the GPU 120 described above.

[0154] The memory 930 can include a volatile memory, such as a random access memory (RAM). The memory 930 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0155] The memory 930 stores executable program code, and the processor 920 executes the executable program code to respectively implement the functions of the CPU 110 and the GPU 120 described above, thereby implementing the image processing method. That is, the memory 930 stores instructions for executing the image processing method.

[0156] Alternatively, the memory 930 stores executable program code, and the processor 920 executes the executable program code to respectively implement the functions of the modules described above, thereby implementing the image processing method. That is, the memory 930 stores instructions for executing the image processing method.

[0157] The communication interface 940 uses a transceiver module such as, but not limited to, a network interface card and a transceiver, to implement communication between the computing device 900 and other devices or communication networks.

[0158] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0159] As Figure 10 illustrated, the computing device cluster includes at least one computing device 900. The same instructions for performing the image processing method can be stored in the memory 930 of one or more computing devices 900 in the computing device cluster.

[0160] In some possible implementation manners, partial instructions for performing the image processing method can also be respectively stored in the memory 930 of one or more computing devices 900 in the computing device cluster. In other words, the combination of one or more computing devices 100 can collectively perform the instructions for performing the image processing method.

[0161] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster can store different instructions, respectively used for performing part of the functions of the CPU 110 and the GPU 120 described above. That is, the instructions stored in the memory 930 in different computing devices 900 can implement the functions of one or more modules in the CPU 110 and the GPU 120 described above.

[0162] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. Figure 11 A possible implementation manner is illustrated. As Figure 11 illustrated, two computing devices are respectively a computing device 900A and a computing device 900B, which are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation manner, the memory 930 in the computing device 900A stores instructions for performing the functions of part of the modules in the CPU 110 and the GPU 120 described above. At the same time, the memory 930 in the computing device 900B stores instructions for performing the functions of another part of the modules in the CPU 110 and the GPU 120 described above.

[0163] Figure 11 The connection manner between the computing device cluster illustrated in the figure can be that considering that the image processing method provided in the present application needs to store a large amount of data, it is considered to perform the functions of another part of the modules in the CPU 110 and the GPU 120 described above by the computing device 900B.

[0164] It should be understood that Figure 11 the functions of the computing device 900A illustrated in the figure can also be completed by multiple computing devices 900. Similarly, the functions of the computing device 900B can also be completed by multiple computing devices 900.

[0165] The embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection relationship between the computing devices in the computing device cluster described above with reference to Figure 9 and Figure 10 The difference is that the memory 930 in one or more computing devices 900 in the computing device cluster can store the same instruction for performing the image processing method.

[0166] In some possible implementation manners, the memory 930 in one or more computing devices 900 in the computing device cluster can also respectively store partial instructions for performing the image processing method. In other words, the combination of the one or more computing devices 900 can collectively execute the instruction for performing the image processing method.

[0167] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster can store different instructions for performing part of the functions of the computing device 900. That is, the instructions stored in the memory 930 in different computing devices 900 can implement the functions of one or more modules in the CPU 110 and the GPU 120 described above.

[0168] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the image processing method.

[0169] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that can be used to store data that can be accessed by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions that instruct the computing device to perform the image processing method.

[0170] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized by, The method comprises the following steps: receiving a plurality of mesh data of a component; the plurality of mesh data is discrete data of a three-dimensional model of the component; transmitting the plurality of mesh data to a graphics processing unit (GPU); clustering the plurality of mesh data to obtain a plurality of mesh cluster data; transmitting the plurality of mesh cluster data to the GPU; the GPU is used for rendering the plurality of mesh data, and rendering the plurality of mesh cluster data after receiving the plurality of mesh cluster data.

2. The method of claim 1, wherein, The method further comprises the following steps: detecting whether the number of the plurality of mesh data is greater than a set number; transmitting the plurality of mesh data to the GPU when the number of the plurality of mesh data is greater than the set number.

3. The method according to claim 1 or 2, characterized in that, The method further comprises the following steps: detecting whether the plurality of mesh data is clustered into the plurality of mesh cluster data within a set time; transmitting the plurality of mesh cluster data to the GPU when the plurality of mesh data is clustered into the plurality of mesh cluster data within the set time.

4. The method of claim 3, wherein, The method further comprises the following steps: transmitting the mesh cluster data that has been clustered to a GPU driver when the plurality of mesh data is not clustered into the plurality of mesh cluster data within the set time; the GPU driver is used for calling a rendering pipeline to render the mesh cluster data that has been clustered.

5. The method of claim 4, wherein, The method further comprises the following steps: receiving deleted mesh data and / or mesh cluster data sent by the GPU driver; the deleted mesh data refers to mesh data and / or mesh cluster data after the GPU driver performs view frustum culling and mesh occlusion culling on the received mesh data and / or mesh cluster data; the deleted mesh cluster data refers to mesh cluster data after the GPU driver performs view frustum culling and mesh occlusion culling on the received mesh cluster data; sending the deleted mesh data and / or mesh cluster data to the GPU.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises the following steps before transmitting the plurality of mesh data to the GPU: identifying surfaces of the component to which each mesh data in the plurality of mesh data belongs; the surface of the component comprises at least one layer; labeling the same layer ID for each mesh data corresponding to the layer of the component; the GPU is used for uniformly rendering mesh data and / or mesh cluster data corresponding to the same layer ID.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises the following steps: sending depth information and layer ID of each mesh data in the plurality of mesh data and depth information and layer ID of each mesh cluster data in the plurality of mesh cluster data to the GPU; the GPU is used for determining the relationship between each layer of the component according to the depth information and the layer ID.

8. An image processing apparatus characterized by comprising: The method comprises the following steps: a first processing unit is used for receiving a plurality of mesh data of a component; the plurality of mesh data is discrete data of a three-dimensional model of the component; a second processing unit is used for transmitting the plurality of mesh data to a graphics processing unit (GPU); a third processing unit, configured to cluster the plurality of mesh data to obtain a plurality of mesh cluster data; a fourth processing unit, configured to transmit the plurality of mesh cluster data to the GPU; the GPU is configured to render the plurality of mesh data, and render the plurality of mesh cluster data after receiving the plurality of mesh cluster data.

9. The apparatus of claim 8, wherein the second processing unit is specifically configured to detect whether the number of the plurality of mesh data is greater than a set number; the second processing unit is specifically configured to transmit the plurality of mesh data to the GPU when the number of the plurality of mesh data is greater than the set number.

10. The apparatus of claim 8 or 9, wherein the fourth processing unit is specifically configured to detect whether the plurality of mesh data are all clustered into the plurality of mesh cluster data within a set time; the fourth processing unit is specifically configured to transmit the plurality of mesh cluster data to the GPU when the plurality of mesh data are all clustered into the plurality of mesh cluster data within the set time.

11. The apparatus of claim 10, wherein the fourth processing unit is further configured to transmit the mesh cluster data that have been clustered to a GPU driver when the plurality of mesh data are not all clustered into the plurality of mesh cluster data within the set time; and the GPU driver is configured to call a rendering pipeline to render the mesh cluster data that have been clustered.

12. The apparatus of claim 11, wherein the fourth processing unit is further configured to receive deleted mesh data and / or mesh cluster data sent by the GPU driver; the deleted mesh data refers to mesh data and / or mesh cluster data after the GPU driver performs view frustum culling and mesh occlusion culling on the received mesh data and / or mesh cluster data; and the deleted mesh cluster data refers to mesh cluster data after the GPU driver performs view frustum culling and mesh occlusion culling on the received mesh cluster data; the fourth processing unit is further configured to send the deleted mesh data and / or mesh cluster data to the GPU.

13. The apparatus of any one of claims 8-12, wherein the first processing unit is further configured to identify surfaces of the components to which each of the plurality of mesh data belongs before transmitting the plurality of mesh data to the GPU; and the surfaces of the components include at least one layer. the first processing unit is further configured to mark the same layer ID for each mesh data corresponding to a layer of the components; and the GPU is configured to uniformly render the mesh data and / or mesh cluster data corresponding to the same layer ID.

14. The apparatus of any one of claims 8-13, wherein The fourth processing unit is further configured to send, to the GPU, depth information and layer ID of each of the plurality of mesh data and depth information and layer ID of each of the plurality of mesh cluster data; and the GPU is configured to determine the relationship between each layer of the component according to the depth information and the layer ID.

15. A computing device, comprising: Comprising: at least one memory; at least one processor configured to execute instructions stored in the memory to cause the computing device to perform the method of any of claims 1-7.

16. A computer-readable storage medium, characterized in that, comprising computer program instructions that, when executed by a computing device, cause the computing device to perform the method of any of claims 1-7.

17. A computer program product comprising instructions, characterized in that, The computer program product stores instructions that, when executed by a computing device, cause the computing device to implement the method of any of claims 1-7.

Images