Data transmission method and device, equipment and storage medium

Abstract

The invention provides a data transmission method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: calling a first graphic programming interface, creating a vertex buffer object, and distributing a first storage space for the vertex buffer object; receiving data transmitted by the data generation device; calling a second graphic programming interface, and creating a first buffer area having a mapping relationship with the first storage space; and according to a mapping relation between the first buffer area and the first storage space, writing the data into the first storage space by utilizing a calculation shader indicated by the second graphic programming interface, wherein the first graphic programming interface and the second graphic programming interface are different types of graphic programming interfaces. And the data transmission efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of graphics programming technology, and in particular to a data transmission method, device, electronic device and computer-readable storage medium. Background Art

[0002] In the related art, graphics rendering application developers will develop graphics rendering applications for one of the graphics programming interfaces (such as the OpenGL interface) and deploy the graphics rendering applications they developed to different operating systems. However, some manufacturers have launched other types of graphics programming interfaces specifically for the characteristics of their own hardware, and gradually stopped following up on the new versions of graphics programming interfaces required by graphics rendering application developers. As a result, some functions in the graphics rendering applications cannot be used after the graphics rendering applications are ported to certain devices. Summary of the invention

[0003] In view of this, the present application provides a data transmission method, device, electronic device and computer-readable storage medium.

[0004] Specifically, the present application is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, a data transmission method is provided, including:

[0006] Calling a first graphics programming interface, creating a vertex buffer object, and allocating a first storage space for the vertex buffer object;

[0007] receiving data transmitted by a data generating device;

[0008] Calling a second graphics programming interface to create a first buffer that has a mapping relationship with the first storage space; and, based on the mapping relationship between the first buffer and the first storage space, using a compute shader indicated by the second graphics programming interface to write the data into the first storage space; wherein the first graphics programming interface and the second graphics programming interface are different types of graphics programming interfaces.

[0009] Optionally, calling the second graphics programming interface to create a first buffer having a mapping relationship with the first storage space includes:

[0010] Calling a first graphics programming interface to establish a mapping relationship between the vertex buffer object and the address space of the application program, and obtaining a first pointer pointing to the address space;

[0011] Calling a second graphics programming interface, taking the first pointer as a parameter, and creating a first buffer;

[0012] The step of writing the data into the first storage space by using a compute shader indicated by the second graphics programming interface according to a mapping relationship between the first buffer and the first storage space comprises:

[0013] The data is written into the first buffer using a compute shader indicated by the second graphics programming interface; wherein, writing the data into the first buffer indicates: based on the first pointer and the mapping relationship, writing the data into the first storage space.

[0014] Optionally, the receiving data transmitted by the data generating device includes:

[0015] receiving the first frame of data transmitted by the data generating device; or

[0016] Receive incremental data transmitted by a data generating device; the incremental data is determined by the data generating device comparing the non-first frame data with the previous frame data after generating the non-first frame data; and / or the incremental data is determined by the data generating device fusing the non-first frame data with the previous frame data to obtain fused data after generating the non-first frame data, and then comparing the previous frame data with the fused data.

[0017] Optionally, the first graphics programming interface includes an OpenGL interface, and the second graphics programming interface includes a Metal interface.

[0018] Optionally, before calling the second graphics programming interface to create a first buffer area that is mapped to the first storage space, the method further includes:

[0019] Detecting whether a first storage space corresponding to the vertex buffer object satisfies a preset condition; the preset condition indicates that a free portion in the first storage space is sufficient to write the data;

[0020] If not, the first storage space corresponding to the vertex buffer object is expanded.

[0021] Optionally, after receiving the data transmitted by the data generating device, the method further includes:

[0022] storing the received data in a second storage space;

[0023] Before the computing shader instructed by the second graphics programming interface writes the data into the first storage space, the method includes:

[0024] Calling a second graphics programming interface, taking a second pointer pointing to the second storage space as a parameter, and creating a second buffer;

[0025] The data is read from the second storage space using a compute shader indicated by the second graphics programming interface according to the second pointer corresponding to the second buffer.

[0026] Optionally, after receiving the data transmitted by the data generating device, the method further includes:

[0027] storing the received data in a second storage space;

[0028] Before the computing shader instructed by the second graphics programming interface writes the data into the first storage space, the method includes:

[0029] Calling a second graphics programming interface to create a second buffer;

[0030] copying the data from the second storage space to the second buffer;

[0031] The data is read from the second buffer using a compute shader indicated by the second graphics programming interface.

[0032] Optionally, also include:

[0033] After the compute shader indicated by the second graphics programming interface completes writing the data, the first buffer and the second buffer created by calling the second graphics programming interface are deleted; wherein the first storage space corresponding to the vertex buffer object is not deleted.

[0034] Optionally, also include:

[0035] After the calculation shader indicated by the second graphics programming interface completes writing the data, the first graphics programming interface is called to render all the data in the first storage space corresponding to the vertex buffer object to obtain a three-dimensional model.

[0036] Optionally, calling the second graphics programming interface includes:

[0037] The second graphics programming interface is called through the conversion interface; wherein the conversion interface includes a Metal-cpp interface.

[0038] Optionally, the minimum storage unit of the first storage space and the second storage space is a memory page, and the starting address and size of the first storage space and the second storage space meet the page alignment rule indicated by the second graphics programming port.

[0039] Optionally, the data includes point cloud data.

[0040] According to a second aspect of an embodiment of the present application, there is provided a data transmission device, including:

[0041] A creation module, used for calling a first graphics programming interface, creating a vertex buffer object, and allocating a first storage space for the vertex buffer object;

[0042] A data receiving module, used for receiving data transmitted by a data generating device;

[0043] A data writing module is used to call a second graphics programming interface to create a first buffer that has a mapping relationship with the first storage space; and, based on the mapping relationship between the first buffer and the first storage space, use a compute shader indicated by the second graphics programming interface to write the data into the first storage space; wherein the first graphics programming interface and the second graphics programming interface are different types of graphics programming interfaces.

[0044] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a memory, a processor, and executable instructions stored in the memory and executable on the processor;

[0045] Wherein, when the processor executes the executable instructions, the steps in the method as described in any one of the first aspects are implemented.

[0046] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0047] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0048] In the embodiment of the present application, by calling the first graphics programming interface and calling the second graphics programming interface provided by the platform which is different from the first graphics programming interface, the two cooperate to speed up the data transfer to the storage space corresponding to the vertex buffer object. Specifically, by calling the second graphics programming interface to create a first buffer that has a mapping relationship with the first storage space, the calculation shader indicated by the second graphics programming interface can write the incremental data to the first storage space with the help of the mapping relationship, which is conducive to improving the data transmission efficiency.

[0049] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1It is a structural schematic diagram of a three-dimensional model generation system shown in an exemplary embodiment of the present application.

[0052] Figure 2 It is a schematic diagram of an application scenario of a three-dimensional model generation system shown in an exemplary embodiment of the present application.

[0053] Figure 3A It is a flowchart of a data transmission method shown in an exemplary embodiment of the present application.

[0054] Figure 3B It is a flowchart of another data transmission method shown in an exemplary embodiment of the present application.

[0055] Figure 4 It is a flowchart of another data transmission method shown in an exemplary embodiment of the present application.

[0056] Figure 5 It is a schematic diagram of a compute shader reading and writing point cloud data, shown in an exemplary embodiment of the present application.

[0057] Figure 6 It is another schematic diagram of a compute shader reading and writing point cloud data according to an exemplary embodiment of the present application.

[0058] Figure 7 It is a structural schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0060] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0061] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0062] In the related art, the graphics rendering application developer will develop the graphics rendering application for one of the graphics programming interfaces (such as the OpenGL interface), and there is a need to deploy it to different operating systems. For example, in the field of dental scanning, the developer needs to provide an application to present the scanned teeth for the next treatment or diagnosis process; and the application needs to be deployed to different operating systems to adapt to devices with different operating systems installed, such as Windows operating system and Apple operating system. However, if it is deployed in a device that does not follow the new version of the graphics programming interface, some functions will become unusable.

[0063] Take OpenGL as an example: OpenGL (Open Graphics Library) is a cross-platform graphics programming interface that provides a set of function libraries for rendering 2D and 3D graphics. The design goal of OpenGL is to provide a general, high-performance graphics programming interface that enables developers to take advantage of hardware-accelerated graphics rendering. It can be used to develop various graphics applications, including computer games, virtual reality, scientific visualization, computer-aided design (CAD), etc. Considering the cross-platform nature of OpenGL, some graphics rendering application developers will consider developing graphics rendering applications based on OpenGL. However, there are currently some manufacturers that have launched graphics programming interfaces specifically for the characteristics of their own hardware, and have gradually stopped following up on new versions of OpenGL. For example, Apple has launched Metal technology.

[0064] The fact that some manufacturers stop following the new version of OpenGL brings difficulties to the application deployment needs of graphics rendering application developers. If the OpenGL application developed according to the latest version of OpenGL or a higher version is ported to the platform that stops following the new version of OpenGL, some functions in the OpenGL application may not be available. If a new graphics rendering application is redeveloped for the new graphics programming interface, the development cycle is long and the workload of R&D personnel is heavy.

[0065] In an exemplary application scenario, such as in the field of dental scanning, see Figure 1 and Figure 2 , Figure 1 A schematic diagram of the structure of a three-dimensional model generation system is shown, which includes a camera 10 and an electronic device 20; wherein the electronic device 20 includes a point cloud generation device and a rendering device. The camera user scans the user's teeth in the user's mouth to collect tooth images, and transmits the tooth images to the electronic device 20. The point cloud generation device in the electronic device 20 is used to perform three-dimensional reconstruction on the tooth image, thereby obtaining point cloud data related to the teeth; and the obtained point cloud data will be transmitted to the rendering device, which receives the point cloud data and copies the point cloud data to the storage space corresponding to the vertex buffer object created based on the first graphics programming interface (such as OpenGL), so as to call the first graphics programming interface to render the point cloud data, obtain a three-dimensional model related to the teeth, and finally present the three-dimensional model on the display of the electronic device 20; or if the electronic device 20 does not include a display, the electronic device can transmit the three-dimensional model to the display device for display.

[0066] It is understandable that the three-dimensional model generation system can also be applied to the field of scanning other environments or objects and generating corresponding three-dimensional models, and this embodiment does not impose any limitation on this.

[0067] The inventors discovered that after a graphics rendering application developed according to the latest version or a higher version of a first graphics programming interface (such as OpenGL) is ported to a platform that stops following the new version of the first graphics programming interface, some functions in the OpenGL application cannot be used. As a result, after receiving the point cloud data, the electronic device copies the point cloud data to the storage space corresponding to the vertex buffer object, which results in low data transmission efficiency.

[0068] In response to the problems in the related art, an embodiment of the present application provides a data transmission method, which can realize the porting of a graphics rendering application developed according to the latest version or a higher version of a first graphics programming interface (such as OpenGL) to a platform that stops following the new version of the first graphics programming interface. By calling the first graphics programming interface and calling a second graphics programming interface provided by the platform that is different from the first graphics programming interface, the two cooperate to speed up the data transfer to the storage space corresponding to the vertex buffer object. From another perspective, there is no need to develop a new graphics rendering application from scratch, and some of the available functions in the graphics rendering application developed according to the target version can continue to be reused, which is conducive to shortening the development cycle and reducing the workload of R&D personnel.

[0069] The data transmission method provided in the embodiments of the present application can be executed by an electronic device, and the electronic device includes but is not limited to a smart phone / mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, a desktop computer, a media content player, a video game station / system, a virtual reality system, an augmented reality system, a wearable device (e.g., a watch, glasses, gloves, headwear (e.g., a hat, a helmet, a virtual reality headset, an augmented reality headset, a head-mounted device (HMD), a headband), a pendant, an armband, a leg ring, a shoe, a vest), a remote control, or any other device with a graphics processing function.

[0070] Exemplarily, the electronic device integrates a computer program product, and when the electronic device executes the computer program product, the data transmission method provided in the embodiment of the present application is implemented.

[0071] In some embodiments, see Figure 3A , Figure 3A A schematic flow chart of a data transmission method provided by an embodiment of the present application is shown. The method is applied to a rendering device in an electronic device, comprising:

[0072] In S101, a first graphics programming interface is called to create a vertex buffer object, and a first storage space is allocated for the vertex buffer object.

[0073] In S102, data transmitted by a data generating device is received.

[0074] In S103, a second graphics programming interface is called to create a first buffer having a mapping relationship with the first storage space; and, based on the mapping relationship between the first buffer and the first storage space, the data is written into the first storage space using a compute shader indicated by the second graphics programming interface; wherein the first graphics programming interface and the second graphics programming interface are different types of graphics programming interfaces.

[0075] In this embodiment, by calling the first graphics programming interface and calling the second graphics programming interface provided by the platform and different from the first graphics programming interface, the two cooperate to speed up the data transmission to the storage space corresponding to the vertex buffer object.

[0076] Exemplarily, the calling of the second graphics programming interface to create a first buffer having a mapping relationship with the first storage space includes: calling the first graphics programming interface to establish a mapping relationship between the vertex buffer object and the address space of the application program to obtain a first pointer pointing to the address space; calling the second graphics programming interface to create the first buffer using the first pointer as a parameter. The writing of the data to the first storage space using the compute shader indicated by the second graphics programming interface according to the mapping relationship between the first buffer and the first storage space includes: writing the data to the first buffer using the compute shader indicated by the second graphics programming interface; wherein, the writing of the data to the first buffer indicates: based on the first pointer and the mapping relationship, writing the data to the first storage space. In this embodiment, by creating a first buffer with a zero-copy function, it is possible to directly write incremental data to the first storage space using the compute shader indicated by the second graphics programming interface, thereby avoiding multiple data copying processes and improving data transmission efficiency.

[0077] Exemplarily, the data transmitted by the receiving data generating device includes: the first frame data transmitted by the receiving data generating device; and / or, the incremental data transmitted by the receiving data generating device; the incremental data is determined by the data generating device comparing the non-first frame data with the previous frame data after generating the non-first frame data; and / or, the incremental data is determined by the data generating device fusing the non-first frame data with the previous frame data to obtain fused data after generating the non-first frame data, and then comparing the previous frame data with the fused data. In this embodiment, only the changed or newly added data needs to be transmitted, and the amount of data that needs to be transmitted is reduced, which is conducive to saving transmission resources and improving transmission efficiency.

[0078] Exemplarily, the first graphics programming interface includes an OpenGL interface, and the second graphics programming interface includes a Metal interface.

[0079] Exemplarily, before calling the second graphics programming interface to create a first buffer zone that has a mapping relationship with the first storage space, it also includes: detecting whether the first storage space corresponding to the vertex buffer object meets a preset condition; the preset condition indicates that the free portion in the first storage space is sufficient to write the data; if not, expanding the first storage space corresponding to the vertex buffer object.

[0080] Exemplarily, after receiving the data transmitted by the data generating device, the method further includes: storing the received data in a second storage space; before the computing shader indicated by the second graphics programming interface writes the data into the first storage space, the method further includes: calling the second graphics programming interface, taking a second pointer pointing to the second storage space as a parameter, and creating a second buffer;

[0081] The data is read from the second storage space using a compute shader indicated by the second graphics programming interface according to the second pointer corresponding to the second buffer.

[0082] Exemplarily, after receiving the data transmitted by the data generating device, the method further includes: storing the received data in a second storage space. Before the computing shader indicated by the second graphics programming interface writes the data into the first storage space, the method further includes: calling a second graphics programming interface to create a second buffer; copying the data from the second storage space to the second buffer; and reading the data from the second buffer using the computing shader indicated by the second graphics programming interface.

[0083] Exemplarily, it also includes: after the compute shader indicated by the second graphics programming interface completes writing the data, deleting the first buffer and the second buffer created by calling the second graphics programming interface; wherein the first storage space corresponding to the vertex buffer object is not deleted.

[0084] Exemplarily, the method further includes: after the compute shader indicated by the second graphics programming interface completes writing the data, calling the first graphics programming interface to render all data in the first storage space corresponding to the vertex buffer object to obtain a three-dimensional model.

[0085] Exemplarily, the calling the second graphics programming interface includes: calling the second graphics programming interface through a conversion interface; wherein the conversion interface includes a Metal-cpp interface.

[0086] Exemplarily, the minimum storage unit of the first storage space and the second storage space is a memory page, and the starting address and size of the first storage space and the second storage space meet the page alignment rule indicated by the second graphics programming port.

[0087] Exemplarily, the data includes point cloud data.

[0088] Wherein, the data generating device includes a point cloud generating device, and the data includes point cloud data. It is understandable that the data generating device may also be other vertex data generating devices, and the data includes other types of vertex data. This embodiment does not impose any restrictions on this, and specific settings can be made according to actual conditions. Any type of vertex data can be quickly transmitted using the data transmission method provided in the embodiment of this application. The following is an exemplary description using the example of the data including point cloud data. The method is applied to Figure 1 In the rendering device in the electronic device, the rendering device is connected to the point cloud generating device, and the method includes:

[0089] In S101, a first graphics programming interface is called to create a vertex buffer object, and a first storage space is allocated for the vertex buffer object.

[0090] Exemplarily, a Vertex Buffer Object (VBO) is a buffer object used to store and manage vertex data in graphics rendering. In an embodiment of the present application, the first storage space corresponding to the Vertex Buffer Object is used to store point cloud data generated by a point cloud generation device, and the point cloud data includes a plurality of three-dimensional points, each of which carries attribute information, including but not limited to the position information, normal vector information, color information, transparency information or grayscale information of the three-dimensional point.

[0091] S102 includes: receiving point cloud data transmitted by a point cloud generating device.

[0092] Exemplarily, each time the point cloud generating device generates a frame of point cloud data, it can transmit it to the rendering device.

[0093] S103 includes: calling a second graphics programming interface to create a first buffer that has a mapping relationship with the first storage space; and, based on the mapping relationship between the first buffer and the first storage space, using a compute shader indicated by the second graphics programming interface to write the point cloud data into the first storage space; wherein the first graphics programming interface and the second graphics programming interface are different types of graphics programming interfaces.

[0094] In this embodiment, a first buffer zone that has a mapping relationship with the first storage space is created by calling the second graphics programming interface, so that the compute shader indicated by the second graphics programming interface can write incremental point cloud data into the first storage space with the help of the mapping relationship, thereby realizing the first graphics programming interface and the second graphics programming interface. The two cooperate to speed up the data transfer to the storage space corresponding to the vertex buffer object, which is conducive to improving data transmission efficiency.

[0095] In some embodiments, see Figure 3B, the S103 includes S1031 and S1032.

[0096] In S1031, a first buffer having a mapping relationship with the first storage space can be created in the following manner: first, a first graphics programming interface can be called to establish a mapping relationship between the vertex buffer object and the address space of the application program to obtain a first pointer pointing to the address space. Then, a second graphics programming interface is called to create a first buffer using the first pointer as a parameter.

[0097] Exemplarily, the rendering device maps the vertex buffer object to the address space of the application program, and in this way obtains the first storage space corresponding to the vertex buffer object for subsequent use. The rendering device calls the second graphics programming interface to create a first buffer based on the first pointer, so that the first buffer becomes a "zero copy buffer", so that the rendering device has the ability to use the compute shader indicated by the second graphics programming interface to write the point cloud data into the first buffer, which is actually the ability to directly write into the first storage space.

[0098] In S1032, the point cloud data can be written into the first buffer using the compute shader indicated by the second graphics programming interface; wherein, writing the point cloud data into the first buffer indicates: based on the first pointer and the mapping relationship, writing the point cloud data into the first storage space.

[0099] In this embodiment, by calling the second graphics programming interface, the first pointer pointing to the address space is used as a parameter to create a first buffer with a zero-copy function, so that the calculation shader indicated by the second graphics programming interface can directly write the incremental point cloud data to the first storage space with the help of the first pointer and the mapping relationship, avoiding multiple data copying processes, which is conducive to improving data transmission efficiency. It should be noted that the current incremental point cloud data is written to the first storage space, and the first graphics programming interface updates the previous data stored in the first storage space based on the current incremental point cloud data, and renders based on the updated point cloud data.

[0100] Exemplarily, when there are multiple compute shaders indicated by the second graphics programming interface, the point cloud data can be written in parallel into the first storage space using the multiple compute shaders indicated by the second graphics programming interface, thereby making use of the high concurrency capability of the compute shaders to further improve the data transmission efficiency.

[0101] In some embodiments, the electronic device is a device installed with iOS, tvOS or macOS. iOS is a mobile operating system developed by Apple. tvOS is a television operating system based on iOS developed by Apple. Mac OS is an operating system developed by Apple that runs on Macintosh series computers. The second graphics programming interface is a Metal interface, and the first graphics programming interface and the second graphics programming interface are different types of graphics programming interfaces, such as the first graphics programming interface is an OpenGL interface.

[0102] Exemplarily, taking the graphics rendering application as OpenGL as an example, for example, an electronic device with macOS currently supports OpenGL 4.1, or an electronic device with iOS currently supports OpenGL ES 3.0, if the graphics rendering application ported to the electronic device is developed according to a target version higher than the version supported by the electronic device (for example, the target version is OpenGL 4.3, or higher than OpenGL 4.3), then the electronic device can implement the data transmission method provided in the embodiment of the present application based on the image rendering application. It is understandable that the embodiment of the present application does not impose any restrictions on the specific version number of the target version, and can be specifically set according to the actual application scenario.

[0103] For S101, an exemplary explanation is given by taking the first graphics programming interface as an OpenGL interface as an example: First, the rendering device can use the glGenBuffers function in the OpenGL interface to generate an identifier of a vertex buffer object. Then, the glBindBuffer function in the OpenGL interface can be used to bind the generated vertex buffer object to the OpenGL context. Then, according to the vertex data structure, sufficient memory space is allocated, that is, the first storage space corresponding to the vertex buffer object. Subsequently, the point cloud data transmitted by the point cloud generation device can be copied to the first storage space corresponding to the vertex buffer object.

[0104] For S102, the point cloud generating device is connected to the rendering device. The point cloud generating device can scan the target environment or target object according to actual needs to obtain point cloud data. For example, refer to Figure 1 and Figure 2 In the oral scanning scenario, after receiving the tooth image captured by the camera, the point cloud generation device performs three-dimensional reconstruction on the tooth image to obtain point cloud data, and transmits the point cloud data to the rendering device, so that the rendering device can use the point cloud data to build a three-dimensional tooth model, and the three-dimensional tooth model can be used for occlusion detection, undercut processing, etc.

[0105] The point cloud generating device may perform three-dimensional reconstruction on at least one received two-dimensional image frame to generate at least one frame of point cloud data. Exemplarily, for the first frame of point cloud data, the point cloud generating device may transmit it to the rendering device, and the rendering device receives the first frame of point cloud data and executes step S103 to copy the first frame of point cloud data to the first storage space corresponding to the vertex buffer object, so as to facilitate subsequent use of the vertex buffer object for data rendering.

[0106] As for the generated non-first frame point cloud data, the current frame point cloud data of the non-first frame point cloud data is used for explanation. It is necessary to splice and / or fuse and / or optimize the current frame point cloud data with the previous frame point cloud data to obtain the current point cloud update data. Considering that the amount of point cloud update data increases with the increase of the number of frames of point cloud data, if the corresponding point cloud update data is transmitted to the rendering device for each frame of point cloud data, the performance requirement of data transmission will be very high, which obviously causes a waste of transmission resources, and the pressure of data transmission will become greater and greater, and the transmission efficiency will become lower and lower; therefore, the point cloud generation device selects the incremental point cloud data that is newly added and / or changed relative to the previous point cloud frame data from the current point cloud update data, and transmits the incremental point cloud data to the rendering device. It should be noted that the previous frame point cloud data can be one or more frames (partial or complete) of point cloud data previously acquired, can be point cloud data before splicing and / or fusion and / or optimization, or can be point cloud data after splicing and / or fusion and / or optimization (i.e., previous point cloud update data). For the generated non-first frame point cloud data, the point cloud generating device can also compare the current frame point cloud data with the previous frame point cloud data to determine the incremental point cloud data generated for the first time in the current frame point cloud data (that is, the point cloud data that has not been transmitted before), and then transmit the incremental point cloud data to the rendering device. It can be seen that only the newly added point cloud data needs to be transmitted, and the amount of data that needs to be transmitted is reduced, which is conducive to saving transmission resources and improving transmission efficiency.

[0107] For the previous frame point cloud data, due to the addition of splicing and / or fusion and / or optimization to the current frame point cloud data, some points in the previous frame point cloud data may change, and the point cloud generation device can also compare the previous frame point cloud data with the current point cloud update data, determine the incremental point cloud data that is newly added and / or changed in the current point cloud update data relative to the previous frame point cloud data, including the first generation of the current frame point cloud data and the changes and updates in the previous frame point cloud data, and transmit this part of the point cloud data as incremental point cloud data to the rendering device. In this embodiment, only the newly added and / or changed point cloud data needs to be transmitted, which is conducive to saving transmission resources and improving transmission efficiency.

[0108] Those skilled in the art can understand that, each time the point cloud generating device generates a frame of point cloud data, it can transmit the frame of point cloud data (specifically: the first frame of point cloud data, or incremental point cloud data) to the rendering device, and the rendering device can execute steps S102 to S103; that is, in the process of the point cloud generating device transmitting the point cloud data, steps S102 to S103 are repeatedly executed until the point cloud generating device stops transmitting the point cloud data.

[0109] In some embodiments, considering that in the process of scanning the target environment or target object, the point cloud data generated by the point cloud generating device is increasing, and accordingly, the point cloud data transmitted by the point cloud generating device to the rendering device is gradually accumulated over time, it may happen that the remaining space in the first storage space corresponding to the vertex buffer object is insufficient. Therefore, please refer to Figure 4 , before S1031, it also includes S105 and S106: in S105, the rendering device needs to detect whether the first storage space corresponding to the vertex buffer object meets the preset condition; the preset condition indicates that the free part in the first storage space is sufficient to write the point cloud data transmitted by the point cloud generation device; if not, in S106, it is necessary to expand the first storage space corresponding to the vertex buffer object, and then execute S1031; if it is satisfied, there is no need to expand the capacity, and S1031 is directly executed. This embodiment implements the expansion of the first storage space so that the point cloud data can be completely written into the first storage space.

[0110] Exemplarily, after receiving the point cloud data transmitted by the point cloud generation device, the rendering device can compare the data volume of the received point cloud data with the free capacity in the first storage space corresponding to the vertex buffer object. If the space capacity is smaller than the data volume of the point cloud data, it is necessary to expand the first storage space, and then execute S1031.

[0111] In some embodiments, for S1031, an exemplary explanation is given by taking the first graphics programming interface as an OpenGL interface: the rendering device can use the glMapBufferRange function in the OpenGL interface to establish a mapping relationship between the vertex buffer object and the address space of the application; in other words, the glMapBufferRange function is used to map part of the data of the vertex buffer object to a memory area accessible to the CPU, which allows the application to directly read or write the data of the vertex buffer object.

[0112] When using the glMapBufferRange function to establish a mapping relationship, you can specify the type of vertex buffer object to be mapped, specify the offset of the vertex buffer object from which to start mapping, specify the length of the data to be mapped, and specify at least one of the access rights to the mapped data. The return value of the glMapBufferRange function is a first pointer to the address space of the application, through which the point cloud data in the first storage space corresponding to the vertex buffer object can be directly read and written.

[0113] Taking the second graphics programming interface as a Metal interface as an example, the rendering device can use the newBufferWithBytesNoCopy function in the Metal interface to create a first buffer with a zero-copy function with a first pointer pointing to the address space as a parameter. When writing point cloud data into the first buffer, the point cloud data is actually written into the first storage space corresponding to the vertex buffer object.

[0114] For S1032, after creating the first buffer, the rendering device can write the point cloud data into the first buffer using the compute shader indicated by the Metal interface based on the high concurrency capability of the compute shader of Metal; in fact, writing the point cloud data into the first buffer means: based on the first pointer corresponding to the first buffer and the above-mentioned mapping relationship, using the compute shader indicated by the Metal interface to write the point cloud data into the first storage space; this avoids the multiple copying process of data and improves the data transmission efficiency.

[0115] In some embodiments, the compute shader indicated by the Metal interface needs to read the point cloud data first before writing the point cloud data into the first storage space. Two possible reading methods are provided below:

[0116] After receiving the point cloud data transmitted by the point cloud generating device, the received point cloud data can be stored in the memory space, that is, the second storage space. The first storage space and the second storage space are different storage spaces; specifically, the first storage space is the video memory space, and the second storage space is the memory space. The rendering device can render the point cloud data by calling the first graphics programming interface, specifically based on the point cloud data in the first storage space.

[0117] In one possible implementation, see Figure 5, the rendering device can call the second graphics programming interface, take the second pointer pointing to the second storage space as a parameter, and create a second buffer with zero copy function. After the first buffer and the second buffer are created, the rendering device can read the point cloud data from the second storage space according to the second pointer corresponding to the second buffer using the calculation shader indicated by the second graphics programming interface, and then write the read point cloud data into the first storage space using the calculation shader indicated by the second graphics programming interface according to the first pointer and the mapping relationship. This embodiment realizes a one-time copy process through the first buffer with zero copy function and the second buffer with zero copy function, which is conducive to improving data transmission efficiency.

[0118] Exemplarily, the rendering device may use the newBufferWithBytesNoCopy function in the Metal interface to create the second buffer with a zero-copy function by taking the first pointer as a parameter.

[0119] In some embodiments, in order to be able to call the second graphics programming interface to perform the point cloud data copy process, that is, to realize the above-mentioned zero copy function, the minimum storage unit of the above-mentioned first storage space and the second storage space is a memory page, and the starting address and size of the first storage space and the second storage space need to meet the page alignment rule indicated by the second graphics programming interface. The page alignment rule means that when allocating and accessing memory, the starting address and size of the data are required to be an integer multiple of the memory page size. The page alignment rule is intended to optimize access and performance, ensure that data is aligned according to the correct boundary, and thus improve processing efficiency.

[0120] In another possible implementation, see Figure 6 , the rendering device may call the second graphics programming interface to directly create a second buffer. After the second buffer is created, the point cloud data is copied from the second storage space to the second buffer. After the first buffer with zero copy function is created and the point cloud data is copied to the second buffer, the rendering device may read the point cloud data from the second buffer according to the calculation shader indicated by the second graphics programming interface, and then, according to the first pointer and the mapping relationship, write the read point cloud data into the first storage space using the calculation shader indicated by the second graphics programming interface.

[0121] Exemplarily, the rendering device may use the newBuffer function in the Metal interface to directly create the second buffer.

[0122] In a possible implementation, when the first graphics programming interface and the second graphics programming interface use different calling languages, a conversion interface is required to perform conversion processing.

[0123] Exemplarily, the first graphics programming interface includes an OpenGL interface and can be called through the C++ language. The second graphics programming interface includes a Metal interface and can be called through the Swift language or the Objective C language. When calling the second graphics programming interface, the rendering device can call the second graphics programming interface through a conversion interface; the conversion interface includes a Metal-cpp interface. The Metal-cpp interface bridges the C++ code with the Objective-C code. The Metal-cpp interface provides a set of classes and functions that encapsulate Metal's Objective-C interface, so that the C++ code can directly call these interfaces without writing Objective-C code in the C++ code. In this way, developers can use the syntax and features of C++ to write Metal applications, and can easily access and use Metal's functions. In other words, the role of the Metal-cpp interface is to convert the operation of calling the Metal interface with C++ into the operation of calling the Metal interface with Objective-C.

[0124] In another possible implementation, when the first graphics programming interface and the second graphics programming interface use the same calling language and the two interfaces support data transfer between each other, data can be directly transferred between the first graphics programming interface and the second graphics programming interface. Exemplarily, the first graphics programming interface includes an OpenGL interface and can be called through Swift or Objective C, and the second graphics programming interface includes a Metal interface and can also be called through Swift or Objective C.

[0125] In some embodiments, after the compute shader indicated by the second graphics programming interface completes writing the point cloud data, the rendering device may delete the first buffer and the second buffer created by calling the second graphics programming interface to release resources; wherein, the first storage space corresponding to the vertex buffer object is used to store the full amount of point cloud data generated by the point cloud generation device, and the first storage space corresponding to the vertex buffer object is not deleted.

[0126] In some embodiments, after the calculation shader indicated by the second graphics programming interface completes writing the point cloud data, the rendering device can call the first graphics programming interface to render all the point cloud data in the first storage space corresponding to the vertex buffer object to obtain a three-dimensional model, as shown in FIG. Figure 2The three-dimensional model of teeth displayed in the rendering device. Exemplarily, taking the first graphics programming interface as an OpenGL interface as an example, the glDrawArrays function in the OpenGL interface can be used to draw the point cloud data.

[0127] It is not difficult to understand that the schemes described in the above embodiments can be combined when there is no conflict, and they are not listed one by one in the embodiments of the present application.

[0128] Figure 7 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 7 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0129] The electronic device includes a point cloud generating device and a rendering device, wherein the rendering device includes a data transmission device. The data transmission device may include:

[0130] A creation module, used for calling a first graphics programming interface, creating a vertex buffer object, and allocating a first storage space for the vertex buffer object;

[0131] A data receiving module, used for receiving data transmitted by a data generating device;

[0132] A data writing module is used to call a second graphics programming interface to create a first buffer that has a mapping relationship with the first storage space; and, based on the mapping relationship between the first buffer and the first storage space, use a compute shader indicated by the second graphics programming interface to write the data into the first storage space; wherein the first graphics programming interface and the second graphics programming interface are different types of graphics programming interfaces.

[0133] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0134] Accordingly, an embodiment of the present application further provides an electronic device, including a memory, a processor, and executable instructions stored in the memory and executable on the processor;

[0135] Wherein, the processor implements the steps in the above method when executing the executable instructions.

[0136] Exemplarily, the processor includes but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0137] Exemplarily, the memory may include at least one type of storage medium, including flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc.

[0138] Accordingly, an embodiment of the present application further provides a computer program product, including a computer program, which is used to implement the above method when executed by a processor.

[0139] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0140] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal, enables the terminal to execute the above method.

[0141] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0142] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0143] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0144] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0145] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0146] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0147] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0148] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data transmission method, characterized in that: include: Calling a first graphics programming interface, creating a vertex buffer object, and allocating a first storage space for the vertex buffer object; receiving data transmitted by a data generating device; Calling a second graphics programming interface to create a first buffer zone that is mapped to the first storage space; And, according to the mapping relationship between the first buffer and the first storage space, the data is written into the first storage space using the compute shader indicated by the second graphics programming interface; wherein the first graphics programming interface and the second graphics programming interface are different types of graphics programming interfaces.

2. The method according to claim 1, characterized in that The calling of the second graphics programming interface to create a first buffer having a mapping relationship with the first storage space includes: Calling a first graphics programming interface to establish a mapping relationship between the vertex buffer object and the address space of the application program, and obtaining a first pointer pointing to the address space; Calling a second graphics programming interface, taking the first pointer as a parameter, and creating a first buffer; The step of writing the data into the first storage space by using a compute shader indicated by the second graphics programming interface according to a mapping relationship between the first buffer and the first storage space comprises: The data is written into the first buffer using a compute shader indicated by the second graphics programming interface; wherein, writing the data into the first buffer indicates: based on the first pointer and the mapping relationship, writing the data into the first storage space.

3. The method according to claim 1, characterized in that The receiving data transmitted by the data generating device comprises: receiving the first frame of data transmitted by the data generating device; or Receive incremental data transmitted by a data generating device; the incremental data is determined by the data generating device comparing the non-first frame data with the previous frame data after generating the non-first frame data; and / or the incremental data is determined by the data generating device fusing the non-first frame data with the previous frame data to obtain fused data after generating the non-first frame data, and then comparing the previous frame data with the fused data.

4. The method according to claim 1, characterized in that: The first graphics programming interface includes an OpenGL interface, and the second graphics programming interface includes a Metal interface.

5. The method according to claim 1, characterized in that Before calling the second graphics programming interface to create a first buffer area that has a mapping relationship with the first storage space, the method further includes: Detecting whether a first storage space corresponding to the vertex buffer object satisfies a preset condition; the preset condition indicates that a free portion in the first storage space is sufficient to write the data; If not, the first storage space corresponding to the vertex buffer object is expanded.

6. The method according to claim 1, characterized in that After receiving the data transmitted by the data generating device, the method further comprises: storing the received data in a second storage space; Before the computing shader instructed by the second graphics programming interface writes the data into the first storage space, the method includes: Calling a second graphics programming interface, taking a second pointer pointing to the second storage space as a parameter, and creating a second buffer; The data is read from the second storage space using a compute shader indicated by the second graphics programming interface according to the second pointer corresponding to the second buffer.

7. The method according to claim 1, characterized in that After receiving the data transmitted by the data generating device, the method further comprises: storing the received data in a second storage space; Before the computing shader instructed by the second graphics programming interface writes the data into the first storage space, the method includes: Calling a second graphics programming interface to create a second buffer; copying the data from the second storage space to the second buffer; The data is read from the second buffer using a compute shader indicated by the second graphics programming interface.

8. The method according to claim 5 or 6, characterized in that: Also includes: After the compute shader indicated by the second graphics programming interface completes writing the data, the first buffer and the second buffer created by calling the second graphics programming interface are deleted; wherein the first storage space corresponding to the vertex buffer object is not deleted.

9. The method according to claim 8, characterized in that Also includes: After the calculation shader indicated by the second graphics programming interface completes writing the data, the first graphics programming interface is called to render all the data in the first storage space corresponding to the vertex buffer object to obtain a three-dimensional model.

10. The method according to any one of claims 1 to 6, characterized in that: The calling of the second graphics programming interface comprises: The second graphics programming interface is called through the conversion interface; wherein the conversion interface includes a Metal-cpp interface.

11. The method according to claim 6, characterized in that The minimum storage unit of the first storage space and the second storage space is a memory page, and the starting address and size of the first storage space and the second storage space meet the page alignment rule indicated by the second graphics programming port.

12. The method according to claim 1, characterized in that The data includes point cloud data.

13. A data transmission device, characterized in that: include: A creation module, used for calling a first graphics programming interface, creating a vertex buffer object, and allocating a first storage space for the vertex buffer object; A data receiving module, used for receiving data transmitted by a data generating device; A data writing module, used for calling a second graphics programming interface to create a first buffer zone having a mapping relationship with the first storage space; And, according to the mapping relationship between the first buffer and the first storage space, the data is written into the first storage space using the compute shader indicated by the second graphics programming interface; wherein the first graphics programming interface and the second graphics programming interface are different types of graphics programming interfaces.

14. An electronic device, characterized in that: including a memory, a processor, and executable instructions stored in the memory and executable on the processor; Wherein, when the processor executes the executable instructions, the steps in the method as described in any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images