A method, device, terminal device, and computer-readable storage medium for estimating memory occupancy

Abstract

The present application relates to a method, apparatus, and terminal device for estimating memory occupancy, including: determining a plurality of models to be rendered in the current scene, where the plurality of models to be rendered includes first-type models and / or second-type models; determining the number of vertices corresponding to the model to be currently rendered; determining the maximum number of fragments corresponding to the current model according to the number of vertices; and, if the current model is a first-type model, determining the memory space required for rendering the current model based on the space occupancy of the rendered first-type models, the maximum number of fragments, and the picture attributes; if the current model is a second-type model, determining the memory space required for rendering the current model based on the space occupancy of the rendered second-type models and the maximum number of fragments. The present application can make a good allocation of the subsequent use of memory by estimating the space occupied by the models to be rendered.

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology and also to the field of software algorithm technology. Specifically, it relates to a method, device, terminal device, computer-readable storage medium, and computer program product for estimating memory occupancy. Background Art

[0002] Augmented Reality (AR) is a relatively new technical content that combines real-world information and virtual-world information content. It simulates and processes entity information that is difficult to experience in the spatial range of the real world on the basis of technologies such as computers, superimposes and effectively applies virtual information content in the real world, and can be perceived by the human senses during this process, thus achieving a sensory experience beyond reality. After the real environment and virtual objects overlap, they can exist simultaneously in the same picture and space.

[0003] Augmented reality technology can not only effectively display the content of the real world but also prompt the display of virtual information content, and these delicate contents complement and superimpose each other. There are mainly new technologies and means such as multimedia, 3D modeling, and scene fusion in augmented reality technology, and there are obvious differences between the information content provided by augmented reality and the information content that humans can perceive.

[0004] At present, browser-based augmented reality (webAR) supports the ability to build 3D scenes and the ability to render the combination of virtual and real. However, for the current performance of mobile phones, the occupancy and optimization of rendering scenes are still the most important issues to be considered.

[0005] In response to the above problems in the related art, no effective solution has been found yet. The above is only the background information known to the inventor related to the present application and does not constitute an admission of the prior art. Summary of the Invention

[0006] In view of the technical problems existing in the prior art, the present invention provides a method for estimating memory occupancy, which is characterized by including: determining a plurality of models to be rendered in the current scene, the plurality of models to be rendered including a first type of model and / or a second type of model, the first type of model including pictures, and the second type of model not including pictures, wherein the rendering of the plurality of models is carried out one by one; determining the number of vertices corresponding to the model to be rendered currently; determining the maximum number of fragments corresponding to the current model according to the number of vertices; and, if the current model is a first type of model, determining the memory space required for rendering the current model based on the space occupancy, maximum number of fragments, and picture attributes of the rendered first type of models, wherein the picture attributes include picture size, picture width, and picture height; if the current model is a second type of model, determining the memory space required for rendering the current model based on the space occupancy and maximum number of fragments of the rendered second type of models; wherein, the picture attributes include: picture size, picture width, and picture height.

[0007] Optionally, determining the maximum number of fragments corresponding to the current model according to the number of vertices includes: calculating according to the first calculation formula: Max(triangle) = s×(s - 1)×(s - 2) / 6 where, Max(triangle) is the maximum number of fragments corresponding to the current model, and s is the number of vertices corresponding to the current model.

[0008] Optionally, in the case of determining that the current model is a first type of model, the determining the memory space required for rendering the current model includes: calculating the memory space required for rendering the structural part of the current model based on the space occupancy, maximum number of fragments, and picture attributes of one or more rendered first type of models; calculating the memory space required for rendering the picture part of the current model based on the space occupancy and picture attributes of one or more rendered first type of models; and determining the sum of the memory spaces required for rendering the structural part and the picture part of the current model as the memory space required for rendering the current model.

[0009] Optionally, calculating the memory space required for rendering the structural part of the current model based on the space occupancy, maximum number of fragments, and picture attributes of one or more rendered first type of models includes: calculating according to the second calculation formula: Memory(s + f) wp ≤K wp ×∑(Max(triangle)×ImgWidth×ImgHeight) where, Memory(s + f) wp is the memory space required for rendering the structural part of the current model, ImgWidth is the picture width, ImgHeight is the picture height, and K wpis the operation efficiency coefficient corresponding to the structural part of the current model, where K wp is the first preset value.

[0010] Optionally, based on the space occupancy and picture attributes of one or more first-class models that have been rendered, calculate the memory space required to render the picture part of the current model, including: calculating according to the third calculation formula: Memory(Img) = K p ×∑(ImgSize + 4×ImgWidth×ImgHight) where, Memory(Img) is the memory space required to render the picture part of the current model, ImgSize is the picture size, and K p is the operation efficiency coefficient corresponding to the picture part of the current model, where K p is the second preset value.

[0011] Optionally, when it is determined that the current model is a second-class model, the determination of the memory space required to render the current model includes: calculating according to the fourth calculation formula:

[0012] Memory(s + f) np = K np ×∑(Max(triangle)×4) where, Memory(s + f) np is the memory space required to render the structural part of the current model, and K np is the operation efficiency coefficient corresponding to the structural part of the current model, where K np is the third preset value.

[0013] Optionally, it further includes: calibrating the operation efficiency coefficient according to the actual space occupancy of the rendered model. If the current model is not the first model to be rendered among the first-class models or not the first model to be rendered among the second-class models, then calculate the operation efficiency coefficient corresponding to each rendered model according to the actual space occupancy of the rendered models of the same class, and determine the operation efficiency coefficient of the current model according to the operation efficiency coefficient corresponding to the rendered model.

[0014] Optionally, if the number of rendered first-class models or the number of rendered second-class models is less than 5, then determining the operation efficiency coefficient of the current model according to the operation efficiency coefficient corresponding to the rendered model includes: when the current model is a first-class model, use the second calculation formula to calculate the multiple operation efficiency coefficients corresponding to the structural parts of the multiple rendered first-class models respectively, and take the average value of the multiple operation efficiency coefficients as the operation efficiency coefficient K wpWhen the current model is a first-class model, the third calculation formula is used to calculate multiple computing efficiency coefficients corresponding to the rendered image parts of multiple first-class models, and the average value of the multiple computing efficiency coefficients is used as the computing efficiency coefficient K p When the current model is a second type model, the fourth calculation formula is used to calculate multiple operation efficiency coefficients corresponding to the structural parts of multiple rendered second type models, and the average value of the multiple operation efficiency coefficients is used as the operation efficiency coefficient K np .

[0015] Optionally, if the number of rendered first-category models or the number of rendered second-category models is greater than or equal to 5, the computational efficiency coefficient of the current model is determined according to the computational efficiency coefficients corresponding to the rendered models, including: eliminating singular samples in the rendered first-category models or second-category models; calculating the average value K′ of the computational efficiency coefficients corresponding to each model in the remaining first-category models or second-category models 平均 Determined as the operational efficiency coefficient of the current model.

[0016] Optionally, the step of removing the singular samples in the rendered first type model or the second type model comprises: Step 1: calculating the actual operation efficiency coefficients K′1 to K′ of the rendered n models n The standard deviation σ of i , satisfying condition 1: K′ i >K′ 平均 And (K′ i -σ) / K′ 平均 >R1, then K′ i Eliminate, among which, 1.1 <R1<1.9;步骤三:若第i个实际运算效率系数K′ i , satisfying condition 2: K′ i <K′ 平均 And (K′ i +σ) / K′ 平均 <R2,则将K′ i Eliminate, where 0.1 <R2<0.9;步骤四:基于剔除后剩余的多个实际运算效率系数,返回步骤一,并执行步骤一至四,直至满足停止条件,所述停止条件为剩余模型的数量小于或等于已渲染模型的数量的70%,或者当前剩余所有模型的运算效率系数都不满足条件一和条件二。

[0017] Optionally, if the current model is the first model rendered in the first type of models or the first model rendered in the second type of models, the computational efficiency coefficient is not calibrated.

[0018] An embodiment of the present application provides a device for estimating memory occupancy, which is characterized in that it is applied to a terminal device and includes: a model determination module for determining a plurality of models to be rendered in the current scene, the plurality of models to be rendered including a first type of model and / or a second type of model, the first type of model including pictures, and the second type of model not including pictures, wherein the rendering of the plurality of models is performed one by one; a vertex number determination module for determining the number of vertices corresponding to the model to be rendered currently; a fragment number confirmation module for determining the maximum number of fragments corresponding to the current model according to the number of vertices; and a first type of model calculation module for, if the current model is a first type of model, determining the memory space required for rendering the current model based on the space occupancy of the rendered first type of model, the maximum number of fragments, and the picture attributes, wherein the picture attributes include the picture size, the picture width, and the picture height; a second type of model calculation module for, if the current model is a second type of model, determining the memory space required for rendering the current model based on the space occupancy of the rendered second type of model and the maximum number of fragments; wherein the picture attributes include: the picture size, the picture width, and the picture height.

[0019] An embodiment of the present application provides a terminal device, the server includes a processor and a memory storing computer program instructions, and when the processor executes the computer program instructions, the steps of the method described above are implemented.

[0020] An embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of the method described above are implemented.

[0021] An embodiment of the present application provides a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the steps of the method described above are implemented.

[0022] In the solution of the present application, after obtaining the relevant resource occupancy of the rendered model, these data can be used to estimate the occupancy situation when the next model is rendered. In this way, the rendering process of the models in the entire scene changes from unknown to estimable, and the estimation result is made more accurate. After obtaining the possible occupancy situation when the next model is rendered, the system can adjust the memory allocation. The present application can make the resource allocation more reasonable, improve the memory usage efficiency, not only reduce unnecessary memory occupancy, but also improve the overall operation efficiency of the system. The present application is particularly applicable to occasions such as AR, VR, and MR that require rapid rendering of virtual models in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the implementation manner of the embodiments of the present application, the following briefly introduces the drawings in the embodiments of the present application.

[0024] Figure 1 It is a schematic diagram of the AR system architecture based on a server and a terminal device according to an embodiment of the present application.

[0025] Figure 2 It is a schematic diagram of a virtual-real fusion image for AR navigation using a mobile phone APP.

[0026] Figure 3 It is a flowchart of a method for estimating memory occupancy according to an embodiment of the present application.

[0027] Figure 4 It is a flowchart for determining the memory space required for rendering the current model in the case where the current model is determined to be a first type of model according to an embodiment of the present application.

[0028] Figure 5 It is a flowchart for removing singular samples in the rendered first type of model or second type of model according to an embodiment of the present application.

[0029] Figure 6 It is a block diagram of the structure of a device for estimating memory occupancy according to an embodiment of the present application.

[0030] Figure 7 It is a schematic diagram of the structure of a terminal device for implementing the method for estimating memory occupancy according to an embodiment of the present application.

[0031] Figure 8 It is a schematic diagram of the software structure of an exemplary terminal device according to an embodiment of the present application. Detailed implementation manners

[0032] The principles and spirit of the present application will be described below with reference to several exemplary implementation manners. It should be understood that the purpose of providing these implementation manners is to make the principles and spirit of the present application clearer and more thorough, so that those skilled in the art can better understand and then implement the principles and spirit of the present application. The exemplary implementation manners provided herein are only a part of the implementation manners of the present application, rather than all of the implementation manners. All other implementation manners obtained by those of ordinary skill in the art based on the implementation manners herein without creative efforts fall within the scope of protection of the present application.

[0033] Those skilled in the art know that the implementation manners of the present application can be realized as a system, a device, an equipment, a method, a computer-readable storage medium or a computer program product. Therefore, the present application can be specifically realized in at least one of the following forms: complete hardware, complete software, or a combination of hardware and software. According to the specific implementation manners of the present application, the present application claims protection for a method, a device, a terminal device, a computer-readable storage medium and a computer program product for estimating memory occupancy.

[0034] In this document, terms such as first, second, and third are only used to distinguish one entity (or operation) from another entity (or operation), and do not imply any order or relationship between these entities (or operations).

[0035] Embodiments of this application can be applied to servers and terminal devices. Please refer to Figure 1 , which schematically shows a schematic diagram of an AR system architecture based on a server and terminal devices. The AR system architecture includes a server 10 and several terminal devices 20. In some examples, the terminal device 20 is an AR device, which can be a dedicated AR device, such as a head-mounted AR device (Head-mounted displays, HMD), smart gloves, clothing and other intelligent wearable electronic devices. In some examples, the terminal device 20 can be a general AR device, such as a mobile phone, a portable computer, a laptop, a tablet computer, a virtual reality (Virtual Reality, VR) device, a vehicle-mounted device, a navigation device, a game device, and so on.

[0036] Taking an AR helmet or AR glasses as an example, a head-mounted display, a machine vision system, a mobile computer, etc. can be integrated and set in a device that can be bound and worn. The device has a display similar to glasses in shape and is worn on the user's head during operation. The device can transmit augmented reality information to the display or project it onto the user's eyeballs, thereby enhancing the user's visual immersion. In some examples, the AR device also has a camera, which can be a wide-angle camera, a telephoto camera, or a structured light camera (also known as a point cloud depth camera, 3D structured light camera, or depth camera). The structured light camera is based on 3D vision technology and can obtain the plane and depth information of an object. The structured light camera can project light with certain structural characteristics onto the object to be photographed through a near-infrared laser, and then the reflected light is collected by an infrared camera and processed by a processor chip. Its calculation principle is to calculate the object position and depth information based on the change of the light signal caused by the object and present a 3D image. Usually, a two-dimensional image is presented on a terminal device such as a mobile phone, and the depth of different positions on the image cannot be displayed. Using a structured light camera, 3D image information data can be captured, that is, not only the information such as the color of different positions in the image can be obtained, but also the depth information of different positions can be obtained, which can be used for AR ranging. Of course, ordinary terminal devices can also obtain the depth information of 2D images based on optical cameras and combined with deep learning algorithms and other methods, and finally 3D images can also be presented.

[0037] In some examples, software or an application program APP with AR function is installed in the terminal device 20. The server 10 can be the management server or application server of the software or APP. The server 10 can be a single server, a server cluster composed of multiple servers, or a cloud server or a cloud-based server, etc. The terminal device 20 is integrated with a module with networking function, such as a Wireless-Fidelity (Wifi) module, a Bluetooth module, a 2G / 3G / 4G / 5G communication module, etc., so as to connect to the server 10 through a network.

[0038] Exemplarily, the user can log in to the user account through the APP installed in the mobile phone, and the user can also log in to the user account through the software installed in the AR glasses.

[0039] Taking the APP with AR navigation function as an example, the APP can have capabilities such as high-precision map navigation, environmental understanding, and virtual-real fusion rendering. The APP can report the current geographical location information to the server 10 through the terminal device 20, and the server 10 provides AR navigation services for the user based on the real-time geographical location information. Exemplarily, taking the terminal device 20 as a mobile phone, in response to the user's operation of starting the APP, the mobile phone can start the camera to collect images of the real environment, and then the system performs AR enhancement on the images of the real environment collected by the camera, and incorporates or superimposes rendered AR effects (such as navigation route signs, road names, merchant information, advertisement displays, etc.) in the images of the real environment, and displays the virtual-real fusion images on the mobile phone screen.

[0040] Figure 2 Schematically shows a virtual-real fusion image for AR navigation using a mobile phone APP, where the indication arrow of AR navigation is superimposed on the real road surface and space in the figure, and the electronic resources of merchant promotions float at a specified position in the space in the form of a parachute carrying a gift box.

[0041] Embodiments of the present application relate to a terminal device and / or a server. The principles and spirits of the present application will be elaborated in detail through several exemplary embodiments or representative implementations below.

[0042] In view of the technical problems existing in the prior art, the present invention proposes a method for estimating memory occupancy. Figure 1 It is a flowchart of a method for estimating memory occupancy according to an embodiment of the present application, as Figure 1 shown, the method includes:

[0043] S101: Determine multiple models to be rendered in the current scene. The multiple models to be rendered include a first type of model and / or a second type of model. The first type of model includes pictures, and the second type of model does not include pictures, where the rendering of the multiple models is performed one by one.

[0044] S102: Determine the number of vertices corresponding to the model to be rendered currently.

[0045] S103: Determine the maximum number of fragments corresponding to the current model according to the number of vertices.

[0046] S104: If the current model is a first - type model, determine the memory space required for rendering the current model based on the space occupancy, maximum number of fragments, and picture attributes of the already - rendered first - type model, where the picture attributes include picture size, picture width, and picture height. If the current model is a second - type model, determine the memory space required for rendering the current model based on the space occupancy and maximum number of fragments of the already - rendered second - type model. Among them, the picture attributes include: picture size, picture width, and picture height.

[0047] In the solution of this application, since the models in the scene are rendered one by one, after a model has been rendered, the occupancy of relevant resources (for example, the space occupancy for rendering the model, the picture attributes in the model, etc.) can be obtained when rendering this model. After obtaining the occupancy of relevant resources of the already - rendered model, these data can be used to estimate the occupancy when rendering the next model. In this way, the rendering process of the models in the whole scene changes from unknown to estimable, and the estimation result becomes more accurate. After obtaining the possible occupancy when rendering the next model, the system can adjust the memory allocation. For example, if the estimated occupancy when rendering the next model is small, more memory can be allocated to other functions to improve the overall operation efficiency of the system. This application can make the resource allocation more reasonable and the memory usage more efficient. It can not only reduce unnecessary memory occupancy, but also improve the overall operation efficiency of the system.

[0048] According to an embodiment of this application, determining the maximum number of fragments corresponding to the current model according to the number of vertices includes: calculating according to the first calculation formula:

[0049] Max(triangle) = s×(s - 1)×(s - 2) / 6

[0050] Among them, Max(triangle) is the maximum number of fragments corresponding to the current model, and s is the number of vertices corresponding to the current model.

[0051] In the solution of this application, the number of fragments of some models can be directly obtained (when it can be directly obtained, Max(triangle) is the actual value), while for some models that cannot be directly obtained, the corresponding maximum number of fragments can be estimated based on the number of their vertices. Generally, the estimated corresponding maximum number of fragments will be larger than the actual number of fragments, which is more conducive to the loading of the model. In this application, model-related rendering is the most important part of the entire system operation and also occupies the largest part. Therefore, during the estimation process, the occupancy of other influencing factors can be ignored, and only the model rendering is estimated.

[0052] Figure 2 It is a flowchart for determining the memory space required for rendering the current model in the case where the current model is determined to be a first-type model in an embodiment of this application, as Figure 2 shown, which includes:

[0053] S201: Calculate the memory space required for rendering the structural part of the current model based on the space occupancy, maximum number of fragments, and picture attributes of one or more first-type models that have been rendered.

[0054] S202: Calculate the memory space required for rendering the picture part of the current model based on the space occupancy and picture attributes of one or more first-type models that have been rendered.

[0055] S203: Determine the sum of the memory spaces required for rendering the structural part and the picture part of the current model as the memory space required for rendering the current model.

[0056] According to an embodiment of this application, the rendering of the model includes the rendering of its structural part and picture part, and the sum of the occupancies of the two renderings is determined as the space occupancy for rendering the model.

[0057] Calculating the memory space required for rendering the structural part of the current model based on the space occupancy, maximum number of fragments, and picture attributes of one or more first-type models that have been rendered includes: calculating according to the second calculation formula:

[0058] Memory(s + f) wp

[0059] ≤K wp ×∑(Max(triangle)×ImgWidth×ImgHeight)

[0060] where Memory(s + f) wp is the memory space required for rendering the structural part of the current model, ImgWidth is the picture width, ImgHeight is the picture height, and K wpis the operation efficiency coefficient corresponding to the structural part of the current model, where K wp is the first preset value.

[0061] According to an embodiment of the present application, ImgWidth and ImgHeight can be directly obtained before rendering the model. According to an embodiment of the present application, when the current model is the first model to be rendered in the first type of models, K wp is the first preset value, where the first preset value can be a reference value based on previously rendered models (in the previously rendered models, Memory(s + f) wp , Max(triangle), ImgWidth, and ImgHeight are all known). For example, it can be the average value of K wp inferred from such previously rendered models. In this way, when rendering the models in the current scene, this preset value can be directly used for estimation processing.

[0062] In the present application, estimating the space occupancy through the maximum number of fragments can not only ensure the smooth progress of the model loading process, but also reasonably allocate the corresponding memory space, adjust the proportion of memory occupancy at all times, and increase the utilization rate of memory.

[0063] According to an embodiment of the present application, based on the space occupancy and picture attributes of one or more first-type models that have been rendered, calculate the memory space required to render the picture part of the current model, including: calculate according to the third calculation formula:

[0064] Memory(Img) = K p × ∑(ImgSize + 4 × ImgWidth × ImgHeight)

[0065] where Memory(Img) is the memory space required to render the picture part of the current model, ImgSize is the picture size, and K p is the operation efficiency coefficient corresponding to the picture part of the current model, where K p is the second preset value.

[0066] In the present application, in addition to the estimation of the structural part, it also includes the estimation of the picture part, and the sum of the two is determined as the space occupancy for rendering the model. Considering the picture occupancy part can also make the estimation of the space occupancy more accurate. Among them, the value of K p is similar to the aforementioned K wp , and will not be elaborated here.

[0067] According to an embodiment of the present application, in the case of determining that the current model is a second-type model, determine the memory space required to render the current model, including: calculate according to the fourth calculation formula:

[0068] Memory(s + f) np = K np × ∑(Max(triangle) × 4)

[0069] Wherein, Memory(s + f) np is the memory space required to render the structural part of the current model, and K np is the operation efficiency coefficient corresponding to the structural part of the current model, wherein, K np is the third preset value.

[0070] In this application, some models do not contain picture resources and are regarded as the second type of models. The estimation of the space occupied by the second type of models is only related to Max(triangle) and K np Related. Classifying the models into the first type of models and the second type of models can make the results more accurate during estimation. Among them, K np has a value similar to the aforementioned K wp and will not be elaborated here.

[0071] According to an embodiment of this application, the method for estimating memory occupancy further includes: calibrating the operation efficiency coefficient according to the actual space occupancy of the rendered model. Among them, if the current model is not the first model to be rendered in the first type of models or not the first model to be rendered in the second type of models, then calculate the operation efficiency coefficient corresponding to each rendered model according to the actual space occupancy of the rendered models of the same type, and determine the operation efficiency coefficient of the current model according to the operation efficiency coefficient corresponding to the rendered models.

[0072] In addition to the aforementioned methods for the value of K wp , K p , K p , the value of K wp , K p , K p can also be calibrated according to the resource occupancy of the rendered models in this scenario. Since the differences between rendered models may be very large in different scenarios, and the complexity of models may be closer in the same scenario, calibrating using the rendered models in this scenario can make the values of K wp , K p , K p more accurate. According to an embodiment of this application, K wp , K p , K p can use the actual values of K wp , K p , K p of the previous rendered model (i.e., the values obtained by inverse deduction of the relevant calculation formula) as the K wp , Kp , K p value.

[0073] According to an embodiment of the present application, if the number of rendered first - type models or the number of rendered second - type models is less than 5, then determine the operation efficiency coefficient of the current model according to the operation efficiency coefficients corresponding to the rendered models, including: when the current model is a first - type model, use the second calculation formula to calculate the multiple operation efficiency coefficients respectively corresponding to the structural parts of the multiple rendered first - type models, and take the average value of the multiple operation efficiency coefficients as the operation efficiency coefficient K wp ; when the current model is a first - type model, use the third calculation formula to calculate the multiple operation efficiency coefficients respectively corresponding to the picture parts of the multiple rendered first - type models, and take the average value of the multiple operation efficiency coefficients as the operation efficiency coefficient K p ; when the current model is a second - type model, use the fourth calculation formula to calculate the multiple operation efficiency coefficients respectively corresponding to the structural parts of the multiple rendered second - type models, and take the average value of the multiple operation efficiency coefficients as the operation efficiency coefficient K np .

[0074] In the present application, when the number of rendered first - type models or the number of rendered second - type models is less than 5, the value of K wp , K p , K np can be directly obtained by using the formula, and the average value is taken as the K wp , K p , K np corresponding to the estimated memory occupation during the rendering of the next model. In this way, the value of K wp , K p , K np obtained by taking the average is more accurate compared to directly using the actual value of K wp , K p , K np of the previous model.

[0075] According to an embodiment of the present application, if the number of rendered first - type models or the number of rendered second - type models is greater than or equal to 5, then determine the operation efficiency coefficient of the current model according to the operation efficiency coefficients corresponding to the rendered models, including: eliminating the singular samples in the rendered first - type models or second - type models; taking the average value K' 平均 of the operation efficiency coefficients corresponding to each model in the remaining first - type models or second - type models as the operation efficiency coefficient of the current model.

[0076] In this application, in the models rendered one by one, there may be individual singular samples (i.e., the complexity of this model is much higher or lower than that of other models), which will seriously affect the magnitude of the average value and make the estimation result deviate greatly from the actual value. By removing the singular samples, the influence on the estimation result can be reduced and the estimation result can be made more accurate.

[0077] Figure 3 It is a flowchart for removing singular samples from the first type of rendered models or the second type of rendered models in an embodiment of this application. As shown in the figure, it includes:

[0078] S301: Calculate the standard deviation σ of the actual operation efficiency coefficients K′1 to K′ of the n rendered models. n of.

[0079] S302: If the i-th actual operation efficiency coefficient K′ i , satisfies Condition 1: K′ i > K′ 平均 and (K′ i -σ) / K′ 平均 > R1, then remove K′, where 1.1 < R1 < 1.9.

[0080] S303: If the i-th actual operation efficiency coefficient K′ i , satisfies Condition 2: K′ i < K′ 平均 and (K′ i +σ) / K′ 平均 < R2, then remove K′ i , where 0.1 < R2 < 0.9.

[0081] S304: Based on the remaining multiple actual operation efficiency coefficients after removal, return to Step 1 and execute Steps 1 to 4 until the stop condition is met. The stop condition is that the number of remaining models is less than or equal to 70% of the number of rendered models, or the operation efficiency coefficients of all currently remaining models do not satisfy Condition 1 and Condition 2.

[0082] According to an embodiment of this application, removing singular samples is achieved by circularly calculating whether the gap between the K value (corresponding to K wp , K p or K np ) of each model and the standard deviation in the current valid samples (i.e., samples without singular samples) is too large or too small, so as to remove the singular samples among them. This can reduce the influence of singular samples on the estimation result.

[0083] According to an embodiment of this application, if the current model is the first rendered model in the first type of models or the first rendered model in the second type of models, the operation efficiency coefficient is not calibrated.

[0084] In this application, the first rendered model may not be estimated to reduce the computational burden.

[0085] A specific embodiment is given below to describe the above estimation process. The specific data is shown in Table 1.

[0086] Table 1

[0087]

[0088] As shown in the table, it contains two sets of data, namely the first rendered model and the second rendered model. Among them, these two models do not contain pictures, so they are both the second type of model. Among them, the number of fragments of the first model is 19388, that is, Max(triangle) = 19388, and the actual memory occupancy Memory(s + f) np = 102. According to the fourth calculation formula

[0089] Memory(s + f) np = K np ×∑(Max(triangle) × 4)

[0090] It can be known that when rendering the first model, K np1 = 0.00132. The number of fragments of the second model is Max(triangle) = 20244. Substituting it into the fourth calculation formula:

[0091] Memory(s + f) np

[0092] = K np ×∑(Max(triangle) × 4) = 0.00132 × 4 × 20244

[0093] = 109.3

[0094] From the data in Table 1, it can be seen that the actual space occupancy for rendering the second model is 105M, which is very close to the estimated occupancy of 109.3M calculated. It can be seen that this application can accurately estimate the space occupancy during model rendering.

[0095] In the solution of this application, since the models in the scene are rendered one by one, after a model has been rendered, it is possible to obtain the situation of relevant resource occupation when rendering this model (for example, the space occupation for rendering this model, the picture attributes in this model, etc.). After obtaining the relevant resource occupation of the rendered model, these data can be used to estimate the occupation situation when rendering the next model. In this way, the rendering process of the models in the entire scene changes from unknown to estimable. After obtaining the possible occupation situation when rendering the next model, the system can adjust the memory allocation. For example, if the estimated occupation when rendering the next model is small, more memory can be allocated to other functions to improve the overall operation efficiency of the system. This application can make the resource allocation more reasonable and the memory usage more efficient. It can not only reduce unnecessary memory occupation, but also improve the overall operation efficiency of the system.

[0096] It should be noted that for each embodiment of this application, for the sake of clear description, they are all expressed as a combination of a series of actions or processes. Those skilled in the art should know that the implementation process is not limited by the order of the described actions or processes, and some steps in the embodiments of this application can be processed in other orders or simultaneously.

[0097] Corresponding to the method provided by this application, this application also provides a device for estimating memory occupation. Figure 6 The structural schematic diagram of a device 100 for estimating memory occupation according to an embodiment of this application is shown, which includes:

[0098] A model determination module 401, configured to determine multiple models that need to be rendered in the current scene, the multiple models that need to be rendered include a first type of model and / or a second type of model, the first type of model includes pictures, and the second type of model does not include pictures, and the rendering of the multiple models is performed one by one;

[0099] A vertex number determination module 402, configured to determine the number of vertices corresponding to the model to be rendered currently;

[0100] A fragment number confirmation module 403, configured to determine the maximum number of fragments corresponding to the current model according to the number of vertices; and, a first type of model calculation module, configured to, if the current model is a first type of model, determine the memory space required for rendering the current model based on the space occupation, the maximum number of fragments, and the picture attributes of the rendered first type of model, where the picture attributes include the picture size, the picture width, and the picture height; a second type of model calculation module, configured to, if the current model is a second type of model, determine the memory space required for rendering the current model based on the space occupation and the maximum number of fragments of the rendered second type of model; where the picture attributes include: the picture size, the picture width, and the picture height.

[0101] Those skilled in the art should understand that the embodiments described herein belong to preferred embodiments, and the actions, steps, modules, units, etc. involved are not necessarily essential for the embodiments of the present application. In the above embodiments, the descriptions of the various embodiments of the present application each have their own emphasis. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0102] Figure 7 FIG. 4 is a schematic structural diagram of an electronic device 60 provided by an embodiment of the present application. The electronic device 60 includes a processor 61, a memory 62, and a communication bus for connecting the processor 61 and the memory 62. A computer program that can run on the processor 61 is stored in the memory 62. When the processor 61 runs the computer program, it can execute or implement the steps in the methods of the various embodiments of the present application. The electronic device 60 further includes a communication interface for receiving and sending data. The electronic device 60 may be the server in the embodiments of the present application, or the electronic device 60 may also be a cloud server. The electronic device 60 may also be the terminal device or AR device in the embodiments of the present application. In a suitable case, the electronic device may also be referred to as a computing device.

[0103] In some embodiments, the processor 61 may be a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a modem processor, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), etc.; the processor 61 may also be other general-purpose processors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. Among them, by learning from the biological neural network structure, the neural-network processing unit NPU can quickly process the input information and can also continuously perform self-learning. Through the NPU, the electronic device 60 can implement applications such as intelligent cognition, such as image recognition, face recognition, semantic recognition, speech recognition, text understanding, etc.

[0104] In some embodiments, the memory 62 may be an internal storage unit of the electronic device 60, such as the hard disk or memory of the electronic device 60; the memory 62 may also be an external storage device of the electronic device 60, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 60. The memory 62 may also include both the internal storage unit and the external storage device of the electronic device 60. The memory 62 can be used to store an operating system, application programs, a BootLoader, data, and other programs, such as program codes of computer programs. The memory 62 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read only memory (EPROM), or a compact disc read-only memory (CD-ROM). The memory 62 is used to store the program codes executed by the electronic device 60 and the data transmitted. The memory 62 can also be used to temporarily store the data that has been output or will be output.

[0105] Those skilled in the art can understand that Figure 7 merely examples of the electronic device 60, which do not constitute a limitation on the electronic device 60. The electronic device 60 may include more or fewer components than those shown in the figure, or combine some components, or include different components. For example, it may also include input / output devices, network access devices, etc.

[0106] Figure 8 is a schematic diagram of the software structure of the terminal device according to the embodiments of the present application. Taking the Android system as an example of the mobile phone operating system, in some embodiments, the Android system is divided into four layers, namely: the application layer, the application framework layer (framework, FWK), the system layer, and the hardware abstraction layer. The layers communicate with each other through software interfaces.

[0107] First of all, the application layer may include multiple application packages. The application packages may be various application programs app such as calls, cameras, videos, navigation, weather, instant messaging, education, etc., or may also be application programs app based on AR technology.

[0108] Second, the application framework layer FWK provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer may include some predefined functions, such as functions for receiving events sent by the application framework layer.

[0109] The application framework layer may include a window manager, a resource manager, a notification manager, etc.

[0110] Among them, the window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data, and make this data accessible to applications. The data may include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0111] Among them, the resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, and so on.

[0112] Among them, the notification manager enables applications to display notification information in the status bar, can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of background-running applications, and can also be a notification that appears on the screen in the form of a dialog window. For example, prompt text information in the status bar, emit a prompt sound, the electronic device vibrates, the indicator light flashes, etc.

[0113] In addition, the application framework layer may also include a view system. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, on the display interface of the SMS notification icon, there may be a view for displaying text and a view for displaying pictures.

[0114] Third, the system layer may include multiple functional modules, such as a sensor service module, a physical state recognition module, a 3D graphics processing library (e.g., OpenGLES), and so on.

[0115] Among them, the sensor service module is used to monitor the sensor data uploaded by various sensors in the hardware layer to determine the physical state of the mobile phone; the physical state recognition module is used to analyze and recognize user gestures, faces, etc.; the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0116] In addition, the system layer may further include a surface manager and a media library. The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media library supports the playback and recording of various common audio and video formats, as well as static image files, etc.

[0117] Finally, the hardware abstraction layer is the layer between the hardware and the software. The hardware abstraction layer may include a display driver, a camera driver, a sensor driver, etc., which are used to drive the relevant hardware in the hardware layer, such as a display screen, a camera, a sensor, etc.

[0118] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program or instructions. When the computer program or instructions are executed, the steps in the method designed in the above embodiment are implemented.

[0119] The embodiment of the present application also provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed, the steps in the method designed in the above embodiment are implemented. Exemplarily, the computer program product may be a software installation package.

[0120] Those skilled in the art should be aware that the methods, steps, or functions of the related modules / units described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product, or by the processor executing computer program instructions. Among them, the computer program product includes at least one computer program instruction, and the computer program instructions can be composed of corresponding software modules. The software modules can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable hard disks, CD-ROMs (Compact Disc Read-Only Memory), or any other form of storage medium well-known in the art. The computer program instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions can be transmitted from a website, a computer, a server, or a data center to another website, a computer, a server, or a data center in a wired or wireless manner. The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (such as an SSD), etc.

[0121] Regarding each device / product described in the above embodiments, the modules / units included therein can be software modules / units, hardware modules / units, or partially software modules / units and partially hardware modules / units. For example, for a device / product applied or integrated into a chip, each of the modules / units included therein can be implemented in a hardware manner such as a circuit, or at least some of the modules / units are implemented in the form of a software program and run on a processor integrated inside the chip, and the remaining modules / units are implemented in a hardware manner such as a circuit. Another example is that for a device / product applied or integrated into a terminal, each of the modules / units included therein can be implemented in a hardware manner such as a circuit, or at least some of the modules / units are implemented in the form of a software program and run on a processor integrated inside the terminal, and the remaining modules / units can be implemented in a hardware manner such as a circuit.

[0122] As described above, the foregoing is only a specific implementation manner of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein. It should be understood that the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should all be covered within the protection scope of the present application.

Claims

1. A method for estimating memory occupancy, characterized in that, Including: Determine multiple models to be rendered in the current scene, where the multiple models to be rendered include first - type models and / or second - type models. The first - type models include pictures, and the second - type models do not include pictures. The rendering of the multiple models is carried out one by one; Determine the number of vertices corresponding to the model to be rendered currently; Determine the maximum number of fragments corresponding to the current model according to the number of vertices; And, If the current model is a first - type model, based on the space occupancy, maximum number of fragments, and picture attributes of the already - rendered first - type models, as well as the maximum number of fragments and picture attributes of the current model, determine the memory space required for rendering the current model, where the picture attributes include picture size, picture width, and picture height; If the current model is a second - type model, based on the space occupancy and maximum number of fragments of the already - rendered second - type models, as well as the maximum number of fragments and picture attributes of the current model, determine the memory space required for rendering the current model; Wherein, the picture attributes include: picture size, picture width, and picture height.

2. The method according to claim 1, characterized in that, Determine the maximum number of fragments corresponding to the current model according to the number of vertices, including: Calculate according to the first calculation formula: Max(triangle)=s×(s - 1)×(s - 2) / 6 Where, Max(triangle) is the maximum number of fragments corresponding to the current model, and s is the number of vertices corresponding to the current model.

3. The method according to claim 2, characterized in that, In the case of determining that the current model is a first - type model, the determination of the memory space required for rendering the current model includes: Based on the space occupancy, maximum number of fragments, and picture attributes of one or more already - rendered first - type models, as well as the maximum number of fragments and picture attributes of the current model, calculate the memory space required for rendering the structural part of the current model; Based on the space occupancy and picture attributes of one or more already - rendered first - type models, as well as the maximum number of fragments and picture attributes of the current model, calculate the memory space required for rendering the picture part of the current model; Determine the sum of the memory spaces required for rendering the structural part and the picture part of the current model as the memory space required for rendering the current model.

4. The method according to claim 3, characterized in that, Based on the space occupancy, maximum number of fragments, and picture attributes of one or more already - rendered first - type models, calculate the memory space required for rendering the structural part of the current model, including: Calculate according to the second calculation formula: Memory(s+f) wp ≤K wp ×∑(Max(triangle)×ImgWidth×ImgHeight) Among them, Memory(s + f) wp is the memory space required for rendering the structural part of the current model. ImgWidth is the image width, ImgHeight is the image height, and K wp is the operation efficiency coefficient corresponding to the structural part of the current model, where K wp is the first preset value.

5. The method according to claim 4, characterized in that, Based on the space occupancy and picture attributes of one or more already - rendered first - type models, calculate the memory space required for rendering the picture part of the current model, including: Calculate according to the third calculation formula: Memory(Img) = K p ×∑(ImgSize + 4 × ImgWidth × ImgHeight) Among them, Memory(Img) is the memory space required to render the image part of the current model, ImgSize is the image size, and K p is the operation efficiency coefficient corresponding to the image part of the current model, where K p is the second preset value.

6. The method according to claim 5, characterized in that, In the case of determining that the current model is a second - type model, the determination of the memory space required for rendering the current model includes: Calculate according to the fourth calculation formula: Memory(s+f) np = K np ×∑(Max(triangle) × 4) Among them, Memory(s + f) np is the memory space required for rendering the structural part of the current model, K np is the operation efficiency coefficient corresponding to the structural part of the current model, where K np is the third preset value.

7. The method according to claim 6, characterized in that, Also include: Calibrate the operation efficiency coefficient according to the actual space occupancy of the already - rendered models, where, If the current model is not the first model to be rendered among the first type of models or the first model to be rendered among the second type of models, then calculate the operation efficiency coefficient corresponding to each rendered model according to the actual space occupancy of the rendered models of the same type, and determine the operation efficiency coefficient of the current model according to the operation efficiency coefficients corresponding to the rendered models.

8. The method according to claim 7, characterized in that, If the number of rendered models of the first type or the number of rendered models of the second type is less than 5, then determine the operation efficiency coefficient of the current model according to the operation efficiency coefficients corresponding to the rendered models, including: When the current model is a first - type model, use the second calculation formula to calculate multiple operation efficiency coefficients corresponding to the structural parts of the rendered first - type models respectively, and take the average value of the multiple operation efficiency coefficients as the operation efficiency coefficient K wp ; When the current model is a first - type model, use the third calculation formula to calculate the multiple operation efficiency coefficients corresponding to the rendered picture parts of the multiple first - type models respectively, and take the average value of the multiple operation efficiency coefficients as the operation efficiency coefficient K p ; When the current model is a second - type model, use the fourth calculation formula to calculate the multiple operation efficiency coefficients corresponding to the structural parts of the multiple rendered second - type models respectively, and take the average value of the multiple operation efficiency coefficients as the operation efficiency coefficient K np .

9. The method according to claim 8, wherein If the number of rendered models of the first type or the number of rendered models of the second type is greater than or equal to 5, then determine the operation efficiency coefficient of the current model according to the operation efficiency coefficients corresponding to the rendered models, including: Eliminate the singular samples in the rendered models of the first type or the second type; The average value K' of the operation efficiency coefficients corresponding to each of the remaining first-type models or second-type models 平均 is determined as the operation efficiency coefficient of the current model.

10. The method according to claim 9, wherein The elimination of the singular samples in the rendered models of the first type or the second type includes: Step 1: Calculate the standard deviation σ of the actual operation efficiency coefficients K′1 to K′ of the n rendered models; n ​ Step 2: If the i-th actual operation efficiency coefficient K′ i , satisfies Condition 1: K′ i >K′ 平均 and (K′ i -σ) / K′ 平均 >R1, then K′ i is eliminated, where 1.1 < R1 < 1.9; Step 3: If the actual operation efficiency coefficient K' of the i-th i , satisfies Condition 2: K' i < K' 平均 and (K' i + σ) / K' 平均 < R2, then K' i will be excluded, where 0.1 < R2 < 0.9; Step Four: Based on the remaining multiple actual operation efficiency coefficients after elimination, return to Step One and execute Steps One to Four until the stop condition is met. The stop condition is that the number of remaining models is less than or equal to 70% of the number of rendered models, or the operation efficiency coefficients of all the currently remaining models do not meet Condition One and Condition Two.

11. The method according to any one of claims 7-10, wherein If the current model is the first model to be rendered among the first type of models or the first model to be rendered among the second type of models, then do not calibrate the operation efficiency coefficient.

12. An apparatus for predicting memory occupancy, wherein Applied to a terminal device, including: A model determination module, configured to determine multiple models to be rendered in the current scene. The multiple models to be rendered include the first type of models and / or the second type of models. The first type of models includes pictures, and the second type of models does not include pictures. The rendering of the multiple models is carried out one by one; A vertex number determination module, configured to determine the number of vertices corresponding to the model to be rendered currently; A fragment number confirmation module, configured to determine the maximum number of fragments corresponding to the current model according to the number of vertices; and, A first type of model calculation module, configured to if the current model is a first type of model, then determine the memory space required for rendering the current model based on the space occupancy, maximum number of fragments, and picture attributes of the rendered first type of models, as well as the maximum number of fragments and picture attributes of the current model, where the picture attributes include picture size, picture width, and picture height; A second type of model calculation module, configured to if the current model is a second type of model, then determine the memory space required for rendering the current model based on the space occupancy and maximum number of fragments of the rendered second type of models, as well as the maximum number of fragments and picture attributes of the current model; Among them, the picture attributes include: picture size, picture width, and picture height.

13. A terminal device, wherein Including a processor and a memory storing computer program instructions. When the processor executes the computer program instructions, the method described in any one of claims 1-11 is implemented.

14. A computer-readable storage medium, wherein Computer program instructions are stored on the computer storage medium. When the computer program instructions are executed by the processor, the method described in any one of claims 1-11 is implemented.

15. A computer program product, wherein It includes computer program instructions which, when executed by a processor, implement the method according to any one of claims 1 to 11.

Images