Data screening method, electronic equipment and storage medium
By obtaining the candidate model list and candidate memory usage list, determining the key model combination and target priority, filtering and deleting low-important data packets, the memory overflow problem caused by excessive data volume is solved, and efficient data filtering and system performance improvement is achieved.
Patent Information
- Application Number
- CN202510342861.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-21
- Publication Date
- 2025-06-20
AI Technical Summary
When the amount of data is too large, memory usage surges, resulting in memory overflow problems. It is difficult for the existing technology to effectively manage memory and improve system performance.
By obtaining the candidate model list and candidate memory usage list, determine the key model combination and target priority, filter and delete data packets with low importance, and realize data filtering.
It effectively avoids memory overflow problems, improves the system's response speed and overall performance, and ensures the accuracy of data screening.
Smart Images
Figure CN120179879A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and particularly to a method for data screening, an electronic device, and a storage medium. Background Art
[0002] Modeling users will push several data for page display. Before page display, the data will be processed. To make the page respond faster, caching the data in memory is a common optimization method. However, when the data volume is too large, it will cause a sharp increase in memory usage, which will in turn lead to a memory overflow problem. To avoid this situation, it is particularly important to use a cache management strategy based on the upper limit of available memory. When the memory space usage reaches the upper limit, selective deletion of data is performed to manage the memory, which can effectively improve the service performance. Summary of the Invention
[0003] In view of the above technical problems, the technical solution adopted by the present invention is as follows: A method for data screening, including the following steps:
[0004] S100, obtaining a candidate model list, where the candidate model list includes several candidate models, and the candidate models are digital models used for computer page display.
[0005] S200, according to the candidate model list, obtaining a list of key model combinations B = {B1,..., B j ,..., B m}, B j is the jth key model combination, j = 1... m, and m is the number of key model combinations in the list of key model combinations. Among them, in S200, the key model combinations are obtained through the following steps:
[0006] S201, obtaining a list of candidate memory occupancies C = {C1,..., C r ,..., C n} corresponding to the candidate model list, where C r is the candidate memory occupancy corresponding to the rth candidate model, r = 1... n, and n is the number of candidate models in the candidate model list.
[0007] S202, according to the candidate model list, obtaining a list of specified model combinations A = {A1,..., A i ,..., A N}, A i is the ith specified model combination, i = 1... N, and N is the number of specified model combinations in the list of specified model combinations. Among them, the specified model combination is a model combination obtained by screening a specified number of candidate models from the candidate model list for combination.
[0008] S203. Obtain a specified memory occupancy list D = {D1,..., D i ,..., D N} corresponding to the specified model combination list according to C and A, where D i is the specified memory occupancy corresponding to A i , and the specified memory occupancy is the sum of the candidate memory occupancies corresponding to the candidate models included in each specified model combination.
[0009] S204. When D i ≥∑ n r＝1 C r - D 0 , determine that A i is the key model combination, and D 0 is the maximum capacity of the data that can be stored in the Node memory.
[0010] S300. Obtain a target priority list F = {F1,..., F j ,..., F m} corresponding to the key model combination list, where F j is the target priority corresponding to B j , and the target priority is a score obtained based on the feature information of each candidate model in the key model combination.
[0011] S400. Determine a number of final models according to F and delete the data packet corresponding to each final model to implement data screening. When F j ≤ F 0 , determine the key models included in B j as the final models, and F 0 is a preset priority threshold.
[0012] The present invention protects a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above data screening method is implemented.
[0013] The present invention protects a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above data screening method is implemented.
[0014] The present invention has at least the following beneficial effects: A method for data screening, comprising the following steps: obtaining a list of candidate models, and based on the list of candidate models, obtaining a list of key model combinations. Among them, obtaining a list of candidate memory occupancies corresponding to the list of candidate models, based on the list of candidate models, obtaining a list of specified model combinations, based on the list of candidate memory occupancies and the list of specified model combinations, obtaining a list of specified memory occupancies corresponding to the list of specified model combinations, determining key model combinations according to the list of specified memory occupancies, obtaining a list of target priorities corresponding to the list of key model combinations, and based on the list of target priorities, determining several final models and deleting the data packets corresponding to each final model to achieve data screening. The present invention analyzes the characteristic data corresponding to the models, screens the data based on the priorities corresponding to the data, removes the data with lower importance, so that the accuracy of data screening is relatively high, can effectively avoid the problem of memory overflow, and improve the response speed and overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.
[0016] Figure 1 It is a flowchart of the method for data screening provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.
[0018] Embodiment
[0019] This embodiment provides a method for data screening. The method includes the following steps, as Figure 1 shown:
[0020] S100, obtaining a list of candidate models, where the list of candidate models includes several candidate models.
[0021] Specifically, the candidate model is a digital model for computer page display, such as: candidate models such as 3D rendering models.
[0022] S200. Obtain a list of key model combinations B = {B1, ……, B j , ……, B m}, where B j is the j-th key model combination, j = 1 …… m, and m is the number of key model combinations in the list of key model combinations.
[0023] Specifically, in S200, the key model combinations are obtained through the following steps:
[0024] S201. Obtain a list of candidate memory occupancies C = {C1, ……, C r , ……, C n} corresponding to the list of candidate models, where C r is the candidate memory occupancy corresponding to the r-th candidate model, r = 1 …… n, and n is the number of candidate models in the list of candidate models.
[0025] Specifically, the candidate memory occupancy is the size of the memory space required to process the data included in the candidate model.
[0026] S202. Obtain a list of specified model combinations A = {A1, ……, A i , ……, A N} according to the list of candidate models, where A i is the i-th specified model combination, i = 1 …… N, and N is the number of specified model combinations in the list of specified model combinations. Among them, the specified model combination is a model combination obtained by screening a specified number of candidate models from the list of candidate models for combination.
[0027] Specifically, the specified number is the number of candidate models included in the specified model combination.
[0028] Furthermore, the value range of the specified number is from 1 to n.
[0029] Specifically, N meets the following conditions:
[0030] N = 2 n - 1.
[0031] S203. Obtain a list of specified memory occupancies D = {D1, ……, D i , ……, D N} corresponding to the list of specified model combinations according to C and A, where D i is the specified memory occupancy corresponding to A i . Among them, the specified memory occupancy is the sum of the candidate memory occupancies corresponding to the candidate models included in each specified model combination.
[0032] S204. When D i≥∑ n r＝1 C r -D 0 When, determine A i as the key model combination, D 0 is the maximum capacity of data that can be stored in the Node memory.
[0033] Specifically, the key model combination includes several key models, where the key models are candidate models in the candidate model list.
[0034] S300, obtain the target priority list F = {F1,..., F j ,..., F m} corresponding to the key model combination list, F j is the target priority corresponding to B j where the target priority is a score obtained based on the feature information of each candidate model in the key model combination.
[0035] Specifically, in S300, F is obtained through the following steps j :
[0036] S301, obtain the key model list TB j corresponding to B j = {TB j1 ,..., TB je ,..., TB jf(j)}, TB je is the e-th key model corresponding to B j , e... f(j), f(j) is the number of key models in the key model list corresponding to B j corresponding.
[0037] S302, according to TB j , obtain the candidate feature data list set EB j corresponding to TB j = {EB j1 ,..., EB je ,..., EB jf(j)}, EB je = {EB 1 je ,..., EB v je ,..., EB b je}, EB v je is the v-th candidate feature data in the candidate feature data list corresponding to TB je , v = 1... b, b is the number of candidate feature data in the candidate feature data list.
[0038] Specifically, the candidate feature data is the data under the candidate feature, where the candidate feature is a performance feature characterized when the backend processes the data corresponding to the key model, such as candidate features like memory occupancy, data upload time, CPU processing time, model usage frequency, etc.
[0039] S303. According to EB j , obtain EB j The corresponding first target priority list set QB j = {QB j1 , ……, QB je , ……, QB jf(j)}, QB je = {QB 1 je , ……, QB v je , ……, QB b je}, QB v je is the first target priority corresponding to EB v je , where the first target priority is a value obtained by normalizing the candidate feature data.
[0040] Specifically, the method of the normalization process is based on the characteristics of the candidate feature data. Those skilled in the art know that the method of the normalization process can be selected according to actual needs, and all fall within the protection scope of the present invention, which will not be elaborated here. It can be understood that when the data corresponding to the candidate feature is larger, resulting in a smaller importance degree of the candidate feature, the normalization process method should ensure that the data after normalization becomes smaller as the data before normalization becomes larger. When the data corresponding to the candidate feature is larger, resulting in a larger importance degree of the candidate feature, the normalization process method should ensure that the data after normalization becomes larger as the data before normalization becomes larger. For example, for memory occupancy and CPU processing time, when the memory occupancy is larger, its corresponding importance degree is smaller, and when the CPU processing time is longer, its corresponding importance degree is smaller. Therefore, when normalizing the memory occupancy, a normalization method such as taking the reciprocal should be selected. For the model usage frequency, when the usage frequency of a model is larger, its corresponding importance degree is larger. Therefore, when normalizing the memory occupancy, a linear normalization method should be selected.
[0041] S304. According to QB j , obtain TB j The corresponding second target priority list PB j = {PB j1 , ……, PB je , ……, PBjf(j)}, PB je is TB je The corresponding second target priority, where PB je meets the following conditions:
[0042] PB je = ∑ b v＝1 (λ v je × QB v je ) / b, λ v je is EB v je The corresponding weight.
[0043] Specifically, λ is obtained through the following steps in S304 v je :
[0044] S3041, obtain the candidate feature data list EB j corresponding to the v-th candidate feature of B v j = {EB v j1 , ……, EB v je , ……, EB v jf(j)}}, EB v j1 is the v-th candidate feature data corresponding to the first key model of B j , EB v jf(j) is the v-th candidate feature data corresponding to the f(j)-th key model of B j .
[0045] S3042, according to EB v j , obtain the number of abnormal data U v corresponding to the v-th candidate feature, and the number of abnormal data U v is the number of abnormal data obtained from EB v j , where when , determine EB v je as the abnormal data in EB v j , σ = ∑ f(j) e＝1 EB v je / f(j).
[0046] S3043, when the v-th candidate feature is the first feature and U v < 2 / f(j), determine λ v je = U v / f(j).
[0047] Specifically, the first feature is a feature in which the candidate feature data corresponding to the candidate feature is negatively correlated with the first target priority. Among them, it can be understood that: when the candidate feature data corresponding to the candidate feature is larger, the obtained first target priority is smaller. For example, candidate features such as memory occupancy and CPU processing time.
[0048] S3044, when the v-th candidate feature is the first feature and U v ≥ 2 / f(j), determine λ v je = 1 - (U v / f(j)).
[0049] S3045, when the v-th candidate feature is the second feature and U v < 2 / f(j), determine λ v je = 1 - (U v / f(j)).
[0050] Specifically, the second feature is a feature in which the candidate feature data corresponding to the candidate feature is positively correlated with the first target priority. Among them, it can be understood that: when the candidate feature data corresponding to the candidate feature is larger, the obtained first target priority is smaller. For example, candidate features such as model usage frequency.
[0051] S3046, when the v-th candidate feature is the second feature and U v ≥ 2 / f(j), determine λ v je = U v / f(j).
[0052] S305, determine F j and PB j , where F j meets the following conditions: j
[0053] F j = (1 / f(j) + ∑ f(i) v＝1 PB je / f(j)) / 2.
[0054] As can be seen from the above, the feature data corresponding to the candidate model is analyzed, and weight settings are made for the candidate feature data based on the importance of the feature data, so that the accuracy of subsequent data screening is relatively high.
[0055] S400, according to F, determine a number of final models and delete the data packets corresponding to each final model to achieve data screening, where when F j ≤F 0 When, determine the key model included in B j as the final model, F 0 is a preset priority threshold.
[0056] Specifically, the value range of F 0 is 0.2 to 0.5. Among them, those skilled in the art know that the selection can be made according to actual needs, and all fall within the protection scope of the present invention, and will not be elaborated here.
[0057] As can be seen from the above, based on the number of key models and the weights of the key models corresponding to the key model combination, the priority of the key model combination is obtained, and the key models are screened according to the importance corresponding to the key model combination, and the data items with lower importance are removed, so that the accuracy of data screening is relatively high, which can effectively avoid the memory overflow problem and improve the response speed and overall performance of the system.
[0058] A data screening method provided in this embodiment, the method includes the following steps: obtaining a candidate model list, obtaining a key model combination list according to the candidate model list, where a candidate memory occupancy list corresponding to the candidate model list is obtained, a specified model combination list is obtained according to the candidate model list, a specified memory occupancy list corresponding to the specified model combination list is obtained according to the candidate memory occupancy list and the specified model combination list, a key model combination is determined according to the specified memory occupancy list, a target priority list corresponding to the key model combination list is obtained, and a number of final models are determined according to the target priority list and the data packets corresponding to each final model are deleted to achieve data screening. The present invention analyzes the feature data corresponding to the model, screens the data based on the priority corresponding to the data, and removes the data with lower importance, so that the accuracy of data screening is relatively high, which can effectively avoid the memory overflow problem and improve the response speed and overall performance of the system.
[0059] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to a method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.
[0060] An embodiment of the present invention further provides an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.
[0061] Although some specific embodiments of the present invention have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and not for limiting the scope of the present invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.
Claims
1. A method for data screening, characterized in that: The method comprises the following steps: S100, obtaining a candidate model list, wherein the candidate model list includes a plurality of candidate models, and the candidate models are digital models used for computer page display; S200, according to the candidate model list, obtain a key model combination list B = {B1, ..., B j , ..., B m }, B j is the jth key model combination, j=1...m, m is the number of key model combinations in the key model combination list, wherein the key model combination is obtained in S200 by the following steps: S201, obtaining a candidate memory usage list C corresponding to the candidate model list = {C1, ..., C r , ..., C n }, where C r is the candidate memory occupancy corresponding to the rth candidate model, r = 1...n, n is the number of candidate models in the candidate model list; S202, according to the candidate model list, obtain a specified model combination list A = {A1, ..., A i , ..., A N }, A i is the i-th specified model combination, i=1...N, N is the number of specified model combinations in the specified model combination list, wherein the specified model combination is a model combination obtained by screening a specified number of candidate models from the candidate model list and combining them; S203, according to C and A, obtain a specified memory occupancy list D corresponding to the specified model combination list = {D1, ..., D i , ..., D N }, D i A i The corresponding specified memory occupancy, wherein the specified memory occupancy is the sum of the candidate memory occupancy corresponding to the candidate models included in each specified model combination; S204, when D i ≥∑ n r＝1 C r -D 0 When A i is the key model combination, D 0 The maximum capacity of data that can be stored in the Node memory; S300, obtaining a target priority list F corresponding to the key model combination list = {F1, ..., F j , ..., F m }, F j For B j The corresponding target priority, wherein the target priority is a score obtained based on the feature information of each candidate model in the key model combination; S400, according to F, determine several final models and delete the data packets corresponding to each final model to achieve data screening, wherein when F j ≤F 0 When B j The included key models are determined as the final models, F 0 is the preset priority threshold.
2. The method for data screening according to claim 1, characterized in that: The candidate memory usage is the size of the memory space required to process the data included in the candidate model.
3. The method for data screening according to claim 1, characterized in that: The specified number is the number of candidate models included in the specified model combination, wherein the value range of the specified number is 1 to n.
4. The method for data screening according to claim 1, characterized in that: N meets the following conditions: N = 2 n -1.
5. The method for data screening according to claim 1, characterized in that: In S300, F is obtained by the following steps: j : S301, obtain B j Corresponding key model list TB j ={TB j1 , ..., TB je , ..., TB jf(j) }, TB je For B j The corresponding e-th key model, e...f(j), f(j) is B j The number of key models in the corresponding key model list; S302, according to TB j , get TB j Corresponding candidate feature data list set EB j ={EB j1 ,……,EB je ,……,EB jf(j) }, EB je ={EB 1 je ,……,EB v je ,……,EB b je }, EB v je TB je The vth candidate feature data in the corresponding candidate feature data list, v=1...b, b is the number of candidate feature data in the candidate feature data list; S303, according to EB j , get EB j The corresponding first target priority list set QB j = {QB j1 , ..., QB je , ..., QB jf(j) }, QB je = {QB 1 je , ..., QB v je , ..., QB b je }, QB v je EB v je a corresponding first target priority, wherein the first target priority is a value obtained by normalizing the candidate feature data; S304, according to QB j , get TB j The corresponding second target priority list PB j = {PB j1 , ..., PB je , ..., PB jf(j) }, PB je TB je The corresponding second target priority, where PB je Meet the following conditions: PB je =∑ b v＝1 (λ v je ×QB v je ) / b,λ v je EB v je The corresponding weights; S305, according to QB j and PB j , determine F j , where F j Meet the following conditions: F j =(1 / f(j)+∑ f(i) v＝1 PB je / f(j)) / 2。 6. The method for data screening according to claim 5, characterized in that: The candidate feature data is data under the candidate feature, wherein the candidate feature is a performance feature characterized by the backend when processing the data corresponding to the key model.
7. The method for data screening according to claim 1, characterized in that: F 0 The value range is 0.2~0.
5.
8. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 7.
9. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 8.