Data loading method based on large model and related device

Abstract

The invention provides a data loading method based on a large model and a related device. Specifically, under the condition that a first data acquisition request sent by a client device is received, if requested data meet a preset abstract generation rule, a server can firstly return a data abstract corresponding to the requested data to the client device. Since the data size of the data abstract is smaller than that of the requested data, the client device can quickly load the data abstract of the requested data and display the probability of the requested data to the user. And under the condition that the requested data is determined to meet the preset abstract generation rule, the server can also predict whether the client equipment needs to load the complete content of the data or not according to the access habit information of the client equipment by adopting a complete loading prediction model. If yes, the requested data with the large data size is returned to the client device, and progressive loading is achieved. Therefore, the loading time consumption of the client equipment can be shortened, and the loading speed can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data loading, and particularly relates to a data loading method based on a large model and a related device. BACKGROUND

[0002] Server storage technology is a technical solution of storing data in a local server or a remote server and realizing centralized storage, centralized management and access protection of data through server hardware devices and their supporting software. Since the server storage technology can be adapted to most data application scenarios, it is widely used in multiple fields and has become one of the mainstream data storage solutions.

[0003] In the case of applying the server storage technology, if a client device needs to obtain specific data, the client device can send a data obtaining request to a server to instruct the server to send the required data to the client device, so that the data can be loaded into the client device. However, the inventors have found that the existing data loading solution has the problems of long loading time and slow loading speed. SUMMARY

[0004] The present application aims to at least solve one of the above technical defects, in particular, the technical defect of long loading time and slow loading speed in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a data loading method based on a large model, comprising:

[0006] In response to a first data obtaining request sent by a client device, determining requested data;

[0007] If the requested data meets a preset summary generation rule, obtaining access habit information of the client device, and returning a target data summary corresponding to the requested data to the client device; wherein the target data summary is generated in advance and is used to describe the gist information of the requested data;

[0008] Generating a loading prediction result according to the access habit information and a complete loading prediction model trained in advance; wherein the complete loading prediction model is used to predict whether the client device needs to load the complete content of the requested data;

[0009] If the loading prediction result is to load, the requested data is sent to the client device under the condition of meeting a preset loading rule.

[0010] In some embodiments, the step of generating a data summary comprises:

[0011] In the case of receiving original data, determining whether the original data meets the summary generation rule;

[0012] If the original data satisfies the summary generation rule and the original data belongs to a structured object, a node name, an original node value and a data type of each data node in the original data are obtained;

[0013] If the node name of the target node does not include a preset key field, summary information describing a content outline of the original node value is generated according to the data type of the target node and the original node value of the target node, and the node value of the target node is replaced with the summary information to obtain a data summary of the original data; wherein the target node is any data node of the original data.

[0014] In some embodiments, the step of generating the summary information describing the content outline of the original node value according to the data type of the target node and the original node value of the target node comprises:

[0015] If the data type of the target node is an array, the first N array elements and element count summary of the target node value are taken as the summary information; wherein the target node value is the original node value of the target node.

[0016] If the data type of the target node is a structured text, a key content of the target node value is extracted, and the summary information is generated according to the key content.

[0017] If the data type of the target node is an unstructured text, part of the characters of the target node value are taken as the summary information.

[0018] If the data type of the target node is a set data, data distribution statistics of the target node value are performed, and the summary information is generated according to the statistical results.

[0019] In some embodiments, the step of generating the data summary further comprises:

[0020] If the original data satisfies the summary generation rule and the original data belongs to a complex nested object, a type description text of the original data is generated, and a data summary of the original data is obtained according to the type description text.

[0021] In some embodiments, the step of judging whether the original data satisfies the summary generation rule comprises:

[0022] An actual data size, an actual data complexity and an actual data priority of the original data are obtained respectively.

[0023] If the actual data size is greater than a preset size threshold, the actual data complexity is higher than a preset complexity threshold, and the actual data priority is lower than a preset priority threshold, it is determined that the summary generation rule is met.

[0024] In some embodiments, the generating of the data summary further includes:

[0025] If the original data meets the summary generation rule, a retrieval identifier of the original data is generated, and the retrieval identifier is stored in association with the original data.

[0026] Metadata is added to the data summary of the original data according to the retrieval identifier.

[0027] In some embodiments, the method further includes:

[0028] In response to a second data acquisition request sent by the client device, a target retrieval identifier is extracted from the second data acquisition request.

[0029] If a preset verification rule is met, target original data is determined according to the target retrieval identifier, and the target original data is sent to the client device; wherein the sending priority of the target original data is higher than the sending priority of the requested data.

[0030] In a second aspect, the embodiments of the present application provide a data loading device based on a large model, comprising:

[0031] A first response module is configured to determine requested data in response to a first data acquisition request sent by a client device.

[0032] A data summary sending module is configured to, if the requested data meets a preset summary generation rule, obtain access habit information of the client device, and return a target data summary corresponding to the requested data to the client device; wherein the target data summary is pre-generated and is used to describe the gist information of the requested data.

[0033] A loading prediction module is configured to generate a loading prediction result according to the access habit information and a complete loading prediction model pre-trained, wherein the complete loading prediction model is used to predict whether the client device needs to load the complete content of the requested data.

[0034] A first complete data sending module is configured to, if the loading prediction result is to load, send the requested data to the client device if a preset loading rule is met.

[0035] In a third aspect, an embodiment of the present application provides a storage medium, the storage medium storing computer readable instructions, the computer readable instructions being executed by one or more processors to cause the one or more processors to perform the steps of the data loading method based on a large model according to any of the above embodiments.

[0036] In a fourth aspect, an embodiment of the present application provides a server, the server comprising: one or more processors, and a memory;

[0037] The memory stores computer readable instructions, the computer readable instructions being executed by the one or more processors to perform the steps of the data loading method based on a large model according to any of the above embodiments.

[0038] In the data loading method based on a large model and the related apparatus provided by some embodiments of the present application, in the case that the first data acquisition request sent by the client device is received, if the requested data meets the preset summary generation rule, the server can first return the data summary corresponding to the requested data to the client device. Since the data amount of the data summary is less than that of the requested data, the client device can quickly load the data summary of the requested data and show the user a gist of the requested data.

[0039] In the case that it is determined that the requested data meets the preset summary generation rule, the server can also use a complete loading prediction model to predict whether the client device needs to load the complete content of the data according to the access habit information of the client device. If so, the server returns the requested data with a larger data amount to the client device, and realizes progressive loading.

[0040] Therefore, the present application realizes progressive loading of data by sending a data summary with a smaller data amount first and then sending complete data with a larger data amount on demand, so that the loading time of the client device can be shortened, the loading speed can be improved, and the client device can show the user the data content in a shorter time. Moreover, the present application uses a complete loading prediction model to predict the complete loading demand of the client device, and returns the complete content of the requested data accordingly, so that the influence of progressive loading on user experience can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0042] Figure 1An application environment diagram of the big model-based data loading method in one embodiment;

[0043] Figure 2 A flowchart of the big model-based data loading method in one embodiment;

[0044] Figure 3 A data example of a structured object in one embodiment;

[0045] Figure 4 A structural diagram of the big model-based data loading device in one embodiment;

[0046] Figure 5 An internal structure diagram of the server in one embodiment. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0048] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of data information involved in the present application comply with relevant laws and regulations and do not violate public order and good customs.

[0049] In some embodiments, the big model-based data loading method provided by the present application can be applied to Figure 1 the application environment shown. The client device 102 refers to a device capable of displaying data content to the user 106, which can be but is not limited to various personal computers, notebook computers, laptop computers, tablet computers, smart phones, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc., and the portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 refers to a device providing data content to the client device 102, which can be a single server or a server cluster.

[0050] Specifically, when the client device 102 needs to display data content to the user 106, the client device 102 may send a data acquisition request to the server 104 to instruct the server 104 to return the requested data to the client device 102. Upon receiving the requested data, the client device 102 may use a specific data display method to display the data content of the requested data to the user 106. For example, the client device 102 may load a page based on the requested data, allowing the user 106 to view the data content of the requested data through the page.

[0051] When returning requested data, the existing technology returns the complete content of each requested data item to the client device 102, enabling the client device 102 to perform operations such as data caching and page generation based on the complete requested data, thereby realizing the data display function. Due to the large amount of requested data, the existing technology suffers from the technical problems of long loading time and slow loading speed.

[0052] However, in most application scenarios, users may not understand the complete content of each requested data in detail, but will only have an in-depth understanding of the data they are interested in. Based on this, in order to solve the above problems, the present application provides a data loading method and related devices based on a large model, which realizes progressive loading of data by first sending a data summary with a smaller amount of data, and then sending the complete data with a larger amount of data on demand, thereby shortening the loading time of the client device and improving the loading speed, so that the client device can display the data content to the user in a shorter time. In addition, the present application uses a complete loading prediction model to predict the complete loading requirements of the client device, and returns the complete content of the requested data accordingly, thereby reducing the impact of progressive loading on the user experience.

[0053] It is understood that the requested data involved in this application can be any type of data, and this application does not impose any specific restrictions on this. In some examples, for ease of description, this application uses browser data as an example. It should be understood that in addition to browser data, the requested data can also be other types of data, or the requested data can also include more types of data.

[0054] In some embodiments, the present application provides a data loading method based on a large model. The following embodiments use this method to Figure 1 The server shown in the figure is used as an example for explanation. Figure 2 As shown, the method may include the following steps:

[0055] S202: In response to a first data acquisition request sent by the client device, determine requested data.

[0056] The first data acquisition request can be sent by the client device and used to instruct the server to return the to-be-loaded data. The requested data refers to the data requested by the first data acquisition request.

[0057] In this step, the server can determine the requested data in any manner in a case where the first data acquisition request sent by the client device is received. For example, the first data acquisition request can carry a data identifier of the requested data, and the server can query the data according to the data identifier, so as to determine the requested data.

[0058] S204: If the requested data meets a preset summary generation rule, access habit information of the client device is acquired, and a target data summary corresponding to the requested data is returned to the client device; the target data summary is generated in advance and is used to describe the outline information of the requested data.

[0059] The summary generation rule refers to a rule used to determine whether the summary-full content mode needs to be applied to data loading, and the specific rule content can be set according to actual conditions, which is not limited in the application. The summary-full content mode refers to a loading mode in which the data summary is first returned to the client device, and then the complete data content is returned on demand.

[0060] The access habit information can be used to describe the data access habit of the client device, for example, can include any one or any combination of context operation information, data access preference information, associated data network information, historical access data and other information data of the client device. The target data summary refers to the data summary corresponding to the requested data, which can describe the outline information of the requested data.

[0061] In this step, the server can determine whether the requested data meets the preset summary generation rule. If yes, it indicates that the summary-full content mode needs to be applied to implement the loading of the requested data, and therefore, the server can return the data summary corresponding to the requested data, that is, the target data summary, to the client device, so that the client device performs data loading according to the target data summary. Moreover, in a case where it is determined that the requested data meets the preset summary generation rule, the server can acquire the access habit information of the client device, so as to perform loading prediction according to the access habit information.

[0062] It needs to be noted that the target data summary can be generated in advance. For example, the server can determine whether the original browser data sent by the browser collection component meets the summary generation rule in a case where the original browser data is received. If yes, the server can generate the data summary of the original browser data without waiting until the original browser data is requested to be acquired by the client device.

[0063] It can be understood that the target data digest can be generated in any manner, which is not specifically limited in the present application, as long as the data digest can describe the summary information of the requested data.

[0064] S206: generating a loading prediction result according to the access habit information and a pre-trained complete loading prediction model, wherein the complete loading prediction model is used to predict whether the client device needs to load the complete content of the requested data.

[0065] In this step, the server can use the complete loading prediction model to predict whether the client device needs to load the complete content of the requested data according to the access habit information of the client device, and obtain a loading prediction result. For example, the server can input the access habit information into the complete loading prediction model to perform loading prediction using the complete loading prediction model, and obtain the loading prediction result output by the complete loading prediction model.

[0066] It should be noted that the model type, model architecture, training method, loss calculation method, etc. of the complete loading prediction model can be determined according to actual conditions, which are not specifically limited in the present application, as long as it is a large model that can predict whether the client device needs to load the complete content of the requested data.

[0067] S208: if the loading prediction result is to load, the server sends the requested data to the client device under the condition that a preset loading rule is met.

[0068] The preset loading rule can be used to determine whether the client device loads the complete content of the requested data, and the specific rule content can be determined according to actual conditions, which is not specifically limited in the present application.

[0069] In some examples, the preset loading rule can be to determine whether the loading prediction result is to load. If the loading prediction result is to load, it can be determined that the preset loading rule is met, and the requested data is sent to the client device. Conversely, if the loading prediction result is not to load, it can be determined that the preset loading rule is not met.

[0070] In other examples, the preset loading rule can be to determine whether the network connection quality between the server and the client device meets a preset quality requirement. In the case of poor network connection quality, it can be determined that the preset loading rule is not met. In the case of good network connection quality, it can be determined that the preset loading rule is met.

[0071] In this step, if the complete loading prediction model predicts that the client device needs to load the complete content of the requested data, the server can return the requested data to the client device, so that the client device loads the data according to the received requested data, thereby displaying the complete data content of the requested data. In this way, the present application can predict the relevant data that the user may need to access, and send these data to the client device in advance, so that the client device can be loaded in advance, thereby reducing the waiting time of the user.

[0072] In this application, by sending a small amount of data summary first and then sending a large amount of complete data on demand, the progressive loading of data is realized, thereby shortening the loading time of the client device, improving the loading speed, and enabling the client device to display the data content to the user in a shorter time. Moreover, the complete loading prediction model is used to predict the complete loading requirement of the client device, and the complete content of the requested data is returned accordingly, thereby reducing the impact of progressive loading on user experience.

[0073] In some embodiments, the generation step of the data summary comprises:

[0074] Step A1: In the case of receiving the original data, it is judged whether the original data meets the summary generation rule;

[0075] Step A3: If the original data meets the summary generation rule and the original data belongs to a structured object, the node name, original node value and data type of each data node in the original data are obtained.

[0076] Step A5: If the node name of the target node does not include the preset key field, the summary information describing the content summary of the original node value is generated according to the data type of the target node and the original node value of the target node, and the node value of the target node is replaced with the summary information to obtain the data summary of the original data; wherein the target node is any data node of the original data.

[0077] In this embodiment, in order to enable the data summary to reduce the data volume on the basis of retaining the key information of the original data, and to reduce or avoid the adverse effects of data volume compression on data loading, for structured objects, the present application can generate a data summary according to the node name, data type and original node value of the data node.

[0078] Specifically, the server can judge whether the original data meets the preset summary generation rule when receiving the original data. If it meets, it can be determined that the original data needs to apply the summary-complete content mode when loading, so the data summary of the original data can be generated in advance.

[0079] During data summary generation, if the original data is a structured object, the server can obtain the node name, original node value, and data type of each data node in the original object. A structured object can be an entity with a clear hierarchical structure and defined attributes, including one or more data nodes. Furthermore, a structured object can be a flat or shallowly nested data object, for example, with a nesting depth of less than or equal to 2.

[0080] A data node is the basic building block of a structured object and can be obtained by dividing raw data into predetermined basic data types. Basic data types may include strings, numbers, Booleans, null values, arrays, objects, structures, and collections. The node name of a data node may include the name of its attribute. The raw node value of a data node refers to the specific data content corresponding to the data node in the raw data and can be used to reflect the actual storage information of the data node.

[0081] For example, when the original data is Figure 3 In the structured object shown, the original data includes three data nodes. The first data node has a node name of name, a node value of Book, and a data type of string. The second data node has a node name of price, a node value of 59.9, and a data type of numeric. The third data node has a node name of isAvailable, a node value of true, and a data type of Boolean.

[0082] Considering that the node name of a data node can reflect the informational meaning of the node value, for each data node in the original data, the server can determine whether the node name of the data node includes a preset key field, and accordingly determine whether the data node is used to store key information. If the node name of the data node does not include the preset key field, it can be determined that the data node is not used to store key information. Therefore, summary information for the data node can be generated based on the data type and original node value of the data node, and the node value of the data node can be replaced from the original node value to the summary information, thereby performing data compression on the node value of the data node.

[0083] For example, if the node name of the target node does not include the preset key field, the original node value is ["New York", "Los Angeles", "Chicago", "Houston", "Phoenix", "Philadelphia", "San Antonio", "San Diego"], and the abstract information is ["New York", "Los Angeles", "Chicago", "Houston", "...+4 more locations"], the node value of the target node can be replaced from ["New York", "Los Angeles", "Chicago", "Houston", "Phoenix", "Philadelphia", "San Antonio", "San Diego"] to "New York", "Los Angeles", "Chicago", "Houston", "...+4 more locations". In this way, in the data abstract, the node value of the target node is "New York", "Los Angeles", "Chicago", "Houston", "...+4 more locations".

[0084] It can be understood that the key field can be preset according to actual conditions such as node naming habits, statistical analysis results, etc., and the present application does not make specific limitations. For example, the key field can include id, name, status, type, etc. It should be noted that in the process of generating the data abstract, the server can only compress the node value of the data node, and keep the complete top structure, field name and structure information of the original data.

[0085] Further, in some embodiments, if the node name of the target node includes the preset key field, the original node value of the target node can be kept. In this way, the node value of the target node in the data abstract is consistent with the node value in the original data, so as to keep the key information.

[0086] The server can obtain the data abstract of the original data by performing the above operations on each data node in the original data, and ensure that the data abstract can describe the outline information of the original data.

[0087] In some embodiments, according to the data type of the target node and the original node value of the target node, the step of generating abstract information for describing the content outline of the original node value includes:

[0088] Step A51: If the data type of the target node is array, the first N array elements and element count digest of the target node value are taken as the digest information; wherein the target node value is the original node value of the target node.

[0089] Step A53: If the data type of the target node is structured text, the key content of the target node value is extracted, and the digest information is generated according to the key content;

[0090] Step A55: If the data type of the target node is unstructured text, part of the characters of the target node value are taken as the digest information;

[0091] Step A57: If the data type of the target node is set data, data distribution statistics are performed on the target node value, and the digest information is generated according to the statistical results.

[0092] Specifically, if the data type of the target node is array, the server can retain the length information of the array and the first N array elements as examples, and generate the digest information of the target node according to the same. Wherein N is a predetermined positive integer, for example, it can be 1, 2, 3, etc., and its specific value can be determined according to the actual situation. Illustratively, for an array with an element number of 50 and N = 3, its digest information can include the first 3 array elements, and the remaining 47 array elements are replaced by a count digest such as "[... and 47 more items]".

[0093] If the data type of the target node is structured text such as log, error message, etc., the server can identify and extract the key content of the original node value, and generate the digest information according to the key content, so that the digest information includes the key content. Wherein the key content can be error code, exception name or state indicator, etc. In some examples, the server can also add meta information in the data digest, which can record any one or any combination of content total length, line number, format type, etc.

[0094] It can be understood that the server can extract the key content in any way, for example, a pre-trained content extraction model can be used for extraction, or a regular expression can be used for pattern matching on the original node value.

[0095] If the data type of the target node is unstructured text, the server can extract part of the characters in the original node value according to the preset character extraction rule as the digest information of the data node. For example, the server can extract the first 1~200 characters and the last 100~200 characters of the original node value, and the middle content is replaced by an ellipsis mark.

[0096] If the target node's data type is set data, the server can generate summary information based on the data distribution statistics of the original node value, so that the summary information can reflect the data distribution of the original node value. For example, the server can count the total number of records in the set data, the number of different values ​​(such as the count of unique IDs), and the distribution statistics of different values ​​(such as the count of different values ​​of the status field). Based on this, the server can select M representative data items from the set data (such as records with typical characteristics or containing all fields) as examples to generate summary information, where M is a preset positive integer.

[0097] In some embodiments, the step of generating the data summary further includes:

[0098] Step A7: If the original data meets the summary generation rules and the original data belongs to a complex nested object, a type description text of the original data is generated, and a data summary of the original data is obtained according to the type description text.

[0099] The complex nested object may be a data object with a nesting depth greater than or equal to 3. The category description text may be used to describe the number of attributes of the object.

[0100] In most cases, data nodes with greater nesting depths are less likely to be used than data nodes with lesser nesting depths. For example, data nodes with greater nesting depths are less likely to be referenced and / or called. Therefore, when generating a data summary, if the original data is a complex nested object, the server may not expand its full content but instead replace it with a type description text such as "[Object with 12 properties]" and may also append the object's size information. This further reduces the size of the data summary.

[0101] In some embodiments, determining whether the original data satisfies summary generation rules includes:

[0102] Step B1: respectively obtaining the actual data size, actual data complexity, and actual data priority of the original data;

[0103] Step B3: If the actual data size is greater than the preset size threshold, the actual data complexity is higher than the preset complexity threshold, and the actual data priority is lower than the preset priority threshold, it is determined that the summary generation rule is satisfied.

[0104] In this embodiment, the server may comprehensively evaluate whether to apply the summary-full content mode to the original data for data loading based on data size, data complexity, and data priority, so as to balance high loading speed and good user experience.

[0105] Specifically, the server, in the case of receiving the original data, can acquire the data size of the original data, the data complexity of the original data and the data priority of the original data respectively, and obtain the actual data size, the actual data complexity and the actual data priority.

[0106] It can be understood that the server can determine the data size, the data complexity and the data priority in any manner, which is not specifically limited in the present application. For example, the server can calculate the total byte size of the original data and obtain the actual data size. For another example, the server can recursively traverse the data structure of the original data and calculate the nesting level depth and the number of application relationships, thereby obtaining the actual data complexity. For another example, the server can check the semantic type label of the original data, and take the data labeled as error information, state notification or key operation result as high-priority data.

[0107] The server can compare the actual data size with a preset size threshold to determine whether the actual data size is greater than the preset size threshold. Similarly, the server can compare the actual data complexity with a preset complexity threshold to determine whether the actual data complexity is higher than the preset complexity threshold. The server can compare the actual data priority with a preset priority threshold to determine whether the actual data priority is lower than the preset priority threshold.

[0108] If the actual data size is greater than the preset size threshold, the actual data complexity is higher than the preset complexity threshold and the actual data priority is lower than the preset priority threshold, it can be determined that the summary generation rule is satisfied. If the actual data size is less than or equal to the preset threshold size, the actual data complexity is lower than the preset complexity threshold or the actual data priority is higher than the preset priority threshold, it can be determined that the summary generation rule is not satisfied, in which case, when the original data is requested by the client device, the server can directly return the complete content of the original data.

[0109] It can be understood that the preset size threshold, the preset complexity threshold and the preset priority threshold can be determined according to actual conditions, which is not specifically limited in the present application. Exemplarily, the preset size threshold can be 1024 bytes, and the preset complexity threshold can be a nesting depth greater than 5 and / or including more than 20 independent objects.

[0110] In some embodiments, the step of generating the data summary further comprises:

[0111] Step A9: If the original data satisfies the summary generation rule, a retrieval identifier of the original data is generated, and the retrieval identifier is stored in association with the original data;

[0112] Step A11: Adding metadata to the data summary of the original data according to the retrieval identifier.

[0113] In this embodiment, when it is determined that the summary-full content mode needs to be applied, the server can generate a retrieval identifier of the original data and store the retrieval identifier in association with the original data, so as to facilitate subsequent determination of the full content of the original data according to the retrieval identifier. It can be understood that the retrieval identifier can be used to uniquely identify the original data, which can be generated in any manner, for example, can be generated based on a hash value and a timestamp of the original data, and the present application does not make a specific limitation hereon.

[0114] In the case where the retrieval identifier is obtained, the server can add metadata to the data summary, so that the metadata of the data summary records the retrieval identifier of the original data. It can be understood that, in addition to the retrieval identifier, the metadata can also include more information, which can be determined according to actual conditions, and the present application does not make a specific limitation hereon.

[0115] For example, the creation time, the last modification time and the data summary generation time of the original data, and the like. Further, the metadata can also include an integrity identifier, which is used to indicate whether the full content of the original data can be used for retrieval. The metadata can also include data format information and version information to help the client device correctly interpret the data summary.

[0116] In some embodiments, the data loading method based on a large model provided by the present application can further include:

[0117] Step C1: in response to the second data acquisition request sent by the client device, extracting the target retrieval identifier from the second data acquisition request;

[0118] Step C3: in the case where the preset verification rule is met, determining the target original data according to the target retrieval identifier, and sending the target original data to the client device; wherein the sending priority of the target original data is higher than the sending priority of the requested data.

[0119] Wherein, the second data acquisition request can be a request for acquiring full data content, and can carry the target retrieval identifier. The target retrieval identifier refers to the retrieval identifier corresponding to the data whose full content needs to be acquired.

[0120] In some examples, when the server receives the data acquisition request sent by the client device, if the data acquisition request carries the target retrieval identifier, it can be determined that the request is the second data acquisition request, otherwise, it can be determined that the request is the first data acquisition request. In other examples, when the server receives the data acquisition request sent by the client device, it can determine whether the request belongs to the first data acquisition request or the second data acquisition request according to the request type carried in the data acquisition request.

[0121] Specifically, in a case where the server receives the second data acquisition request sent by the client device, the server can extract the target search identifier from the second data acquisition request, and perform data search and data query according to the target search identifier, so as to obtain the original data corresponding to the target search identifier, that is, the target original data. In a case where a preset verification rule is met, the server can return the target original data to the client device with a high sending priority.

[0122] In this way, the server can preferentially send and respond to the data acquisition request triggered by the user to directly acquire complete content, and for predictive loading, the server can return the requested data using a background low-priority task. In this way, the loading time can be further shortened and the loading speed can be improved, the user waiting time can be shortened, and the user experience can be improved.

[0123] It can be understood that the specific rule content of the preset verification rule can be set according to actual conditions, and the present application does not make specific limitations thereto. In some examples, the preset verification rule can be to verify whether the user has corresponding data access permission and / or whether the target search identifier is a valid identifier.

[0124] In some examples, in a case where the client device receives complete data, the client device can integrate the complete data with the data digest. The client device can maintain the consistency of the data model, and ensure the smooth transition of the UI from the digest view to the complete content view. The client device can keep the current position and state of the user, and avoid view jumping or resetting caused by loading complete content. The client device can issue a data update notification and refresh the display accordingly.

[0125] In some embodiments, the client device can detect an implicit trigger condition generated by user interaction. For example, the client device can monitor the operation performed by the user on the data digest, and send a second data acquisition request to the server according to the user operation. For example, when the user clicks the "expand" button, scrolls to the edge of the data view, or stays on the digest item for more than a predetermined time threshold (such as 3 seconds), the client device can determine the data part that needs to be loaded according to the specific position and type of the interaction, and generate a second data acquisition request accordingly, and send the second data acquisition request to the server.

[0126] The following describes a large model-based data loading apparatus provided by an embodiment of the present application. The large model-based data loading apparatus described below can be referred to in correspondence with the large model-based data loading method described above.

[0127] In some embodiments, as shown in Figure 4 The present application provides a large model-based data loading apparatus 300, which includes:

[0128] The first response module 302 is configured to determine the requested data in response to the first data acquisition request sent by the client device.

[0129] The data digest sending module 304 is configured to acquire the access habit information of the client device and return a target data digest corresponding to the requested data to the client device if the requested data meets the preset digest generation rule; the target data digest is pre-generated and is used to describe the outline information of the requested data.

[0130] The loading prediction module 306 is configured to generate a loading prediction result according to the access habit information and a pre-trained complete loading prediction model; the complete loading prediction model is used to predict whether the client device needs to load the complete content of the requested data.

[0131] The first complete data sending module 308 is configured to send the requested data to the client device if the loading prediction result is to load and a preset loading rule is met.

[0132] In some embodiments, the apparatus 300 of the present application further comprises:

[0133] The digest generation judgment module is configured to judge whether the original data meets the digest generation rule if the original data is received.

[0134] The data node acquisition module is configured to acquire the node name, the original node value and the data type of each data node in the original data if the original data meets the digest generation rule and the original data belongs to a structured object.

[0135] The first data digest generation module is configured to generate digest information used to describe the content outline of the original node value according to the data type of the target node and the original node value of the target node and replace the node value of the target node with the digest information to obtain the data digest of the original data if the node name of the target node does not include a preset key field; the target node is any data node of the original data.

[0136] In some embodiments, the first data digest generation module of the present application comprises:

[0137] The first digest generation unit is configured to take the first N array elements and element count digest of the target node value as the digest information if the data type of the target node is an array; the target node value is the original node value of the target node.

[0138] The second abstract generation unit is configured to extract key content of the target node value if the data type of the target node is structured text, and generate the abstract information according to the key content.

[0139] The third abstract generation unit is configured to take part of characters of the target node value as the abstract information if the data type of the target node is unstructured text.

[0140] The fourth abstract generation unit is configured to perform data distribution statistics on the target node value if the data type of the target node is collection data, and generate the abstract information according to the statistical result.

[0141] In some embodiments, the apparatus 300 of the present application further comprises:

[0142] The second data abstract generation module is configured to generate type description text of the original data if the original data satisfies the abstract generation rule and the original data belongs to a complex nested object, and obtain a data abstract of the original data according to the type description text.

[0143] In some embodiments, the abstract generation judgment module of the present application comprises:

[0144] The actual data acquisition unit is configured to acquire actual data size, actual data complexity and actual data priority of the original data respectively.

[0145] The judgment unit is configured to determine that the abstract generation rule is satisfied if the actual data size is greater than a preset size threshold, the actual data complexity is higher than a preset complexity threshold, and the actual data priority is lower than a preset priority threshold.

[0146] In some embodiments, the apparatus 300 of the present application further comprises:

[0147] The association storage module is configured to generate a retrieval identifier of the original data if the original data satisfies the abstract generation rule, and store the retrieval identifier and the original data in association.

[0148] The metadata addition module is configured to add metadata to the data abstract of the original data according to the retrieval identifier.

[0149] In some embodiments, the apparatus 300 of the present application further comprises:

[0150] The second response module is configured to extract a target retrieval identifier from a second data acquisition request sent by the client device in response to the second data acquisition request.

[0151] The second complete data sending module is configured to, when a preset verification rule is met, determine target original data according to the target search identifier, and send the target original data to the client device; wherein the sending priority of the target original data is higher than the sending priority of the requested data.

[0152] In one embodiment, the present application also provides a storage medium having computer readable instructions stored therein, which, when executed by one or more processors, cause the one or more processors to perform the steps of the data loading method based on a large model in any embodiment.

[0153] In one embodiment, the present application also provides a server having computer readable instructions stored therein, which, when executed by one or more processors, cause the one or more processors to perform the steps of the data loading method based on a large model in any embodiment.

[0154] Schematically, Figure 5 An internal structure schematic diagram of a server provided by an embodiment of the present application is shown in FIG. 9. Referring to FIG. 9, Figure 5 The server 900 includes a processing component 902, which further includes one or more processors, and a memory resource represented by a memory 901, for storing instructions executable by the processing component 902, such as an application program. The application program stored in the memory 901 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 902 is configured to execute the instructions to perform the steps of the data loading method based on a large model described in any embodiment above.

[0155] The server 900 can also include a power supply component 903 configured to perform power management of the server 900, a wired or wireless network interface 904 configured to connect the server 900 to a network, and an input / output (I / O) interface 905. The server 900 can operate based on an operating system stored in the memory 901, such as Windows Server TM, MacOS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0156] Those skilled in the art can understand that the internal structure of the server shown in the present application is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the server to which the scheme of the present application is applied. Specifically, the server can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] Finally, it should be noted that the terms "first" and "second", and the like, herein do not denote any order, quantity, combination or importance, but are used to identify one element from another, and do not imply that the specific identities thereof are essential or that the identities are chronological or related in their occurrence. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element. Also, "a", "an", "the", and "said" are used to refer to one or more than one (i.e., to "at least one") of the referenced elements, unless otherwise specified. A plurality also means two or more, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of one or more of the associated listed items.

[0158] The various embodiments described in this specification are presented by way of example, and are not intended to limit the scope of the application. Each embodiment is presented in a way that emphasizes the differences between the embodiments and the other embodiments, but the embodiments can be combined according to the needs of the user.

[0159] The above description of disclosed embodiments provides enabling disclosure sufficient for one of ordinary skill in the art to practice the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data loading method based on a large model, characterized in that: include: In response to a first data acquisition request sent by a client device, determining requested data; If the requested data satisfies a preset summary generation rule, obtaining access habit information of the client device and returning a target data summary corresponding to the requested data to the client device; wherein the target data summary is pre-generated and used to describe the outline information of the requested data; Generate a loading prediction result based on the access habit information and a pre-trained complete loading prediction model; wherein the complete loading prediction model is used to predict whether the client device needs to load the complete content of the requested data; If the load prediction result indicates that loading is required, the requested data is sent to the client device if a preset loading rule is met.

2. The method according to claim 1, characterized in that The steps for generating a data summary include: Upon receiving the original data, determining whether the original data satisfies the summary generation rule; If the original data satisfies the summary generation rule and the original data belongs to a structured object, obtaining the node name, original node value and data type of each data node in the original data; If the node name of the target node does not include a preset key field, summary information is generated based on the data type of the target node and the original node value of the target node to describe the content outline of the original node value, and the node value of the target node is replaced with the summary information to obtain a data summary of the original data; wherein the target node is any data node of the original data.

3. The method according to claim 2, characterized in that The step of generating summary information for describing the content outline of the original node value according to the data type of the target node and the original node value of the target node comprises: If the data type of the target node is an array, the first N array elements and the element count summary of the target node value are used as the summary information; wherein the target node value is the original node value of the target node; If the data type of the target node is structured text, extracting key content of the target node value and generating the summary information according to the key content; If the data type of the target node is unstructured text, part of the characters of the target node value is used as the summary information; If the data type of the target node is set data, data distribution statistics are performed on the target node value, and the summary information is generated according to the statistical results.

4. The method according to claim 2, characterized in that The step of generating the data summary further includes: If the original data meets the summary generation rule and the original data belongs to a complex nested object, a type description text of the original data is generated, and a data summary of the original data is obtained according to the type description text.

5. The method according to claim 2, characterized in that The determining whether the original data satisfies the summary generation rule includes: respectively obtaining the actual data size, actual data complexity, and actual data priority of the original data; If the actual data size is greater than a preset size threshold, the actual data complexity is higher than a preset complexity threshold, and the actual data priority is lower than a preset priority threshold, it is determined that the summary generation rule is satisfied.

6. The method according to any one of claims 2 to 5, characterized in that The step of generating the data summary further includes: If the original data satisfies the summary generation rule, generating a search identifier for the original data, and storing the search identifier in association with the original data; Metadata is added to the data summary of the original data according to the retrieval identifier.

7. The method according to claim 6, characterized in that The method further comprises: In response to a second data acquisition request sent by the client device, extracting a target retrieval identifier from the second data acquisition request; When preset verification rules are met, target original data is determined according to the target retrieval identifier, and the target original data is sent to the client device; wherein the sending priority of the target original data is higher than the sending priority of the requested data.

8. A data loading device based on a large model, characterized in that: include: A first response module, configured to determine requested data in response to a first data acquisition request sent by a client device; a data summary sending module, configured to obtain access habit information of the client device and return a target data summary corresponding to the requested data to the client device if the requested data satisfies a preset summary generation rule; wherein the target data summary is pre-generated and used to describe the outline information of the requested data; a loading prediction module, configured to generate a loading prediction result based on the access habit information and a pre-trained complete loading prediction model; wherein the complete loading prediction model is configured to predict whether the client device needs to load the complete content of the requested data; The first complete data sending module is configured to send the requested data to the client device if the loading prediction result indicates that loading is required and a preset loading rule is met.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, which, when executed by one or more processors, enable the one or more processors to perform the steps of the large model-based data loading method as described in any one of claims 1 to 7.

10. A server, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, execute the steps of the large model-based data loading method according to any one of claims 1 to 7.

Images