Method and related device for storing and processing data for large model scenes

Abstract

The present disclosure provides a method and related device for storing and processing data for large model scenarios, and involves data processing, data storage, cloud platform, cloud service, deep learning, large model, large language model and other artificial intelligence technology fields. A specific implementation of the method includes: in response to receiving data to be stored in a target data lake, detecting the data type of the data to be stored; in response to the data type being an unstructured data type, generating a feature vector of the data to be stored; and storing the feature vector as a substitute for the data to be stored in the target data lake. This implementation can enhance the data support capabilities of the data lake and expand the application scenarios of the data lake.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as data processing, data storage, cloud platforms, cloud services, deep learning, large models, large language models, and especially to methods, devices, electronic devices, computer-readable storage media, and computer program products for storing and processing data for large model scenarios. Background Art

[0002] With the development of society, the types of data are becoming more and more diverse, among which the more prominent data types include structured data and unstructured data. Unstructured data, such as multimedia data generated by application systems such as transportation systems and security systems, etc., in order to facilitate the storage and analysis of these data, special data management equipment is needed to manage these data.

[0003] Data lake is a big data architecture that can be used to store and process large-scale, diverse data. Data lake can eliminate the problems of data isolation and data dispersion, making data easier to access and analyze. Unlike traditional data warehouses, data lakes do not require predefined data structures or data models. Data lakes use a flat data model, store all data in its original format, and allow users to convert and process data on demand when needed. Therefore, how to improve the application and use value of data lakes is a problem worthy of attention. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, electronic device, computer-readable storage medium, and computer program product for storing and processing data for a large model scene.

[0005] In a first aspect, an embodiment of the present disclosure proposes a method for storing data for a large model scenario, including: in response to receiving data to be stored in a target data lake, detecting a data type of the data to be stored; in response to the data type being an unstructured data type, generating a feature vector of the data to be stored; and storing the feature vector as a substitute for the data to be stored in the target data lake.

[0006] In a second aspect, an embodiment of the present disclosure proposes a method for processing data for a large model scenario, comprising: in response to receiving a call request from a target device to call a second data processing model to process target input data, obtaining the target input data, wherein the second data processing model is obtained by adjusting the model data source of the first data processing model from the local data of the first data processing model to a target data lake, and the target data lake is obtained based on the data storage method for large model scenarios of the first aspect above; in response to the data type of the target input data being an unstructured data type, determining whether a target feature vector corresponding to the target input data can be extracted from the target data lake; in response to the target feature vector corresponding to the target input data being able to be extracted from the target data lake, extracting the target feature vector from the target data lake, and using the target feature vector as the actual input of the second data processing model, and using the second data processing model to generate a target output result corresponding to the target feature vector; and providing the target output result to the target device.

[0007] In a third aspect, an embodiment of the present disclosure proposes a device for storing data for a large model scenario, comprising: a data type detection unit, configured to detect the data type of the data to be stored in response to receiving the data to be stored in a target data lake; a feature vector conversion unit, configured to generate a feature vector of the data to be stored in response to the data type being an unstructured data type; and a feature vector storage unit, configured to store the feature vector as a substitute for the data to be stored in the target data lake.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a device for processing data for a large model scenario, comprising: a call request receiving unit, configured to obtain the target input data in response to receiving a call request from a target device to call a second data processing model to process target input data, wherein the second data processing model is obtained by adjusting the model data source of the first data processing model from the local data of the first data processing model to a target data lake, and the target data lake is obtained based on the device for storing data in the third aspect; a target feature vector detection unit, configured to determine whether a target feature vector corresponding to the target input data can be extracted from the target data lake in response to the data type of the target input data being an unstructured data type; a target feature vector extraction unit, configured to extract a target feature vector from the target data lake in response to the target feature vector corresponding to the target input data being able to be extracted from the target data lake, and use the target feature vector as the actual input of the second data processing model, and use the second data processing model to generate a target output result corresponding to the target feature vector; a target result output unit, configured to provide the target output result to the target device.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the method for storing data for large model scenarios as described in any implementation method in the first aspect and / or the method for processing data for large model scenarios as described in any implementation method in the second aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement a method for storing data for a large model scenario as described in any implementation method in the first aspect and / or a method for processing data for a large model scenario as described in any implementation method in the second aspect when executed.

[0011] In the seventh aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the method for storing data for large model scenarios as described in any implementation manner in the first aspect and / or the method for processing data for large model scenarios as described in any implementation manner in the second aspect.

[0012] The data storage method, device, electronic device, computer-readable storage medium, and computer program product for large model scenarios provided by the embodiments of the present disclosure detect the data type of the data to be stored in response to receiving the data to be stored in the target data lake. If the data type is an unstructured data type, a feature vector of the data to be stored is generated, and the feature vector is used as a substitute for the data to be stored and stored in the target data lake.

[0013] The data storage method for large model scenarios provided by the present invention converts unstructured data into feature vectors for storage when storing data in a data lake, so that the data lake can use the form of feature vectors to provide "unstructured data" that can be directly used to support model reasoning and training. This approach can enhance the data support capabilities of the data lake and expand the application scenarios of the data lake.

[0014] Furthermore, based on the above-mentioned data storage method for large model scenarios, the method for processing data for large model scenarios provided by the present disclosure can utilize the target data lake that stores unstructured data in the form of feature vectors as the data source of the data processing model, which can simplify the configuration requirements of the data processing model while improving the processing capabilities of the data processing model.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0017] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0018] Figure 2 A flowchart of a method for storing data in a large model scenario provided by an embodiment of the present disclosure;

[0019] Figure 3 A flowchart of a process for constructing a data processing queue provided in an embodiment of the present disclosure;

[0020] Figure 4 A flowchart of a method for processing data for a large model scenario provided by an embodiment of the present disclosure;

[0021] Figure 5 A flowchart of a method for storing data and a method for processing data in an application scenario combined with a large model scenario provided by an embodiment of the present disclosure;

[0022] Figure 6 A structural block diagram of a device for storing data for a large model scene provided in an embodiment of the present disclosure;

[0023] Figure 7 A structural block diagram of a device for processing data for a large model scene provided in an embodiment of the present disclosure;

[0024] Figure 8 A schematic diagram of the structure of an electronic device suitable for executing a method of storing data for a large model scenario and / or processing data for a large model scenario provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0026] In addition, in the technical solutions involved in this disclosure, the acquisition, storage, use, processing, transportation, provision and disclosure of user personal information (such as the data to be stored later involved in this disclosure) are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0027] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, electronic device, and computer-readable storage medium for storing and processing data for a large model scenario of the present disclosure can be applied.

[0028] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0029] The user can use the terminal devices 101, 102 and 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102 and 103 and the server 105 can be installed with various applications for realizing information communication between the two, such as online storage applications, cloud service applications, instant messaging applications, etc.

[0030] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.; when terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, which is not specifically limited here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, which is not specifically limited here.

[0031] The server 105 can provide various services through various built-in applications. Taking an online storage application that can provide data lake storage services in a large model scenario as an example, the server 105 can achieve the following effects when running the online storage application: First, in response to receiving the data to be stored in the target data lake from the terminal devices 101, 102 and 103 through the network 104, the server 105 detects the data type of the data to be stored; then, in response to the data type being an unstructured data type, the server 105 generates a feature vector of the data to be stored; finally, the server 105 uses the feature vector as a substitute for the data to be stored and stores it in the target data lake.

[0032] It should be noted that the data to be stored can be obtained from the terminal devices 101, 102 and 103 through the network 104, or can be pre-stored in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally (for example, when starting to process the previously reserved data storage task), it can choose to directly obtain the data from the local.

[0033] Since building a data lake, maintaining a data lake, and using a data lake may require more computing resources, storage resources, and stronger computing power, the data storage method and data processing method for large model scenarios provided in the subsequent embodiments of the present disclosure are generally executed by the server 105 with stronger computing power and more computing resources. Correspondingly, the device for storing data for large model scenarios is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned various operations handed over to the server 105 through the online storage class applications installed thereon, and then output the same results as the server 105. Especially when there are multiple terminal devices with different computing capabilities at the same time, but when the online storage application determines that the terminal device has strong computing capabilities and more remaining computing resources, the terminal device can be allowed to perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105. Correspondingly, the device for storing data for large model scenarios can also be set in the terminal devices 101, 102 and 103. For example, in some scenarios, the server 105 can be used as the execution subject of the method for storing data for large model scenarios, and the terminal devices 101, 102 and 103 can be used as the execution subject of the method for processing data for large model scenarios.

[0034] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0035] First, please refer to Figure 2 , Figure 2 A flowchart of a method for storing data in a large model scenario provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0036] Step 201: In response to receiving data to be stored in a target data lake, detecting a data type of the data to be stored;

[0037] This step is intended to be performed by the execution subject of the storage data method for large model scenarios (such as Figure 1 The server 105 shown in the figure responds when receiving the data to be stored in the target data lake, and detects the data type of the data to be stored. The data type of the data to be stored may be, for example, structured data, unstructured data, etc.

[0038] Structured data, also known as row data, is data that is logically expressed and implemented by a two-dimensional table structure. It strictly follows the data format and length specifications and is mainly stored and managed through a relational database. In contrast to structured data, unstructured data is not suitable for being represented by a two-dimensional database table. Unstructured data is data with an irregular or incomplete data structure, no predefined data model, and is not convenient to be represented by a two-dimensional database logical table. Unstructured data can include office documents in various formats, XML, HTML, various reports, pictures, audio, and video information.

[0039] It should be noted that the data to be stored can be obtained by the above-mentioned execution subject directly from a local storage device, or from a non-local storage device (such as Figure 1 The local storage device may be a data storage module provided in the above-mentioned execution subject, such as a server hard disk, in which case the data to be stored can be quickly read locally; the non-local storage device may also be any other electronic device provided for storing data, such as some user terminals, in which case the above-mentioned execution subject may obtain the required data to be stored by sending an acquisition command to the electronic device.

[0040] Step 202: In response to the data type being an unstructured data type, generating a feature vector of the data to be stored;

[0041] Based on step 201, this step is intended to be responded by the above-mentioned execution subject when it is determined that the data type of the data to be stored is an unstructured data type (referred to as unstructured data), and to process the data to be stored and generate a feature vector of the data to be stored.

[0042] For example, the execution subject can process the data to be stored into feature vectors based on feature engineering (embedding). In some embodiments, the execution subject can process the data to be stored into feature vectors by calling a data processing model. It should be understood that the corresponding data processing model can usually be configured according to the specific type of unstructured data (e.g., text, picture, etc.) to process it and generate feature vectors. For example, when the data to be stored is unstructured data in the form of text, the execution subject can process it through, for example, a word vector model, and process it into a corresponding feature vector (e.g., a word feature vector, a semantic feature vector, etc.). Similarly, when the data to be stored is unstructured data in the form of a picture, the execution subject can process it into a corresponding feature vector (e.g., a graphic semantic feature vector) through an image feature extraction model.

[0043] In some optional implementations of this embodiment, if the execution entity determines that the data to be stored is not unstructured data, for example, the data to be stored is structured data, the execution entity may directly store it in the target data lake without processing it into a feature vector.

[0044] Step 203: The feature vector is used as a substitute for the data to be stored and stored in the target data lake.

[0045] On the basis of step 202, this step is intended to be used by the execution subject to store the feature vector generated in the above step 202 as a substitute for the data to be stored in the target data lake. For example, the execution subject can obtain the location (or storage space) expected to be used to store the data to be stored, and then store the feature vector in the location instead of storing the data to be stored in the location, so that the feature vector can be used as a substitute for the data to be stored. In some optional implementations of this embodiment, the execution subject can maintain the original access path, access identifier, keyword information, etc. used for the data to be stored, so that the feature vector can be retrieved as a substitute based on the original method of retrieving the data to be stored. As a result, when other devices subsequently expect to retrieve and use unstructured data, the data lake can use the feature vector corresponding to the unstructured data as a substitute and provide it to other devices.

[0046] The data storage method for large model scenarios provided by the embodiments of the present disclosure converts unstructured data into feature vectors for storage when storing data in a data lake, so that the data lake can use the feature vector form to provide "unstructured data" that can be directly used to support model reasoning and training. This approach can enhance the data support capabilities of the data lake and expand the application scenarios of the data lake.

[0047] In some scenarios, the execution subject may continuously receive multiple data to be stored, or allow multiple data to be continuously received, and the execution subject may not have the ability to process and store these data to be stored in the target data lake at the same time. Or, considering the cost of resource configuration and other reasons, it is not desirable for the execution subject to provide the ability to process and store these data to be stored in the target data lake at the same time. In this case, the execution subject can process multiple data to be stored in sequence by means of data processing queues. That is, the execution subject can store these data to be stored in the data processing queue, and based on the order indicated in the data processing queue, sequentially execute the processing tasks for the data to be stored, and then continuously, stably and orderly process multiple data to be stored, so as to ensure the stability of the system while reducing the configuration requirements of the execution subject for computing resources. For example, even if the execution subject can only execute one data to be stored alone, or can only process other fixed number of data to be stored (the fixed number is less than the number of data to be stored that has been received) at the same time, it can still complete the processing of multiple data to be stored by means of multiple rounds of processing.

[0048] In some embodiments, when the execution entity receives the data to be stored, it can also process only the "incremental" data to be stored through the "duplication checking and deduplication" method, or in other words, only store the incremental data to be stored in the data processing queue to avoid repeatedly executing the same task and causing a waste of computing resources.

[0049] For easier understanding, you can also refer to Figure 3 . Figure 3 A flowchart of a process for constructing a data processing queue provided in an embodiment of the present disclosure, wherein process 300 specifically includes the following steps:

[0050] Step 301: in response to receiving a data providing request sent by a data source, obtaining a set of initial data to be stored provided by the data source;

[0051] Specifically, the execution subject obtains a set of initial data to be stored provided by the data source in response to receiving a data provision request sent by the data source. The data source may be pre-configured and has the authority to store the data to be stored in the target data lake.

[0052] Step 302: Determine a set of difference data to be stored in the data processing queue based on a difference comparison between a set of initial data to be stored and a set of data to be stored in the data processing queue;

[0053] Specifically, after the execution subject obtains a set of initial data to be stored based on the above step 301, it can compare the difference between it and the set of data to be stored in the data processing queue to determine the initial data to be stored that exists in the set of initial data to be stored but does not exist in the set of data to be stored. In short, the execution subject can determine the initial data to be stored as an increment by comparing the difference between a set of initial data to be stored and a set of data to be stored in the data processing queue, and then obtain a set of difference data to be stored. The difference data to be stored in a set of difference data to be stored is actually the initial data to be stored that exists in the set of initial data to be stored and does not exist in the set of data to be stored.

[0054] Step 303: Store a group of difference data to be stored into a data processing queue.

[0055] Specifically, the execution subject may store a set of difference data to be stored in a data processing queue to wait for processing.

[0056] Typically, the execution subject can access a new "data task to be processed" by storing a set of difference data to be stored after the last data to be processed in the data processing queue.

[0057] It should be understood that process 300 can be performed in, for example, Figure 2 The step 201 shown is previously executed to construct a data processing queue for providing and sending data to be stored. Accordingly, the execution subject can provide the data to be stored, or start the above process 200, by controlling the sending timing of the data processing queue.

[0058] As an alternative or in addition, the execution subject may also execute process 300 in parallel while executing process 200 to update the content in the data processing queue, so that process 200 can be executed continuously and in multiple stages to process multiple data to be stored.

[0059] In some optional implementations of this embodiment, in order to avoid misjudgment of difference data to be stored due to elimination of data to be stored in the data processing queue after the execution subject has completed processing of the data to be stored in the data processing queue, the execution subject may also respond in the presence of processed data to be stored, and detect whether there is target difference data to be stored corresponding to the processed data to be stored in a group of difference data to be stored. The processed data to be stored includes historical data to be stored for which corresponding feature vectors have been generated, that is, the processed data to be stored may be data to be stored in the data processing queue that has been processed.

[0060] For easier understanding, you can continue to refer to Figure 3For example, between step 302 and step 303 in process 300, steps 304 to 307 may also be included.

[0061] Step 304: Detect whether there is processed data to be stored;

[0062] Specifically, the execution subject may detect whether the above-mentioned processed data to be stored exists. If the execution subject detects that the processed data to be stored exists, step 305 is executed, otherwise, step 303 is continued.

[0063] Step 305: Detect whether there is target difference data to be stored corresponding to the processed data to be stored in a group of difference data to be stored;

[0064] If the execution subject detects that there is target difference data to be stored corresponding to the processed data to be stored in a group of difference data to be stored, step 306 is executed; otherwise, step 303 is continued.

[0065] Step 306: Delete the target difference data to be stored, and obtain a set of updated difference storage data;

[0066] Step 307: Store the updated set of differential storage data into the data processing queue.

[0067] Therefore, the execution subject can screen again when there is processed data to be stored, avoiding misjudgment of different data to be stored caused by eliminating the data to be stored in the data processing queue after the execution subject has completed the processing of the data to be stored in the data processing queue.

[0068] On the basis of any of the above embodiments, as described above, in some embodiments of the present disclosure, the data to be stored can also be processed and the feature vector of the data to be stored can be generated by introducing a data processing model. For ease of understanding, the data processing model for generating feature vectors can be described as a first data processing model. Typically, the first data processing model can be trained using data in a local database associated with it, and after the training is completed, reasoning is performed using the data in the local database. In some optional implementations of this embodiment, the first data processing model is configured based on a generative language model.

[0069] The general language model (GLM) belongs to a large language model (LLM), which can also be called a large language model. It is an artificial intelligence model that aims to understand and generate human language. In addition, the generative language model can perform processing operations accordingly based on the content it understands to obtain corresponding processing results. For example, when receiving data to be stored of unstructured data types, the generative language model can understand the specific meaning of "converting the data to be stored into a feature vector" after understanding the instructions, and perform the operation of "converting the data to be stored into a feature vector". The characteristic of LLM is that it is large in scale, and it can usually include a large number of parameters to help them learn complex patterns in language data. These models are usually based on deep learning architectures, such as converters, which help them provide better processing performance on various NLP tasks. In an embodiment of the present disclosure, the execution subject can also choose to obtain a generative language model. The generative language model is at least configured to process the data to be stored of unstructured data types into feature vectors. For example, the generative language model can be pre-trained to have the ability to generate feature vectors of data to be stored of unstructured data types. In this case, the execution subject can instruct the generative language model to generate feature vectors of the data to be stored of the unstructured data type based on pre-configured guide words, guide tags, etc. For example, the guide word can be "please generate feature vectors of the data to be stored of the unstructured data type". In some scenarios, in order to improve the generation efficiency and generation quality, auxiliary guide words such as "the feature vector should be in the form of XX" can also be entered at the same time to guide the generative language model more finely and facilitate the generative language model to obtain output content that meets expectations.

[0070] Similarly, for the generative language model, it can be configured by default to omit the "guide word". For example, for the purpose of generating feature vectors for data to be stored of unstructured data types, the generative language model can be based on the default configuration, and of course understand the need to generate feature vectors for data to be stored of unstructured data types. Thus, by means of the default configuration, the generative language model can stably and directionally process the data to be stored of unstructured data types and generate corresponding feature vectors. Thus, the generative language model can be used to generate feature vectors for data to be stored of unstructured data types more efficiently and with higher quality.

[0071] As explained above, since unstructured data cannot be directly used by, for example, data processing models, data analysis models, etc., the data lake cannot directly provide support for data processing models and data analysis models. Based on the method of storing data provided by the present disclosure, feature vectors are used as a substitute for the data to be stored, so that the data lake can directly provide support for data processing models and data analysis models, which not only improves the data support capability of the data lake, but also expands the application scenarios of the data lake. Specifically, the target data lake obtained by using feature vectors to replace the data to be stored of unstructured data types provided in the embodiments of the present disclosure can be directly used as the data source of the online model to provide data support for it.

[0072] For example, the execution entity may provide a second data processing model online to provide services to users. The second data processing model is obtained by adjusting the model data source of the first data processing model from the local data of the first data processing model to the target data lake. That is, the second data processing model can use the target data lake as a data support source to be used by the target device (e.g., terminal devices 101, 102, and 103) used by the user.

[0073] Correspondingly, in some scenarios, the execution entity can also use the target data lake to support other model data provided by the user. For example, when the feature vectors stored in the target data lake can be recognized and used by other models provided by the user, the user can use the data stored in the data lake to support other models that are expected to be used by connecting the target data lake to the other model.

[0074] For easier understanding, please refer to Figure 4 . Figure 4 The present disclosure provides a flowchart of a method for processing data for a large model scenario, wherein process 400 includes:

[0075] Step 401: In response to receiving a call request for calling a second data processing model to process target input data from a target device, acquiring the target input data;

[0076] This step is intended to respond when the execution subject receives a call request from the target device to call the second data processing model to process the target input data, so as to obtain the target input data. For example, the target input data can be the text to be processed, the image to be processed, etc. entered by the user. As described above, the second data processing model can be a data processing model obtained by adjusting the model data source of the first data processing model from the local data of the first data processing model to the target data lake. Accordingly, compared with the first data processing model, the second data processing model can utilize, for example, the above Figure 2The target database obtained in the illustrated embodiment is used as a data source to obtain data support for model reasoning and training.

[0077] Step 402: In response to the data type of the target input data being an unstructured data type, determining whether a target feature vector corresponding to the target input data can be extracted from the target data lake;

[0078] The purpose of this step is to determine whether a target feature vector corresponding to the target input data can be extracted from the target data lake and used as a substitute for the unstructured data type provided by the user when the execution entity determines that the data type of the target input data is an unstructured data type, that is, the target input data cannot be directly used by the second data processing model.

[0079] In practice, the target feature vector corresponding to the target input data can be a feature vector obtained based on the target input data, or it can be the supporting data needed to process the target feature vector into a feature vector (for example, the supporting data can be a feature vector used as a sample, etc.).

[0080] If the execution subject can extract the target feature vector corresponding to the target input data from the target data lake as a substitute for the unstructured data, step 403 is executed.

[0081] It should be understood that, similar to storing content in the target data lake, the execution entity can also provide the target input data directly from the target data lake to the second data processing model for processing when the data type of the target input data is not an unstructured data type, which will not be repeated here.

[0082] Step 403: extracting a target feature vector from the target data lake, and using the target feature vector as the actual input of the second data processing model, and using the second data processing model to generate a target output result corresponding to the target feature vector;

[0083] The purpose of this step is that, based on the above step 402, if the execution entity determines that the target feature vector can be provided in the target data lake, the execution entity can extract the target feature vector from the target data lake and input it into the second data processing model for processing, so as to use the second data processing model to generate a target output result corresponding to the target feature vector.

[0084] Step 404: Provide the target output result to the target device.

[0085] This step aims to, based on the above step 403, have the execution subject provide the output target output result to the target device to complete the data processing flow.

[0086] The method for processing data for large model scenarios provided by the present disclosure can utilize a target data lake storing unstructured data in the form of feature vectors as the data source of a data processing model, thereby simplifying the configuration requirements of the data processing model while improving the processing capabilities of the data processing model.

[0087] In some optional implementations of this embodiment, in response to failure to extract a target feature vector corresponding to the target input data from the target data lake, process 400 may further include step 405. Step 405 may be performed when the execution subject determines that a target feature vector corresponding to the target input data cannot be extracted from the target data lake as a substitute for the unstructured data after executing step 402.

[0088] Step 405: Mark the target feature vector;

[0089] Specifically, the execution subject may mark the required target feature vector. For example, the execution subject may maintain the description information of the target feature vector as a mark.

[0090] Step 406: Monitor the updated feature vectors stored in the target data lake, and detect whether there is a target updated feature vector corresponding to the target feature vector;

[0091] Specifically, based on the above step 405, the execution subject can continuously monitor the content of the update feature vector stored in the target data lake, and detect whether the update feature vector includes or exists the target update feature vector. That is, the execution subject can continuously detect whether the required target update feature vector has been updated and stored in the target data lake.

[0092] Furthermore, if the execution subject detects that the required target update feature vector has been updated and stored in the target data lake, step 407 may be selected for execution.

[0093] Step 407: Send the first prompt information to the target device.

[0094] Specifically, after the execution subject detects that the required target feature vector has been updated and stored in the target data lake (that is, the target feature vector is updated to the target data lake by adding the target update feature vector), the first prompt information can be used to prompt the execution subject that the target data lake has the support capability (that is, the target data lake can be provided). In other words, the first prompt information can indicate that the target device is allowed to use the data processing model to process the target input data. Accordingly, the user of the target device can choose whether to issue a request to use the target feature vector again based on demand. Therefore, such an event notification mechanism can be used to send a notification message after the upstream stores the data to be stored in the target data lake, so that the downstream can consume the notification message regularly and process the incremental data. This method decouples the upstream and downstream and can process incremental data in a streaming manner.

[0095] In some optional implementations of the present embodiment, as described above, when using a data processing queue to provide data to be processed, the execution entity can also send a second prompt message to the target device in response to the absence of target data to be stored associated with the target input data in the data processing queue, and the second prompt message indicates that the target device is not allowed to process the target input data using the second data processing model.

[0096] Specifically, if the data processing queue is used to provide data to be processed, and the target feature vector does not exist in the target data lake, the execution entity can determine whether the "target feature vector" will be added in the future by detecting the data to be processed in the data processing queue. If the execution entity determines that there is a lack of data to be processed in the data processing queue for adding the "target feature vector", it can send a second prompt message to the target device. The second prompt message may indicate that the target device is not allowed to use the second data processing model to process the target input data. As a result, the target data lake refuses the target device to call the second data model indicated by the target data lake when it lacks support capabilities, which can avoid processing errors while providing feedback to the user and preventing the user from incorrectly waiting for a response.

[0097] In addition, in some optional implementations of the present embodiment, when using a data processing queue to provide data to be stored, if the execution subject determines that there is target data to be stored associated with the target input data in the data processing queue, the execution subject may also choose to adjust the data output priority of the target data to be stored in the data processing queue to the target priority. For example, based on a pre-set processing priority rule, priorities 1, 2, and 3 are configured based on the priority processing order, and the processing order may be: priority 1>priority 2>priority 3. The execution subject may determine priority 1 as the target priority to adjust the processing priority of the target data to be stored so that the target data to be stored can be processed as quickly as possible. As a result, the data to be stored with the target feature vector can be processed preferentially to reduce the waiting time on the user side.

[0098] In some embodiments, the execution subject may also choose to directly adjust the sequence of the target data to be stored in the data processing queue to achieve a similar purpose. For example, the execution subject may adjust the processing order of the target data to be stored to the Xth position to speed up the processing. Exemplarily, the execution subject may adjust the position of the target data to be stored in the sequence of the data processing queue to the "next position" to be processed immediately after the current processing is completed, so that after the processing of the current data to be processed is completed, the target data to be stored can be processed immediately to reduce the waiting time on the user side.

[0099] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario, see Figure 5 The process 500 shown in FIG. 500 may include two processes of storing data and processing data for large model scenarios as described above. For example, the server 105 may be used as the execution subject of storing data and processing data, and the terminal device 103 may provide the server 105 with data to be stored (e.g., data to be stored 510) for storage in the data lake (e.g., target data lake 530). The terminal device 101 may call the generative language model 525 to process the target input data (e.g., target input data 540).

[0100] The following will explain in detail, the specific process is as follows:

[0101] In response to receiving the to-be-stored data 510 to be stored in the target data lake 530 from the terminal device 103 , the server 105 may detect the data type of the to-be-stored data 530 .

[0102] In response to determining that the data type of the data 510 to be stored is an unstructured data type, the server 105 processes it using a generative language model 520 to generate a feature vector 515 of the data 510 to be stored.

[0103] Furthermore, the server 105 may store the feature vector 515 in the target data lake 530 as a replacement for the data to be stored 510 .

[0104] Next, when the terminal device 101 desires to call the generative language model 525 to process the target input data 540, it may send a call request including a call request 545 of the target input data 540 to the server 105. The generative language model 525 is obtained by adjusting the model data source of the generative language model 520 from the local data of the generative language model 520 to the target data lake 530.

[0105] In response to receiving a call request 545 for calling the generative language model 525 to process the target input data 540 from the terminal device 101 , the server 105 obtains the target input data 540 .

[0106] Next, in response to the data type of the target input data 540 being an unstructured data type, the server 105 determines whether a target feature vector 550 corresponding to the target input data 540 can be extracted from the target data lake 530 .

[0107] For ease of understanding, illustratively in process 500 , the server 105 can extract a target feature vector 550 corresponding to the target input data 540 from the target data lake 530 .

[0108] Next, the server 105 may use the target feature vector 550 as the actual input of the generative language model 525 , and use the generative language model 525 to generate a target output result 560 corresponding to the target feature vector 550 .

[0109] Finally, the server 105 can provide the target output result 560 to the terminal device 101, so as to utilize the target data lake 530 as the model data source of the generative language model 525, and provide the user (or the terminal device 101) with a data processing service based on the generative language model 525.

[0110] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for storing data for a large model scene. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0111] like Figure 6As shown, the device 600 for storing data for a large model scenario of this embodiment may include: a data type detection unit 601, a feature vector conversion unit 602, and a feature vector storage unit 603. Among them, the data type detection unit 601 is configured to detect the data type of the data to be stored in response to receiving the data to be stored in the target data lake; the feature vector conversion unit 602 is configured to generate a feature vector of the data to be stored in response to the data type being an unstructured data type; the feature vector storage unit 603 is configured to store the feature vector as a substitute for the data to be stored in the target data lake.

[0112] In the device 600 for storing data in a large model scene, the specific processing of the data type detection unit 601, the feature vector conversion unit 602 and the feature vector storage unit 603 and the technical effects thereof can be referred to in detail. Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0113] In some optional implementations of this embodiment, in response to continuously receiving multiple data to be stored, the multiple data to be stored are processed in sequence based on a data processing queue, and the device 600 also includes: an initial data acquisition unit, configured to acquire a group of initial data to be stored provided by the data source in response to receiving a data provision request issued by the data source; a data difference comparison unit, configured to determine a group of difference data to be stored to be stored in the data processing queue based on a difference comparison between a group of initial data to be stored and a group of data to be stored, wherein the difference data to be stored in a group of difference data to be stored includes the initial data to be stored that exists in a group of initial data to be stored and does not exist in a group of data to be stored; a difference data storage unit, configured to store a group of difference data to be stored in the data processing queue.

[0114] In some optional implementations of the present embodiment, the device 600 also includes: a duplicate data detection unit, configured to detect, in response to the existence of processed data to be stored, whether there is target difference data to be stored corresponding to the processed data to be stored in a group of difference data to be stored, wherein the processed data to be stored includes historical data to be stored for which a corresponding feature vector has been generated; a difference data updating unit, configured to, in response to the existence of target difference data to be stored in a group of difference data to be stored, delete the target difference data to be stored to obtain an updated group of difference storage data; and a difference data storage unit, further configured to store the updated group of difference storage data into the data processing queue.

[0115] In some optional implementations of this embodiment, the feature vector conversion unit 602 is further configured to, in response to the data type being an unstructured data type, process the data to be stored using the first data processing model to generate a feature vector of the data to be stored.

[0116] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for processing data. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0117] like Figure 7 As shown, the apparatus 700 for processing data for a large model scenario of this embodiment may include: a call request receiving unit 701, a target feature vector detecting unit 702, a target feature vector extracting unit 703 and a target result outputting unit 704. The call request receiving unit 701 is configured to obtain the target input data in response to receiving a call request for calling a second data processing model to process target input data from a target device, wherein the second data processing model is obtained by adjusting the model data source of the first data processing model from the local data of the first data processing model to the target data lake, and the target data lake is based on Figure 6 The device 600 for storing data shown is obtained; a target feature vector detection unit 702 is configured to determine whether a target feature vector corresponding to the target input data can be extracted from the target data lake in response to the data type of the target input data being an unstructured data type; a target feature vector extraction unit 703 is configured to extract a target feature vector from the target data lake in response to the target feature vector corresponding to the target input data being able to be extracted from the target data lake, and use the target feature vector as the actual input of the second data processing model, and use the second data processing model to generate a target output result corresponding to the target feature vector; a target result output unit 704 is configured to provide the target output result to the target device.

[0118] In this embodiment, in the device 700 for processing data for a large model scene, the specific processing of the call request receiving unit 701, the target feature vector detection unit 702, the target feature vector extraction unit 703 and the target result output unit 704 and the technical effects thereof can be referred to respectively. Figure 4 The relevant descriptions of steps 401-404 in the corresponding embodiment are not repeated here.

[0119] In some optional implementations of this embodiment, the device 700 also includes: a call request receiving unit, configured to obtain the target input data in response to receiving a call request from the target device to call the second data processing model to process the target input data, wherein the second data processing model is obtained by adjusting the model data source of the first data processing model from the local data of the first data processing model to the target data lake; a target feature vector detection unit, configured to determine whether a target feature vector corresponding to the target input data can be extracted from the target data lake in response to the data type of the target input data being an unstructured data type; a target feature vector extraction unit, configured to extract the target feature vector from the target data lake in response to the target feature vector corresponding to the target input data being able to be extracted from the target data lake, and use the target feature vector as the actual input of the second data processing model, and use the second data processing model to generate a target output result corresponding to the target feature vector; a target result output unit, configured to provide the target output result to the target device.

[0120] In some optional implementations of the present embodiment, in response to failing to extract a target feature vector corresponding to the target input data from the target data lake, the device 700 further includes: a target feature vector marking unit, configured to mark the target feature vector; a data lake update monitoring unit, configured to monitor the update feature vectors stored in the target data lake, and detect whether there is a target update feature vector corresponding to the target feature vector; and a first prompt information sending unit, configured to send a first prompt information to the target device in response to the presence of a target update feature vector corresponding to the target feature vector, the first prompt information being used to indicate that the target device is allowed to process the target input data using the second data processing model.

[0121] In some optional implementations of this embodiment, in response to continuously receiving multiple data to be stored, the multiple data to be stored are processed sequentially based on a data processing queue, and the device 700 also includes: a queue processing order adjustment unit, configured to adjust the data output priority of the target data to be stored in the data processing queue to the target priority in response to the presence of target data to be stored associated with the target input data in the data processing queue.

[0122] In some optional implementations of this embodiment, the device 700 also includes: a second prompt information sending unit, configured to send a second prompt information to the target device in response to the absence of target to-be-stored data associated with the target input data in the data processing queue, the second prompt information indicating that the target device is not allowed to process the target input data using the second data processing model.

[0123] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The apparatus for processing data for large model scenarios provided by this embodiment can utilize a target data lake storing unstructured data in the form of feature vectors as the data source of a data processing model, and can simplify the configuration requirements of the data processing model while improving the processing capabilities of the data processing model.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0125] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0126] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0127] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0128] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a method for storing data and / or a method for processing data for a large model scene. For example, in some embodiments, a method for storing data and / or a method for processing data for a large model scene may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method for storing data and / or the method for processing data for the large model scene described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured in any other appropriate manner (for example, by means of firmware) to execute a method for storing data and / or a method for processing data for a large model scenario.

[0129] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0131] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0134] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services. The server can also be divided into a server of a distributed system, or a server combined with a blockchain.

[0135] According to the technical solution of the embodiment of the present disclosure, when storing data in the data lake, the unstructured data is converted into feature vectors for storage, so that the data lake can use the form of feature vectors to provide "unstructured data" that can be used directly to support model reasoning and training. This approach can enhance the data support capabilities of the data lake and expand the application scenarios of the data lake.

[0136] Furthermore, based on the above-mentioned data storage method for large model scenarios, the method for processing data for large model scenarios provided by the present disclosure can utilize the target data lake that stores unstructured data in the form of feature vectors as the data source of the data processing model, which can simplify the configuration requirements of the data processing model while improving the processing capabilities of the data processing model.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data storage method for a large model scenario, comprising: In response to receiving the data to be stored in the target data lake, detecting a data type of the data to be stored; In response to the data type being an unstructured data type, generating a feature vector of the data to be stored; Using the feature vector as a substitute for the data to be stored, and storing it in the target data lake; In response to continuously receiving a plurality of data to be stored, the plurality of data to be stored are processed sequentially based on a data processing queue, further comprising: In response to receiving a data provision request issued by a data source, obtaining a group of initial data to be stored provided by the data source; based on a difference comparison between a group of initial data to be stored and a group of data to be stored in the data processing queue, determining a group of difference data to be stored to be stored in the data processing queue, wherein the difference data to be stored in the group of difference data to be stored includes the initial data to be stored that exists in the group of initial data to be stored and does not exist in the group of data to be stored; in response to the existence of processed data to be stored, detecting whether there is target difference data to be stored corresponding to the processed data to be stored in the group of difference data to be stored, wherein the processed data to be stored includes historical data to be stored for which corresponding feature vectors have been generated; in response to the existence of the target difference data to be stored in the group of difference data to be stored, deleting the target difference data to be stored to obtain an updated group of difference stored data; storing the updated group of difference stored data in the data processing queue.

2. The method according to claim 1, wherein: Generating a feature vector of the data to be stored includes: The data to be stored is processed using a first data processing model to generate a feature vector of the data to be stored.

3. The method according to claim 2, wherein: The first data processing model is configured based on a generative language model.

4. A method for processing data for a large model scenario, comprising: In response to receiving a call request from a target device to call a second data processing model to process target input data, the target input data is obtained, wherein the second data processing model is obtained by adjusting a model data source of the first data processing model from local data of the first data processing model to a target data lake, and the target data lake is obtained based on the data storage method for large model scenarios according to any one of claims 1 to 3; In response to the data type of the target input data being an unstructured data type, determining whether a target feature vector corresponding to the target input data can be extracted from the target data lake; In response to being able to extract a target feature vector corresponding to the target input data from the target data lake, extracting the target feature vector from the target data lake, taking the target feature vector as an actual input of the second data processing model, and using the second data processing model to generate a target output result corresponding to the target feature vector; The target output result is provided to the target device.

5. The method according to claim 4, wherein: In response to failure to extract a target feature vector corresponding to the target input data from the target data lake, the method further includes: marking the target feature vector; Monitoring the updated feature vector stored in the target data lake, and detecting whether there is a target updated feature vector corresponding to the target feature vector; In response to the existence of a target update feature vector corresponding to the target feature vector, first prompt information is sent to the target device, where the first prompt information is used to indicate that the target device is allowed to process the target input data using the second data processing model.

6. The method according to claim 5, wherein: In response to continuously receiving a plurality of data to be stored, the plurality of data to be stored are processed sequentially based on a data processing queue, further comprising: In response to the existence of target to-be-stored data associated with the target input data in the data processing queue, a data output priority of the target to-be-stored data in the data processing queue is adjusted to a target priority.

7. The method according to claim 6, further comprising: In response to the absence of target to-be-stored data associated with the target input data in the data processing queue, second prompt information is sent to the target device, wherein the second prompt information indicates that the target device is not allowed to process the target input data using the second data processing model.

8. A device for storing data for a large model scene, comprising: a data type detection unit, configured to detect a data type of the data to be stored in response to receiving the data to be stored in the target data lake; a feature vector conversion unit, configured to generate a feature vector of the data to be stored in response to the data type being an unstructured data type; a feature vector storage unit, configured to store the feature vector as a substitute for the data to be stored in the target data lake; In response to continuously receiving a plurality of data to be stored, the plurality of data to be stored are processed sequentially based on a data processing queue, further comprising: An initial data acquisition unit is configured to, in response to receiving a data provision request issued by a data source, acquire a group of initial data to be stored provided by the data source; a data difference comparison unit is configured to determine a group of difference data to be stored to be stored in the data processing queue based on a difference comparison between a group of initial data to be stored and the group of data to be stored, wherein the difference data to be stored in the group of difference data to be stored includes the initial data to be stored that exists in the group of initial data to be stored and does not exist in the group of data to be stored; a duplicate data detection unit is configured to, in response to the existence of processed data to be stored, detect whether there is target difference data to be stored corresponding to the processed data to be stored in the group of difference data to be stored, wherein the processed data to be stored includes historical data to be stored for which corresponding feature vectors have been generated; a difference data update unit is configured to, in response to the existence of the target difference data to be stored in the group of difference data to be stored, delete the target difference data to be stored to obtain an updated group of difference storage data; a difference data storage unit is configured to store the updated group of difference storage data in the data processing queue.

9. The apparatus according to claim 8, wherein the feature vector conversion unit is further configured to, in response to the data type being an unstructured data type, process the data to be stored using a first data processing model to generate a feature vector of the data to be stored.

10. The device according to claim 9, wherein: The first data processing model is configured based on a generative language model.

11. A device for processing data for a large model scene, comprising: A call request receiving unit, configured to obtain the target input data in response to receiving a call request for calling a second data processing model to process target input data from a target device, wherein the second data processing model is obtained by adjusting a model data source of the first data processing model from local data of the first data processing model to a target data lake, and the target data lake is obtained based on the device for storing data according to any one of claims 8 to 10; a target feature vector detection unit, configured to determine, in response to the data type of the target input data being an unstructured data type, whether a target feature vector corresponding to the target input data can be extracted from the target data lake; a target feature vector extraction unit, configured to extract the target feature vector from the target data lake in response to being able to extract the target feature vector corresponding to the target input data from the target data lake, and use the target feature vector as the actual input of the second data processing model, and generate a target output result corresponding to the target feature vector by using the second data processing model; The target result output unit is configured to provide the target output result to the target device.

12. The device according to claim 11, wherein In response to failure to extract a target feature vector corresponding to the target input data from the target data lake, the method further includes: a target feature vector marking unit, configured to mark the target feature vector; a data lake update monitoring unit, configured to monitor the update feature vector stored in the target data lake, and detect whether there is a target update feature vector corresponding to the target feature vector; The first prompt information sending unit is configured to send first prompt information to the target device in response to the existence of a target update feature vector corresponding to the target feature vector, wherein the first prompt information is used to indicate that the target device is allowed to process the target input data using the second data processing model.

13. The device according to claim 12, wherein: In response to continuously receiving a plurality of data to be stored, the plurality of data to be stored are processed sequentially based on a data processing queue, further comprising: The queue processing order adjustment unit is configured to adjust the data output priority of the target to-be-stored data in the data processing queue to a target priority in response to the existence of target to-be-stored data associated with the target input data in the data processing queue.

14. The apparatus according to claim 13, further comprising: A second prompt information sending unit is configured to send a second prompt information to the target device in response to the absence of target to-be-stored data associated with the target input data in the data processing queue, wherein the second prompt information indicates that the target device is not allowed to process the target input data using the second data processing model.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data storage method for large model scenarios described in any one of claims 1-3 and / or the data processing method for large model scenarios described in any one of claims 4-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the data storage method for large model scenarios described in any one of claims 1 to 3 and / or the data processing method for large model scenarios described in any one of claims 4 to 7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the data storage method for large model scenarios according to any one of claims 1 to 3 and / or the data processing method for large model scenarios according to any one of claims 4 to 7.

Images