Data acquisition method and computing device

Abstract

The invention discloses a data acquisition method and computing equipment, relates to the technical field of data processing, and can improve the efficiency of acquiring shared data convenient to use. The method comprises the following steps: receiving a data request message, wherein the data request message is used for indicating to obtain first shared data in a target format; afterwards, first shared data can be obtained from a database, and the database comprises shared data of multiple formats and data formats of the shared data. Then, it can be determined whether the data format of the first shared data is a target format, and if the data format of the first shared data is not the target format, the data format of the first shared data is converted into the target format. The first shared data in the target format may then be transmitted.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data acquisition method and computing device. Background Technology

[0002] In the development of artificial intelligence technology, various algorithm frameworks have emerged, such as the widely used TensorFlow, PyTorch, and MXNet. Developers can build network models based on these frameworks to meet the needs of tasks such as image recognition, natural language processing, and speech recognition. Furthermore, developers can train models using shared data from data sharing platforms to ensure the accuracy of the model's output.

[0003] However, currently, after obtaining shared data from data sharing platforms, the data still needs to be processed, making the process of using shared data rather cumbersome. Therefore, how to improve the efficiency of obtaining easily usable shared data has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a data acquisition method and computing device that can improve the efficiency of acquiring easily accessible shared data.

[0005] Firstly, this application provides a data acquisition method applicable to a computing device. In this method, a data request message is received, indicating the acquisition of first shared data in a target format. Then, the first shared data is acquired from a database, which includes shared data in various formats and their shared formats. Next, it is determined whether the data format of the first shared data is the target format; if the data format of the first shared data is not the target format, the data format of the first shared data is converted to the target format. Finally, the first shared data in the target format is sent.

[0006] Based on the above technical solution, after receiving a data request message for obtaining first shared data in the target format, the first shared data can be retrieved from the database. Furthermore, if the data format of the first shared data is not the target format, the data format of the first shared data is converted to the target format and then sent. In this way, developers do not need to adjust the data format after obtaining the shared data, reducing the complexity of data acquisition and processing, thereby improving the efficiency of obtaining easily usable shared data. Moreover, compared to manual data format conversion, having the computing device convert the data format of the shared data reduces the probability of problems occurring during the data conversion process, thereby improving the quality of data conversion.

[0007] In conjunction with the first aspect, in one possible implementation, the shared data includes training data used to train the model. The data request message includes target format information, which characterizes the target algorithm framework and / or the target format, where the target format is a data format supported by the target algorithm framework; the first shared data is used to train the model of the target algorithm framework.

[0008] In conjunction with the first aspect, in another possible implementation, the data request message further includes: conversion requirement information, which indicates the data quality after format conversion and / or the efficiency of the format conversion. If the data format of the first shared data is not the target format, the data format of the first shared data is converted to the target format, including: obtaining the initial format of the first shared data; determining multiple first conversion algorithms, which are used to convert the data from the initial format to the target format; determining a second conversion algorithm from the multiple first conversion algorithms that matches the conversion requirement information; and performing format conversion on the first shared data based on the second conversion algorithm to obtain the first shared data in the target format.

[0009] In conjunction with the first aspect, in another possible implementation, the multiple first transformation algorithms include: a first type of algorithm and a second type of algorithm. The data quality obtained based on the first type of algorithm is higher than the data quality obtained based on the second type of algorithm, and the data transformation efficiency of the second type of algorithm is higher than that of the first type of algorithm.

[0010] In conjunction with the first aspect, in another possible implementation, a second conversion algorithm matching the conversion requirement information is determined from a plurality of first conversion algorithms. This includes: if the conversion requirement information indicates that the data quality after format conversion is greater than a preset quality threshold, then one of the first type of algorithms is selected as the second conversion algorithm. If the conversion requirement information indicates that the efficiency of the format conversion is greater than a preset efficiency threshold, then one of the second type of algorithms is selected as the second conversion algorithm.

[0011] In conjunction with the first aspect, in another possible implementation, if the conversion requirement information indicates that the data quality after format conversion is greater than a preset quality threshold, then one of the algorithms of the first type is used as the second conversion algorithm, including: if the data quality of the first shared data after format conversion indicates that the conversion requirement information indicates that the data quality is greater than the preset quality threshold, then the quality information of the first shared data and / or the available resources of the training device are obtained, and the training device is used to train a model based on the first shared data. If the quality information of the first shared data and / or the available resources of the electronic device meet the high-quality conversion conditions, then the algorithm of the first type is used as the second conversion algorithm; the high-quality conversion conditions include at least one of the following: the quality information of the first shared data is greater than the preset quality threshold, and the available resources of the training device are greater than the resources required by the first shared data.

[0012] In conjunction with the first aspect, in another possible implementation, the method further includes: receiving a query request message, the query request message including query keywords; the query keywords are used to represent the data content and / or data type of the data to be queried. In response to the query request message, based on the query keywords and the metadata of each shared data in the database, determining first shared data whose metadata matches the query keywords. Sending a query result, the query result including at least the data identifier of the first shared data. A data request message including the data identifier of the first shared data.

[0013] In conjunction with the first aspect, in another possible implementation, the method further includes: receiving the uploaded second shared data; and storing the second shared data and its metadata in a database.

[0014] In conjunction with the first aspect, in another possible implementation, the metadata includes at least one of the following: data sample, quality information, data size, data format, data type, and data description information, wherein the quality information is used to indicate the data quality of the shared data.

[0015] In a second aspect, this application provides a computing device comprising: a memory and one or more processors, wherein the memory is coupled to the processors; the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the one or more processors, the computing device causes the computing device to perform the method as described in the first aspect or the second aspect and any possible design thereof.

[0016] Thirdly, this application provides a computer-readable storage medium including computer instructions that, when executed on a computing device, cause the computing device to perform the method as described in the first aspect or the second aspect and any possible design thereof.

[0017] Fourthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect or the second aspect and any possible design thereof.

[0018] It is understood that the beneficial effects achieved by the computing device described in the second aspect and any possible design thereof, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect can be referred to as the beneficial effects in the first aspect and any possible design thereof, which will not be repeated here. Attached Figure Description

[0019] Figure 1 A schematic diagram illustrating the composition of a data acquisition system architecture provided in an embodiment of this application;

[0020] Figure 2 A flowchart of a data acquisition method provided in an embodiment of this application;

[0021] Figure 3 A flowchart illustrating another data acquisition method provided in this application embodiment;

[0022] Figure 4 A flowchart illustrating another data acquisition method provided in this application embodiment;

[0023] Figure 5 This is a schematic diagram illustrating the composition of a data acquisition device provided in an embodiment of this application;

[0024] Figure 6 A schematic diagram illustrating the composition of another data acquisition device provided in this application embodiment;

[0025] Figure 7 This is a schematic diagram of the structural composition of a chip system provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In this application, the character " / " generally indicates that the objects before and after it are in an "or" relationship. For example, A / B can be understood as A or B.

[0028] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0029] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include other steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or device.

[0030] Furthermore, in the embodiments of this application, the words "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design options. Specifically, the use of the words "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0031] To facilitate understanding of the technical solutions of this application, before providing a detailed description of the data acquisition method in the embodiments of this application, the technical terms mentioned in the embodiments of this application will be introduced first.

[0032] 1. Algorithm Framework: An algorithm framework is a software library used to build, train, and deploy artificial intelligence models. For example, artificial intelligence (AI) algorithm frameworks can include TensorFlow, PyTorch, Caffe, etc.

[0033] 2. Data Format: Data format refers to the encoding method used to store, process, and transmit data. Different AI frameworks may require different data formats. For example, TensorFlow's data formats can be TFRecord, NumPy arrays, or Python lists. PyTorch's data formats can be Tensor or NumPy arrays. Caffe's data formats can be LevelDB or LMDB.

[0034] 3. Meta-information: Meta-information is used to describe some basic characteristics of data, such as size, type, quality indicators, etc.

[0035] 4. Application Programming Interface (API): An API is a set of predefined rules and specifications that allow different software applications to interact by calling APIs.

[0036] 5. Dependency packages: Dependency packages are a set of code libraries or modules required to implement certain functions.

[0037] This application provides a data acquisition method. When building a network model using an algorithm framework, developers can obtain shared data from a data sharing platform to train the model. Considering that models built using different algorithm frameworks require input data in different formats, this method can receive a data request message for obtaining shared data in the target format. Then, stored shared data can be retrieved from a database. Next, the shared data can be converted to the target format and sent. In this way, by pre-setting the required data format, the data sharing platform can directly convert the data to the required format and return the shared data in the target format. This eliminates the need for developers to adjust the data format after acquiring the shared data, reducing the complexity of data acquisition and improving the efficiency of obtaining easily usable shared data. Furthermore, compared to manual data format conversion, having the data sharing platform convert the shared data format reduces the probability of problems during the conversion process, thereby improving the quality of data conversion.

[0038] The implementation environment of the embodiments of this application is described below.

[0039] like Figure 1 As shown, a data acquisition system provided in an embodiment of this application is provided. The system includes: server 101, training device 102, terminal device 103, and terminal device 104.

[0040] The server 101 stores multiple shared data sets, including shared data in different formats. The server 101 can receive data request messages from the training device 102, which request the acquisition of shared data in a target format. The server 101 can then send the shared data in the target format to the training device 102. Furthermore, the server 101 can convert the data format.

[0041] In some embodiments, server 101 may also be used to receive query request messages from training device 102 and to provide query results to training device 102, the query results of which may include shared data and / or data identifiers of shared data.

[0042] In some embodiments, server 101 can also be used to receive shared data uploaded by terminal devices 103 and 104, and store the shared data and its metadata, which describes the data attributes of the shared data. Server 101 is equipped with a database. This database can be used to store the shared data and metadata uploaded by each terminal device.

[0043] In this embodiment of the application, server 101 may include at least one computing device. Different computing devices within the at least one computing device may communicate with each other.

[0044] It should be noted that server 101 can be a single physical or logical server, or it can consist of two or more physical or logical servers that share different responsibilities and work together to implement the various functions of the server.

[0045] For example, server 101 may be a blade server, high-density server, rack server, or tower server, etc.

[0046] In this embodiment of the application, the training device 102 can query shared data from the server 101 and obtain shared data in the target format from the server 101.

[0047] In some embodiments, the training device 102 deploys a model built based on a target algorithm framework, where the target format is a data format supported by the target algorithm framework. The training device 102 can train the model based on shared data in the target format.

[0048] Optionally, the training device 102 may include at least one computing device. Different computing devices within the at least one computing device may communicate with each other.

[0049] Optionally, the training device 102 can be a terminal device, such as a mobile phone, tablet computer, handheld computer, personal computer (PC), cellular phone, personal digital assistant (PDA), wearable device (such as smartwatch, smart bracelet, etc.), smart home device (such as TV, etc.), in-vehicle system (such as in-vehicle computer, etc.), smart screen, game console, headphones, AI speaker, augmented reality (AR) / virtual reality (VR) device, ultra-mobile personal computer (UMPC), laptop computer, netbook, desktop computer or all-in-one computer, etc.

[0050] It should be noted that the embodiments of this application do not limit the device form of the training device 102; the above is merely an illustrative example.

[0051] After introducing the application scenarios and system architecture of the embodiments of this application, the methods of the embodiments of this application will be described below. The methods in the following embodiments can all be implemented in servers and training devices with the above-described system architecture.

[0052] The data acquisition method provided in this application can be implemented by a data acquisition device, which can be... Figure 1 The data acquisition device can be the central processing unit (CPU) of the server (or training device), or the data acquisition module within the server. Alternatively, the data acquisition device can be a data sharing platform deployed on the server. This application embodiment uses a server executing a data acquisition method as an example to illustrate the data acquisition method provided in this application embodiment.

[0053] This application provides a data acquisition method, such as... Figure 2 As shown, the data acquisition method may include:

[0054] S201, Receive data request message.

[0055] The data request message indicates the need to retrieve first shared data in a target format, which is used to train the model of the target algorithm framework. The target format is a data format supported by the target algorithm framework.

[0056] It should be noted that the embodiments of this application do not limit the target algorithm framework and target format. For example, when the target algorithm framework is TensorFlow, the target format can be TFRecord format, NumPy array, or Python list. When the target algorithm framework is PyTorch, the target format can be Tensor format or NumPy array. When the target algorithm framework is Caffe, the target format can be LevelDB format or LMDB format.

[0057] In one possible implementation, a data request message from a training device can be received. The training device is used to train a model based on the first shared data.

[0058] In one possible implementation, the data request message may include target format information, which characterizes the target algorithm framework and / or the target format. The target format information may be a framework identifier for the target algorithm framework, or it may be a format identifier for the target format. Alternatively, the target format information may include both a framework identifier for the target algorithm framework and a format identifier for the target format.

[0059] It should be understood that this application does not limit the representation of the framework identifier. For example, the framework identifier can be the name of an algorithm framework. Alternatively, the framework identifier can be represented by letters (such as a, b, c, etc.), allowing the server to find the corresponding algorithm framework using the letters. Furthermore, this application does not limit the representation of the format identifier. For example, the format identifier can be the name of a data format. Alternatively, the format identifier can be represented by numbers (such as 1, 2, 3, etc.), allowing the server to find the corresponding data format using the numbers.

[0060] In one possible implementation, the training device can send a data request message to the server. The server can then receive the data request message sent by the training device.

[0061] For example, developers can add dependency packages and data acquisition interfaces provided by a server (or data sharing platform) to the model training code (such as Python code). These dependency packages include code that enables the acquisition of shared data. Then, when the training device runs the model training code, it can call the dependency packages and data acquisition interfaces provided by the server (or data sharing platform), allowing the server to receive data request messages sent by the training device.

[0062] Furthermore, target format information can be passed as parameters to the data acquisition interface. For example, "PyTorch" (i.e., the frame identifier) ​​can be passed as a parameter to the data acquisition interface. Another example is passing "Tensor" (i.e., the format identifier) ​​as a parameter to the data acquisition interface. Yet another example is passing both "PyTorch" and "Tensor" as parameters to the data acquisition interface.

[0063] Understandably, the data request message includes target format information. When the target format information is used to characterize the target algorithm framework, the server can determine the data formats supported by that algorithm framework, thereby ensuring that the model can be trained based on the shared data. Similarly, when the target format information is used to characterize the target format, the server can determine the shared data in the target format to ensure that the data sent to the training device is in a format supported by the target algorithm framework, thus ensuring that the model can be trained based on the shared data.

[0064] S202. Retrieve the first shared data from the database.

[0065] In this embodiment, the database includes shared data in various formats and the data format of the shared data. The shared data in the database can be shared data uploaded by multiple terminal devices. The shared data may include training data used to train a model.

[0066] It should be understood that, in this embodiment, the server can receive and store shared data uploaded by any terminal device. Thus, when other training devices need to train a model, they can obtain the shared data from the server to complete the model training.

[0067] It should be noted that this application embodiment does not limit the data type of the shared data in the database. For example, the shared data can be images, audio, video, text, etc. Furthermore, this application embodiment does not limit the data format of the shared data in the database. For example, the data format can be TFRecord format, NumPy array, Python list, Tensor format, etc. In other words, in this application embodiment, the server can store data of any data type and any data format.

[0068] In one possible implementation, the data request message includes a data identifier for the first shared data. The server can then retrieve the first shared data from the database based on this data identifier.

[0069] It should be noted that the data identifier is not limited in the embodiments of this application. For example, the data identifier can be the storage address of shared data on the server. Another example is that the data identifier can be a Uniform Resource Locator (URL). Yet another example is that the data identifier can be a number, such as "1", "2", "3", etc.

[0070] S203. Determine whether the data format of the first shared data is the target format.

[0071] In one possible implementation, the server stores an initial format of the shared data, which is the data format in which the shared data is stored in the database. The server can obtain the initial format of the first shared data. Then, the server can compare the initial format of the first shared data with a target format to determine whether the initial format of the first shared data is the target format.

[0072] In some embodiments, if the data format of the first shared data is the target format, the server can execute S205. If the data format of the first shared data is not the target format, the server can execute S204-S205.

[0073] S204. Convert the data format of the first shared data into the target format.

[0074] In this embodiment of the application, if the data format of the first shared data is not the target format, the server can convert the data format of the first shared data into the target format.

[0075] In one possible implementation, the server can determine the target format based on the target format information. Then, based on the target format and the initial format of the first shared data, the server determines multiple first conversion algorithms, which are used to convert the data from the initial format to the target format. Afterward, the server can convert the data format of the first shared data to the target format based on any of the first conversion algorithms.

[0076] In one possible implementation, the server can determine the data format supported by the target algorithm framework based on the framework identifier of the target algorithm framework, and use the data format supported by the target algorithm framework as the target format.

[0077] S205, Send the first shared data in the target format.

[0078] In one possible implementation, first shared data in the target format can be sent to the training device.

[0079] Based on the above technical solution, after receiving a data request message from the training device requesting first shared data in the target format, the first shared data can be retrieved from the database. Furthermore, if the format of the first shared data is not the target format, the data format of the first shared data is converted to the target format and sent to the training device. This eliminates the need for developers to adjust the data format after retrieving the shared data, reducing the complexity of data acquisition and thus improving the efficiency of model training. Moreover, compared to manual data format conversion, having the data sharing platform convert the shared data format reduces the probability of problems occurring during the conversion process, thereby improving the quality of data conversion.

[0080] In some embodiments, the training device may also send a data request message to the server, which further includes: conversion requirement information, indicating the data quality after format conversion and / or the efficiency of the format conversion. The server can convert the shared data based on the conversion requirement information.

[0081] It should be understood that data quality refers to the degree of accuracy, completeness, and other aspects of data performance. For example, the quality of an image can be determined by its resolution, richness of detail, and completeness. The quality of text can be determined by its source, timeliness, and other factors.

[0082] This application provides a data acquisition method, such as... Figure 3 As shown, in this data acquisition method, S204 may include:

[0083] S301. Obtain the initial format of the first shared data.

[0084] S302, Determine multiple first conversion algorithms.

[0085] The first conversion algorithm is used to convert the data from the initial format to the target format.

[0086] It should be noted that the specific process of determining multiple first conversion algorithms can refer to the method of determining the first conversion algorithm in S204 of the above embodiment, which will not be repeated here.

[0087] S303. Determine a second conversion algorithm that matches the conversion requirement information from multiple first conversion algorithms.

[0088] In one possible implementation, the server stores multiple sets of conversion algorithms, each set corresponding to an initial format and a target format. Each set of conversion algorithms includes multiple conversion algorithms, each corresponding to different conversion requirement information. The server can select a second conversion algorithm from among the multiple first conversion algorithms based on the conversion requirement information.

[0089] It should be noted that the embodiments of this application do not limit the conversion requirement information. For example, when the conversion requirement information is used to represent the data quality after format conversion, the conversion requirement information can be a specific quality value (such as 1, 5, 8), a quality level (such as quality level 1, quality level 2, quality level 3), or a quality requirement (such as high quality, low quality). When the conversion requirement information is used to represent the efficiency of the format conversion, the conversion requirement information can be a specific efficiency value (such as 1, 5, 8), a quality level (such as efficiency level 1, efficiency level 2, efficiency level 3), or an efficiency requirement (such as high efficiency, low efficiency).

[0090] For example, suppose multiple first conversion algorithms include: conversion algorithm a, conversion algorithm b, and conversion algorithm c. Conversion algorithm a has a quality value of 1, conversion algorithm b has a quality value of 2, and conversion algorithm c has an efficiency value of 1. If the conversion requirement information is "quality value 1", then conversion algorithm a can be used as the second conversion algorithm.

[0091] In another possible implementation, the multiple first transformation algorithms include: a first type of algorithm and a second type of algorithm. The data quality obtained based on the first type of algorithm is higher than that obtained based on the second type of algorithm, and the data transformation efficiency of the second type of algorithm is higher than that of the first type of algorithm.

[0092] If the conversion request indicates that the quality of the converted data is greater than a preset quality threshold, the server will use one of the algorithms from the first category as the second conversion algorithm. If the conversion request indicates that the efficiency of the conversion is greater than a preset efficiency threshold, the server will use one of the algorithms from the second category as the second conversion algorithm.

[0093] In other words, the first type of algorithm produces data of higher quality, while the second type of algorithm is more efficient at converting data.

[0094] It should be noted that the embodiments of this application do not limit the preset quality threshold and preset efficiency threshold. For example, the preset quality threshold can be 0, 2, 3, etc. The preset efficiency threshold can be 0, 1, 2, etc.

[0095] For example, if the conversion requirement information is "quality value is 3" and the preset quality threshold is 1, then one of the first type of algorithms can be used as the second conversion algorithm.

[0096] In this way, by setting up both the first and second types of algorithms, we can ensure that the data is transformed according to the transformation requirements, thereby improving the user experience.

[0097] S304. Based on the second conversion algorithm, the first shared data is converted to obtain the first shared data in the target format.

[0098] Understandably, the data request message also includes conversion requirement information, which indicates the quality of the data after format conversion and / or the efficiency of the conversion. After obtaining the initial format of the first shared data, multiple first conversion algorithms can be determined to convert the data from the initial format to the target format. Then, a second conversion algorithm matching the conversion requirement information can be selected from the multiple first conversion algorithms to complete the data format conversion. In this way, the training device can autonomously set conversion requirements, and the server can determine the corresponding conversion algorithm based on the conversion requirements, thereby enabling the data to be converted according to the conversion requirements.

[0099] In some embodiments, before using the first type of algorithm to perform format conversion on shared data, it can be determined whether to use the first type of algorithm as the second conversion algorithm to perform high-quality data conversion based on high-quality conversion conditions.

[0100] In one possible implementation, if the data quality of the first shared data after format conversion indicates that the conversion requirement information is greater than a preset quality threshold, the server obtains the quality information of the first shared data and / or the available resources of the training device. Then, the server can determine whether the quality information of the first shared data and / or the available resources of the training device meet the high-quality conversion conditions. If the quality information of the first shared data and / or the available resources of the electronic device meet the high-quality conversion conditions, the server uses the first type of algorithm as the second conversion algorithm. In this embodiment, the high-quality conversion conditions include at least one of the following: the quality information of the first shared data is greater than the preset quality threshold, and the available resources of the training device are greater than the resources required by the first shared data.

[0101] It should be understood that the embodiments of this application do not limit the available resources of the training device. For example, the available resources may be storage resources, computing resources (such as graphics processing unit (GPU) resources), etc.

[0102] In this embodiment, the server stores quality information and the size of each shared data item. The server can obtain the quality information of the first shared data from the stored quality information of each shared data item. Furthermore, the server can determine the amount of resources required for the first shared data based on its size.

[0103] Understandably, if the quality of the first shared data exceeds a preset quality threshold, it indicates that the first shared data is high-quality data. Therefore, the first type of conversion algorithm can be used to convert the data format, thus avoiding data loss during the conversion process. Furthermore, since high-quality data typically requires a large amount of resources, if the available resources on the training device exceed the resources required for the first shared data, the first type of conversion algorithm can be used to convert the data format, ensuring that the converted first shared data can be used correctly subsequently.

[0104] In some embodiments, if the quality information of the first shared data and / or the amount of available resources of the electronic device do not meet the conditions for high-quality conversion, the server will use the second type of algorithm as the second conversion algorithm.

[0105] In other words, if the quality information of the first shared data is less than or equal to a preset quality threshold, and / or the available resources of the training device are less than or equal to the resources required for the first shared data, a conversion algorithm with higher conversion efficiency can be used as the second conversion algorithm.

[0106] Understandably, if the quality information of the first shared data does not meet the conditions for high-quality conversion, it indicates that the data quality of the first shared data is low. Even if the data is converted using the second type of conversion algorithm, there will be no data loss during the conversion process, and the conversion efficiency can be improved. Alternatively, since high-quality data usually requires a large amount of resources, if the available resources of the training device are less than or equal to the resources required by the first shared data, the required resources can be reduced by lowering the quality of the converted data, thus ensuring that the training device can use the converted shared data normally. Furthermore, using the second type of conversion algorithm can also improve conversion efficiency, thereby improving the efficiency of acquiring shared data.

[0107] In some embodiments, before receiving a data request message from the training device, the terminal device may send a query request message to the server. The query request message includes query keywords, which represent the data content and / or data type of the data to be queried. Then, the server may receive the query request message from the terminal device. In response to the query request message, the server determines the first shared data whose metadata matches the query keywords, based on the query keywords and the metadata of each shared data in the database. The server may then send a query result to the terminal device, the query result including at least the data identifier of the first shared data.

[0108] In this embodiment, the server stores metadata about each piece of shared data. This metadata may include at least one of the following: data sample, quality information, data size, data format, data type, and data description information. The data sample is used to display on the terminal device's page, and the quality information indicates the data quality of the shared data.

[0109] For example, suppose terminal device a uploads a set of shared data a, which includes: shared data 1 to shared data 10. The data examples can be shared data 1 and shared data 2, the quality information can be 2, the data format of shared data a is Tensor format, the data type is video data, and the data description information is "This shared data is used for autonomous driving scenarios".

[0110] It should be understood that the embodiments of this application do not limit the method of obtaining the metadata of the shared data. For example, when a terminal device uploads shared data, it can upload the metadata of the shared data. As another example, a server can generate the metadata of the shared data based on the shared data. Furthermore, the embodiments of this application do not simplify the application scenarios of the shared data. For example, the shared data can be development code to train a model for application development. As another example, the shared data can be educational training data to train a model for educational training. As yet another example, the shared data can be company data, meteorological data, traffic data, medical data, etc.

[0111] Understandably, after receiving a query request message from a terminal device, the system can determine the first shared data that matches the query keywords in the database, and then send the query results to the terminal device. The query results must include at least the data identifier of the first shared data. This allows users to select appropriate shared data before using it. Then, the data identifier can be sent to the terminal, enabling the terminal to obtain the shared data the user wishes to use, thereby improving the accuracy of data retrieval.

[0112] In some embodiments, the server may receive uploaded second shared data.

[0113] For example, a terminal device can send second shared data to a server. The server can receive the second shared data uploaded by the terminal device. Afterward, the server can store the second shared data and its metadata in a database.

[0114] In other words, the server can receive and store shared data uploaded by terminal devices. This increases the amount of data on the server, making it easier for other users to retrieve and obtain the data they need more efficiently.

[0115] In some embodiments, the terminal device may send a data request message to the server. The terminal device may then receive first shared data in a target format sent by the server.

[0116] This eliminates the need for developers to adjust the data format after acquiring shared data; terminal devices can directly obtain shared data in the target format from the server, thereby improving the efficiency of model training. Furthermore, compared to manual data format conversion, having the data sharing platform convert the shared data format reduces the probability of problems occurring during the conversion process, thus improving the quality of the data conversion.

[0117] This application provides a data acquisition method, such as... Figure 4 As shown, the data acquisition method may include:

[0118] S401, Send a data request message to the server.

[0119] S402, Receive data request message sent by training equipment.

[0120] S403. Retrieve the first shared data from the database.

[0121] S404. Convert the data format of the first shared data into the target format.

[0122] In this embodiment of the application, if the data format of the first shared data is not the target format, the data format of the first shared data is converted into the target format.

[0123] S405. Send the first shared data in the target format to the training device.

[0124] S406, Receive first shared data in the target format sent by the server.

[0125] It should be noted that for a detailed description of S401-406, please refer to the description of S201-S205 in the above embodiments, which will not be repeated here.

[0126] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of servers and training devices. It is understood that, in order to achieve the above functions, servers and training devices include corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the data acquisition method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computing device software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] This application embodiment can divide the data acquisition device into functional modules or functional units according to the above method examples. For example, each function can be divided into a separate functional module or functional unit, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or in software functional modules or functional units. The module or unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0128] Please refer to Figure 5 This diagram illustrates a data acquisition device provided in an embodiment of this application. This data acquisition device can be a functional module within the aforementioned server used to implement the method described in the embodiments of this application. Figure 5 As shown, the data acquisition device may include: a receiving module 501, a processing module 502, and a sending module 503.

[0129] The receiving module 501 is used to receive a data request message, which indicates the need to obtain first shared data in a target format. The processing module 502 is used to obtain the first shared data from a database, which includes shared data in various formats and the data format of the shared data. The processing module 502 is also used to convert the data format of the first shared data into the target format if the data format of the first shared data is not the target format. The sending module 503 is used to send the first shared data in the target format.

[0130] In one possible implementation, the shared data includes training data used to train the model. The data request message includes target format information, which characterizes the target algorithm framework and / or the target format, which is a data format supported by the target algorithm framework; the first shared data is used to train the model of the target algorithm framework.

[0131] In another possible implementation, the data request message further includes: conversion requirement information, which indicates the data quality after format conversion and / or the efficiency of the format conversion. Processing module 502 is used to obtain the initial format of the first shared data. Processing module 502 is also used to determine a plurality of first conversion algorithms, which are used to convert the data from the initial format to the target format. Processing module 502 is also used to determine a second conversion algorithm from the plurality of first conversion algorithms that matches the conversion requirement information. Processing module 502 is also used to perform format conversion on the first shared data based on the second conversion algorithm to obtain the first shared data in the target format.

[0132] In another possible implementation, the multiple first transformation algorithms include: a first type of algorithm and a second type of algorithm. The data quality obtained based on the first type of algorithm is higher than that obtained based on the second type of algorithm, and the data transformation efficiency of the second type of algorithm is higher than that of the first type of algorithm.

[0133] In another possible implementation, processing module 502 is further configured to, if the data quality after format conversion of the conversion requirement information representation is greater than a preset quality threshold, use one of the first type of algorithms as the second conversion algorithm. Processing module 502 is also configured to, if the efficiency of the format conversion of the conversion requirement information representation is greater than a preset efficiency threshold, use one of the second type of algorithms as the second conversion algorithm.

[0134] In another possible implementation, processing module 502 is further configured to, if the data quality of the first shared data after the conversion of the conversion requirement information representation format is greater than a preset quality threshold, obtain the quality information of the first shared data and / or the available resource quantity of the training device, and the training device is used to train a model based on the first shared data. Processing module 502 is further configured to, if the quality information of the first shared data and / or the available resource quantity of the electronic device meet the high-quality conversion conditions, use the first type of algorithm as the second conversion algorithm; the high-quality conversion conditions include at least one of the following: the quality information of the first shared data is greater than the preset quality threshold, and the available resource quantity of the training device is greater than the resource quantity required by the first shared data.

[0135] In another possible implementation, the receiving module 501 is further configured to receive a query request message, which includes query keywords; the query keywords represent the data content and / or data type of the data to be queried. The processing module 502 is further configured to, in response to the query request message, determine, based on the query keywords and the metadata of each shared data in the database, the first shared data whose metadata matches the query keywords. The sending module 503 is further configured to send a query result, which includes at least the data identifier of the first shared data. The data request message includes the data identifier of the first shared data.

[0136] In another possible implementation, the receiving module 501 is further configured to receive second shared data uploaded by the terminal device. The processing module 502 is further configured to store the second shared data and its metadata in a database.

[0137] In another possible implementation, the metadata includes at least one of the following: data sample, quality information, data size, data format, data type, and data description information, wherein the quality information is used to indicate the data quality of the shared data.

[0138] Please refer to Figure 6 This diagram illustrates another data acquisition device provided in an embodiment of this application. This data acquisition device can be a functional module in the aforementioned training device used to implement the method of the embodiments of this application. Figure 6 As shown, the data acquisition device may include: a sending module 601 and a receiving module 602.

[0139] The sending module 601 is used to send a data request message to the server. The data request message indicates the acquisition of first shared data in a target format. The data request message includes target format information, which characterizes the target algorithm framework or target format. The target format is a data format supported by the target algorithm framework. The first shared data is used to train the model of the target algorithm framework. The receiving module 602 is used to receive the first shared data in the target format sent by the server.

[0140] Other embodiments of this application provide a computing device. The computing device may include a memory and one or more processors. The memory and processors are coupled. The memory stores computer program code, including computer instructions. When the processor executes the computer instructions, the computing device can perform the various functions or steps described in the above method embodiments.

[0141] This application also provides a chip system, such as... Figure 7 As shown, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 are interconnected via lines. For example, the interface circuit 702 can be used to receive signals from other devices (e.g., the memory of a computing device). As another example, the interface circuit 702 can be used to send signals to other devices (e.g., the processor 701). Exemplarily, the interface circuit 702 can read instructions stored in memory and send those instructions to the processor 701. When the instructions are executed by the processor 701, the computing device can perform the steps described in the above embodiments.

[0142] This application also provides a computer-readable storage medium including computer instructions that, when executed on the aforementioned computing device, cause the computing device to perform various functions or steps described in the method embodiments.

[0143] This application also provides a computer program product that, when run on a computer, causes the computer to perform the functions or steps described in the above method embodiments.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0146] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data acquisition method, characterized in that, Applied to computing devices, including: Receive a data request message, the data request message being used to indicate the acquisition of first shared data in a target format; The first shared data is obtained from a database, wherein the database includes shared data in various formats and the data format of the shared data; If the data format of the first shared data is not the target format, the data format of the first shared data will be converted into the target format; Send the first shared data in the target format.

2. The method according to claim 1, characterized in that, The shared data includes training data, which is used to train the model; the data request message includes target format information, which is used to characterize the target algorithm framework and / or the target format, wherein the target format is a data format supported by the target algorithm framework. The training data is used to train the model of the target algorithm framework.

3. The method according to claim 2, characterized in that, The data request message further includes: conversion requirement information, which indicates the data quality after format conversion and / or the efficiency of the format conversion; the step of converting the data format of the first shared data to the target format when the data format of the first shared data is not the target format includes: Obtain the initial format of the first shared data; A plurality of first conversion algorithms are determined, the first conversion algorithms being used to convert data from the initial format to the target format; A second conversion algorithm that matches the conversion requirement information is determined from the plurality of first conversion algorithms; The first shared data is format-converted based on the second conversion algorithm to obtain the first shared data in the target format.

4. The method according to claim 3, characterized in that, The plurality of first conversion algorithms include: a first type of algorithm and a second type of algorithm; Among them, the data quality obtained based on the first type of algorithm is higher than that obtained based on the second type of algorithm, and the data conversion efficiency of the second type of algorithm is higher than that of the first type of algorithm.

5. The method according to claim 4, characterized in that, The step of determining the second conversion algorithm that matches the conversion requirement information from the plurality of first conversion algorithms includes: If the conversion requirement information indicates that the quality of the data after format conversion is greater than a preset quality threshold, then one of the first type of algorithms will be used as the second conversion algorithm. If the conversion requirement information indicates that the efficiency of the conversion format is greater than a preset efficiency threshold, then one of the second type of algorithms will be used as the second conversion algorithm.

6. The method according to claim 5, characterized in that, If the conversion requirement information indicates that the data quality after format conversion is greater than a preset quality threshold, then one of the first type of algorithms will be used as the second conversion algorithm, including: If the conversion requirement information indicates that the data quality of the first shared data after format conversion is greater than a preset quality threshold, then the quality information of the first shared data and / or the available resources of the training device are obtained, and the training device is used to train a model based on the first shared data; If the quality information of the first shared data and / or the available resources of the training device meet the high-quality conversion conditions, then the first type of algorithm is used as the second conversion algorithm; the high-quality conversion conditions include at least one of the following: the quality information of the first shared data is greater than the preset quality threshold, and the available resources of the training device are greater than the resources required by the first shared data.

7. The method according to any one of claims 1-6, characterized in that, Prior to receiving the data request message, the method further includes: Receive a query request message, the query request message including query keywords; the query keywords are used to represent the data content and / or data type of the data to be queried. In response to the query request message, based on the query keywords and the metadata of each shared data in the database, the first shared data that matches the query keywords is determined; Send the query results, which include at least the data identifier of the first shared data; The data request message includes the data identifier of the first shared data.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: Receive second shared data; The second shared data and its metadata are stored in the database.

9. The method according to claim 8, characterized in that, The metadata includes at least one of the following: data sample, quality information, data size, data format, data type, and data description information, wherein the quality information is used to indicate the data quality of the shared data.

10. A computing device, characterized in that, The computing device includes a memory and a processor; the memory and the processor are electrically connected; the memory is used to store program instructions, and the processor is used to execute the program instructions to enable the computing device to implement the method as described in any one of claims 1-8.

Images