Sampling method and device, electronic equipment and storage medium

Abstract

The invention provides a sampling method and device, electronic equipment and a storage medium, a large top heap structure is used in the process of a K center greedy sampling algorithm, each candidate sample is added into a distance large top heap according to the distance between the candidate sample and a center point, and the candidate sample at a root node in the large top heap structure is the sample with the maximum similar distance, so that the sampling efficiency is improved. When the sample with the largest distance is selected to be added to the central point, only heap building time O (n) is consumed, all similar distances do not need to be traversed, the root node of the large top heap can be directly obtained to be added to the central point, the calculation complexity of the node selected to be added to the central point is reduced from O (nk) to O (n), then the calculation complexity of the sampling process is reduced from O (n * k2 * d) to O (k * n * d), the calculation complexity is reduced, and the calculation efficiency is improved. And the sampling efficiency is improved. And weight information is introduced, so that the sampling method is more targeted, and the sampling probability of high-quality samples is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a sampling method, device, electronic equipment and storage medium. Background Art

[0002] In the process of model training, a large number of training samples are usually used. For example, when training an image recognition model, a large number of labeled or unlabeled images are required as training data, and when training a text recognition model, a large number of labeled or unlabeled text data are required as training data. For another example, in the process of training a large AI model, it is usually necessary to fine-tune the large AI model through labeled instruction fine-tuning data to make the large model more suitable for the target scenario.

[0003] Using a large number of training samples to train a model greatly affects the training speed of the model, and the relevance of a large number of training samples to the model tasks may be uneven, which also affects the accuracy of model training to a certain extent. Therefore, it is necessary to select the training samples participating in the training by sampling in the training sample data set.

[0004] In the related art, the K-center greedy sampling algorithm is a widely used sampling algorithm, which generates a set of center points by repeatedly selecting the point farthest from the selected center point set. However, it needs to calculate the maximum distance from all points to the existing center point at each step, resulting in a computational complexity of O(n*k 2 *d), where n is the number of points, k is the number of selected center points, and d is the vector dimension. This will bring a large computational burden on large-scale data sets, which in turn affects the sampling efficiency and model training efficiency. Summary of the invention

[0005] In view of this, embodiments of the present invention provide a sampling method, device, electronic device and storage medium to improve sampling efficiency.

[0006] According to one aspect of the present invention, a sampling method is provided, the method comprising:

[0007] Obtain candidate samples and a target sampling quantity, wherein the candidate samples include one or more of image data and text data;

[0008] Calculate the similarity distance between each remaining candidate sample and the initial center point, and construct a distance max-heap based on the similarity distance, wherein the initial center point is any candidate sample, the remaining candidate samples are candidate samples other than the center point in the candidate samples, the distance max-heap is a complete binary tree, and the distance of each node in the distance max-heap is greater than or equal to the distance of its child node;

[0009] Obtain the remaining candidate samples corresponding to the root node of the distance heap as the newly added center point;

[0010] Determine whether the number of center points reaches the target sampling number. If not, calculate the similarity distance between each remaining candidate sample and each center point, and update the distance max heap based on the similarity distance, and return to the step of obtaining the remaining candidate samples corresponding to the root node of the distance max heap as the newly added center points until the number of center points reaches the target sampling number.

[0011] In a possible embodiment, the method further includes:

[0012] Calculating a sample weight of each candidate sample using a data quality model, wherein the sample weight is used to identify the relevance of the candidate sample to a target task; the target task includes image recognition or text prediction;

[0013] Determine the candidate sample with the largest sample weight among the candidate samples as the initial center point.

[0014] In a possible embodiment, constructing a distance max-heap based on the similarity distance includes:

[0015] For each of the remaining candidate samples, calculating a similarity distance between the remaining candidate sample and the initial center point;

[0016] Obtaining a value corresponding to the remaining candidate sample based on the product of the similarity distance of the remaining candidate sample and the sample weight of the remaining candidate sample;

[0017] A distance max-heap is constructed based on the values ​​corresponding to the remaining candidate samples, wherein the root node of the distance max-heap is the remaining candidate sample corresponding to the maximum value.

[0018] In a possible embodiment, the method further includes:

[0019] In the case of generating newly added central points and / or candidate samples, the steps of calculating the similarity distances between each remaining candidate sample and each of the central points and updating the distance max-heap based on the similarity distances are performed.

[0020] In a possible embodiment, the candidate samples are stored via a sample data group, the center points are stored via a center point data group, and the distance heap is stored via a heap top data group; the method further includes:

[0021] Moving the newly added center point from the sample data group to the center point data group;

[0022] Determining whether the number of center points reaches the target sampling number includes:

[0023] Determine whether the number of center points in the center point data set reaches the target sampling number.

[0024] According to another aspect of the present invention, there is provided a sampling device, the device comprising:

[0025] An acquisition module, used to acquire candidate samples and a target sampling quantity, wherein the candidate samples include one or more of image data and text data;

[0026] A calculation module, used for calculating the similarity distance between each remaining candidate sample and the initial center point, and constructing a distance max-heap based on the similarity distance, wherein the initial center point is any candidate sample, the remaining candidate samples are candidate samples other than the center point in the candidate samples, the distance max-heap is a complete binary tree, and the distance of each node in the distance max-heap is greater than or equal to the distance of its child node;

[0027] A new module is added, which is used to obtain the remaining candidate samples corresponding to the root node of the distance heap as the newly added center point;

[0028] An updating module is used to determine whether the number of center points reaches the target sampling number. If not, the similarity distance between each remaining candidate sample and each center point is calculated, and the distance max heap is updated based on the similarity distance, and the step of obtaining the remaining candidate samples corresponding to the root node of the distance max heap as the newly added center points is returned until the number of center points reaches the target sampling number.

[0029] In a possible embodiment, the calculation module is also used to calculate the sample weight of each candidate sample using a data quality model, and the sample weight is used to identify the relevance of the candidate sample to a target task; the target task includes image recognition or text prediction; and the candidate sample with the largest sample weight among the candidate samples is determined as the initial center point.

[0030] In a possible embodiment, constructing a distance max-heap based on the similarity distance includes:

[0031] For each of the remaining candidate samples, calculating a similarity distance between the remaining candidate sample and the initial center point;

[0032] Obtaining a value corresponding to the remaining candidate sample based on the product of the similarity distance of the remaining candidate sample and the sample weight of the remaining candidate sample;

[0033] A distance max-heap is constructed based on the values ​​corresponding to the remaining candidate samples, wherein the root node of the distance max-heap is the remaining candidate sample corresponding to the maximum value.

[0034] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0035] Processor; and

[0036] Memory for storing programs,

[0037] The program includes instructions, which, when executed by the processor, enable the processor to perform any of the above-mentioned sampling methods.

[0038] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned sampling methods.

[0039] One or more technical solutions provided in the embodiments of the present invention use a max-heap structure in the process of using the K-center greedy sampling algorithm to add each candidate sample to the distance max-heap according to the distance from the center point. Since the value represented by a node in the max-heap structure is greater than its child node, the candidate sample at the root node is the sample with the largest distance from the center point. Accordingly, in the process of executing the K-center greedy sampling algorithm, when selecting the sample with the largest distance to add to the center point, there is no need to traverse the distances between each candidate sample and each center point. Instead, the root node of the max-heap can be directly obtained and added to the center point, that is, the computational complexity of selecting the node to be added to the center point is reduced from O(nk) to O(1), thereby reducing the computational complexity of the sampling process from O(n*k). 2 *d) is reduced to O(k*n*d), which reduces the computational complexity and improves the sampling efficiency, where n is the number of candidate samples, k is the number of center points, and d is the vector dimension of the candidate samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 A schematic diagram of a flow chart of a sampling method provided by an embodiment of the present invention;

[0042] Figure 2 A process diagram of the K-center greedy sampling algorithm;

[0043] Figure 3 A schematic diagram of a max-heap structure used in a sampling method provided in an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of updating a max-heap structure in a sampling method provided in an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of another flow chart of a sampling method provided by an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of the structure of a sampling device provided in an embodiment of the present invention;

[0047] Figure 7 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0049] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0050] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0051] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0052] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0053] In order to improve the sampling efficiency, the embodiments of the present invention provide a sampling method, an apparatus, an electronic device and a storage medium. The sampling method provided by the embodiments of the present invention can be applied to any electronic device with a sampling function, and the electronic device can be a server, a computer or a mobile terminal, etc. The scheme of the present invention is described below with reference to the accompanying drawings:

[0054] Figure 1 A flow chart of a sampling method provided in an embodiment of the present invention may include the following steps:

[0055] S101, obtaining candidate samples and a target sampling quantity, wherein the candidate samples include one or more of image data and text data;

[0056] S102, calculating the similarity distance between each remaining candidate sample and the initial center point, and constructing a distance max-heap based on the similarity distance, wherein the initial center point is any candidate sample, the remaining candidate samples are candidate samples other than the center point among the candidate samples, the distance max-heap is a complete binary tree, and the distance of each node in the distance max-heap is greater than or equal to the distance of its child node;

[0057] S103, obtaining the remaining candidate samples corresponding to the root node of the distance heap as the newly added center point;

[0058] S104, determining whether the number of center points reaches the target sampling number, if not, executing S105;

[0059] S105, calculating the similarity distance between each remaining candidate sample and each of the center points, and updating the distance max heap based on the similarity distance, returning to the step of obtaining the remaining candidate samples corresponding to the root node of the distance max heap as the newly added center points, until the number of center points reaches the target sampling number.

[0060] By using the embodiment of the present invention, by using the max-heap structure in the process of using the K-center greedy sampling algorithm, each candidate sample is added to the distance max-heap according to the distance from the center point. Since the value represented by the node in the max-heap structure is greater than its child node, the candidate sample at the root node is the sample with the largest distance from the center point. Accordingly, in the process of executing the K-center greedy sampling algorithm, when selecting the sample with the largest distance to add to the center point, it only takes O(n) time to establish the max-heap structure. There is no need to traverse the distance between each candidate sample and each center point. Instead, the root node of the max-heap can be directly obtained and added to the center point, that is, the computational complexity of selecting the node to be added to the center point is reduced from O(nk) to O(n), thereby reducing the computational complexity of the sampling process from O(n*k). 2*d) is reduced to O(k*n*d), which reduces the computational complexity and improves the sampling efficiency, where n is the number of candidate samples, k is the number of center points, and d is the vector dimension of the candidate samples.

[0061] The above S101-S104 are exemplarily described below:

[0062] In S101, candidate samples may be selected according to actual application scenarios. For example, if the application scenario is object detection, image recognition, etc., the candidate samples may be image data. If the application scenario is text recognition, text prediction, etc., the candidate samples may be text data.

[0063] The above-mentioned candidate samples can be labeled data or unlabeled data. Based on the above examples, for application scenarios such as object detection and image recognition, the labels corresponding to the training image data used can be the object information contained in the image. For application scenarios such as text recognition and text prediction, the labels corresponding to the training text data used can be the category to which the text belongs, the text recognition result, etc.

[0064] In a possible embodiment, the candidate sample data may be SFT data, which refers to labeled data for supervised fine-tuning (SFT). Supervised fine-tuning refers to further training a pre-trained model on a smaller, specific data set to improve the performance of the model in a specific task or field. For example, if the application scenario is an AI text map in the animation field, the candidate sample may be an animated image and a description text of the animated image to form an image-text pair.

[0065] The K-center greedy sampling algorithm is a sampling algorithm based on a greedy strategy. Its goal is to select K representative points (called center points) from a data set to minimize the maximum distance between the unselected points and their nearest center points. It is often used in diversity sampling and coverage optimization problems, and is particularly suitable for scenarios where the most representative data needs to be obtained under a limited budget.

[0066] like Figure 2 As shown in the figure, {A, B} is the center point set in the current state, and the third point needs to be selected to be added to the center point set. The process is to first traverse the points other than the center point set, such as C and D, and calculate the distances from C and D to the center point set. The distances from C and D to the center point set are defined as the shortest distances to the center point set. In the figure, AC is the distance from C to the center point set, and BD is the distance from D to the center point set. At this time, the distance from C to the center point set is the largest, so C is selected as the next point to be added.

[0067] The above-mentioned target sampling number refers to the number of the above-mentioned center points, that is, the number of target samples participating in model training. The target sampling number can be selected according to the actual application scenario, which is usually smaller than the value of the candidate sample. In a possible embodiment, the target sampling number can be determined based on the historical training records. For example, the historical training records contain five training records, and the number of training samples used each time is A, B, C, D, and E, respectively. Then, the statistical value of the number of each training sample can be determined as the target sampling number. The statistical value can be the maximum value, minimum value, median, average value, etc.

[0068] In a possible embodiment, an initial center point may be first selected from the candidate samples. The initial center point may be selected arbitrarily from the candidate samples. As a possible implementation, the weight of each candidate sample may be calculated, and the initial center point may be selected based on the sample weight. The above sample weight may characterize the relevance of the sample to the target task. The target task may be a model application scenario, such as image recognition or text prediction, etc. As a possible implementation, the sample weight of each candidate sample may be obtained through a data quality model such as a reward model, and the candidate sample with the largest sample weight may be selected as the initial center point.

[0069] In a possible embodiment, after determining the initial center point, the distance between each remaining candidate sample and the initial center point can be calculated. The distance generally refers to the similarity distance between the remaining candidate samples and the initial center point. The similarity distance can be Euclidean distance, Manhattan distance, etc. The above-mentioned remaining candidate samples refer to the candidate samples other than the center point in the candidate samples. Exemplarily, if the number of candidate samples is n, then it is necessary to calculate the similarity distance between n-1 remaining candidate samples and the initial center point.

[0070] In a possible embodiment, a distance Max-Heap structure can be constructed based on each similar distance. The Max-Heap is a complete binary tree structure in which the value of each node is greater than or equal to the value of its child node. The Max-Heap is usually used to implement a priority queue and can efficiently support operations such as inserting and deleting the maximum value. Through the heapization process, the maximum element is always retained at the root node of the heap, so that the time complexity of obtaining the maximum value is O(1).

[0071] In this embodiment, the point with the largest similarity distance between the remaining candidate samples and the initial center point can be used as the root node of the max-heap, and the remaining candidate samples can be inserted in order of distance from large to small to obtain a distance max-heap. As a possible implementation method, after calculating the similarity distance between a remaining candidate sample and the initial center point each time, the remaining candidate sample can be added to the max-heap structure.

[0072] Exemplarily, after calculating the similarity distance between the first remaining candidate sample and the initial center point, the remaining candidate sample can be added to the root node of the big top heap structure, and after calculating the similarity distance between the second remaining candidate sample and the initial center point, the two similarity distances are compared, and the larger value is used as the root node, and the smaller value is used as the child node of the root node, and so on, until all the remaining candidate samples are added to the big top heap structure. The first remaining candidate sample and the second remaining candidate sample can be randomly selected.

[0073] In a possible embodiment, the distance max-heap structure can also be obtained based on the similarity distance between the remaining candidate samples and the initial center point and the sample weights of the remaining candidate samples. Exemplarily, after calculating the similarity distance between the remaining candidate samples and the initial center point, the max-heap structure can be constructed based on the product of the similarity distance and the sample weight, and specifically, the maximum value of the product is the root node of the max-heap structure.

[0074] That is, for each of the remaining candidate samples, the similarity distance between the remaining candidate sample and the initial center point may be calculated;

[0075] Obtaining a value corresponding to the remaining candidate sample based on the product of the similarity distance of the remaining candidate sample and the sample weight of the remaining candidate sample;

[0076] A distance max-heap is constructed based on the values ​​corresponding to the remaining candidate samples, wherein the root node of the distance max-heap is the remaining candidate sample corresponding to the maximum value.

[0077] After obtaining the distance heap structure, the root node of the heap structure can be used as the newly added center point, such as Figure 3 As shown, the similarity distance corresponding to the root node in the max-heap structure is 45, which is the maximum value among all similarity distances. Therefore, the candidate sample corresponding to the root node can be used as the newly added center point.

[0078] Through the above technical solution, sample weight information is introduced during the sampling process. When the candidate sample is at the same distance from the existing center point, the point with a higher weight is preferentially selected as the new center point, thereby improving the representativeness and rationality of the sampling.

[0079] After the center point is updated, the similarity distance between the remaining candidate samples and the center point after removing the newly added center point can be calculated, and the distance heap can be updated based on the similarity distance between the remaining candidate samples and the center point, which includes the above initial center point and the newly added center point. Figure 4As shown in the figure, the values ​​contained in the original big heap structure are 40, 20, 30, 25, 15, and 5, and the newly obtained value is 34. Therefore, it can be used as the parent node of 30 and 25 to update the original big heap structure. The above values ​​can be similarity distances or the product of similarity distances and sample weights. The numerical contents of each candidate sample in a big heap structure are consistent, that is, the values ​​of each node in the big heap structure are similarity distances or the product of similarity distances and sample weights.

[0080] In a possible embodiment, the candidate samples, center points and max-heap can all be stored in an array. Accordingly, when a new value is calculated, the max-heap array can be updated with the value, and the maximum value is obtained based on the max-heap array to obtain a newly added center point. When the newly added center point is obtained, the center point can be moved from the candidate sample array to the center point array. Accordingly, it can be determined whether the number of center points in the center point data group reaches the target sampling number to determine whether the number of samples collected reaches the target sampling number. If it reaches, the process can be terminated. If not, it returns to S104.

[0081] By using a max-heap, the center point selection process is accelerated. After the heap is built, the center point selection can be realized with only O(1) complexity, and the entire complexity can be reduced to O(k*n*d), where n is the number of candidate samples, k is the number of center points, and d is the vector dimension of the candidate samples.

[0082] like Figure 5 As shown, Figure 5 Another flow chart of a sampling method provided by an embodiment of the present invention may include the following steps:

[0083] S501, input point set points, sampling number k and weights.

[0084] The above point sets and weights are input in the form of data sets.

[0085] S502: Select a point from the points corresponding to the maximum weight and add it to the central point set.

[0086] S503, initialize a max-heap to store the distances * weights of all points to the center point set.

[0087] S504, determine whether the central point set format is less than K, if so, execute S505, if not, execute S507.

[0088] S505, select the top element of the stack and add it to the center point set.

[0089] S506, traverse points to update the distance from the max heap to the center point set * weight, and return to S504.

[0090] S507: Output the center point set.

[0091] By using the embodiment of the present invention, the sampling speed of the K-center greedy sampling algorithm is improved by using a large top heap data structure, and the sampling process is made more targeted by introducing weight information, which can better reflect the importance or priority of the data. The algorithm shows higher efficiency and flexibility in the application of large-scale data sets, and can reduce the calculation time while ensuring accuracy.

[0092] In the process of large model training, it is crucial to select high-quality SFT data. High-quality SFT data can not only accelerate the convergence of the model, but also significantly improve the performance and generalization ability of the model. This algorithm can significantly reduce the time to find high-quality SFT data on large-scale data sets.

[0093] Based on the same inventive concept, the embodiment of the present invention also provides a sampling device, such as Figure 6 As shown, the apparatus 600 may include:

[0094] An acquisition module 601 is used to acquire candidate samples and a target sampling quantity, wherein the candidate samples include one or more of image data and text data;

[0095] A calculation module 602 is used to calculate the similarity distance between each remaining candidate sample and the initial center point, and construct a distance max-heap based on the similarity distance, wherein the initial center point is any candidate sample, the remaining candidate samples are candidate samples other than the center point in the candidate samples, the distance max-heap is a complete binary tree, and the distance of each node in the distance max-heap is greater than or equal to the distance of its child node;

[0096] A new module 603 is added, which is used to obtain the remaining candidate samples corresponding to the root node of the distance heap as the newly added center point;

[0097] The updating module 604 is used to determine whether the number of center points reaches the target sampling number. If not, the similarity distance between each remaining candidate sample and each center point is calculated, and the distance max heap is updated based on the similarity distance, and the step of obtaining the remaining candidate samples corresponding to the root node of the distance max heap as the newly added center points is returned until the number of center points reaches the target sampling number.

[0098] In a possible embodiment, the calculation module is also used to calculate the sample weight of each candidate sample using a data quality model, and the sample weight is used to identify the relevance of the candidate sample to a target task; the target task includes image recognition or text prediction; and the candidate sample with the largest sample weight among the candidate samples is determined as the initial center point.

[0099] In a possible embodiment, constructing a distance max-heap based on the similarity distance includes:

[0100] For each of the remaining candidate samples, calculating a similarity distance between the remaining candidate sample and the initial center point;

[0101] Obtaining a value corresponding to the remaining candidate sample based on the product of the similarity distance of the remaining candidate sample and the sample weight of the remaining candidate sample;

[0102] A distance max-heap is constructed based on the values ​​corresponding to the remaining candidate samples, wherein the root node of the distance max-heap is the remaining candidate sample corresponding to the maximum value.

[0103] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0104] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0105] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0106] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0107] refer to Figure 7, a block diagram of an electronic device 700 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0108] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0109] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0110] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, any of the above-described sampling methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform any of the above-described sampling methods in any other appropriate manner (e.g., by means of firmware).

[0111] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A sampling method, characterized in that: The method comprises: Obtain candidate samples and a target sampling quantity, wherein the candidate samples include one or more of image data and text data; Calculate the similarity distance between each remaining candidate sample and the initial center point, and construct a distance max-heap based on the similarity distance, wherein the initial center point is any candidate sample, the remaining candidate samples are candidate samples other than the center point in the candidate samples, the distance max-heap is a complete binary tree, and the distance of each node in the distance max-heap is greater than or equal to the distance of its child node; Obtain the remaining candidate samples corresponding to the root node of the distance heap as the newly added center point; Determine whether the number of center points reaches the target sampling number. If not, calculate the similarity distance between each remaining candidate sample and each center point, and update the distance max heap based on the similarity distance, and return to the step of obtaining the remaining candidate samples corresponding to the root node of the distance max heap as the newly added center points until the number of center points reaches the target sampling number.

2. The method according to claim 1, characterized in that The method further comprises: Calculating a sample weight of each candidate sample using a data quality model, wherein the sample weight is used to identify the relevance of the candidate sample to a target task; the target task includes image recognition or text prediction; Determine the candidate sample with the largest sample weight among the candidate samples as the initial center point.

3. The method according to claim 2, characterized in that The constructing a distance max-heap based on the similarity distance comprises: For each of the remaining candidate samples, calculating a similarity distance between the remaining candidate sample and the initial center point; Obtaining a value corresponding to the remaining candidate sample based on the product of the similarity distance of the remaining candidate sample and the sample weight of the remaining candidate sample; A distance max-heap is constructed based on the values ​​corresponding to the remaining candidate samples, wherein the root node of the distance max-heap is the remaining candidate sample corresponding to the maximum value.

4. The method according to claim 1, characterized in that: The method further comprises: In the case of generating newly added central points and / or candidate samples, the steps of calculating the similarity distances between each remaining candidate sample and each of the central points and updating the distance max-heap based on the similarity distances are performed.

5. The method according to claim 1, characterized in that The candidate samples are stored by a sample data group, the center point is stored by a center point data group, and the distance heap is stored by a heap top data group; the method further includes: Moving the newly added center point from the sample data group to the center point data group; Determining whether the number of center points reaches the target sampling number includes: Determine whether the number of center points in the center point data set reaches the target sampling number.

6. A sampling device, characterized in that: The device comprises: An acquisition module, used to acquire candidate samples and a target sampling quantity, wherein the candidate samples include one or more of image data and text data; A calculation module, used for calculating the similarity distance between each remaining candidate sample and the initial center point, and constructing a distance max-heap based on the similarity distance, wherein the initial center point is any candidate sample, the remaining candidate samples are candidate samples other than the center point in the candidate samples, the distance max-heap is a complete binary tree, and the distance of each node in the distance max-heap is greater than or equal to the distance of its child node; A new module is added, which is used to obtain the remaining candidate samples corresponding to the root node of the distance heap as the newly added center point; An updating module is used to determine whether the number of center points reaches the target sampling number. If not, the similarity distance between each remaining candidate sample and each center point is calculated, and the distance max heap is updated based on the similarity distance, and the step of obtaining the remaining candidate samples corresponding to the root node of the distance max heap as the newly added center points is returned until the number of center points reaches the target sampling number.

7. The device according to claim 6, characterized in that The calculation module is also used to calculate the sample weight of each candidate sample using a data quality model, and the sample weight is used to identify the relevance of the candidate sample to a target task; the target task includes image recognition or text prediction; and determine the candidate sample with the largest sample weight among the candidate samples as the initial center point.

8. The device according to claim 7, characterized in that The constructing a distance max-heap based on the similarity distance comprises: For each of the remaining candidate samples, calculating a similarity distance between the remaining candidate sample and the initial center point; Obtaining a value corresponding to the remaining candidate sample based on the product of the similarity distance of the remaining candidate sample and the sample weight of the remaining candidate sample; A distance max-heap is constructed based on the values ​​corresponding to the remaining candidate samples, wherein the root node of the distance max-heap is the remaining candidate sample corresponding to the maximum value.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-5.

Images