Data screening program, data screening method, and information processing device.

Abstract

To streamline data extraction. [Solution] The processing unit 12 receives a specified number of data to be extracted from a plurality of data, each representing a point in an N-dimensional space 20 (where N is an integer greater than or equal to 1). The processing unit 12 divides the N-dimensional space 20 into a first number of areas. For each of the first number of areas, the processing unit 12 performs a selection process to select one data from among the plurality of data present in that area. The processing unit 12 determines whether the total number of selected data is the same as the specified number. If the total number is not the same as the specified number, the processing unit 12 changes the number of areas dividing the N-dimensional space 20 to a second number different from the first number and executes the selection process.

Description

Technical Field

[0001] The present invention relates to a data screening program, a data screening method, and an information processing apparatus.

Background Art

[0002] Data screening may be performed by an information processing apparatus. Data screening is a technique for selecting data from a huge data group and reducing the total number of data. For example, data screening may improve the accuracy of data by deleting missing values and outliers, or may improve the processing speed for 3D point cloud data of a 3D model.

[0003] Here, based on the information of a point cloud obtained by measuring the real space, a region containing the point cloud is divided into a plurality of regions by a BVH (Bounding Volume Hierarchy) method, and the point cloud is thinned out at a certain ratio for each region according to the number and density of the point cloud contained in each divided region. There is a proposed system.

[0004] Also, in a system for forming an integrated 3D model of a scene by combining a plurality of data of point clouds obtained from a plurality of scanners such as LiDAR (Light Detection And Ranging), a method for deleting redundant points in an overlapping region in each data has been proposed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] In data screening, it is possible to reduce data while ensuring the diversity of the original dataset. For example, in artificial intelligence (AI) technology, the diversity of the training data used in machine learning affects the generalization performance of the machine learning model created based on that training data.

[0007] In this case, the method for extracting data from the original dataset becomes the problem. For example, one method could be to calculate the distance between each pair of data, extract one pair of data that is relatively close in distance, and discard the other. However, this method would require calculating the distance for every single pair of data in the dataset. Therefore, the larger the dataset being screened, the greater the processing cost required to extract data from it.

[0008] In one aspect, the present invention aims to improve the efficiency of data extraction. [Means for solving the problem]

[0009] In one embodiment, a data screening program is provided. This data screening program causes a computer to perform the following processes: The computer accepts a specified number of data to be extracted from a plurality of data, each representing a point in an N-dimensional space (where N is an integer greater than or equal to 1). The computer divides the N-dimensional space into a first number of areas. For each of the first number of areas, the computer performs a selection process to select one data from the plurality of data present in that area. The computer determines whether the total number of selected data is equal to the specified number. If the total is not equal to the specified number, the computer changes the number of areas dividing the N-dimensional space to a second number different from the first number and performs the selection process again.

[0010] In one embodiment, a data screening method performed by a computer is provided. Also, in one aspect, an information processing apparatus having a memory unit and a processing unit is provided.

Advantages of the Invention

[0011] In one aspect, the extraction of data can be made more efficient.

Brief Description of the Drawings

[0012] [Figure 1] It is a diagram for explaining the information processing apparatus of the first embodiment. [Figure 2] It is a diagram showing a hardware example of the information processing apparatus of the second embodiment. [Figure 3] It is a diagram showing an example of data screening. [Figure 4] It is a diagram showing a functional example of the information processing apparatus. [Figure 5] It is a flowchart showing an example of data screening. [Figure 6] It is a diagram showing an example of data screening. [Figure 7] It is a flowchart showing a comparative example. [Figure 8] It is a diagram showing an example of data screening results. [Figure 9] It is a diagram showing an example of data screening results. [Figure 10] It is a diagram showing an example of data screening results. [Figure 11] It is a diagram for explaining the computational complexity of data screening. [Figure 12] It is a diagram showing another comparative example. [Figure 13] It is a diagram showing an example of data selection of the information processing apparatus of the second embodiment. [Figure 14] It is a flowchart showing the first modification example. [Figure 15] It is a flowchart showing the second modification example.

Modes for Carrying Out the Invention

[0013] Hereinafter, this embodiment will be described with reference to the drawings. [First Embodiment] The first embodiment will be described.

[0014] FIG. 1 is a diagram for explaining an information processing apparatus according to the first embodiment. The information processing apparatus 10 performs data screening on a data set including a plurality of data. Each of the plurality of data indicates a point in an N-dimensional space 20. N is an integer of 1 or more. In FIG. 1, the N-dimensional space 20 is represented on a two-dimensional plane for convenience. The points indicated by each of the plurality of data may be points projected onto the N-dimensional space by dimensionality reduction for each of the plurality of original data indicating points in an M-dimensional space. M is an integer larger than N. For dimensionality reduction, existing methods such as principal component analysis and singular value decomposition are used, for example. The information processing apparatus 10 can use the data set after data screening as, for example, training data in machine learning.

[0015] The information processing apparatus 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile semiconductor memory such as a RAM (Random Access Memory), or a non-volatile storage such as an HDD (Hard Disk Drive) or a flash memory.

[0016] The processing unit 12 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). However, the processing unit 12 may include an electronic circuit for specific use such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processor executes a program stored in a memory such as a RAM (which may also be the storage unit 11). A collection of a plurality of processors may be referred to as a "multiprocessor" or simply a "processor".

[0017] The memory unit 11 stores multiple data points. These multiple data points correspond to the dataset to be screened. These multiple data points represent multiple points in the N-dimensional space 20 (step ST1). Figure 1 illustrates the multiple points corresponding to the multiple data points plotted in the N-dimensional space 20. Note that the N-dimensional space 20 is illustrated in two dimensions for convenience.

[0018] The processing unit 12 accepts a specified number of data to be extracted from multiple data sets. For example, the processing unit 12 accepts a specified number entered by the user into the information processing device 10. This specified number corresponds to the number of data to be retained as a result of the data screening.

[0019] The processing unit 12 divides the N-dimensional space into multiple areas by dividing it into a first number of N-dimensional grids. Each grid is an N-dimensional hypercube. For example, when N=1, the grid is a line segment. When N=2, the grid is a square. When N=3, the grid is a cube. For N=4 or greater, a hypercube with the same side length can also be defined. Each grid is also the same size. The processing unit 12 divides the N-dimensional space into the same number of areas as the number of grids by arranging each grid adjacent to each other without gaps.

[0020] For example, the processing unit 12 divides the N-dimensional space 20 into four areas 21a, 22a, 23a, and 24a by dividing the N-dimensional space 20 into four grids 21, 22, 23, and 24. For each of the multiple areas, the processing unit 12 performs a selection process to select one data from among multiple data that exist in that area (step ST2). From step ST2 onward, points corresponding to the data selected by the selection process are shown as black circles, and points corresponding to data that are not selected are shown as dotted circles. Also, if no data belongs to a certain area, no data is selected from that area.

[0021] For example, each of multiple data points may have an index. The index is a number that identifies the data, such as 1, 2, 3, ... The processing unit 12 may select one data point from each area based on the index. In one example, the processing unit 12 may select the data point with the smallest index among the data points belonging to a certain area. Alternatively, the processing unit 12 may select the data point with the largest index, or the data point with the median index, among the data points belonging to a certain area.

[0022] In another example, the processing unit 12 may select data from among the data belonging to a certain area that corresponds to the point closest to the center of that area, or data that corresponds to the point closest to the centroid of each point belonging to that area. However, the processing unit 12 can reduce the cost of the selection process by selecting data using a simpler method, such as selection based on an index, compared to selecting data by calculating the centroid or the like.

[0023] The processing unit 12 determines whether the total number of data selected in the current selection process is the same as the specified number. If the total number is the same as the specified number, the processing unit 12 outputs information indicating the data selected in the current selection process as the extraction result of data screening. If the total number is not the same as the specified number, the processing unit 12 changes the number of grids dividing the N-dimensional space to a second number different from the first number and executes the selection process.

[0024] For example, suppose the specified number received is "8". The processing unit 12 determines that the total number of data selected in the selection process of step ST2, "4", is different from the specified number "8". The processing unit 12 compares the total number "4" with the specified number "8" and detects that the total number "4" is smaller than the specified number "8". Then, the processing unit 12 changes the number of grids to a number greater than the number of grids "4" in step ST2 and executes the selection process (step ST3). In step ST3, as an example, the case where the number of grids is "16" is shown. For example, in step ST3, the processing unit 12 selects 15 data. The processing unit 12 can reduce the number of grids by, for example, increasing the length of one side of each grid, or increase the number of grids by shortening the length of one side of each grid.

[0025] Next, the processing unit 12 determines whether the total number of data selected in the selection process of step ST3, "15", is the same as the specified number, "8". The processing unit 12 determines that the total number "15" is different from the specified number, "8". The processing unit 12 compares the total number "15" with the specified number, "8", and detects that the total number "15" is greater than the specified number, "8". Then, the processing unit 12 changes the number of grids to a number smaller than the number of grids "16" in step ST3 and executes the selection process (step ST4). At this time, the processing unit 12 changes the number of grids to a number larger than the number of grids "4" in step ST2. In step ST4, as an example, the case where the number of grids is "9" is shown. For example, in step ST4, the processing unit 12 selects 8 data.

[0026] The processing unit 12 determines whether the total number of data selected in the selection process of step ST4, which is "8", is the same as the specified number "8". The processing unit 12 determines that the total number "8" is the same as the specified number "8". Therefore, the processing unit 12 outputs information indicating the data selected in the selection process of step ST4 as the extraction result of data screening.

[0027] As described above, the processing unit 12 may generate multiple data representing points in N-dimensional space by projecting multiple original data representing points in M-dimensional space onto N-dimensional space through dimensionality reduction. In this case, the multiple N-dimensional data are associated one-to-one with the multiple M-dimensional original data. Therefore, the final extraction result from the above selection process (for example, the index of the extracted data) can be used as the result of data screening on the multiple M-dimensional original data.

[0028] In this way, the information processing device 10 obtains multiple data points, each representing a point in an N-dimensional space, and a specified number of data points to be extracted from among the multiple data points. The N-dimensional space is divided into a first number of areas. For each of the first number of areas, a selection process is performed to select one data point from the data points present in that area. It is determined whether the total number of selected data points is equal to the specified number. If the total number is not equal to the specified number, the number of areas dividing the N-dimensional space is changed to a second number different from the first number, and the selection process is performed again.

[0029] This allows the information processing device 10 to extract data more efficiently. For example, since the information processing device 10 extracts one data point from each non-overlapping area of ​​the N-dimensional space, it can ensure data diversity in the extraction results.

[0030] Here, as a comparative example, one could consider a method that calculates the distance between each pair of data, extracts one of the pairs with relatively close distances, and discards the other. For example, Euclidean distance can be used to calculate the distance between data. However, in the comparative example method, the distance is calculated for each and every pair of data. The computational complexity of this comparative example is O(n) for the total number of data points n. 2 Therefore, the larger the size of the data being screened, the greater the computational cost becomes.

[0031] In contrast, the information processing device 10 only needs to calculate which grid (area) each of the n data points belongs to. Therefore, the computational complexity of data screening by the information processing device 10 is O(n). As a result, the information processing device 10 can reduce the computational complexity and speed up data extraction compared to the comparative example method. Furthermore, the information processing device 10 can prevent the computational complexity from becoming excessive even if the size of the data to be screened becomes large.

[0032] Furthermore, the methods in the above comparative examples often fail to extract the same number of data points as specified by the user. In contrast, the information processing device 10 can extract the same number of data points as specified by the user.

[0033] As mentioned above, the information processing device 10 can generate multiple data points in an N-dimensional space by projecting multiple original data points, each representing a point in an M-dimensional space, onto an N-dimensional space through dimensionality reduction, and can perform selection processing on these multiple data points. This allows the information processing device 10 to reduce computational costs and speed up data extraction compared to performing selection processing using multiple original data points in an M-dimensional space.

[0034] [Second Embodiment] Next, a second embodiment will be described. Figure 2 shows an example of the hardware of the information processing device according to the second embodiment.

[0035] The information processing device 100 performs data screening on a dataset containing multiple data. The information processing device 100 may also be called a computer. The information processing device 100 has a processor 101, RAM 102, HDD 103, GPU 104, input interface 105, media reader 106, and communication interface 107. These units of the information processing device 100 are connected to a bus inside the information processing device 100. The processor 101 corresponds to the processing unit 12 of the first embodiment. The RAM 102 or HDD 103 corresponds to the storage unit 11 of the first embodiment.

[0036] The processor 101 is an arithmetic unit that executes program instructions. The processor 101 is, for example, a CPU. The processor 101 loads at least a portion of the program and data stored in the HDD 103 into the RAM 102 and executes the program. The processor 101 may include multiple processor cores. The information processing device 100 may have multiple processors. The processor that executes one of the multiple processes performed by the information processing device 100 may be different from the processor that executes a different process from the multiple processes. In addition, at least a portion of the processes described below may be executed in parallel using multiple processors or processor cores. A collection of multiple processors is sometimes called a "multiprocessor" or simply a "processor". A processor may also be called a "processor circuitry".

[0037] RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the processor 101 and data used by the processor 101 for calculations. The information processing device 100 may also be equipped with other types of memory, and may be equipped with multiple types of memory.

[0038] HDD103 is a non-volatile storage device that stores software programs such as the OS (Operating System), middleware, and application software, as well as data. The information processing device 100 may also be equipped with other types of storage devices such as flash memory or SSD (Solid State Drive), and may be equipped with multiple non-volatile storage devices.

[0039] The GPU 104 outputs an image to the display 111 connected to the information processing unit 100, according to instructions from the processor 101. Any type of display can be used as the display 111, such as a CRT (Cathode Ray Tube) display, an LCD (Liquid Crystal Display), a plasma display, or an organic electro-luminescence (OEL) display.

[0040] The input interface 105 acquires input signals from the input device 112 connected to the information processing device 100 and outputs them to the processor 101. The input device 112 can be a pointing device such as a mouse, touch panel, touchpad, or trackball, or a keyboard, remote controller, or button switch. Furthermore, multiple types of input devices may be connected to the information processing device 100.

[0041] The media reader 106 is a reading device that reads programs and data recorded on the recording medium 113. The recording medium 113 can be, for example, a magnetic disk, an optical disk, a magneto-optical disk (MO), or semiconductor memory. Magnetic disks include flexible disks (FD) and HDDs. Optical disks include CDs (Compact Discs) and DVDs (Digital Versatile Discs).

[0042] The media reader 106 copies programs and data read from the recording medium 113 to other recording media such as RAM 102 or HDD 103. The read programs are executed by the processor 101, for example. The recording medium 113 may be a portable recording medium and may be used for distributing programs and data. The recording medium 113 and HDD 103 are sometimes referred to as computer-readable recording media.

[0043] The communication interface 107 is connected to the network 114 and communicates with other information processing devices via the network 114. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or router, or a wireless communication interface connected to a wireless communication device such as a base station or access point.

[0044] Figure 3 shows an example of data screening. The information processing device 100 improves the quality of training data by performing data screening on datasets used, for example, as training data in machine learning. For example, the diversity of data in training data affects the generalization performance of machine learning models created based on the training data. Therefore, in order to improve the quality of training data, the information processing device 100 uniformly screens datasets in a way that ensures data diversity during data screening.

[0045] Figure 3 illustrates the original dataset 30 and dataset 31 after data screening of dataset 30. Each data point in dataset 30 represents a point in an N-dimensional space, where N is an integer greater than or equal to 1. In Figure 3, points corresponding to each data point in datasets 30 and 31 are shown as solid circles. For example, the total number of data points in dataset 30 is 45. The information processing device 100 obtains dataset 31 by uniformly thinning the data in dataset 30. For example, the total number of data points in dataset 31 is 15. In the figure for dataset 31, data points from dataset 30 that are not included in dataset 31 are represented by dotted circles. The information processing device 100 provides a function to efficiently perform data screening.

[0046] Figure 4 shows an example of the functions of an information processing device. The information processing device 100 includes a data storage unit 120, a dimensionality reduction processing unit 130, and a data extraction unit 140. The data storage unit 120 uses the storage area of ​​RAM 102 or HDD 103. The dimensionality reduction processing unit 130 and the data extraction unit 140 are realized by the execution of a program stored in RAM 102 by the processor 101.

[0047] The data storage unit 120 stores multiple original data points representing points in an M-dimensional space. M is an integer greater than N. These multiple original data points are datasets before dimensionality reduction, input to the information processing device 100 by a user or the like. Each of the multiple original data points has an index that identifies it. The indices are numbers 1, 2, 3, ...

[0048] Furthermore, the data storage unit 120 stores multiple data points representing points in an N-dimensional space, which are generated by the dimensionality reduction processing unit 130 based on the multiple original data points. These multiple data points are dimensionality-reduced datasets. The data projected into the N-dimensional space has the same index as the original data corresponding to that data.

[0049] The dimensionality reduction processing unit 130 performs dimensionality reduction on multiple M-dimensional original data to generate multiple data by projecting the multiple original data into an N-dimensional space. Methods such as principal component analysis and singular value decomposition are used for dimensionality reduction.

[0050] The data extraction unit 140 receives the number of data to be extracted (specified number) input by the user, and extracts the specified number of data from multiple N-dimensional data output by the dimensionality reduction processing unit 130.

[0051] Specifically, the data extraction unit 140 divides the N-dimensional space into multiple areas by dividing it into a first number of N-dimensional grids. The data extraction unit 140 can divide the N-dimensional space into the same number of areas as the number of grids by arranging each grid adjacently and without gaps in the N-dimensional space. For each of the multiple areas, the data extraction unit 140 performs a selection process to select one data from among multiple data present in that area.

[0052] The data extraction unit 140 determines whether the total number of data selected in the current selection process is equal to a specified number. If the total number is equal to the specified number, the data extraction unit 140 outputs information indicating the data selected in the current selection process as the extraction result of data screening. If the total number is not equal to the specified number, the data extraction unit 140 changes the number of grids dividing the N-dimensional space to a second number of grids different from the first number and executes the selection process.

[0053] Next, the procedure for data screening using the information processing device 100 will be explained. In the following explanation, a point in N-dimensional space that a given data represents will be referred to as a data point. Selecting a data point and selecting data are essentially the same thing.

[0054] First, let's consider the case where N=2. The two-dimensional space is represented by orthogonal X and Y axes. Data points are represented by X and Y coordinates. The grid dividing the two-dimensional space is a square. Figure 5 is a flowchart showing an example of data screening.

[0055] (S10) The data extraction unit 140 receives information on the specified number of data to be extracted by data screening and information specifying the dataset to be screened. In this example, the number of dimensions M of the dataset specified in step S10 is 3 or greater. The data extraction unit 140 stores the information on the specified number of data in the data storage unit 120 and notifies the dimensionality reduction processing unit 130 of the information on the specified dataset. The dimensionality reduction processing unit 130 retrieves the specified dataset from the data storage unit 120 and projects the dataset onto a two-dimensional space. Specifically, the dimensionality reduction processing unit 130 generates multiple data representing points in a two-dimensional space by performing dimensionality reduction processing on multiple original data representing points in an M-dimensional space.

[0056] (S11) The data extraction unit 140 sets the variable D_low to 0 and the variable D_high to the maximum length of the X-axis and Y-axis. The length of the X-axis is, for example, the difference between the maximum and minimum values ​​of the X coordinate at each data point. The length of the Y-axis is, for example, the difference between the maximum and minimum values ​​of the Y coordinate at each data point.

[0057] (S12) The data extraction unit 140 divides the 2D space into a grid, with the length S of one side of the grid being (D_low+D_high) / 2. (S13) The data extraction unit 140 selects the data point with the smallest index in each grid.

[0058] (S14) The data extraction unit 140 determines whether the number of data points selected in step S13 is the same as the number specified in step S10 (specified number). If the number of data points selected in step S13 is different from the specified number, the process proceeds to step S15. If the number of data points selected in step S13 is the same as the specified number, the data extraction unit 140 outputs the index corresponding to each data point selected in step S13 as the data screening result and terminates the data screening process.

[0059] (S15) The data extraction unit 140 determines whether the number of data points selected in step S13 is less than the number specified in step S10 (specified number). If the number of data points selected in step S13 is less than the specified number, the process proceeds to step S16. If the number of data points selected in step S13 is greater than the specified number, the process proceeds to step S17.

[0060] (S16) The data extraction unit 140 updates D_high=S. Then, the process proceeds to step S12. (S17) The data extraction unit 140 updates D_low=S. Then, the process proceeds to step S12.

[0061] In step S13, the data extraction unit 140 may select the data point with the highest index in each grid, or it may select the data point with the median index in that grid. Alternatively, the data extraction unit 140 may select the data closest to the center point of the grid, or it may select the data closest to the centroid of each data point included in the grid.

[0062] Furthermore, the data extraction unit 140 can efficiently determine the grid size by automatically calculating it using binary search, as shown in steps S15 to S17. Furthermore, for example, the data extraction unit 140 may obtain data corresponding to the index ultimately output as a result of data screening from the original M-dimensional dataset, and use each of the obtained data as training data in machine learning. This allows the information processing device 100 to improve the quality of the training data. For example, the information processing device 100 can improve the generalization performance of the machine learning model created by machine learning based on the training data.

[0063] Next, we will explain a specific example of data screening using the information processing device 100. Figure 6 shows an example of data screening. Multiple data points are plotted in a two-dimensional space 40. The two-dimensional space 40 is a rectangular region that encloses the multiple data points. The length of the X-axis of the two-dimensional space 40 is A. The length of the Y-axis of the two-dimensional space 40 is B. The maximum value of the X-axis length A and Y-axis length B is assumed to be B. Also, the number of data points specified by the user is assumed to be "10".

[0064] When step S12 is executed for the first time, i.e., in iteration "0", D_low=0 and D_high=B. Therefore, the data extraction unit 140 sets S=(0+B) / 2=B / 2. The data extraction unit 140 then divides the 2D space 40 into 4 areas using 4 grids, each with a side length S of S=B / 2. The data extraction unit 140 selects the data point with the minimum index from each of these 4 areas. The data points selected from each area are called "representative points". In the figure, the representative points are shown as black circles plotted in the corresponding grid area.

[0065] In the example in Figure 6, the number of representative points in iteration "0" is 4. The number of representative points "4" is less than the specified number "10". Therefore, the data extraction unit 140 updates D_high=S=B / 2.

[0066] Next, when step S12 is executed, i.e., in iteration "1", D_low=0 and D_high=B / 2. Therefore, the data extraction unit 140 sets S=(0+B / 2) / 2=B / 4. The data extraction unit 140 then divides the 2D space 40 into 16 areas using 16 grids, each with a side length S of S=B / 4. The data extraction unit 140 selects the data point with the minimum index from each of these 16 areas.

[0067] In the example in Figure 6, the number of representative points in iteration "1" is 14. The number of representative points "14" is greater than the specified number "10". Therefore, the data extraction unit 140 updates D_low=S=B / 4.

[0068] Next, when step S12 is executed, i.e., in iteration "2", D_low = B / 4 and D_high = B / 2. Therefore, the data extraction unit 140 sets S = (B / 4 + B / 2) / 2 = (3 / 8)B. The data extraction unit 140 then divides the 2D space 40 into 9 areas using 9 grids, each with a side length S of S = (3 / 8)B. The data extraction unit 140 selects the data point with the minimum index from each of these 9 areas.

[0069] In the example in Figure 6, the number of representative points in iteration "2" is 8. The number of representative points "8" is less than the specified number "10". Therefore, the data extraction unit 140 updates D_high=S=(3 / 8)B. Subsequently, the data extraction unit 140 repeats the above procedure, and when the number of representative points matches the specified number "10", it outputs the index of each finally selected representative point and terminates the data screening.

[0070] By the way, the following comparative example method can be considered as a way to perform screening while ensuring data diversity. The device used to execute the comparative example will be called an "information processing device." Figure 7 is a flowchart showing a comparative example.

[0071] (S100) The information processing device receives information on the specified number of data to be extracted by data screening and information specifying the dataset to be screened. The information processing device projects the dataset onto a two-dimensional space.

[0072] (S101) The information processing device calculates the Euclidean distance between each data projected onto the two-dimensional space and other data. That is, the information processing device calculates the Euclidean distance for each pair of data.

[0073] (S102) The information processing device calculates the minimum value of the Euclidean distance calculated in step S101, and sets this minimum value as D_threshold. (S103) The information processing device deletes one data point from a pair of data points that have a Euclidean distance of (D_threshold) ~ (D_threshold + α). The threshold α is a positive real number. The value of α is predetermined. In one example, α = 0.01.

[0074] (S104) The information processing device determines whether the number of selected data points, i.e., the number of data points that have not been deleted through the repetition of step S103, is less than or equal to the specified number of data points (specified number). If the number of selected data points is less than or equal to the specified number, the information processing device outputs the index of each data point that has not been deleted and terminates the data screening. If the number of selected data points is greater than the specified number of data points, the process proceeds to step S105.

[0075] (S105) The information processing device updates D_threshold to += α. That is, the information processing device adds α to the current value of D_threshold and sets that value as the new value of D_threshold. Then the process proceeds to step S103.

[0076] For the comparative example in Figure 7, please refer to Reference 1. Reference 1: K. Shimizu and 5 others, “Phase stability of Au-Li binary systems studying using neural network”, [online], 2021, Phys. Rev. B 103, 094112 (2021), [Retrieved October 9, 2020], Internet<URL:https: / / journals.aps.org / prb / pdf / 10.1103 / PhysRevB.103.094112> In the above comparative example, the Euclidean distance is calculated for every pair of data. Therefore, if the size of the dataset being screened, i.e., the number of data points in the dataset, is large, the computational cost of data extraction becomes excessive.

[0077] Next, we will illustrate the data screening results obtained by the information processing device 100 and the data screening results obtained by the comparative example method. Figure 8 shows an example of data screening results.

[0078] Table 50 shows the experimental results of data screening using each method of the information processing device 100 and the comparative example, when the number of data points n in the dataset is n=10000 and n=100000. The number of data points after screening (specified number) is 1000. The processor used in the experiment was an Intel® Xeon® CPU E5-2698 v4, with 1 core and 1 thread.

[0079] In both cases, n=10000 and n=100000, each data point (population) in the dataset was generated using NumPy 1.26.4 so that it follows a normal distribution with mean 0 and standard deviation 1. In Table 50, the rows for the normal population show the standard deviation values ​​for each data point in both n=10000 and n=100000, while other items (number of extracted data points and execution time) are shown with a "-" (hyphen) to indicate no value.

[0080] For comparison, Table 50 also includes the results obtained using random sampling. Random sampling is a method of randomly selecting data from n items. Table 50 shows the number of extracted data, standard deviation, and execution time for each row of the Random Sampling, Comparative Example Method, and Information Processing Device 100. The number of extracted data is the number of data points ultimately extracted from the normally occurring population through data screening. The standard deviation is the standard deviation of each data point extracted through data screening. The execution time is the time required to perform the data screening. The unit of execution time is seconds (s).

[0081] The data screening results for each method when n=10000 are as follows: For random sampling, the number of data points extracted was "1000", the standard deviation was "0.9974", and the execution time was "0.0011" seconds.

[0082] In the comparative example method illustrated in Figure 7, the number of extracted data points was "911", the standard deviation was "1.5368", and the execution time was "3.6239" seconds. In the method of the information processing device 100 illustrated in Figure 5, the number of extracted data points is "1000", the standard deviation is "1.5265", and the execution time is "0.4794" seconds.

[0083] The data screening results for each method when n=100,000 are as follows: For random sampling, the number of data points extracted was "1000", the standard deviation was "1.0051", and the execution time was "0.0127" seconds.

[0084] In the comparative example method illustrated in Figure 7, the number of extracted data points was "927", the standard deviation was "1.9069", and the execution time was "419.7760" seconds. In the method of the information processing device 100 illustrated in Figure 5, the number of extracted data points is "1000", the standard deviation is "1.9101", and the execution time is "4.4169" seconds.

[0085] According to Table 50, the method of the information processing device 100 yields a larger standard deviation for each data point obtained than random sampling. In other words, the method of the information processing device 100 allows for the extraction of more diverse data than random sampling.

[0086] Furthermore, according to Table 50, the method of the information processing device 100 has a shorter execution time than the method of the comparative example. In other words, the method of the information processing device 100 can reduce the execution time of data sampling compared to the method of the comparative example. Also, in the comparative example method, the number of extracted data is less than the specified number. On the other hand, the method of the information processing device 100 can appropriately extract the number of data specified by the user.

[0087] Thus, according to the data screening method of the information processing device 100, a number of data specified by the user can be obtained in a short time while maintaining diversity. Figure 9 shows an example of data screening results.

[0088] Graph 60 is a scatter plot showing the normal population used in the experiment for n=10000 in Figure 8. Here, a scatter plot plots each data point in a two-dimensional space. Graph 61 shows the data set extracted as a result of random sampling of the normal population in Graph 60. Graph 62 shows the data set extracted as a result of data screening using the comparative example method for the normal population in Graph 60. Graph 63 shows the data set extracted as a result of data screening using the information processing device 100 method for the normal population in Graph 60.

[0089] According to graphs 61, 62, and 63, the information processing device 100 can extract diverse data from the original dataset. Figure 10 shows an example of data screening results.

[0090] Graph 70 shows the normal population used in the experiment for n=100,000 in Figure 8. Graph 71 shows the data set extracted as a result of random sampling from the normal population in Graph 70. Graph 72 shows the data set extracted as a result of data screening using the comparative example method for the normal population in Graph 70. Graph 73 shows the data set extracted as a result of data screening using the information processing device 100 method for the normal population in Graph 70.

[0091] According to graphs 71, 72, and 73, the information processing device 100 can extract diverse data from the original dataset. Figure 11 illustrates the computational complexity of data screening.

[0092] The data point cluster 80 represents n=6 data points. The comparative example method in Figure 7 calculates the Euclidean distance for each combination of data points. The computational complexity of the comparative example method is O(n 2 ) This is the result. In the comparative example method, for example, 36 elements in a 6x6 Euclidean distance matrix 81 are calculated for the data point cloud 80.

[0093] On the other hand, the method of the information processing device 100 calculates which grid each point is contained in. The computational complexity of the method of the information processing device 100 is O(n). For example, the data extraction unit 140 may generate grid index information 82 for each of the six data points, indicating the index of the grid in which the data point is contained. In this case, the data extraction unit 140 scans the grid index information 82 starting from the smallest data point index, and when the index of the first occurrence of a grid is found, it determines that the index of the data point at that time as the index of the representative point in that grid. The grid index information 82 may be stored in the data storage unit 120.

[0094] Thus, the information processing device 100 can reduce the computational load compared to the comparative example method and speed up the data screening process. Furthermore, the information content of the grid index information 82 is smaller than that of the Euclidean distance matrix 81. Therefore, the information processing device 100 can reduce the memory usage, such as RAM 102, used for computation.

[0095] Another comparative example of data screening is a technique called Voxelization Down-sampling, which is used, for example, to reduce the data volume of 3D point cloud data such as 3D models. Reference 2 is a useful reference for Voxelization Down-sampling.

[0096] Reference 2: Dawei Li, 3 other names, "PSegNet: Simultaneous Semantic and Instance Segmentation for Point Clouds of Plants", [online], 2022, Plant Phenomics, [searched on October 9, 2024], Internet <URL: https: / / spj.science.org / doi / 10.34133 / 2022 / <9787643> FIG. 12 is a diagram showing other comparative examples.

[0097] In Voxelization Down-sampling, the data point cloud is divided into three-dimensional grids (voxels), and the centroid of each data point within a voxel is calculated. Then, each data point in the voxel is replaced with the centroid point. That is, in Voxelization Down-sampling, new data points (centroid points) are generated, and the information of each original data point is deleted. Voxelization Down-sampling does not select existing data points.

[0098] Therefore, Voxelization Down-sampling cannot be applied to cases where "data needs to be selected from a data point cloud". For example, Voxelization Down-sampling cannot be applied for "improving the quality of training data in machine learning".

[0099] Also, in Voxelization Down-sampling, the voxel size is predetermined. The number of data points after screening is determined depending on the voxel size. In Voxelization Down-sampling, the voxel size is set manually, and the number of data points after screening cannot be determined in advance.

[0100] FIG. 13 is a diagram showing an example of data selection of the information processing apparatus according to the second embodiment. The information processing device 100 selects data points from areas divided by each grid, for example, based on an index. In one example, the information processing device 100 selects the data point with the smallest index among the data points belonging to that area. Therefore, the information processing device 100 can select existing data points. Furthermore, the information processing device 100 can pre-specify the number of data points to be extracted and extract the same number of data points as the specified number while maintaining the diversity of data points.

[0101] The information processing device 100 may also select the data point with the highest index among the data points belonging to the area, or the data point with the median index among the data points belonging to the area. The information processing device 100 may also select the data point closest to the centroid of the data points belonging to the area, or the data point closest to the center of the area.

[0102] [Differentiation] Next, a modified version of the second embodiment will be described. Figure 5 illustrates the processing of data points in a two-dimensional space, but the information processing device 100 can perform similar processing on data points in one-dimensional or three-dimensional or more-dimensional spaces.

[0103] For example, when projecting data points into three-dimensional space, the following procedure is followed: Three-dimensional space is represented by orthogonal X, Y, and Z axes. Data points are represented by X, Y, and Z coordinates. The grid dividing the three-dimensional space is a cube.

[0104] Figure 14 is a flowchart of the first modified example. (S20) The data extraction unit 140 receives information on the specified number of data to be extracted by data screening and information specifying the dataset to be screened. In this example, the number of dimensions M of the dataset specified in step S20 is 4 or greater. The data extraction unit 140 stores the information on the specified number of data in the data storage unit 120 and notifies the dimensionality reduction processing unit 130 of the information on the specified dataset. The dimensionality reduction processing unit 130 retrieves the specified dataset from the data storage unit 120 and projects the dataset onto a three-dimensional space. Specifically, the dimensionality reduction processing unit 130 generates multiple data representing points in a three-dimensional space by performing dimensionality reduction processing on multiple original data representing points in an M-dimensional space.

[0105] (S21) The data extraction unit 140 sets the variable D_low to 0 and the variable D_high to the maximum length of the X, Y, and Z axes. The length of the X axis is, for example, the difference between the maximum and minimum values ​​of the X coordinate at each data point. The length of the Y axis is, for example, the difference between the maximum and minimum values ​​of the Y coordinate at each data point. The length of the Z axis is, for example, the difference between the maximum and minimum values ​​of the Z coordinate at each data point.

[0106] (S22) The data extraction unit 140 divides the 3D space into a grid, with the length S of one side of the grid being (D_low+D_high) / 2. (S23) The data extraction unit 140 selects the data point with the smallest index in each grid.

[0107] (S24) The data extraction unit 140 determines whether the number of data points selected in step S23 is the same as the number specified in step S20 (specified number). If the number of data points selected in step S23 is different from the specified number, the process proceeds to step S25. If the number of data points selected in step S23 is the same as the specified number, the data extraction unit 140 outputs the index corresponding to each data point selected in step S23 as the data screening result and terminates the data screening process.

[0108] (S25) The data extraction unit 140 determines whether the number of data points selected in step S23 is less than the number specified in step S20 (specified number). If the number of data points selected in step S23 is less than the specified number, the process proceeds to step S26. If the number of data points selected in step S23 is greater than the specified number, the process proceeds to step S27.

[0109] (S26) The data extraction unit 140 updates D_high=S. Then, the process proceeds to step S22. (S27) The data extraction unit 140 updates D_low=S. Then, the process proceeds to step S22.

[0110] For example, when projecting data points into an N-dimensional space, the following procedure is followed: If N≧2, the N-dimensional space is represented by N mutually orthogonal axes. If N=1, the N-dimensional space is represented by one axis. Data points are represented by N coordinates corresponding to the N axes. The grid that divides the N-dimensional space is an N-dimensional hypercube.

[0111] Figure 15 is a flowchart showing a second modified example. (S30) The data extraction unit 140 receives information on the specified number of data to be extracted by data screening and information specifying the dataset to be screened. In this example, the number of dimensions M of the dataset specified in step S20 is greater than N. The data extraction unit 140 stores the information on the specified number of data in the data storage unit 120 and notifies the dimensionality reduction processing unit 130 of the information on the specified dataset. The dimensionality reduction processing unit 130 retrieves the specified dataset from the data storage unit 120 and projects the dataset onto an N-dimensional space. Specifically, the dimensionality reduction processing unit 130 generates multiple data representing points in an N-dimensional space by performing dimensionality reduction processing on multiple original data representing points in an M-dimensional space.

[0112] (S31) The data extraction unit 140 sets the variable D_low to 0 and the variable D_high to the value of the longest edge among the edges corresponding to the N-dimensional axes. The length of an edge corresponding to a certain axis is, for example, the difference between the maximum and minimum values ​​of the coordinates of that axis at each data point.

[0113] (S32) The data extraction unit 140 divides the N-dimensional space into a grid, with the length S of one side of the grid being (D_low+D_high) / 2. (S33) The data extraction unit 140 selects the data point with the smallest index in each grid.

[0114] (S34) The data extraction unit 140 determines whether the number of data points selected in step S33 is the same as the number specified in step S30 (specified number). If the number of data points selected in step S33 is different from the specified number, the process proceeds to step S35. If the number of data points selected in step S33 is the same as the specified number, the data extraction unit 140 outputs the index corresponding to each data point selected in step S33 as the data screening result and terminates the data screening process.

[0115] (S35) The data extraction unit 140 determines whether the number of data points selected in step S33 is less than the number specified in step S30 (specified number). If the number of data points selected in step S33 is less than the specified number, the process proceeds to step S36. If the number of data points selected in step S33 is greater than the specified number, the process proceeds to step S37.

[0116] (S36) The data extraction unit 140 updates D_high=S. Then, the process proceeds to step S32. (S37) The data extraction unit 140 updates D_low=S. Then, the process proceeds to step S32.

[0117] In steps S23 and S33, the data extraction unit 140 may select the data point with the highest index in each grid, or it may select the data point with the median index in the relevant grid. Alternatively, the data extraction unit 140 may select the data closest to the center point of the relevant grid, or it may select the data closest to the centroid of each data point included in the relevant grid.

[0118] For example, the data extraction unit 140 may obtain data corresponding to the index ultimately output as a result of data screening from the original M-dimensional dataset, and use each of the obtained data as training data in machine learning.

[0119] According to the information processing device 100 of the second embodiment, data extraction can be made more efficient. For example, the information processing device 100 can extract the same number of data as a specified number (specified number) while ensuring diversity. In addition, the information processing device 100 can speed up the extraction of such data.

[0120] The information processing device 100 performs, for example, the following process: The data extraction unit 140 accepts a specified number of data to be extracted from a plurality of data, each representing a point in an N-dimensional space. The data extraction unit 140 divides the N-dimensional space into a first number of areas. For each of the first number of areas, the data extraction unit 140 performs a selection process to select one data from among the plurality of data present in that area. The data extraction unit 140 determines whether the total number of selected data is the same as the specified number. If the total number is not the same as the specified number, the data extraction unit 140 changes the number of areas dividing the N-dimensional space to a second number different from the first number and performs the selection process.

[0121] This allows the information processing device 100 to extract data more efficiently. For example, the information processing device 100 can extract data at high speed. Also, for example, the information processing device 100 can extract the same number of data as a specified number while maintaining the data diversity of the original data set. Furthermore, the information processing device 100 can reduce the amount of memory used for calculations, such as RAM 102.

[0122] For example, the area of ​​the first number may be a partitioned area of ​​N-dimensional space divided by an N-dimensional grid of the first number. Each of the multiple data points may have an index that identifies it. In the selection process, the data extraction unit 140 may select one data point from among the data points present in the area based on the index of each data point present in that area.

[0123] This allows the information processing device 100 to extract data more efficiently. For example, the information processing device 100 can extract data at high speed. As a method for selecting data based on an index, one could consider selecting the data with the smallest or largest index, or the data with the median index, from among the data present in the area. However, it is preferable to minimize the amount of computation required for data selection. For this reason, the method of selecting the data with the smallest or largest index from among the data present in the area is more preferable. The data extraction unit 140 can easily obtain the data with the smallest or largest index for each area by scanning the aforementioned grid index information 82 in ascending or descending order of the data index.

[0124] For example, when changing the number of grid cells, the data extraction unit 140 compares the total number of selected data with the specified number. If the total number is less than the specified number, the data extraction unit 140 sets the second number to a number greater than the first number. If the total number is greater than the specified number, the data extraction unit 140 sets the second number to a number less than the first number.

[0125] This allows the information processing device 100 to efficiently bring the number of data selected in the selection process closer to a specified number. As mentioned above, the data extraction unit 140 can increase the number of grids by shortening the length of one side of the grid. Conversely, the data extraction unit 140 can decrease the number of grids by lengthening the length of one side of the grid.

[0126] Furthermore, as shown in Figure 6, when the data extraction unit 140 decreases the number of grids after increasing them, for example, it determines the number of grids to be greater than the number of grids before the previous increase. When the data extraction unit 140 increases the number of grids after decreasing them, it determines the number of grids to be smaller than the number of grids before the previous decrease. More specifically, in iteration "2" in Figure 6, the number of grids is determined to be "9", which is greater than the number of grids "4" (in iteration "0") before the previous increase. Also, if the number of grids is increased in the iteration following iteration "2" in Figure 6, the number of grids in that next iteration will be determined to be smaller than the number of grids "14" (in iteration "1") before the previous decrease. Alternatively, if the number of grids is reduced in the iteration following iteration "2" in Figure 6, the number of grids in that next iteration will be determined to be greater than the number of grids "4" before the previous increase (in iteration "0").

[0127] This allows the information processing device 100 to efficiently determine the number of grid cells. For example, the information processing device 100 can quickly approach a specified number of selected data cells, thereby speeding up data extraction.

[0128] The data extraction unit 140 outputs information indicating the selected data as the extraction result if the total number of data selected in the selection process is the same as the specified number. This allows the information processing device 100 to easily identify each data point after data screening of multiple data points, and to make each data point after data screening available for use as training data in machine learning, etc.

[0129] The dimensionality reduction processing unit 130 may generate multiple data by projecting each of the multiple original data, each representing a point in M-dimensional space, onto N-dimensional space. As a result, the information processing device 100 can reduce the computational cost of data screening and speed up processing compared to performing data screening using multiple original data in M-dimensional space as is. In addition, the information processing device 100 can reduce the amount of data used for calculations by reducing dimensionality, and can reduce memory usage such as RAM 102. For example, by making the number of dimensions relatively small, such as N=1 or N=2, further speed improvements and memory savings can be achieved. For projection from M-dimensional space to N-dimensional space, methods such as principal component analysis and singular value decomposition can be used.

[0130] Furthermore, each of the processes in the dimensionality reduction processing unit 130 and the data extraction unit 140 of the information processing device 100 may be executed by one or more processors. For example, the processor executing one of these processes may be different from the processor executing another. Furthermore, at least a portion of each process of the information processing device 100 may be executed in parallel by multiple processor cores or multiple processors. For example, the selection process may be executed in parallel by multiple processor cores or multiple processors.

[0131] The information processing in the first embodiment can be achieved by having the processing unit 12 execute a program. The information processing in the second embodiment can be achieved by having the processor 101 execute a program. The program can be recorded on a computer-readable recording medium 113.

[0132] For example, a program can be distributed by distributing a recording medium 113 on which the program is stored. Alternatively, the program may be stored on another computer and distributed via a network. A computer may, for example, store (install) a program stored on the recording medium 113 or a program received from another computer into a storage device such as RAM 102 or HDD 103, and then read and execute the program from that storage device. [Explanation of symbols]

[0133] 10 Information Processing Devices 11 Storage section 12 Processing Units 20 N-dimensional space 21,22,23,24 grid Areas 21a, 22a, 23a, 24a

Claims

1. On the computer, Each accepts a specified number of data points to be extracted from a set of data points, each representing a point in an N-dimensional space (where N is an integer greater than or equal to 1). The aforementioned N-dimensional space is divided into a first number of areas, For each of the aforementioned first number areas, a selection process is performed to select one data from among the plurality of data present in that area. Determine whether the total number of selected data items is the same as the specified number. If the sum is not the same as the specified number, the number of areas dividing the N-dimensional space is changed to a second number different from the first number, and the selection process is executed. A data screening program that executes processing.

2. The area of ​​the first number is obtained by dividing the N-dimensional space into an N-dimensional grid of the first number. Each of the aforementioned plurality of data has an index that identifies the data, In the selection process, one data item is selected from the data items present in the area based on the index of each data item present in the area. A data screening program according to claim 1, which causes the computer to perform the processing.

3. In the selection process, the data with the smallest or largest index is selected from among the data present in the area. A data screening program according to claim 2, which causes the computer to perform the processing.

4. When changing the number of the aforementioned areas, the total number and the specified number are compared, If the sum is less than the specified number, the second number is set to a number greater than the first number. If the aforementioned total is greater than the aforementioned specified number, the second number is set to a number smaller than the first number. A data screening program according to claim 1, which causes the computer to perform the processing.

5. If the total number is the same as the specified number, the information indicating the selected data is output as the extraction result. A data screening program according to claim 1, which causes the computer to perform the processing.

6. The plurality of data is generated by projecting each of the plurality of original data, each representing a point in an M-dimensional space (where M is an integer greater than N), onto the N-dimensional space. A data screening program according to claim 1, which causes the computer to perform the processing.

7. Computers Each accepts a specified number of data points to be extracted from a set of data points, each representing a point in an N-dimensional space (where N is an integer greater than or equal to 1). The aforementioned N-dimensional space is divided into a first number of areas, For each of the aforementioned first number areas, a selection process is performed to select one data from among the plurality of data present in that area. Determine whether the total number of selected data items is the same as the specified number. If the sum is not the same as the specified number, the number of areas dividing the N-dimensional space is changed to a second number different from the first number, and the selection process is executed. Data screening methods.

8. A memory unit that stores multiple data points, each representing a point in an N-dimensional space (where N is an integer greater than or equal to 1), A processing unit that receives a specified number of data to be extracted from the aforementioned multiple data, divides the N-dimensional space into a first number of areas, performs a selection process for each of the first number of areas to select one data from the data present in that area, determines whether the total number of selected data is the same as the specified number, and if the total number is not the same as the specified number, changes the number of areas dividing the N-dimensional space to a second number different from the first number and executes the selection process, An information processing device having

Images