Training data processing system, training data processing method, and program

Abstract

To provide a training data processing system, training data processing method, and program that can shorten the training time of machine learning models. [Solution] The learning data processing system according to this disclosure is used in one or more machine learning models for recognizing objects captured by a camera, and includes a data adjustment unit that adjusts the number of learning data to suppress parameter bias in learning data that includes at least one of the parameters of relative position and relative orientation between the camera and the object.

Description

Technical Field

[0001] The present disclosure relates to a learning data processing system, a learning data processing method, and a program.

Background Art

[0002] In recent years, technologies for recognizing objects using machine learning (including deep learning) have been developed. For example, Patent Document 1 discloses a technology for a forklift learned using deep learning to estimate the position and orientation of a pallet. The controller of the forklift according to Patent Document 1 recognizes a pallet by an image processing technology using deep learning based on the captured image data of a camera. The controller designates a frame line surrounding the pallet in the captured image data of the camera and compares it with learning data to recognize the pallet. The forklift can estimate the position and orientation of the pallet based on the plane equation of the front surface of the pallet calculated from the laser measurement point data corresponding to the recognized pallet.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the deep learning technology described in Patent Document 1, the deep learning model can recognize a pallet from image data captured by a forklift camera by training with a sufficient amount of training data. In other words, a large amount of training data is required for machine learning models such as deep learning to recognize objects. If the processing performance of the machine learning model remains constant, the training time required to train the machine learning model increases as the amount of training data increases. In technologies like that described in Patent Document 1, there is a risk that the training time required for the machine learning model to recognize objects will increase.

[0005] This disclosure is made to solve these problems and aims to provide a training data processing system, training data processing method, and program that can shorten the training time of machine learning models. [Means for solving the problem]

[0006] The learning data processing system described herein is used in one or more machine learning models for recognizing objects captured by a camera, and includes a data adjustment unit that adjusts the number of learning data to suppress bias in parameters, with respect to learning data that includes at least one of the parameters of relative position and relative orientation between the camera and the object.

[0007] The training data processing method relating to this disclosure is used by a computer in one or more machine learning models for recognizing an object captured by a camera, and analyzes training data that includes at least one of the parameters of relative position and relative orientation between the camera and the object, and adjusts the number of training data to suppress bias in the parameters.

[0008] The program relating to this disclosure is used in one or more machine learning models for recognizing objects captured by a camera, and causes a computer to perform the steps of: analyzing training data that includes at least one of the parameters of relative position and relative orientation between the camera and the object; and adjusting the number of training data to suppress bias in the parameters. [Effects of the Invention]

[0009] This disclosure provides a training data processing system, a training data processing method, and a program that can shorten the training time of machine learning models. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram showing the configuration of the learning data processing system 1 related to this disclosure. [Figure 2] Figure 2 is a flowchart showing an example of the processing operation of the learning data processing system 1. [Figure 3] Figure 3 is a block diagram showing the configuration of the learning data processing system 10 related to this disclosure. [Figure 4] Figure 4 is a schematic diagram illustrating an example of a method for collecting training data based on real images. [Figure 5] Figure 5 is a table showing an example of 6D true value data. [Figure 6] Figure 6 is a schematic diagram showing an example of an object that is imaged when collecting real-image-based training data. [Figure 7] Figure 7 is a flowchart showing an example of the processing operation of the learning data processing system 10. [Figure 8] Figure 8 shows an example of a histogram of the frequency of yaw angles before adjusting for the number of training data points. [Figure 9] Figure 9 shows an example of a histogram of the frequency of yaw angles after adjusting for the number of training data points. [Figure 10]Figure 10 shows an example of the hardware configuration of the learning data processing system 20 related to this disclosure. [Modes for carrying out the invention]

[0011] (Embodiment 1) Embodiment 1 of the present disclosure will be described below with reference to the drawings. Figure 1 is a block diagram showing the configuration of the learning data processing system 1 according to the present disclosure. The learning data processing system 1 includes a data adjustment unit 2. The data adjustment unit 2 is used for one or more machine learning models for recognizing objects captured by a camera. The data adjustment unit 2 has a function to adjust the number of learning data to suppress parameter bias in learning data that includes at least one of the parameters of relative position and relative orientation between the camera and the object.

[0012] In this embodiment 1, the machine learning model includes a deep learning model. The machine learning model according to this embodiment 1 refers to a regression model that acquires the features of an object from an image captured by a camera according to an algorithm and setting parameters, and estimates the object being captured. There may be one or more machine learning models used to recognize an object. The machine learning model may be, for example, a CNN (Convolutional Neural Network), a KRR (Kernel Ridge Regression), or a combination thereof. Furthermore, the machine learning model may be a combination of these models with YOLO (You Only Look Once).

[0013] The training data used for the machine learning model includes at least one parameter of the relative position and the relative orientation between the camera and the object. That is, the training data may include only the relative position as a parameter, may include only the relative orientation as a parameter, or may include both of them. The relative position and the relative orientation are also collectively referred to as the relative position and orientation. That is, the training data may include the relative position and orientation.

[0014] The relative position and the relative orientation are the relative position and orientation between the camera and the object. That is, the relative position represents the position of the other with respect to one of the camera or the object, and the relative orientation represents the orientation of the other with respect to one of them. Considering that the camera and the object each have a coordinate system, the relative position and the relative orientation may be, for example, the position and orientation of the camera viewed from the object coordinate system, or the position and orientation of the object viewed from the camera coordinate system.

[0015] The relative position and the relative orientation may be represented by any of a table, a vector, or a matrix. Also, the relative position and the relative orientation may be collectively represented by one table, one vector, or one matrix. Here, when the relative position is represented by a vector, the relative position may be referred to as a translation vector. Also, when the relative orientation is represented by a matrix, the relative orientation may be referred to as a rotation matrix.

[0016] The training data used by the machine learning model may directly or indirectly include at least one parameter of the relative position and the relative orientation between the camera and the object. That is, the machine learning model may learn by receiving an input of the parameter, or may learn by receiving an input of another data calculated from the parameter.

[0017] The data adjustment unit 2 adjusts the number of learning data so as to suppress the bias of parameters for the learning data used by the machine learning model. The data adjustment unit 2 does not have to adjust the number of learning data so as to eliminate the bias of parameters, or may adjust the number of learning data so as to eliminate the bias. In other words, as a result of the data adjustment unit 2 adjusting the number of learning data, the bias of parameters in the learning data does not have to be eliminated, or the bias may be eliminated.

[0018] The data adjustment unit 2 adjusts the number of learning data by deleting the learning data. Here, the data adjustment unit 2 may adjust the number of learning data by deleting unnecessary learning data and then supplementing necessary learning data. Further, the data adjustment unit 2 may adjust the number of learning data by temporarily increasing the number of learning data and then deleting unnecessary learning data. Even when performing such adjustment, the data adjustment unit 2 adjusts so that the finally adjusted number of learning data is less than the number of learning data before adjustment.

[0019] When adjusting the learning data, the data adjustment unit 2 analyzes the bias of parameters in the learning data to adjust the number of learning data. That is, the data adjustment unit 2 analyzes necessary data and unnecessary data in the learning data in order to suppress the bias of parameters in the learning data, and adjusts the number of learning data by deleting unnecessary data and the like.

[0020] Next, the flow of the processing operation by the learning data processing system 1 will be described. FIG. 2 is a flowchart showing an example of the processing operation of the learning data processing system 1. First, the data adjustment unit 2 analyzes learning data including parameters of at least one of the relative position and the relative orientation (S101). Next, the data adjustment unit 2 adjusts the number of learning data so as to suppress the bias of parameters (S102).

[0021] As described above, the learning data processing system 1 according to this embodiment 1 can shorten the training time of a machine learning model by adjusting the number of training data so as to suppress bias in at least one of the parameters of relative position and relative posture. When the processing performance of a machine learning model is constant, the training time required to train the machine learning model increases as the number of training data increases. In order to obtain a machine learning model with an accuracy that can be used in actual operation, it becomes necessary to train the model with a considerable amount of training data. Therefore, training the model takes a considerable amount of time. Here, there may be bias in parameters such as relative position and relative posture included in the collected training data. The learning data processing system 1 analyzes such bias in parameters and adjusts the number of training data so as to reduce the amount of training data used by the machine learning model while maintaining the accuracy of the machine learning model. As a result, the learning data processing system 1 can shorten the training time of the machine learning model.

[0022] (Embodiment 2) Next, Embodiment 2 of the present disclosure will be described. Figure 3 is a block diagram showing the configuration of the learning data processing system 10 according to the present disclosure. The learning data processing system 10 includes a learning data list acquisition unit 11, a relative position and orientation statistical information calculation unit 12, a data adjustment unit 13, a batch data extraction unit 14, and a machine learning unit 15. The learning data processing system 10 is an enhanced version of the learning data processing system 1 according to Embodiment 1, with the addition of the learning data list acquisition unit 11, the relative position and orientation statistical information calculation unit 12, the batch data extraction unit 14, and the machine learning unit 15.

[0023] The learning data processing system 10, like the learning data processing system 1, processes training data used for one or more machine learning models for recognizing objects captured by the camera. The learning data processing system 10 has a function to adjust the number of training data to suppress parameter bias for training data that includes at least one of the parameters of relative position and relative orientation between the camera and the object. The machine learning models targeted by the learning data processing system 10 are the same as the machine learning models according to Embodiment 1. Regarding the learning data processing system 10, explanations that overlap with the learning data processing system 1 will be omitted as appropriate.

[0024] In this embodiment, a typical object has a plane on part of it, but is not limited to this, and does not have to have a plane. Objects with a plane include, for example, a rectangular prism, a cylinder, a hemisphere, and other three-dimensional shapes having a plane. Objects without a plane include, for example, a sphere or an ellipsoid, and may also be other curved surfaces.

[0025] The orientation of an object relative to the camera may be determined by the planes that the object possesses. Specifically, the yaw angle, pitch angle, and roll angle of the plane relative to the camera's imaging direction may be uniquely determined by the camera imaging the object. For example, if the object is a rectangular prism, the yaw angle, pitch angle, and roll angle of the plane facing the camera on the rectangular prism may be uniquely determined by the camera imaging the rectangular prism.

[0026] The objects to be recognized may be, for example, pallets, cardboard boxes, or trolleys. The objects may be perfect rectangular prisms or approximate rectangular prisms. Furthermore, the size of the objects may not differ from one imaging scene to the next. In other words, the size of the objects may be standardized. For example, the size of the objects used in training data may be the same as, or different from, the size of the objects to be recognized during actual operation.

[0027] A camera for imaging an object may be mounted on a mobile device that recognizes the object and places or grasps it. For example, if the object is a pallet, the camera may be mounted on a forklift that places the pallet on it. If the object is a cardboard box, the camera may be mounted on a drone that grasps the cardboard box. Alternatively, the camera may be mounted on a picking arm robot that grasps the cardboard box.

[0028] The training data list acquisition unit 11 acquires a list of training data to be used in the machine learning model. The training data list acquisition unit 11 may acquire pre-prepared training data from the storage unit (not shown) of the training data processing system 10, or it may acquire training data from another server (not shown).

[0029] Here, we will explain the training data acquired by the training data list acquisition unit 11. The training data may include images of the target object as part of it. These images may be images captured by a camera or images generated using a simulator. In other words, the method of collecting images in the training data may be based on real images or on a simulator. That is, the training data may include images captured in the real world as part of the training data, or images generated using a simulator as part of the training data. Any publicly available simulator may be used as the simulator.

[0030] An example of a method for collecting training data based on real images is shown in the diagram. Figure 4 is a schematic diagram illustrating an example of a method for collecting training data based on real images. In Figure 4, the object is a pallet depicted in the upper part of the figure. The camera for acquiring images of the pallet is mounted on a forklift equipped with forks for placing the pallet. The forklift is depicted in the center of Figure 4. The width of the aisle is considerably wider than the width of the pallet. Therefore, three patterns of pallet placement can be considered: the left end, the center, and the right end of the aisle. For each placement location, the forklift is moved back and forth while the camera captures images. This allows for the acquisition of images of the pallet. In Figure 4, the arrows between pallets represent changes in the pallet placement location. The arrow to the left of the forklift represents an example of the forklift's movement pattern.

[0031] The explanation of the training data collected by the training data list acquisition unit 11 shown in Figure 3 continues. The training data includes at least one parameter of the relative position and relative orientation between the camera and the object. If the training data further includes an captured image, the parameters in the training data are parameters for that captured image. The relative position and relative orientation of the object with respect to the captured image in the training data are also referred to as 6D true value data.

[0032] Here, an example of 6D true data is shown in the table. Figure 5 is a table showing an example of 6D true data. Figure 5 shows the relative position and relative orientation of the camera with respect to the object. In Figure 5, cam_tx, cam_ty, and cam_tz represent the relative position parameters, and cam_rx, cam_ry, and cam_rz represent the relative orientation parameters. Specifically, cam_tx is the movement of the camera in the X-axis direction (left-right direction) relative to the object, cam_ty is the movement in the Y-axis direction (up-down direction), and cam_tz is the movement in the Z-axis direction (depth direction). Also, cam_rx is the pitch angle of the camera with respect to the object, cam_ry is the yaw angle, and cam_rz is the roll angle.

[0033] In the case of real-image-based data, 6D true value data may be obtained through cooperation with an external system of the training data processing system 10. For example, 6D true value data can be obtained by combining AR (Augmented Reality) markers and VSLAM (Visual Simultaneous Localization and Mapping).

[0034] The method for acquiring 6D true value data based on real images will be explained using diagrams. Figure 6 is a schematic diagram showing an example of an object to be imaged when collecting training data based on real images. In Figure 6, the object is a pallet. Two stacks of goods are placed on the pallet in two rows. An AR marker is attached to the center of one location on the side of the pallet in Figure 6. The AR marker is captured in the image acquired by the camera. The pallet specifications are standardized. Therefore, the size of the pallet is known. In this case, the camera's own position and orientation can be estimated by applying VSLAM to the image acquired while the camera is moving. The 6D true value data may be the camera's own position and orientation relative to the pallet estimated in this way. On the other hand, in a simulator-based approach, the 6D true value data may be output together with the generated image.

[0035] Let's continue the explanation of the learning data collected by the learning data list acquisition unit 11 shown in Figure 3. The learning data may include bounding boxes for the object. If the object is a pallet, the bounding box may be, for example, one that surrounds the entire pallet, one that surrounds the insertion openings on the pallet, or both.

[0036] Furthermore, the training data may include 2D coordinate data of any point on the object captured in the image. Here, the 2D coordinate data may be calculated from camera intrinsic parameters, 6D true data, and information relating to the size of the object. If the object is a palette, the 2D coordinate data may be coordinate data indicating the vertices of the palette. Here, the 2D coordinate data may be coordinate data indicating all the vertices of the palette, or it may be coordinate data indicating some of the vertices. Alternatively, the 2D coordinate data may be coordinate data indicating the vertices of the slots on the palette.

[0037] The learning data list acquisition unit 11 transmits the acquired learning data to the relative position and orientation statistical information calculation unit 12 and the data adjustment unit 13. The learning data list acquisition unit 11 may also transmit the learning data to the batch data extraction unit 14.

[0038] The relative position and orientation statistics calculation unit 12 calculates statistical information for at least one of the parameters of relative position and relative orientation between the camera and the object, based on the learning data received from the learning data list acquisition unit 11. Specifically, the learning data list acquisition unit 11 calculates statistical information about the frequency of the parameter. The relative position and orientation statistics calculation unit 12 may calculate statistical information about the frequency of some of the parameters of relative position and relative orientation, or it may calculate statistical information about the frequency of all parameters. For example, the relative position and orientation statistics calculation unit 12 may calculate statistical information about the yaw angle between the camera and the object.

[0039] The relative position and orientation statistics calculation unit 12 typically creates a histogram of the frequency of relative position and orientation parameters, but it may also create a table of the frequency of each class of the parameters. The relative position and orientation statistics calculation unit 12 transmits the calculated statistics to the data adjustment unit 13 and the batch data extraction unit 14.

[0040] The data adjustment unit 13 has the functions of the data adjustment unit 2 of the learning data processing system 1. Functions of the data adjustment unit 13 that overlap with those of the data adjustment unit 2 will not be explained.

[0041] The data adjustment unit 13 receives training data from the training data list acquisition unit 11 and statistical information about parameters from the relative position and orientation statistical information calculation unit 12. The data adjustment unit 13 adjusts the number of training data to suppress parameter bias for training data that includes at least one of the parameters of relative position and relative orientation between the camera and the object. For example, the data adjustment unit 13 adjusts the number of training data to suppress bias in the yaw angle among the parameters. If part of the object has a plane, the data adjustment unit 13 may adjust the number of training data to suppress bias in the yaw angle of that plane with respect to the camera's imaging direction. If the object does not have a plane, the data adjustment unit 13 may adjust the number of training data to suppress bias in the yaw angle with respect to an arbitrary reference point set on the object.

[0042] The data adjustment unit 13 may adjust the number of training data based on statistical information about the frequency of parameters received from the relative position and orientation statistical information calculation unit 12. For example, the data adjustment unit 13 adjusts the number of training data by setting an upper limit for the frequency of the parameter. Specifically, the data adjustment unit 13 adjusts the number of training data by deleting similar training data from training data that includes parameters with a frequency exceeding a predetermined upper limit.

[0043] Assume that the relative position and orientation statistics calculation unit 12 has created a histogram of the frequency of parameters. Here, the data adjustment unit 13 sets, for example, the average value of the parameter frequency as the upper limit of frequency. That is, the data adjustment unit 13 adjusts the number of training data so that the frequency of each class in the histogram does not exceed its average value. If the parameter is an angle such as the yaw angle, the classes may be every 1 degree or not. The data adjustment unit 13 adjusts the number of training data by, for example, deleting training data that contains parameters with a frequency exceeding the average value of frequency.

[0044] In this case, the data adjustment unit 13 may adjust the number of training data by deleting similar training data from among the training data that include parameters having a frequency exceeding the average frequency. For example, the data adjustment unit 13 defines training data whose relative position vector distance is less than or equal to a predetermined value as similar training data. If there are two similar training data, the data adjustment unit 13 deletes one of them. If there are three or more similar training data, the data adjustment unit 13 may delete two or more of them.

[0045] Furthermore, the data adjustment unit 13 may calculate the similarity of the training data by analyzing the images included in the training data, and may delete training data whose similarity is equal to or greater than a predetermined value. The data adjustment unit 13 calculates the similarity of the training data using, for example, SSIM (Structural Similarity Index Measure). If the object is a pallet, the data adjustment unit 13 determines that training data containing images of identical pallets or similar packaging of goods placed on the pallets are similar.

[0046] If, as a result of deleting similar training data, the frequency of parameters for all classes falls below the former mean, the data adjustment unit 13 terminates the adjustment of the number of training data. If the frequency of parameters for all classes does not fall below the former mean, the data adjustment unit 13 may or may not delete further training data containing parameters that do not fall below the mean. In other words, the data adjustment unit 13 may terminate the adjustment of the number of training data regardless of whether the frequency of parameters for all classes falls below the former mean.

[0047] If the number of training data points needs to be further adjusted, the data adjustment unit 13 may randomly delete training data points that contain parameters that are not below the average value. Furthermore, if the data adjustment unit 13 performed the previous adjustment by calculating the distance between position vectors, the data adjustment unit 13 may further adjust the number of training data points by calculating image similarity using SSIM.

[0048] If the relative position and orientation statistics calculation unit 12 calculates statistics for two or more parameters, the data adjustment unit 13 may set an upper limit for each of the statistics for the multiple parameters. The data adjustment unit 13 may also adjust the number of training data by deleting training data containing parameters whose frequency exceeds the upper limit for each set of statistics using the method described above.

[0049] Assuming that the distribution of parameter frequencies follows a normal distribution, the data adjustment unit 13 may determine the upper limit of parameter frequencies by the following method. First, the data adjustment unit 13 calculates a weighted mean of the parameters weighted by their frequency and uses this weighted mean as the representative parameter. The data adjustment unit 13 may also use the parameter with the highest frequency as the representative parameter. The data adjustment unit 13 sets the frequency of the representative parameter to mean μ. Next, the data adjustment unit 13 calculates the standard deviation σ of the frequencies. Subsequently, the data adjustment unit 13 calculates the average value of the frequencies within the range of mean μ ± standard deviation σ. The data adjustment unit 13 sets this average value as the upper limit. The data adjustment unit 13 adjusts the number of training data using this upper limit in the method described above.

[0050] The data adjustment unit 13 may adjust the number of training data by deleting unnecessary training data and then supplementing it with necessary training data. Alternatively, the data adjustment unit 13 may adjust the number of training data by temporarily increasing the number of training data and then deleting unnecessary training data. For example, the data adjustment unit 13 sets half the average value of the parameter frequency as a provisional upper limit for frequency. In this case, the data adjustment unit 13 deletes similar training data from the training data that includes parameters with frequencies exceeding the upper limit. Subsequently, the data adjustment unit 13 may adjust the final number of training data by supplementing it with necessary training data from the deleted training data. In this case, the data adjustment unit 13 adjusts the number of training data so that the final adjusted number is less than the number of training data before adjustment.

[0051] The data adjustment unit 13 may calculate a lower limit for the frequency of parameters and adjust the number of training data containing parameters with frequencies exceeding this lower limit. In other words, the data adjustment unit 13 may only delete training data containing parameters with frequencies exceeding this lower limit. To put it another way, training data containing parameters with frequencies below this lower limit may not be deleted. Assuming that the distribution of parameters follows a normal distribution, the lower limit for the frequency of parameters may be the frequency of μ ± 3σ, where μ is the mean and 3σ is the three standard deviations for any representative parameter.

[0052] The data adjustment unit 13 transmits the adjusted training data to the machine learning unit 15. The data adjustment unit 13 may or may not transmit the adjusted training data to the batch data extraction unit 14.

[0053] The batch data extraction unit 14 has the function of collecting training data within a batch by extracting training data from part or all of the batch in a manner that suppresses parameter bias when a machine learning model performs training using one or more batches. In other words, the batch data extraction unit 14 is used when a machine learning model performs batch learning or mini-batch learning. Here, a batch is a collection of multiple training data used when performing batch learning or mini-batch learning. The batch size can be arbitrarily set according to the number of training data.

[0054] The batch data extraction unit 14 may use the training data transmitted from the training data list acquisition unit 11, or it may use the training data transmitted from the data adjustment unit 13. Furthermore, when creating batch data, the batch data extraction unit 14 may or may not extract the same training data.

[0055] The batch data extraction unit 14 may classify the training data into multiple categories based on the frequency of parameters and extract training data from these categories. For example, the batch data extraction unit 14 may classify the training data into training data containing high-frequency parameters and training data containing low-frequency parameters, and extract the required number of training data from each category. Here, the batch data extraction unit 14 may extract training data from these categories in such a way that the parameter distribution within the batch is even. For example, the batch data extraction unit 14 may extract a certain number of training data from the training data containing high-frequency parameters and the same number from the training data containing low-frequency parameters. The training data processing system 1 may also extract more training data from the training data containing high-frequency parameters than from the training data containing low-frequency parameters.

[0056] For example, suppose the frequency distribution of a certain parameter follows a normal distribution. In this case, the batch data extraction unit 14 can classify the training data as follows, where μ is the mean of the frequency of the parameter and σ is the standard deviation. That is, the batch data extraction unit 14 can classify the data into "parameters with a frequency less than μ-σ", "parameters with a frequency greater than or equal to μ-σ and less than or equal to μ+σ", and "parameters with a frequency greater than μ+σ". The batch data extraction unit 14 then creates batches by extracting training data equally from each classification. The batch data extraction unit 14 may also create batches by extracting training data such that the ratio of the number of training data points is, for example, "1:3:1". That is, the batch data extraction unit 14 may weight the number of extractions from each classification within a batch according to the bias of the parameters.

[0057] The batch data extraction unit 14 sends the created batch data to the machine learning unit 15.

[0058] The machine learning unit 15 trains a machine learning model using the training data received from the data adjustment unit 13 and the batch data received from the batch data extraction unit 14. For deep learning models that perform batch learning or mini-batch learning, such as CNNs, the batch data received from the batch data extraction unit 14 is input. For other machine learning models that do not perform batch learning or mini-batch learning, the training data received from the data adjustment unit 13 is input. The number of training iterations (epochs) performed by the machine learning unit 15 can be set arbitrarily.

[0059] Next, the processing flow of the learning data processing system 10 will be explained. Figure 7 is a flowchart showing an example of the processing operation of the learning data processing system 10. Note that in Figure 7, the transmission or reception processes for each configuration are omitted. First, the learning data list acquisition unit 11 acquires a list of learning data to be used in the machine learning model in the machine learning unit 15 (S201). Next, the relative position and orientation statistics calculation unit 12 calculates statistical information about the parameters based on the learning data received from the learning data list acquisition unit 11 (S202). After that, the data adjustment unit 13 adjusts the number of learning data based on the learning data received from the learning data list acquisition unit 11 and the statistical information received from the relative position and orientation statistics calculation unit 12 (S203).

[0060] Subsequently, the batch data extraction unit 14 collects the learning data within a batch based on the learning data received from the learning data list acquisition unit 11 or the learning data received from the data adjustment unit 13 (S204). Here, if the batch data extraction unit 14 receives the learning data from the learning data list acquisition unit 11, the batch data extraction unit 14 may collect the learning data within a batch at the same time as the adjustment of the number of learning data by the data adjustment unit 13. Alternatively, the batch data extraction unit 14 may collect the learning data within a batch before the adjustment of the number of learning data by the data adjustment unit 13.

[0061] Subsequently, the machine learning unit 15 performs training of a machine learning model using the training data received from the data adjustment unit 13 and the batch data received from the batch data extraction unit 14 (S205). Here, if the machine learning unit 15 receives training data from the data adjustment unit 13, it may perform training of a machine learning model that does not perform batch training, etc., before receiving batch data from the batch data extraction unit 14. Alternatively, if the machine learning unit 15 receives batch data from the batch data extraction unit 14, it may perform training of a machine learning model that performs batch training, etc., before receiving training data from the data adjustment unit 13.

[0062] Thus, the learning data processing system 10 according to this second embodiment can shorten the training time of the machine learning model by specifically implementing the learning data processing system 1 according to the first embodiment. If the training time is suppressed by simply reducing the number of training data, the machine learning model will not be able to properly recognize the object because its accuracy in recognizing the object will be insufficient. For example, if a machine learning model is used to control a forklift and move a pallet, the machine learning model cannot be used to control the forklift if its accuracy is insufficient.

[0063] Therefore, to obtain sufficient recognition accuracy, it is necessary to increase the amount of training data. However, if the processing performance of the machine learning model remains constant, increasing the amount of training data will increase the training time. For example, if a machine learning model is used to control equipment within a factory, machine learning will be required for each factory, which will affect the lead time for introducing the equipment to new locations.

[0064] Furthermore, the learning methods using related technologies also present the problem of reduced generalization performance. Specifically, these methods result in a bias in the parameters of the collected training data. Consequently, the machine learning model is not sufficiently trained on parameters with relatively low frequency in the training data, leading to the inability to properly recognize objects at such relative positions or orientations, and resulting in equipment control failure. One possible solution is to train the machine learning model with additional training data containing such parameters. However, this requires acquiring even more training data, which increases the training time.

[0065] According to the training data processing system 10, the number of training data can be adjusted to suppress parameter bias in the training data, thereby shortening the training time and reducing the lead time for deployment to new locations. Furthermore, by suppressing parameter bias, the decline in generalization performance can be suppressed. As a result, the need to acquire new training data is reduced, further shortening the training time.

[0066] For example, suppose there is a bias in relative pose in the collected training data. That is, consider a case where training data can be collected in such a way that there is no bias in the relative position between the camera and the object, or where relative position does not need to be considered, but a bias occurs in relative pose. In such a case, by using the training data processing system 10, training data with suppressed relative pose bias can be input into the machine learning model. The distance between the camera and the object can be freely set by moving the camera that images the object, but the training data processing system 10 is effective when there are restrictions on the angle at which the object is imaged or when there are restrictions on the vertical movement of the camera.

[0067] Furthermore, for example, suppose there is a bias in the relative positions of the collected training data. This can occur, for example, when a camera that images an object can move to image the object from various angles, but there are limitations on the positional relationship between the camera and the object when imaging it. Even in such cases, the training data processing system 10 is effective.

[0068] For example, suppose there is a bias in the yaw angle of the collected training data. This can occur when the object is a pallet, the camera is mounted on a forklift used to place the pallet, and training data is to be collected using the camera in a situation where the aisle where the pallet is placed is narrow. In other words, the yaw angle that the camera can move relative to the pallet is limited, resulting in a bias in the yaw angle of the training data. Even in such cases, the training time can be shortened by using the training data processing system 10.

[0069] Such yaw angle bias can occur if an object, like a pallet, has a flat surface in part, and the yaw angle is the yaw angle of that surface with respect to the camera's imaging direction. In other words, even if the object is not a roughly rectangular parallelepiped like a pallet, if the object has a flat surface in part, the training time can be shortened by using the training data processing system 10.

[0070] Furthermore, the training data processing system 10 can adjust the number of training data based on statistical information about the frequency of parameters. This allows the training data processing system 10 to adjust the number of training data by deleting training data containing high-frequency parameters. As a result, the training data processing system 10 can shorten the training time while ensuring the generalization performance of the machine learning model.

[0071] Furthermore, the training data processing system 10 can adjust the number of training data by deleting similar training data from among the training data that include parameters with a frequency exceeding a predetermined upper limit. This allows the training data processing system 10 to balance the training data, thereby further improving generalization performance.

[0072] Here, the learning data processing system 10 can determine that learning data whose relative position vector distance is less than or equal to a predetermined value is similar learning data and delete such learning data. As a result, the learning data processing system 10 can easily determine the similarity of the learning data, thereby reducing the computational cost when calculating the similarity.

[0073] Furthermore, the learning data processing system 10 can calculate a lower limit for the frequency of parameters and adjust the number of learning data points that include parameters with frequencies exceeding this lower limit. This allows the learning data processing system 10 to maintain a certain number of learning data points while suppressing parameter bias, thereby ensuring a certain level of recognition accuracy while preventing excessive deletion of learning data.

[0074] Here, a specific example of the effect of the learning data processing system 10 will be explained using diagrams. Figure 8 is an example of a histogram of the frequency of yaw angles before adjusting the number of learning data. Figure 9 is an example of a histogram of the frequency of yaw angles after adjusting the number of learning data. In Figures 8 and 9, the object is a pallet, and a camera for imaging the pallet is mounted on a forklift. Furthermore, in Figures 8 and 9, learning data is collected by imaging the pallet using real images. In addition, Figures 8 and 9 represent histograms of the camera's yaw angle relative to the pallet. That is, Figures 8 and 9 are histograms with the horizontal axis representing the yaw angle and the vertical axis representing the frequency.

[0075] Figure 8 shows a histogram of 20,000 training data points. As shown in Figure 8, there is a bias in the yaw angles in the training data, with yaw angles around -5 degrees being particularly frequent. On the other hand, the number of training data points for yaw angles smaller than -15 degrees and larger than 0 degrees is relatively small. This is likely because when the pallet is facing the forklift, the camera's yaw angle relative to the pallet is approximately -5 degrees, resulting in a high frequency of training data around -5 degrees. If training were to use all of this training data, the training time would increase significantly.

[0076] In Figure 9, the training data processing system 10 adjusted the number of training data. Specifically, the training data processing system 10 set 300 as the upper limit for frequency and reduced the number of training data for yaw angles with a frequency exceeding 300, thereby reducing the number of training data from 20,000 to 12,000. As a result, the training data processing system 10 can shorten the training time, assuming a constant processing performance for the machine learning model. Furthermore, as shown in Figure 8, by reducing the training data around -5 degrees, which is the most frequent yaw angle, the generalization performance can be improved compared to using the data before the training data adjustment.

[0077] Next, we will explain the effects of the training data processing system 10 when a machine learning model performs batch learning or mini-batch learning. When a machine learning model performs learning using one or more batches, the training data processing system 10 can collect training data within a batch by extracting training data from part or all of the batch in a way that suppresses parameter bias. As a result, even if the machine learning model is a deep learning model such as a CNN that performs batch learning or mini-batch learning, the generalization performance can be improved by suppressing bias in the batch data.

[0078] Furthermore, the training data processing system 10 can extract batch data in a way that suppresses parameter bias in the adjusted training data. This makes it possible to further improve the generalization performance of machine learning models that perform batch learning or mini-batch learning.

[0079] Furthermore, the learning data processing system 10 can classify the learning data into multiple categories based on frequency and extract learning data from these categories. This allows the learning data processing system 10 to easily create batch data with suppressed parameter bias, thereby reducing the processing cost associated with batch data creation.

[0080] Furthermore, the training data processing system 10 can extract training data in such a way that the parameter bias within a batch is evenly distributed. This allows the training data processing system 10 to further improve the generalization performance of machine learning models that perform batch learning or mini-batch learning.

[0081] (Example hardware configuration) Figure 10 shows an example of the hardware configuration of a learning data processing system 20 according to this disclosure. In Figure 10, the learning data processing system 20 includes a processor 21 and a memory 22. The processor 21 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 21 may include multiple processors. The memory 22 is composed of a combination of volatile memory and non-volatile memory. The memory 22 may include storage located away from the processor 21. In this case, the processor 21 may access the memory 22 via an I / O (Input / Output) interface, which is not shown.

[0082] In the above example, the program can be stored and provided to the computer using various types of non-transitory computer-readable medium. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable medium include magnetic storage media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). Alternatively, the program may be provided to the computer using various types of transient computer-readable medium. Examples of transient computer-readable medium include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable medium can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels. Computers include various information processing devices such as PCs, servers, CPUs, MPUs, FPGAs (Field Programmable Gate Arrays), and ASICs (Application Specific Integrated Circuits).

[0083] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0084] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments rather than with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate.

[0085] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) A machine learning model used for recognizing an object captured by a camera is provided, and includes a data adjustment unit that adjusts the number of training data to suppress bias in the parameters, with respect to training data that includes at least one of the parameters of relative position and relative orientation between the camera and the object. A learning data processing system. (Note 2) The data adjustment unit adjusts the number of training data so as to suppress bias in the yaw angle among the parameters. The learning data processing system described in Appendix 1. (Note 3) The aforementioned object has a flat surface in part, The yaw angle is the yaw angle of the plane with respect to the imaging direction of the camera. The learning data processing system described in Appendix 2. (Note 4) The data adjustment unit adjusts the number of training data based on statistical information regarding the frequency of the parameters. A learning data processing system as described in any one of the items 1 to 3 in the appendix. (Note 5) The data adjustment unit adjusts the number of training data by deleting similar training data from the training data that includes the parameter having a frequency exceeding a predetermined upper limit. The learning data processing system described in Appendix 4. (Note 6) The data adjustment unit deletes the learning data in which the distance of the position vector of the relative position is less than or equal to a predetermined value. The learning data processing system described in Appendix 5. (Note 7) When the machine learning model performs training using one or more batches, the system further includes a batch data extraction unit that collects the training data within a batch by extracting the training data from part or all of the batch in a manner that suppresses bias in the parameters. A learning data processing system as described in any one of the items 1 to 6 of the appendix. (Note 8) The batch data extraction unit extracts the training data in such a way as to suppress the bias of the parameters in the training data adjusted by the data adjustment unit. The learning data processing system described in Appendix 7. (Note 9) The aforementioned object is approximately a rectangular parallelepiped. A learning data processing system as described in any one of the items 3 to 8 of the appendix. (Note 10) The aforementioned object is a pallet. The learning data processing system described in Appendix 9. (Note 11) The data adjustment unit adjusts the number of training data so as to suppress the bias in the relative posture. The learning data processing system described in Appendix 1. (Note 12) The data adjustment unit adjusts the number of training data so as to suppress the bias in the relative positions. The learning data processing system described in Appendix 1. (Note 13) The data adjustment unit calculates the lower limit of the frequency and adjusts the number of training data that include the parameter having a frequency exceeding the lower limit. A learning data processing system as described in any one of the items 3 to 12 of the appendices. (Note 14) The batch data extraction unit classifies the training data into a plurality of classifications based on the frequency, and extracts the training data from the plurality of classifications. A learning data processing system as described in any one of the items 7 to 13 of the appendices. (Note 15) The batch data extraction unit extracts the training data such that the bias of the parameters within the batch is evenly distributed. A learning data processing system as described in any one of the items 7 to 14 of the appendices. (Note 16) Computers Used in one or more machine learning models for recognizing objects captured by a camera, and analyzing training data that includes at least one of the parameters of relative position and relative orientation between the camera and the object, The number of training data is adjusted to suppress bias in the parameters. Method for processing training data. (Note 17) A step of analyzing training data used in one or more machine learning models for recognizing objects captured by a camera, which includes at least one of the parameters of relative position and relative orientation between the camera and the object, A step of adjusting the number of training data so as to suppress bias in the parameters, A program that causes a computer to execute something.

[0086] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 15 that are dependent on Appendice 1 may also be dependent on Appendices 16 and 17 in the same manner as those described in Appendices 2 to 15. Some or all of the elements described in any appendice may be applicable to various hardware, software, recording means, systems, and methods for recording software. [Explanation of symbols]

[0087] 1. Training Data Processing System 2. Data Adjustment Unit 10. Training Data Processing System 11. Learning Data List Acquisition Unit 12. Relative Position and Orientation Statistical Information Calculation Unit 13 Data Adjustment Section 14. Batch Data Extraction Unit 15 Machine Learning Department 20. Training Data Processing System 21 processors 22 memory

Claims

1. A machine learning model used for recognizing an object captured by a camera, comprising a data adjustment unit that adjusts the number of training data points to suppress bias in the parameters, with respect to training data that includes at least one of the parameters of relative position and relative orientation between the camera and the object. A learning data processing system.

2. The data adjustment unit adjusts the number of training data so as to suppress bias in the yaw angle among the parameters. The learning data processing system according to claim 1.

3. The aforementioned object has a flat surface in a part thereof, The yaw angle is the yaw angle of the plane with respect to the imaging direction of the camera. The learning data processing system according to claim 2.

4. The data adjustment unit adjusts the number of training data based on statistical information regarding the frequency of the parameters. The learning data processing system according to claim 1.

5. The data adjustment unit adjusts the number of training data by deleting similar training data from the training data that includes the parameter having a frequency exceeding a predetermined upper limit. The learning data processing system according to claim 4.

6. The data adjustment unit deletes the learning data in which the distance of the position vector of the relative position is less than or equal to a predetermined value. The learning data processing system according to claim 5.

7. When the machine learning model performs training using one or more batches, the system further includes a batch data extraction unit that collects the training data within a batch by extracting the training data from part or all of the batch in a manner that suppresses bias in the parameters. A learning data processing system according to any one of claims 1 to 6.

8. The batch data extraction unit extracts the training data in such a way as to suppress the bias of the parameters in the training data adjusted by the data adjustment unit. The learning data processing system according to claim 7.

9. Computers Used in one or more machine learning models for recognizing objects captured by a camera, and analyzing training data that includes at least one of the parameters of relative position and relative orientation between the camera and the object, The number of training data is adjusted to suppress bias in the parameters. Method for processing training data.

10. A step of analyzing training data used in one or more machine learning models for recognizing objects captured by a camera, which includes at least one of the parameters of relative position and relative orientation between the camera and the object, A step of adjusting the number of training data so as to suppress bias in the parameters, A program that causes a computer to execute something.

Images