Inference calculation processing device and inference calculation processing method

Abstract

The robot does not remain idle for extended periods but performs inference calculations within short timeframes. An inference calculation processing device inputs inference data into a learned model to perform inference calculations on the inference data. This device includes: an acquisition unit that acquires the inference data and the learned model; a preprocessing unit that divides the acquired inference data into multiple inference sub-data sets through batch processing; and an execution unit that optimizes the inference calculation processing order of the multiple inference sub-data sets and performs inference calculations on the inference data based on at least a portion of each of the multiple inference sub-data sets and the learned model, according to the optimized inference calculation processing order.

Description

Technical Field

[0001] This invention relates to an inference calculation processing apparatus and an inference calculation processing method. Background Technology

[0002] To enable high-speed computational processing for applications utilizing deep learning, the use of most GPUs (Graphics Processing Units) is essential. However, the rapid pace of GPU updates poses challenges in integrating them into commercial products due to long-term maintenance considerations. Furthermore, the high cost of GPUs also increases implementation costs. Additionally, for applications utilizing deep learning for the retrieval of bulk workpieces on factory production lines, achieving the target cycle time at low cost requires the use of conventional, inexpensive CPUs (Central Processing Units) for high-speed inference execution.

[0003] Regarding this, the following techniques are known: generating and displaying a distance image of a bulk workpiece; performing machine learning by using 3D point group data near a taught removal position in the displayed distance image as input data and using evaluation values ​​corresponding to the teaching or the success or failure of the removal action as labels; generating a learned model that outputs evaluation values ​​of the input 3D point group data; and selecting a removal position corresponding to the selected distance image with a high evaluation value based on the evaluation values ​​output by inputting 3D point group data of a distance image cropped to a specified area size into the generated learned model. For example, see Patent Document 1.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-58960 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, although a distance image of a specified region is cropped from the distance image, once the size of the input data to be used in machine learning is determined, it is necessary to crop the distance image from the distance image so that the size of the distance image cropped in the specified region input to the learned model (hereinafter also referred to as the "cropped image") is the same as the size of the input data to be used in machine learning.

[0009] Furthermore, each of the 3D point clusters of all cropped images (candidate extraction locations) is input into the learning model, and its respective evaluation value is output. The extraction location corresponding to the cropped image with the highest evaluation value is then selected. That is, since all cropped images (candidate extraction locations) are input into the learned model for inference calculations, useless inference calculations are performed on cropped images (candidate extraction locations) with low evaluation values ​​that are not selected / used even if inference calculations are performed.

[0010] In particular, for workpieces with complex shapes, high-resolution (i.e., large data size) distance images and 3D point group data are required. Furthermore, the data size of distance images and 3D point group data also increases with the size of the workpiece. Therefore, when performing inference calculations on large-scale data, the method for calculating all cropped images (extracting position candidates) includes useless computational processing time for cropped images (extracting position candidates) with low evaluation values ​​that are not selected or used even when performing inference calculations. Consequently, the total inference time increases, during which time robots on the factory production line become idle, reducing production efficiency.

[0011] Therefore, it is expected that the robot will not remain idle for long periods of time but will perform inference calculations within a short period of time.

[0012] Methods for solving problems

[0013] One aspect of the inference calculation processing apparatus disclosed herein is an inference calculation processing apparatus that performs inference calculation processing on inference data input to a learned model. The apparatus comprises: an acquisition unit that acquires the inference data and the learned model; a preprocessing unit that divides the acquired inference data into multiple inference sub-data using batch processing; and an execution unit that optimizes the inference calculation processing order of the multiple inference sub-data and performs inference calculation processing on the inference data based on at least a portion of each of the multiple inference sub-data and the learned model, according to the optimized inference calculation processing order.

[0014] One aspect of the inference computation processing method disclosed herein is a computer-implemented inference computation processing method that inputs inference data into a learned model to perform inference computation processing of the inference data. The inference computation processing method comprises the following steps: an acquisition step, acquiring the inference data and the learned model; a preprocessing step, dividing the acquired inference data into multiple inference sub-data through batch processing; and an execution step, optimizing the inference computation processing order of the multiple inference sub-data, and performing inference computation processing of the inference data based on at least a portion of each of the multiple inference sub-data and the learned model according to the optimized inference computation processing order.

[0015] Invention Effects

[0016] According to one method, the robot can perform inferential calculations in a short period of time without being idle for a long time. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating an example of the structure of the robot system according to the first embodiment.

[0018] Figure 2 This is a functional block diagram illustrating a functional structural example of the learned model execution device of the first embodiment.

[0019] Figure 3 This is a flowchart illustrating the inference calculation processing of the learned model execution device.

[0020] Figure 4 This is a functional block diagram illustrating a functional structural example of the learned model execution device in the second embodiment.

[0021] Figure 5 This is a flowchart illustrating the inference calculation process of the learned model execution device.

[0022] Figure 6 This is a diagram illustrating an example of the structure of the robot system according to the third embodiment.

[0023] Figure 7 This is a functional block diagram illustrating a functional structural example of the learned model execution device in the third embodiment.

[0024] Figure 8 This is a flowchart illustrating the inference calculation process of the learned model execution device.

[0025] Figure 9 This is a functional block diagram illustrating the functional structure of a learned model execution device in a modified example of the third embodiment, where training image data has also been acquired.

[0026] Figure 10This is a functional block diagram representing a functional structure example of a learned model execution device when the inference data is sound data.

[0027] Figure 11 This is a functional block diagram representing a functional structure example of a learned model execution device when the inference data is character data.

[0028] Figure 12 This is a functional block diagram representing a functional structure example of a learned model execution device when the inference data is sound data.

[0029] Figure 13 This is a functional block diagram representing a functional structure example of a learned model execution device when the inference data is character data.

[0030] Figure 14 This is a functional block diagram representing a functional structure example of a learned model execution device when training image data has not yet been obtained. Detailed Implementation

[0031] The first to third embodiments will be described in detail with reference to the accompanying drawings.

[0032] Here, the commonality among the various embodiments lies in the structure in which the robot performs inference calculations in a short time without idling for a long time, using a learned model that determines the removal positions of multiple workpieces in a bulk overlapping state.

[0033] However, in the inference calculation processing, in the first embodiment, the inference image data obtained by batch processing multiple workpieces in a scattered and overlapping state is segmented according to the size of the training image data used in the machine learning that generates the learned model. Feature quantities are extracted by image feature analysis of the multiple inference sub-image data generated by the segmentation. Based on the matching results of the feature quantities of each of the multiple inference sub-image data with the feature quantities from the image feature analysis of the training image data, an evaluation score is assigned to each of the multiple inference sub-image data. The inference calculation processing order of the multiple inference sub-image data is optimized according to the priority order of the assigned evaluation scores. In contrast, in the second embodiment, feature points are extracted by image processing of the inference image data, the inference image data is segmented by batch processing according to the number of feature points, and an evaluation score is assigned to each of the multiple inference sub-image data according to the number of feature points. This is different from the first embodiment. Furthermore, in the third embodiment, it differs from the first and second embodiments in the following aspects: based on the three-dimensional point group data (or distance image data) of the workpieces overlapping in a bulk manner obtained by a three-dimensional measuring instrument, the inference image data is segmented by batch processing, and an evaluation score is assigned to each of the multiple inference sub-image data based on the height from the bottom of the container (hereinafter also referred to as "the specified height") in each of the segmented multiple inference sub-image data.

[0034] Hereinafter, the first embodiment will be described in detail first, and then the parts of the second and third embodiments that differ from the first embodiment will be described.

[0035] <First Implementation>

[0036] Figure 1 This diagram illustrates an example of the structure of the robot system 1 according to the first embodiment. Here, it illustrates the case where a learned model generated by machine learning based on image data is executed when the robot removes workpieces that are loosely packed in a container. However, the present invention is not limited to the case where a learned model generated by machine learning based on image data is executed when the robot removes workpieces that are loosely packed in a container. For example, it is not limited to the robot's actions; it can also be applied to the case of executing a learned model generated by machine learning based on image data in a system that performs inferences for performing arbitrary tasks based on image data.

[0037] Furthermore, for example, as described later, in a system that performs inference based on sound data for performing arbitrary tasks, it can also be applied to situations where a learned model generated by machine learning based on sound data is executed. Similarly, in a system that performs inference based on character data for performing arbitrary tasks, it can also be applied to situations where a learned model generated by machine learning based on character data is executed.

[0038] like Figure 1 As shown, the robot system 1 includes a learned model execution device 10 as an inference calculation processing device, a robot control device 20, a robot 30, a camera device 40, multiple workpieces 50, and a container 60.

[0039] The learned model execution device 10, robot control device 20, robot 30, and imaging device 40 can also be directly connected to each other via a connection interface (not shown). Furthermore, the learned model execution device 10, robot control device 20, robot 30, and imaging device 40 can also be interconnected via a network (not shown), such as a LAN (Local Area Network) or the Internet. In this case, the learned model execution device 10, robot control device 20, robot 30, and imaging device 40 are equipped with a communication unit (not shown) for communicating with each other via this connection. Additionally, for ease of explanation, Figure 1 The learned model execution device 10 and the robot control device 20 are depicted independently. In this case, the learned model execution device 10 may also be constructed from a computer, for example. It is not limited to such a structure. For example, the learned model execution device 10 may also be installed inside the robot control device 20 and integrated with the robot control device 20.

[0040] The robot control device 20 is a device known to those skilled in the art for controlling the actions of the robot 30. The robot control device 20 receives, for example, information from the learned model execution device 10 regarding the removal position of a workpiece 50 selected by the learned model execution device 10 from a batch of workpieces 50. The robot control device 20 generates control signals for controlling the actions of the robot 30 to remove the workpiece 50 located at the removal position received from the learned model execution device 10. The robot control device 20 then outputs the generated control signals to the robot 30.

[0041] Furthermore, as described later, the robot control device 20 may also include a learned model execution device 10.

[0042] Robot 30 is a robot that operates under the control of robot control device 20. Robot 30 includes a base for rotating about a vertical axis, a moving and rotating arm, and a handle 31 mounted on the arm to hold the workpiece 50. Furthermore, in Figure 1 In the robot 30, the extraction hand 31 is equipped with an air-adsorption type extraction hand, but it can also be equipped with a gripping type extraction hand, or a magnetic hand that uses magnetic force to extract iron workpieces.

[0043] According to the control signal output by the robot control device 20, the robot 30 drives the arm and the take-out hand 31 to move the take-out hand 31 to the take-out position selected by the learned model execution device 10, and holds the bulk workpiece 50 and takes it out from the container 60.

[0044] Furthermore, the destination of the removed workpiece 50 is not shown in the diagram. Also, the specific structure of the robot 30 is well known to those skilled in the art, therefore a detailed description is omitted.

[0045] In addition, the learned model execution device 10 and robot control device 20 establish a correspondence between the mechanical coordinate system used to control the robot 30 and the camera coordinate system representing the removal position of the workpiece 50 through pre-performed calibration.

[0046] The imaging device 40 is a digital camera or similar device that captures two-dimensional image data by projecting the bulk workpiece 50 inside the container 60 onto a plane perpendicular to the optical axis of the imaging device 40. The image data captured by the imaging device 40 can also be visible light images such as RGB color images, grayscale images, or depth images. Furthermore, the imaging device 40 can be configured to include an infrared sensor to capture thermal images, or it can be configured to include an ultraviolet sensor to capture ultraviolet images for inspecting damage, spots, etc., on the surface of an object. Additionally, the imaging device 40 can be configured to include an X-ray camera sensor to capture X-ray images, or it can be configured to include an ultrasonic sensor to capture ultrasonic images.

[0047] Furthermore, as described later, the imaging device 40 may also be a stereo camera or other three-dimensional measuring instrument.

[0048] The workpiece 50 is placed haphazardly in a bulk state within the container 60. The workpiece 50 is not particularly limited in shape, as long as it can be held by the take-out hand 31 mounted on the arm of the robot 30.

[0049] <Learned Model Execution Device 10>

[0050] Figure 2 This is a functional block diagram illustrating a functional structural example of the learned model execution device 10 in the first embodiment.

[0051] The learned model execution device 10 is a computer known to those skilled in the art, such as... Figure 2 As shown, the system includes a control unit 11. Furthermore, the control unit 11 includes an acquisition unit 110, a preprocessing unit 120, and an execution unit 130. The acquisition unit 110 also includes a data storage unit 111. The preprocessing unit 120 includes a batch processing unit 121. The execution unit 130 includes a feature analysis unit 131, an evaluation score calculation unit 132, an optimization calculation unit 133, an inference calculation processing unit 134, and an inference result storage unit 135.

[0052] <Control Unit 11>

[0053] The control unit 11 includes a CPU (Central Processing Unit), ROM, RAM (Random Access Memory), CMOS (Complementary Metal Oxide Semiconductor) memory, etc., which are configured to communicate with each other via a bus, as is known to those skilled in the art.

[0054] The CPU is the processor that controls the learned model execution device 10 as a whole. The CPU reads the system program and application program stored in ROM via the bus, and controls the learned model execution device 10 as a whole according to the system program and application program. Thus, as... Figure 2 As shown, the control unit 11 is configured to perform the functions of the acquisition unit 110, the preprocessing unit 120, and the execution unit 130. The acquisition unit 110 is configured to perform the function of the data storage unit 111. The preprocessing unit 120 is configured to perform the function of the batch processing unit 121. The execution unit 130 is configured to perform the functions of the feature analysis unit 131, the evaluation score calculation unit 132, the optimization calculation unit 133, the inference calculation processing unit 134, and the inference result storage unit 135. Temporary calculation data, display data, and other data are stored in RAM. Furthermore, the CMOS memory is backed up by a battery (not shown) and is configured as a non-volatile memory that retains its storage state even when the power to the learned model execution device 10 is disconnected.

[0055] <Acquisition Section 110>

[0056] The acquisition unit 110 acquires, for example, image data from the imaging device 40 as inference data, and acquires the learned model and the training image data used when generating the learned model through machine learning from the database 70 on the cloud or edge device.

[0057] The acquisition unit 110 may also be configured to include a data storage unit 111 such as an HDD or USB memory, and to store the acquired learned model in the data storage unit 111. For example, the acquisition unit 110 may also acquire the learned model recorded in a recording medium such as an HDD or USB memory from a database 70 on a cloud or edge device via a network such as a LAN, and copy the acquired learned model to the data storage unit 111 for storage.

[0058] Alternatively, the acquisition unit 110 may acquire training image data recorded on recording media such as HDD or USB memory from the database 70 on the cloud or edge device via a network such as LAN, and copy the acquired training image data to the data storage unit 111 for storage.

[0059] Furthermore, for example, the acquisition unit 110 can acquire image data captured by the imaging device 40, and copy the acquired image data as inference image data to the data storage unit 111 for storage.

[0060] In addition, the acquisition unit 110 acquires image data from the imaging device 40, but as will be described later, it can also acquire three-dimensional point group data or distance image data, etc.

[0061] <Preprocessing Section 120>

[0062] The preprocessing unit 120 may also include a batch processing unit 121, which is configured to perform batch processing on the inference data based on the training image data obtained by the acquisition unit 110 and divide it into multiple inference sub-image data.

[0063] Specifically, the batch processing unit 121 may, for example, divide the inference image data into multiple inference sub-image data by batch processing based on the data size of the training image data used in machine learning.

[0064] <Execution Department 130>

[0065] The execution unit 130 may also be configured to optimize the inference calculation processing order of the multiple inference sub-image data segmented by the preprocessing unit 120 through batch processing, and to perform inference calculation processing of the inference data according to the optimized inference calculation processing order, based on each of the multiple inference sub-image data and the learned model, before reaching the pre-specified search target.

[0066] For example, the execution unit 130 may also be configured to perform image feature analysis on each of the nearby images of the labeled teaching positions on the training image data and each of the multiple inference sub-image data segmented by the preprocessing unit 120 in batch processing, assign an evaluation score to each of the multiple inference sub-image data based on the matching results of the extracted feature quantities, and optimize the inference calculation processing order of the multiple inference sub-image data based on the priority order determined by the magnitude of the evaluation scores assigned to each of the multiple inference sub-image data.

[0067] Specifically, the feature analysis unit 131 of the execution unit 130, for example, attaches labels to the training image data indicating the positions where the workpiece 50 can be removed, and therefore performs image processing on the nearby image regions in the training image data that contain the removal positions indicated by the labels. The feature analysis unit 131 extracts specific feature quantities (hereinafter also referred to as "local feature quantities") A through image processing. In addition, the feature analysis unit 131 also performs image processing on the n inference sub-image data IMG1, IMG2, ..., IMGn segmented in the batch processing of the preprocessing unit 120, and extracts local feature quantities (n is an integer of 2 or more). For example, the feature analysis unit 131 extracts local feature quantities A11 and A12 from the inference sub-image data IMG1, local feature quantities A21, A22, and A23 from the inference sub-image data IMG2, and local feature quantities A31, A32, A33, and A34 from the inference sub-image data IMG3. The feature analysis unit 131 performs matching processing on each of the extracted local feature quantities A11, A12, A21, A22, A23, A31, A32, A33 and A34 with the local feature quantity A of the training image data, and outputs the analysis result data of the matching processing to the evaluation score calculation unit 132 described later.

[0068] The evaluation score calculation unit 132 of the execution unit 130 receives the analysis result data output by the feature analysis unit 131. For example, if the inference sub-image data IMG2 contains local features with high matching degree (e.g., A22), the evaluation score calculation unit 132 assigns a higher evaluation score (e.g., 70 points) to the inference sub-image data IMG2. Furthermore, if, for example, a single inference sub-image data IMG3 contains multiple local features with high matching degree (e.g., A32, A34), the evaluation score calculation unit 132 assigns a higher evaluation score (e.g., 80 points) to the inference sub-image data IMG3. The evaluation score calculation unit 132 then outputs the assigned evaluation scores to the optimization calculation unit 133, which will be described later.

[0069] The optimization calculation unit 133 of the execution unit 130 assigns a high priority order to the inference calculations based on the evaluation score information output by the evaluation score calculation unit 132, starting with the inference sub-image data with the highest evaluation score. The optimization calculation unit 133 optimizes the inference calculation processing order of the n inference sub-image data segments obtained in the batch processing of the preprocessing unit 120, generating a processing order list of the optimized inference calculation processing order. The optimization calculation unit 133 outputs the generated processing order list to the inference calculation processing unit 134, which will be described later.

[0070] In addition, the optimization calculation unit 133 can also remove inference sub-image data with low evaluation scores from the processing order list so that inference calculation processing is not performed on them.

[0071] The inference calculation processing unit 134 of the execution unit 130 performs the required amount of inference calculation processing on the n inference sub-image data in descending order of the processing order list, based on the information of the processing order list received from the optimization calculation unit 133, the n inference sub-image data segmented in the batch processing of the preprocessing unit 120, and the learned model obtained by the acquisition unit 110, before reaching the pre-specified search target (e.g., finding the removal positions of 10 workpieces 50). The inference calculation processing unit 134 outputs the inference result data of the inference calculation processing to the inference result storage unit 135, which will be described later.

[0072] The inference result storage unit 135 of the execution unit 130 receives and stores the inference result data from the inference calculation and processing unit 134.

[0073] In this way, the probability of finding a better candidate position with the same features as the taught position on the training image data is high among the inference sub-image data with high evaluation scores. Therefore, the learned model execution device 10 prioritizes inference calculation processing based on the inference sub-image data with high evaluation scores. As a result, the learned model execution device 10 quickly searches for the target number of retrieved items 50 (hereinafter also referred to as the "prescribed number of candidates") and quickly ends the inference calculation processing, thereby shortening the inference calculation processing time. In other words, the probability of finding a better candidate position with the same features as the taught position on the training image data among the inference sub-image data with low evaluation scores is low. Therefore, by not performing inference calculation processing on the inference sub-image data with low evaluation scores, useless inference calculation processing can be eliminated, thereby shortening the inference calculation processing time. That is, the learned model execution device 10 optimizes the inference calculation processing order based on the priority order of evaluation scores, thereby achieving high-speed inference calculation processing.

[0074] Furthermore, the number of candidates should preferably be determined appropriately based on the required accuracy and processing speed of the inference calculation, but it can also be determined based on the production requirements of the factory's production line.

[0075] <Inference calculation processing of learned model execution device 10>

[0076] Next, the operation of the inference calculation processing of the learned model execution device 10 in this embodiment will be explained.

[0077] Figure 3 This is a flowchart illustrating the inference calculation processing of the learned model execution device 10.

[0078] In step S11, the acquisition unit 110 acquires the learned model and training image data from the database 70.

[0079] In step S12, the acquisition unit 110 acquires inference image data from the imaging device 40.

[0080] In step S13, the batch processing unit 121 of the preprocessing unit 120 divides the inference image data obtained in step S12 into n inference sub-image data by batch processing based on the training image data.

[0081] In step S14, the feature analysis unit 131 of the execution unit 130 performs image feature analysis on the training image data and the n inference sub-image data, and extracts local feature quantities from the training image data and the n inference sub-image data.

[0082] In step S15, the feature analysis unit 131 performs matching processing between the local feature quantities of the training image data and the local feature quantities of each inference sub-image data, and outputs the analysis result data of the matching processing to the evaluation score calculation unit 132.

[0083] In step S16, the evaluation score calculation unit 132 assigns an evaluation score corresponding to the matching degree between the local feature quantities of the training image data to each of the n inference sub-image data based on the analysis result data output from step S15.

[0084] In step S17, the optimization calculation unit 133 optimizes the inference calculation processing order of multiple inference sub-image data for inference calculation processing based on the evaluation score information given in step S16, and generates a processing order list.

[0085] In step S18, the inference calculation processing unit 134 performs inference calculation processing based on the information of the processing order list generated in step S17, the inference sub-image data, and the learned model.

[0086] In step S19, the inference calculation processing unit 134 determines whether the number of candidate extraction positions searched through the inference calculation processing in step S18 has reached a predetermined number of candidates. If the number of candidate extraction positions has reached the predetermined number of candidates, the inference calculation processing ends. On the other hand, if the number of candidate extraction positions is less than the predetermined number of candidates, the processing returns to step S18.

[0087] Through the above, the learned model execution device 10 of the first embodiment obtains inference data from the imaging device 40, a learned model from the database 70, and training image data. Based on the scale of the training image data, the learned model execution device 10 divides the inference image data into n inference sub-image data through batch processing. The learned model execution device 10 extracts local features from each of the training image data and the n inference sub-image data, and performs matching processing between the extracted local features of the training image data and the local features of each inference sub-image data. Based on the analysis results of the matching processing, the learned model execution device 10 assigns an evaluation score to each of the n inference sub-image data corresponding to the matching degree with the local features of the training image data, and optimizes the inference calculation processing order of the multiple inference sub-image data for inference calculation processing based on the evaluation score information.

[0088] Thus, the learned model execution device 10 is able to perform inference calculations in a short time without keeping the robot 30 idle for a long time.

[0089] Furthermore, the learned model execution device 10 can execute inferences at high speed using common, inexpensive CPU devices, enabling the high production efficiency required by the production line to be achieved at low cost.

[0090] The first embodiment has been described above.

[0091] <Second Implementation>

[0092] Next, the second embodiment will be described. As described above, in the inference calculation process, in the first embodiment, the inference image data is segmented by batch processing according to the size of the training image data. Feature quantities are extracted by image feature analysis for each of the multiple inference sub-image data generated by the segmentation. Based on the matching result of the feature quantities of each of the multiple inference sub-image data and the feature quantities of the training image data, an evaluation score is assigned to each of the multiple inference sub-image data. The inference calculation processing order of the multiple inference sub-image data is optimized according to the priority order of the assigned evaluation score values. In contrast, in the second embodiment, the difference from the first embodiment is that image processing is performed on the inference image data, for example, specific feature points are extracted. The inference image data is segmented by batch processing according to the number of extracted feature points. An evaluation score is assigned to each of the multiple inference sub-image data according to the number of feature points contained in each region.

[0093] Therefore, the learned model execution device 10a is able to perform inference calculations in a short time without keeping the robot 30 idle for a long time.

[0094] The second embodiment will be described below.

[0095] and Figure 1 Similarly to the first embodiment, the robot system 1 of the second embodiment has a learned model execution device 10a, a robot control device 20, a robot 30, a camera device 40, multiple workpieces 50, and a container 60.

[0096] <Learned Model Execution Device 10a>

[0097] Figure 4 This is a functional block diagram illustrating a functional structural example of the learned model execution device 10a according to the second embodiment. Furthermore, for devices having... Figure 2 The elements of the learned model execution device 10 that have the same functions are labeled with the same reference numerals, and detailed descriptions are omitted.

[0098] Similar to the learned model execution device 10 of the first embodiment, the learned model execution device 10a includes a control unit 11a. The control unit 11a also includes an acquisition unit 110a, a preprocessing unit 120a, and an execution unit 130a. The acquisition unit 110a includes a data storage unit 111. The preprocessing unit 120a includes a batch processing unit 121a and an image processing unit 122. Furthermore, the execution unit 130a includes an evaluation score calculation unit 132a, an optimization calculation unit 133, an inference calculation processing unit 134, and an inference result storage unit 135.

[0099] <Acquisition Section 110a>

[0100] The acquisition unit 110a acquires image data, for example, from the imaging device 40 as inference data, and acquires the learned model from the database 70 on the cloud or edge device. The acquisition unit 110a saves the acquired learned model and image data in the data storage unit 111.

[0101] In addition, the data storage unit 111 has the same functions as the data storage unit 111 in the first embodiment.

[0102] <Preprocessing Section 120a>

[0103] The preprocessing unit 120a may, for example, acquire image data from the data storage unit 111 of the acquisition unit 110a as inference image data, perform image processing on the acquired inference image data, and divide the inference image data into multiple inference sub-image data by batch processing based on the image processing results.

[0104] Specifically, for example, the image processing unit 122 of the preprocessing unit 120a can perform image processing on the inference image data, extracting features such as edges, corners, and feature points, and outputting them as image processing result data. The following is an example illustrating how the image processing unit 122 extracts specific feature points from the entire region of the inference image data. Figure 1 As shown, for example, when the inference image data is an image obtained by taking pictures of the workpieces 50 in bulk inside the container 60, it is highly likely that there are almost no workpieces 50 in the local image area with a small number of extracted feature points, and that a large plane with uniform brightness and pixel values ​​(e.g., the bottom of the container 60) is captured. In this case, even if the learned model execution device 10a performs inference calculation processing on this image area, it is highly likely that it will not be able to find the workpieces 50 that were taken out, and thus time is wasted on useless inference calculation processing. That is, by analyzing the positional distribution of feature points extracted by the image processing unit 122, the learned model execution device 10a can determine local image areas with a larger concentration of feature points, i.e., local image areas with a larger number of workpieces 50.

[0105] Furthermore, the example described for the image processing unit 122 is that it extracts feature points by performing image processing on image data for inference, but it is not limited to this. For example, the image processing unit 122 can also improve efficiency by changing the features extracted in image processing according to the actual shape of the workpiece 50.

[0106] The batch processing unit 121a performs the following processing: It segments the inference image by making a sub-image for inference if the feature points are concentrated in a local image region on the inference image; and it segments the inference image by making a sub-image for inference if the feature points are few or none. Therefore, it is also possible to generate multiple sub-image data for inference by segmenting the inference image data into multiple sub-images, with the latter being a sub-image for inference if the local image region is highly likely to contain many workpieces 50, and a sub-image for inference if the local image region contains few or none workpieces 50. These sub-images are then output to the execution unit 130a. Furthermore, regarding whether the feature points are concentrated or few in the inference image data, the batch processing unit 121a can also determine whether the number of feature points exceeds a threshold D1 as "many" and the number is less than a threshold D2 (D2 < D1) as "few".

[0107] Therefore, it is possible to distinguish between local image regions that should be prioritized for inference calculations and local image regions that cannot find the object even if inference calculations are performed, thus becoming useless. This enables batch processing optimization, allowing for efficient and smooth optimization of the inference calculation processing order, which will be described later.

[0108] That is, by distinguishing between inference sub-image data that are highly likely to be used for inference calculations and workpieces 50 that should be prioritized for inference calculations during the batch processing stage of inference image data, and inference sub-image data that are useless because they cannot be used for inference calculations even if inference calculations are performed, the execution unit 130a described later can efficiently and smoothly optimize the inference calculation processing order.

[0109] <Execution Department 130a>

[0110] The execution unit 130a may also be configured to optimize the inference calculation processing order of multiple inference sub-image data based on the image processing result data output by the image processing unit 122 of the preprocessing unit 120a, and perform inference calculation processing of the inference data according to the optimized inference calculation processing order, based on each of the required amount of multiple inference sub-image data and the learned model, before reaching the pre-specified search target.

[0111] Specifically, as described above, the batch processing unit 121a of the preprocessing unit 120a extracts feature points from the inference image obtained by photographing the workpiece 50 bulk inside the container 60, divides the inference image data into multiple segments, and generates and outputs multiple inference sub-image data. Therefore, the execution unit 130a may also include an evaluation score calculation unit 132a, which assigns a high evaluation score to the inference sub-image data with a large number of feature points and a low evaluation score to the inference sub-image data with a small number of feature points.

[0112] Therefore, with Figure 2 Similarly, the optimization calculation unit 133 of the execution unit 130a generates and outputs a processing order list in descending order of the evaluation scores assigned by the evaluation score calculation unit 132a.

[0113] and Figure 2 Similarly, the inference calculation processing unit 134 of the execution unit 130a performs inference calculation processing based on the information of the processing order list, multiple inference sub-image data, and the learned model, and saves the inference result data in the inference result storage unit 135.

[0114] In this way, the learned model execution device 10a prioritizes performing inference calculation processing on inference sub-image data with a large number of extracted feature points and a high probability of the existence of a large number of workpieces 50, and does not perform inference calculation processing on inference sub-image data with a small number of feature points, few workpieces 50, or no workpieces 50 at all. This can shorten the inference calculation processing time for finding the predetermined number of candidate workpieces 50 to be extracted from the inference image data.

[0115] Therefore, the learned model execution device 10a can find the predetermined number of candidate workpieces 50 to be taken out from the inference image data, and can quickly end the inference calculation process, thereby shortening the inference calculation processing time.

[0116] <Inference calculation processing of learned model execution device 10a>

[0117] Next, the operations involved in the inference calculation processing of the learned model execution device 10a in this embodiment will be explained.

[0118] Figure 5 This is a flowchart illustrating the inference calculation processing of the learned model execution device 10a.

[0119] Furthermore, the processing in steps S26 to S28 is similar to... Figure 3 Steps S17 to S19 of the first embodiment are the same, and the description is omitted.

[0120] In step S21, the acquisition unit 110a retrieves the learned model from the database 70.

[0121] In step S22, the acquisition unit 110a acquires inference image data from the imaging device 40.

[0122] In step S23, the image processing unit 122 of the preprocessing unit 120a performs image processing on the inference image data obtained in step S22 and extracts feature points from the entire region of the inference image data.

[0123] In step S24, the batch processing unit 121a of the preprocessing unit 120a performs the following processing: based on the feature points extracted in step S23, the inference image is segmented in such a way that local image regions in the feature point set on the inference image data obtained in step S22 are taken as one inference sub-image, and the inference image data is segmented in such a way that local image regions with few or no feature points are taken as one inference sub-image.

[0124] In step S25, the evaluation score calculation unit 132a assigns an evaluation score to each inference sub-image data segmented in step S24 based on the number of feature points.

[0125] Through the above, the learned model execution device 10a of the second embodiment acquires inference image data from the imaging device 40 and a learned model from the database 70. The learned model execution device 10a performs image processing on the acquired inference image data, extracting feature points from the entire region of the inference image data. Based on the extracted feature points, it segments the inference image data into inference sub-image data with a set of feature points and inference sub-image data with few or no feature points. The learned model execution device 10a assigns an evaluation score to each inference sub-image data generated by segmenting the inference image data according to the number of feature points in each inference sub-image region, and optimizes the inference calculation processing order of the multiple inference sub-image data for inference calculation processing based on the assigned evaluation scores.

[0126] Therefore, the learned model execution device 10a is able to perform inference calculations in a short time without keeping the robot 30 idle for a long time.

[0127] Furthermore, the learned model execution device 10a can execute inferences at high speed using common, inexpensive CPU devices, enabling the high production efficiency required by the production line to be achieved at low cost.

[0128] The second embodiment has been described above.

[0129] <Third Implementation Method>

[0130] Next, the third embodiment will be described. As described above, in the inference calculation process, in the first embodiment, the inference image data is segmented by batch processing to match the size of the training image data. Feature quantities are extracted by image feature analysis for each of the segmented inference sub-image data. Based on the matching result between the feature quantities of each of the inference sub-image data and the feature quantities of the training image data, an evaluation score is assigned to each of the inference sub-image data. The inference calculation processing order of the multiple inference sub-image data is optimized based on the priority order of the assigned evaluation score values. In the second embodiment, feature points are extracted by image processing of the inference image data. The inference image data is segmented by batch processing according to the number of feature points. An evaluation score is assigned to each of the multiple inference sub-image data according to the number of feature points. In contrast, the third embodiment differs from the first and second embodiments in the following aspects: based on the three-dimensional point group data (or distance image data) of the workpiece in a scattered and overlapping state obtained by the three-dimensional measuring instrument 45, the inference image data is segmented by batch processing, and an evaluation score is assigned to each of the multiple inference sub-image data according to the specified height of each of the segmented multiple inference sub-image data.

[0131] Therefore, the learned model execution device 10b of the third embodiment can perform inference calculation processing in a short time without making the robot 30 standby for a long time.

[0132] The third embodiment will be described below.

[0133] Figure 6 This is a diagram illustrating an example of the structure of the robot system 1A according to the third embodiment. Furthermore, for systems having... Figure 1 The elements of robot system 1 that have the same functions are labeled with the same reference numerals, and detailed descriptions are omitted.

[0134] like Figure 6 As shown, the robot system 1A includes a learned model execution device 10b, a robot control device 20, a robot 30, a three-dimensional measuring instrument 45, multiple workpieces 50, and a container 60.

[0135] The robot control device 20 and the robot 30 have the same functions as the robot control device 20 and the robot 30 of the first embodiment.

[0136] The 3D measuring instrument 45 can also be configured to acquire 3D information (hereinafter also referred to as a "distance image") by using values ​​calculated from the distances between points on the surface of the bulk workpiece 50 within the container 60 and the plane perpendicular to the optical axis of the 3D measuring instrument 45 as pixel values. For example, Figure 6As shown, the pixel value of point A on workpiece 50 in the distance image is calculated from the distance (height of the 3D measuring instrument 45) between the 3D measuring instrument 45 and point A on workpiece 50 along the Z-axis of the 3D coordinate system (X, Y, Z). That is, the Z-axis of the 3D coordinate system is the optical axis of the 3D measuring instrument 45. Furthermore, the 3D measuring instrument 45 can also acquire 3D point group data of multiple workpieces 50 loaded in the container 60, for example, by using a stereo camera, a camera fixed to the fingertip of the robot 30, a combination of a camera and distance sensors such as a laser scanner and an acoustic sensor. This 3D point group data can be displayed in a 3D view that can be viewed from all viewpoints in 3D space, and is discretized data that allows for 3D confirmation of the overlap state of the multiple workpieces 50 loaded in the container 60.

[0137] Additionally, the 3D measuring instrument 45 can also acquire 3D point group data or distance images and 2D images such as grayscale images or RGB images. Furthermore, the robot system 1A may also include an imaging device (not shown) such as a digital camera, which is different from the 3D measuring instrument 45. The learned model execution device 10b acquires 3D point group data or distance images from the 3D measuring instrument 45 and acquires 2D images from the imaging device (not shown).

[0138] <Learned Model Execution Device 10b>

[0139] Figure 7 This is a functional block diagram illustrating a functional structural example of the learned model execution device 10b according to the third embodiment. Furthermore, for devices having... Figure 1 The elements of the learned model execution device 10 that have the same functions are labeled with the same reference numerals, and detailed descriptions are omitted.

[0140] The learned model execution device 10b includes a control unit 11b. The control unit 11b further includes an acquisition unit 110b, a preprocessing unit 120b, and an execution unit 130b. The acquisition unit 110b also includes a data storage unit 111. Furthermore, the preprocessing unit 120b includes a batch processing unit 121b and a 3D processing unit 123. The execution unit 130b further includes an evaluation score calculation unit 132b, an optimization calculation unit 133, an inference calculation processing unit 134, and an inference result storage unit 135.

[0141] <Acquisition Section 110b>

[0142] The acquisition unit 110b acquires, for example, 3D point group data or distance images and image data used for inference from the 3D measuring instrument 45. Additionally, the acquisition unit 110b acquires the learned model from the database 70 on a cloud or edge device. The acquisition unit 110b saves the acquired learned model, 3D point group data or distance images, and image data to the data storage unit 111.

[0143] In addition, the data storage unit 111 has the same functions as the data storage unit 111 in the first embodiment.

[0144] <Preprocessing Section 120b>

[0145] The preprocessing unit 120b can also be configured to optimize the batch processing of inference image data based on three-dimensional point group data or distance image data, generating multiple inference sub-image data.

[0146] Specifically, the 3D processing unit 123 of the preprocessing unit 120b compares, for example, inference image data obtained by measuring multiple workpieces 50 bulkly disposed within the container 60 in two and three dimensions with 3D point group data (or distance image data). As a result, the 3D processing unit 123 can analyze the distribution of the height (also called "prescribed height") of the real-world 3D position corresponding to each pixel position on the inference image data from the bottom of the container 60. The 3D processing unit 123 outputs the analysis results of the prescribed height distribution as 3D processing result data to the batch processing unit 121b.

[0147] Here, the batch processing unit 121b can also accept three-dimensional processing result data containing distribution information of a specified height, and divide the inference image data into multiple inference sub-image data by reflecting the difference between the specified height corresponding to each pixel position on the inference image data.

[0148] That is, in the batch processing stage of the inference image data, the inference image data is divided into inference sub-image data with more workpieces 50 that should be prioritized for inference calculation processing, and inference sub-image data with fewer workpieces 50 or none of them, even if inference calculation is performed. As a result, the execution unit 130b described later can optimize the inference calculation processing order efficiently and smoothly.

[0149] Furthermore, the aforementioned thresholds are preferably determined appropriately based on the required accuracy and processing speed of the inference calculation, but they can also be determined based on the production requirements of the factory's production line.

[0150] <Execution Department 130b>

[0151] The execution unit 130b may also be configured to optimize the inference calculation processing order of multiple inference sub-image data based on three-dimensional point group data or distance image data, and perform inference calculation processing of inference data according to the optimized inference calculation processing order, based on each of the required amount of multiple inference sub-image data and the learned model, before reaching the pre-specified search target.

[0152] For example, when multiple workpieces 50 are bulk-packed in the container 60, the evaluation score calculation unit 132b of the execution unit 130b can assign a high evaluation score to inference sub-image data with a specified height above a threshold, and prioritize inference calculation processing for such inference sub-image data. On the other hand, the evaluation score calculation unit 132b can also assign a low evaluation score to inference sub-image data with a specified height below the threshold, and optimize the inference calculation processing without performing inference calculation processing for such inference sub-image data. As a result, the learned model execution device 10b can identify the workpieces 50 of a pre-specified target candidate number from a large amount of image data in a shorter inference calculation processing time.

[0153] However, depending on the performance of the 3D measuring instrument 45 and the lighting conditions, there are sometimes areas where 3D measurement fails and 3D data cannot be obtained (data omission).

[0154] Therefore, the evaluation score calculation unit 132b can also reduce the priority of inference calculation processing for inference sub-image data with larger and more data omission areas in the corresponding 3D data, for multiple inference sub-image data output by the batch processing unit 121b of the preprocessing unit 120b. As a result, the learned model execution device 10b can eliminate useless inference calculation processing and shorten the inference calculation processing time.

[0155] Furthermore, the optimization calculation unit 133, the inference calculation processing unit 134, and the inference result storage unit 135 have the same functions as those of the optimization calculation unit 133, the inference calculation processing unit 134, and the inference result storage unit 135 in the first embodiment.

[0156] <Inference calculation processing of learned model execution device 10b>

[0157] Next, the operation of the inference calculation processing of the learned model execution device 10b in this embodiment will be explained.

[0158] Figure 8 This is a flowchart illustrating the inference calculation processing of the learned model execution device 10b.

[0159] Furthermore, the processing in steps S36 to S38 is similar to... Figure 3Steps S17 to S19 of the first embodiment are the same, and the description is omitted.

[0160] In step S31, the acquisition unit 110b retrieves the learned model from the database 70.

[0161] In step S32, the acquisition unit 110b acquires inference image data and three-dimensional point group data or distance image from the three-dimensional measuring instrument 45.

[0162] In step S33, the three-dimensional processing unit 123 analyzes the prescribed height distribution of the inference image data based on the three-dimensional point group data or distance image obtained in step S32, and outputs three-dimensional processing result data.

[0163] In step S34, the batch processing unit 121b divides the inference image data into multiple inference sub-image data based on the three-dimensional processing result data output in step S33.

[0164] In step S35, the evaluation score calculation unit 132b assigns an evaluation score to each inference sub-image data based on the three-dimensional point group data or distance image.

[0165] Through the above, the learned model execution device 10b of the third embodiment acquires inference image data and 3D point group data or distance image from the 3D measuring instrument 45, and acquires the learned model from the database 70. The learned model execution device 10b analyzes the distribution of a specified height in the inference image data based on the 3D point group data or distance image, and divides the inference image data into multiple inference sub-image data based on the analyzed 3D processing result data. The learned model execution device 10b assigns an evaluation score to each inference sub-image data based on the 3D point group data or distance image, and optimizes the inference calculation processing order of the multiple inference sub-image data based on the evaluation score information.

[0166] Thus, the learned model execution device 10b is able to perform inference calculations in a short time without keeping the robot idle for an extended period of time.

[0167] Furthermore, the learned model execution device 10b can execute inferences at high speed using common, inexpensive CPU devices, enabling the high production efficiency required by the production line to be achieved at low cost.

[0168] The third embodiment has been described above.

[0169] <Modification 1 of the Third Embodiment>

[0170] The third embodiment describes a scenario where the learned model execution device 10b also acquires and utilizes inference image data and three-dimensional point group data or distance images from the three-dimensional measuring instrument 45 to extract multiple bulk workpieces 50, but is not limited to this. It is also not limited to the type, shape, size, color, number, or loading state of the workpieces 50.

[0171] For example, it can also be applied to systems that infer the task of robot 30 to remove workpiece 50 from a flat state where multiple workpieces 50 do not overlap or from a loaded state of box-shaped workpieces 50 (e.g., a stacked state of corrugated cardboard).

[0172] <Modification 2 of the Third Embodiment>

[0173] In the third embodiment, the learned model execution device 10b only obtains the learned model from the database 70, but the training image data can also be obtained from the database 70.

[0174] Figure 9 This is a functional block diagram illustrating a modified example of the learned model execution device 10b in the case where training image data is also acquired. Furthermore, for devices having... Figure 7 The learned model execution device 10b and Figure 4 The elements of the learned model execution device 10a have the same functions as the elements in the drawings, and the same reference numerals are used, and detailed descriptions are omitted.

[0175] like Figure 9 As shown, the acquisition unit 110b acquires the learned model and training image data from the database 70.

[0176] In addition, the image processing unit 122 of the preprocessing unit 120a can perform image processing on the training image data and the inference image data in the same way as in the second embodiment, and the batch processing unit 121a divides the inference image data into multiple inference sub-image data by batch processing based on the image processing results.

[0177] Furthermore, the execution unit 130b, the evaluation score calculation unit 132b, the optimization calculation unit 133, the inference calculation processing unit 134, and the inference result storage unit 135 have the same functions as the evaluation score calculation unit 132b, the optimization calculation unit 133, the inference calculation processing unit 134, and the inference result storage unit 135 in the third embodiment.

[0178] Therefore, even when the learned model execution device 10b retrieves training image data from the database 70, the robot 30 will not remain idle for an extended period and can perform inference calculations in a short time. Furthermore, the learned model execution device 10b can execute inferences at high speed using common, inexpensive CPU devices, enabling the high production efficiency required by the production line to be achieved at low cost.

[0179] The first, second, and third embodiments have been described above. However, the learned model execution devices 10, 10a, and 10b are not limited to the above embodiments and include variations and improvements within the scope of achieving the objective.

[0180] <Variation Example 1>

[0181] In the first, second, and third embodiments described above, the learned model execution devices 10, 10a, and 10b are exemplified as devices different from the robot control device 20, but the robot control device 20 may also be configured to have some or all of the functions of the learned model execution devices 10, 10a, and 10b.

[0182] Alternatively, for example, the server may also include part or all of the acquisition unit 110, preprocessing unit 120, and execution unit 130 of the learned model execution device 10. Furthermore, for example, the server may also include part or all of the acquisition unit 110a, preprocessing unit 120a, and execution unit 130a of the learned model execution device 10a. Furthermore, for example, the server may also include part or all of the acquisition unit 110b, preprocessing unit 120b, and execution unit 130b of the learned model execution device 10b. Additionally, the functions of the learned model execution devices 10, 10a, and 10b can also be implemented using virtual server functions or the like in the cloud.

[0183] Furthermore, the learned model execution devices 10, 10a, and 10b can also be distributed processing systems that appropriately distribute the functions of the learned model execution devices 10, 10a, and 10b to multiple servers.

[0184] <Variation Example 2>

[0185] Furthermore, for example, the learned model execution device 10 of the first embodiment described above executes a learned model generated by machine learning based on image data when the robot 30 removes multiple workpieces 50 that are loosely packed in the container 60. It is configured to divide the inference image data into multiple inference sub-image data through batch processing based on the scale of the training image data used during machine learning, and perform matching processing between the training image data and each inference sub-image data based on local feature quantities extracted from the training image data and each of the multiple inference sub-image data. Furthermore, an embodiment is described in which the learned model execution device 10 assigns an evaluation score to each of the multiple inference sub-image data according to the matching degree, optimizes the inference calculation processing order of the multiple inference sub-image data according to the evaluation score, performs inference calculation processing on the multiple inference sub-image data according to the optimized inference calculation processing order, and calculates candidate removal positions for the workpieces 50 removed by the robot 30. However, this is not limited to the case where the learned model generated by machine learning based on image data is executed when the robot 30 removes multiple workpieces 50 that are loosely packed in the container 60. It is not limited to the type, shape, size, color, number, or loading status of workpiece 50.

[0186] For example, it can also be applied to systems that infer the task of robot 30 to remove workpiece 50 from a flat state where multiple workpieces 50 do not overlap or from a loaded state of box-shaped workpieces 50 (e.g., a stacked state of corrugated cardboard).

[0187] For example, the learned model execution device 10 can also replace the robot 30 in a system that retrieves multiple workpieces 50 loosely packed in a container 60, and be applied to a system for performing inference for, for example, performing arbitrary tasks based on sound data in a multi-person conversation or meeting. In this case, image data can also be replaced with sound data, and based on the sound training data (hereinafter also referred to as "training sound data"), the sound inference data (hereinafter also referred to as "inference sound data") is divided into multiple inference sub-sound data by batch processing, and the training sound data and each inference sub-sound data are matched based on the feature quantities extracted from the training sound data and the multiple inference sub-sound data. Furthermore, the learned model execution device 10 can also apply the same method as in the first embodiment, assigning an evaluation score to each of the multiple inference sub-voice data based on the matching degree, optimizing the inference calculation processing order of the multiple inference sub-voice data based on the evaluation scores, and performing inference calculation processing on the multiple inference sub-voice data based on the optimized inference calculation processing order. This allows for the identification of specified conversation content (e.g., content associated with specific target keywords such as "dog (inu)," "cat (neko)," and "weather (tenki)") from a large amount of sound data within a shorter inference calculation processing time. Therefore, in systems that infer and identify conversation content based on sound data from multi-person conversations or meetings, the learned model execution device 10 can avoid useless inference calculation processing on inference sub-voice data that includes or does not include specified sound data such as human voice data (in the case of dividing sound data into cells, these become multiple cell groups, hereinafter also referred to as "cell groups"), thus shortening the inference calculation processing time.

[0188] Alternatively, the learned model execution device 10 can also replace the robot 30 in a system that removes multiple workpieces 50 loosely packed in the container 60, for example, as a system for performing inferences for performing arbitrary tasks based on character data, applied to the case of executing a learned model generated by machine learning based on character data. Specifically, image data can also be replaced with character data, and the same method as in the first embodiment can be applied, based on the character training data (hereinafter also referred to as "training character data"), the character inference data (hereinafter also referred to as "inference character data") is divided into multiple inference sub-character data by batch processing, and the training character data and each inference sub-character data are matched based on the feature amount extracted from the training character data and each of the multiple inference sub-character data. Then, the learned model execution device 10 can also assign an evaluation score to each of the multiple inference sub-character data based on the matching degree, optimize the inference calculation processing order of the multiple inference sub-character data based on the evaluation scores, and perform inference calculation processing of the multiple inference sub-character data based on the optimized inference calculation processing order. This allows for the identification of specified training character data (e.g., content associated with specific target keywords such as "year", "month", and "day") from a large amount of character data within a short inference calculation processing time. Thus, when determining specified faults (e.g., the number of faulty parts, fault time, and location of the reducer) based on the fault history data (character data) of the robot 30, the learned model execution device 10 can avoid useless inference calculation processing of inference sub-character data in regions containing or not containing specified character data (e.g., target keywords such as "reducer") (which, in the case of unitizing character data, become multiple unit groups, hereinafter also referred to as "unit groups"), thereby shortening the inference calculation processing time.

[0189] The following, more specifically, will explain the cases where the data used for inference is (a) sound data and (b) character data.

[0190] (a) Regarding the case where the inference uses sound data.

[0191] Figure 10 This is a functional block diagram illustrating a functional structure example of a learned model execution device 10 when the inference data is sound data. Furthermore, for devices with... Figure 2 The elements of the learned model execution device 10 that have the same functions are labeled with the same reference numerals, and detailed descriptions are omitted.

[0192] In addition, in this case, the learned model takes in sub-voice data for each inference and outputs information such as the proportion of specified conversational content (e.g., specific target keywords such as "dog (inu)", "cat (neko)", "weather (tenki)") attached as teaching labels in the training voice data.

[0193] The acquisition unit 110 may acquire sound data as inference data from a combination of a microphone and a computer, or from a recording device 80 such as a computer, smartphone, tablet terminal, or camera with a built-in microphone.

[0194] and Figure 2 Similarly, in the case of inference image data, the batch processing unit 121 of the preprocessing unit 120 can also set the size of the training sound data to the minimum size, and generate and output the inference sound data as multiple inference sub-sound data by batch processing segmentation.

[0195] and Figure 2 Similarly, in the case of inference image data, the execution unit 130 can also be configured to assign an evaluation score to each of the multiple inference sub-sound data segmented in the batch processing of the preprocessing unit 120 based on the matching degree with the training sound data, and optimize the inference calculation processing order of the multiple inference sub-sound data based on the priority order determined by the magnitude of the assigned multiple evaluation scores.

[0196] Specifically, the feature analysis unit 131 performs feature analysis (e.g., frequency feature analysis) on the training audio data, extracting feature quantities (hereinafter also referred to as "frequency feature analysis results") B of the audio data of the prescribed conversation content (e.g., specific target keywords such as "dog (inu)", "cat (neko)", "weather (tenki)") attached as teaching tags. Furthermore, the feature analysis unit 131 also performs frequency feature analysis on the m inference sub-audio data ABD1, ABD2, ... AUDm segmented in the batch processing of the preprocessing unit 120, extracting frequency feature analysis results (m is an integer of 2 or more). For example, the feature analysis unit 131 extracts frequency feature analysis results B11 and B12 from the inference sub-audio data ABD1, B21, B22, and B23 from the inference sub-audio data ABD2, and B31, B32, B33, and B34 from the inference sub-audio data ABD3. The feature analysis unit 131 performs matching processing on each of the extracted frequency feature analysis results B11, B12, B21, B22, B23, B31, B32, B33, and B34 with the frequency feature analysis result B of the training sound data, and outputs the matching processing analysis result data to the evaluation score calculation unit 132.

[0197] The evaluation score calculation unit 132 receives the analysis result data output by the feature analysis unit 131. For example, if the inference sub-voice data ABD2 contains frequency feature analysis results with high matching degree (e.g., B22), the evaluation score calculation unit 132 assigns a higher evaluation score (e.g., 70 points) to the inference sub-voice data ABD2. Furthermore, if, for example, a single inference sub-voice data ABD3 contains multiple frequency feature analysis results with high matching degree (e.g., B32, B34), the evaluation score calculation unit 132 assigns an even higher evaluation score (e.g., 80 points) to the inference sub-voice data ABD3. The evaluation score calculation unit 132 then outputs the assigned evaluation scores to the optimization calculation unit 133.

[0198] The optimization calculation unit 133 assigns a high priority order to the inference sub-voice data, starting with the sub-voice data with the highest evaluation score, based on the evaluation score information output by the evaluation score calculation unit 132. The optimization calculation unit 133 optimizes the inference calculation processing order of the m sub-voice data segments divided in the batch processing of the preprocessing unit 120, generating a processing order list of the optimized inference calculation processing order. The optimization calculation unit 133 outputs the generated processing order list to the inference calculation processing unit 134.

[0199] In addition, the optimization calculation unit 133 can also remove inference sub-voice data with low evaluation scores from the processing order list so that inference calculation processing is not performed on them.

[0200] The inference calculation processing unit 134 performs inference calculation processing based on the processing order list information received from the optimization calculation unit 133, the m inference sub-voice data segmented in the batch processing of the preprocessing unit 120, and the learned model. The inference calculation processing unit 134 stores the inference result data of the inference calculation processing in the inference result storage unit 135.

[0201] Thus, the probability that specific target keywords with teaching tags attached to the target are contained in the inference sub-voice data with high evaluation scores is high. Therefore, the learned model execution device 10 prioritizes inference calculation processing from the inference sub-voice data with high evaluation scores. As a result, the learned model execution device 10 can quickly identify the predefined conversation content and quickly end the inference calculation processing, thereby shortening the inference calculation processing time. In other words, the probability that the learned model execution device 10 contains target keywords in the inference sub-voice data with low evaluation scores is low. Therefore, by not performing inference calculation processing on the inference sub-voice data with low evaluation scores, useless calculation processing time can be eliminated. That is, the learned model execution device 10 can identify the prescribed conversation content from a large amount of audio data in a shorter inference calculation processing time.

[0202] (b) Regarding the case where the inference uses character data.

[0203] Figure 11 This is a functional block diagram illustrating a functional structure example of the learned model execution device 10 when the inference data is character data. Furthermore, for devices with... Figure 2 The elements of the learned model execution device 10 that have the same functions are labeled with the same reference numerals, and detailed descriptions are omitted.

[0204] In addition, in this case, the learned model takes in sub-character data for each inference and outputs information such as the proportion of specified character data (e.g., specific target keywords such as "year", "month", "day" representing time) attached as teaching labels in the training character data.

[0205] The acquisition unit 110 may acquire character data as inference data from scanning devices 90, such as scanners, cameras, printers with scanning functions, or touch panels capable of handwriting input, which acquire images of character data recorded on paper.

[0206] The batch processing unit 121 of the preprocessing unit 120 can also be connected with Figure 2 Similarly, for the inference using image data, the size of the training character data is set to a minimum, and the inference character data is segmented by batch processing to generate and output multiple inference sub-character data.

[0207] and Figure 2 Similarly, in the case of inference image data, the execution unit 130 can also be configured to assign an evaluation score to each of the multiple inference sub-character data segmented in the batch processing of the preprocessing unit 120 based on the matching degree with the training character data, and optimize the inference calculation processing order of the multiple inference sub-character data based on the priority order determined by the magnitude of the assigned multiple evaluation scores.

[0208] Specifically, the feature analysis unit 131 performs feature analysis on the training character data (e.g., character aspect ratio, symmetry centered on the X-axis, symmetry centered on the Y-axis, etc.) and extracts feature quantities (hereinafter also referred to as "feature analysis results") C from the prescribed character data (e.g., specific target keywords representing time such as "year", "month", "day") attached as teaching labels. Furthermore, the feature analysis unit 131 performs the same feature analysis on the k inference sub-character data MOJI1, MOJI2, ... MOJIk segmented in the batch processing of the preprocessing unit 120, and extracts feature analysis results (k is an integer of 2 or more). For example, the feature analysis unit 131 extracts feature analysis results C11 and C12 from the inference sub-character data MOJI1, C21, C22, and C23 from the inference sub-character data MOJI2, and C31, C32, C33, and C34 from the inference sub-character data MOJI3. The feature analysis unit 131 performs matching processing on each of the extracted feature analysis results C11, C12, C21, C22, C23, C31, C32, C33, and C34 with the feature analysis result C of the training character data, and outputs the matching analysis result data to the evaluation score calculation unit 132.

[0209] The evaluation score calculation unit 132 receives the analysis result data output by the feature analysis unit 131. For example, if the inference sub-character data MOJI2 contains feature analysis results with high matching degree (e.g., C22), the evaluation score calculation unit 132 assigns a high evaluation score (e.g., 70 points) to the inference sub-character data MOJI2. Furthermore, if, for example, a single inference sub-character data MOJI3 contains multiple feature analysis results with high matching degree (e.g., C32, C34), the evaluation score calculation unit 132 assigns an even higher evaluation score (e.g., 80 points) to the inference sub-character data MOJI3. The evaluation score calculation unit 132 then outputs the assigned evaluation scores to the optimization calculation unit 133.

[0210] The optimization calculation unit 133 assigns a high priority order to the inference calculations based on the evaluation score information output by the evaluation score calculation unit 132, starting with the inference sub-character data with the highest evaluation score. The optimization calculation unit 133 optimizes the inference calculation processing order of the k inference sub-character data segments divided in the batch processing of the preprocessing unit 120, generating a processing order list of the optimized inference calculation processing order. The optimization calculation unit 133 outputs the generated processing order list to the inference calculation processing unit 134.

[0211] In addition, the optimization calculation unit 133 can also remove inference sub-character data with low evaluation scores from the processing order list so that inference calculation processing is not performed on them.

[0212] The inference calculation processing unit 134 performs inference calculation processing based on the processing order list information received from the optimization calculation unit 133, the k inference sub-character data segmented in the batch processing of the preprocessing unit 120, and the learned model. The inference calculation processing unit 134 stores the inference result data of the inference calculation processing in the inference result storage unit 135.

[0213] Therefore, the probability that specific target keywords attached to the target are contained in the inference sub-character data with high evaluation scores is high. Thus, the learned model execution device 10 prioritizes inference calculation processing from the inference sub-character data with high evaluation scores. Consequently, the learned model execution device 10 can quickly identify character data such as "the failure of robot 30 occurred on [date]" based on the prescribed character data attached as teaching labels (e.g., specific target keywords representing time such as "year," "month," and "day"), thereby shortening the inference calculation processing time. In other words, the probability that the learned model execution device 10 contains target keywords in the inference sub-character data with low evaluation scores is low. Therefore, by not performing inference calculation processing on the inference sub-character data with low evaluation scores, useless calculation processing time can be eliminated. That is, the learned model execution device 10 can identify the prescribed character data from a large amount of character data in a shorter inference calculation processing time.

[0214] <Variation Example 3>

[0215] Additionally, for example, the learned model execution device 10a of the second embodiment described above executes a learned model generated by machine learning based on image data when the robot 30 removes multiple workpieces 50 that are bulk in the container 60. It obtains inference image data from the imaging device 40 and the learned model from the database 70, performs image processing on the obtained inference image data, and divides the inference image data into inference sub-image data with a feature point set and inference sub-image data with few or no feature points based on the feature points extracted from the inference image data. Furthermore, an embodiment of the learned model execution device 10a is described, which assigns an evaluation score to each inference sub-image data based on the number of feature points of each inference sub-image data, optimizes the inference calculation processing order of multiple inference sub-image data based on the evaluation score, performs inference calculation processing on multiple inference sub-image data according to the optimized inference calculation processing order, and calculates the candidate position for the workpiece 50 retrieved by the robot 30. However, this embodiment is not limited to the case where the robot 30 retrieves multiple workpieces 50 that are bulk in the container 60, and executes the learned model generated by machine learning based on the image data. It is also not limited to the type, shape, size, color, number, loading state, etc. of the workpieces 50.

[0216] For example, it can also be applied to systems that infer the task of robot 30 to remove workpiece 50 from a flat state where multiple workpieces 50 do not overlap or from a loaded state of box-shaped workpieces 50 (e.g., a stacked state of corrugated cardboard).

[0217] Similarly to Modification 2, this can also be applied to systems that perform arbitrary tasks using audio data from recordings of multi-person conversations or meetings. For example, the learned model execution device 10a can acquire inference data (hereinafter also referred to as "inference audio data") from a recording device such as a microphone and a learned model from a database 70, perform feature analysis (e.g., frequency analysis) on the acquired inference audio data, segment it into multiple inference sub-audio data based on the frequency analysis results extracted from the inference audio data, and assign an evaluation score to each inference sub-audio data. Then, the learned model execution device 10a can optimize the inference calculation processing order of the multiple inference sub-audio data based on the evaluation scores, perform inference calculation processing on the multiple inference sub-audio data based on the optimized inference calculation processing order, and find the specified conversation content (e.g., audio data of "dog (inu)," "cat (neko)," "weather (tenki)," etc.) from a large amount of audio data in a shorter inference calculation processing time. Therefore, the learned model execution device 10a, for example, when the specified conversation content is determined, can avoid useless inference calculation processing for the specified sound data, that is, the sub-sound data of the region (cell group) containing or not containing human voices, and can shorten the inference calculation processing time.

[0218] Alternatively, the learned model execution device 10a can also be applied to a scenario of executing a learned model generated by machine learning based on character data, instead of a system in which the robot 30 takes out a plurality of workpieces 50 loosely placed in the container 60, for example, as a system for performing inference for executing an arbitrary task based on character data. Specifically, image data may be replaced with character data, the same method as in the second embodiment may be applied, character inference data (hereinafter also referred to as "inference character data") may be acquired from a scanning device such as a scanner, and the learned model may be acquired from the database 70, feature analysis is performed on the acquired inference character data, the inference character data is divided into a plurality of inference sub-character data based on a feature analysis result extracted from the inference character data, and an evaluation score is assigned to each inference sub-character data. Then, the learned model execution device 10a may optimize the inference calculation processing order of the plurality of inference sub-character data according to the evaluation scores, perform the inference calculation processing of the plurality of inference sub-character data according to the optimized inference calculation processing order, and find out specified character data (for example, "capital", "prefecture", "province", "county", "city", "town", "village", etc.) from huge character data in a short inference calculation processing time. Accordingly, for example, in a case where the learned model execution device 10a determines the transmission destination of mail and distributes the mail for each transmission destination, it does not perform inference calculation processing on inference sub-character data of regions (cell groups) that do not include the specified character data from "capital" to "village", whereby unnecessary inference calculation processing can be avoided, and the inference calculation processing time can be shortened.

[0219] Hereinafter, more specifically, a case where the inference data is (a) sound data and a case where the inference data is (b) character data will be described respectively.

[0220] (a) Regarding the case where the inference data is sound data

[0221] Figure 12 is a functional block diagram illustrating an example of a functional configuration of the learned model execution device 10a in a case where the inference data is sound data. In addition, elements having the same functions as those of the Figure 4 of the learned model execution device 10a are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0222] In addition, the learned model in this case, for example, inputs each inference sub-sound data, and outputs information indicating a proportion or the like of whether or not a predetermined conversation content (for example, human voice data such as "dog (inu)", "cat (neko)", "weather (tenki)") is included.

[0223] The acquisition unit 110a may acquire sound data as inference data from a combination of a microphone and a computer, or from a recording device 80 such as a computer, smartphone, tablet terminal, or camera with a built-in microphone.

[0224] The preprocessing unit 120a can also be configured to perform feature analysis on the inference sound data, and based on the feature analysis results, divide the inference sound data into multiple inference sub-sound data through batch processing.

[0225] Specifically, the feature analysis unit 122a of the preprocessing unit 120a and Figure 10 Corresponding to the feature analysis unit 131, feature analysis (e.g., frequency feature analysis) can also be performed on the inference sound data, and feature analysis result data can be output. As feature analysis result data (frequency analysis result), in areas of low amplitude in the inference sound data, the prescribed conversational content of the voice of the person to be identified (e.g., "dog (inu)") is not included, and can be regarded as data containing only ambient noise at the time of recording. In this case, even if inference calculation processing is performed on the sound data in this area to find the prescribed conversational content of the voice of the person to be identified, inference calculation processing time will be wasted. In addition, in areas of high amplitude in the inference sound data, such as parts of sound data that exceed the recording range of the recording device 80, there is a high possibility that the sound data cannot be successfully obtained and is missing. Even if inference calculation processing is performed on the sound data in such areas, there is a high possibility that inference (recognition) cannot be successfully performed.

[0226] The batch processing unit 121a of the preprocessing unit 120a can also divide the inference sound data into multiple inference sub-sound data, such as cutting out regions of sound data as described above, without performing inference calculation processing.

[0227] In addition, in the frequency analysis results of the inference sound data, if a certain waveform appears periodically, it can be inferred that the same person continuously sings the same words for a certain period of time. The periodically appearing waveform can also be used as the cut-off point (cut-off point). The batch processing unit 121a divides the inference sound data into multiple inference sub-sound data through batch processing.

[0228] The execution unit 130a may also be configured to optimize the inference calculation processing order of multiple inference sub-voice data based on the feature analysis result data output by the feature analysis unit 122a of the preprocessing unit 120a.

[0229] For example, if the batch processing unit 121a generates multiple inference sub-voice data based on the frequency analysis results, the evaluation score calculation unit 132a may also assign low evaluation scores to multiple inference sub-voice data with excessively low or high amplitudes based on the frequency analysis results, and the optimization calculation unit 133 may reduce the priority of these inference calculation processes and not perform inference calculation processes.

[0230] Therefore, when searching for specific conversation content from inference audio data of a longer conversation, useless inference calculations are avoided for inference sub-audio data containing ambient noise where human voices are not present, and for inference sub-audio data that cannot be successfully obtained due to exceeding the recording range of the recording device 80. This shortens the inference calculation processing time for inference audio data and allows for the rapid retrieval of specific conversation content. In other words, the learned model execution device 10a, for example, when the specified conversation content is determined, can avoid useless inference calculations for inference sub-audio data in regions (unit groups) that contain or do not contain the specified audio data (i.e., human voices), thus shortening the inference calculation processing time.

[0231] (b) Regarding the case where the inference uses character data.

[0232] Figure 13 This is a functional block diagram representing a functional structure example of a learned model execution device 10a when the inference data is character data. Furthermore, for devices with... Figure 4 The elements of the learned model execution device 10a that have the same functions are labeled with the same reference numerals, and detailed descriptions are omitted.

[0233] In addition, in this case, the learned model takes as input inference sub-character data containing the destination of the email and outputs information indicating whether it contains the proportion of predefined character data (e.g., specific target keywords such as "city" to "village" used to determine the address).

[0234] The acquisition unit 110a may acquire character data as inference data from scanning devices 90, such as scanners, cameras, printers with scanning functions, or touch panels capable of handwriting input, which acquire images of character data recorded on paper.

[0235] The preprocessing unit 120a can also be configured to perform feature analysis on the character data for inference, and to divide the character data for inference into multiple sub-character data for inference through batch processing based on the feature analysis results.

[0236] Specifically, the feature analysis unit 122a of the preprocessing unit 120a and Figure 11corresponds to the feature analysis unit 131, and may also perform feature analysis on the character data for inference to output feature analysis result data. For example, a task of performing character recognition on handwritten addresses and automatically classifying mails according to their destinations is taken as an example for description. The acquisition unit 110a includes a data storage unit 111, for example, using a scanning device 90 to register character data obtained by scanning the area where the destination of a mail is written as image data, and store it in a recording medium such as an HDD of a personal computer (not shown). For example, the feature analysis unit 122a units the entire area of the thus acquired image data of a series of handwritten characters. The feature analysis unit 122a digitizes the presence or absence state of a character in each small cell, for example, attaches a label of "1" to a cell where a character exists in the area, and attaches a label of "0" to a cell where no character exists in the area. In this way, the feature analysis unit 122a can use the feature map of labels extracted by attaching labels of "0" and "1" to the entire area of the character image data, and recognize an area where labels of "0" are continuously attached as a space between characters to attach a character delimiter. Furthermore, the feature analysis unit 122a can determine the area (cell group) where the characters "都", "道", "府", "县", "市", "群", and "村" exist from a series of handwritten character data by matching each separated independent character with the printed characters "都", "道", "府", "县", "市", "群", and "村". The feature analysis unit 122a outputs the thus obtained label feature map and the existence area information of the specific characters (from "都" to "村") as the feature analysis result data.

[0237] The batch processing unit 121a of the preprocessing unit 120a can receive the feature analysis result data (the label feature map and the existence area information of the specific characters from "都" to "村") output by the feature analysis unit 122a, use the area (cell group) where the specific characters from "都" to "村" exist as a delimiter, split the character data for inference through batch processing to generate a plurality of sub-character data for inference and output them.

[0238] The execution unit 130a may also be configured to optimize the order of inference calculation processing for the plurality of sub-character data for inference based on the feature analysis result data output by the feature analysis unit 122a of the preprocessing unit 120a.

[0239] For example, when a plurality of sub-character data for inference are generated by the batch processing unit 121a of the preprocessing unit 120a, even if inference calculation processing is performed on the sub-character data for inference that includes a region (cell group) where no character exists and a region (cell group) that does not include specific characters from "to" to "village", the inference calculation processing is useless and does not contribute to the target of determining the mailing destination of the mail. Therefore, the evaluation score calculation unit 132a may assign a low evaluation score to the sub-character data for inference that includes a region (cell group) where no character exists and a region (cell group) that does not include the specific characters from "capital" to "village", lower the priority of these inference calculation processes and skip the inference calculation processing. In addition, the evaluation score calculation unit 132a assigns a high evaluation score to the sub-character data for inference that includes a region (cell group) containing the specific characters from "capital" to "village", and preferentially performs the inference calculation processing assigned with the high evaluation score, whereby the mailing destination of the mail can be quickly determined, and the inference calculation processing time of the automatic mail classification task can be shortened.

[0240] <Modification Example 4>

[0241] In addition, for example, although the learned model execution device 10a of the second embodiment acquires only the learned model from the database 70, training image data may also be acquired from the database 70.

[0242] Figure 14 is a functional block diagram showing an example of the functional configuration of the learned model execution device 10a in a case where training image data is also acquired. In addition, elements having the same functions as those of Figure 4 the elements of the learned model execution device 10a are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0243] As shown in Figure 14 , the acquisition unit 110a acquires the learned model and the training image data from the database 70.

[0244] The preprocessing unit 120a may also be configured to perform image processing on the training image data and the inference-use image data, and perform batch processing on the inference-use image data based on the image processing result to divide the inference-use image data into a plurality of sub-image data for inference.

[0245] Specifically, the image processing unit 122 of the preprocessing unit 120a serves as Figure 2The feature analysis unit 131 of the execution unit 130 performs image processing on the neighboring image regions, including the extraction positions indicated by the labels attached to the training image data, and the inference image data, for example, extracting specific local feature quantities. The image processing unit 122 may also match the local feature quantities of the neighboring images of the labeled teaching positions on the extracted training image data with the local feature quantities of each of multiple parts on the inference image data, calculate the matching degree, and output it as image processing result data.

[0246] The batch processing unit 121a of the preprocessing unit 120a segments the inference image data, making local image regions on the inference image data with high matching degrees to the nearby images of the teaching positions on the training image data independent sub-image data for inference. Additionally, the batch processing unit 121a segments the inference image data, making local image regions on the inference image data with low matching degrees to the nearby images of the teaching positions on the training image data independent sub-image data for inference.

[0247] In other words, the batch processing unit 121a can also segment and output the inference image data in a way that makes a large number of local image regions of the extracted object workpiece 50 a sub-image data for inference, and makes a few or none of the local image regions of the extracted object workpiece 50 a sub-image data for inference.

[0248] That is, by distinguishing between inference sub-image data that are highly likely to find workpieces 50 that should be prioritized for inference calculation processing and inference sub-image data that are highly likely to not find workpieces 50 that can be objects even if inference calculation is performed, the execution unit 130a can efficiently and smoothly optimize the inference calculation processing order.

[0249] The above describes the case of retrieving multiple bulk workpieces 50, but is not limited to this. It is also not limited to the type, shape, size, color, number, or loading state of the workpieces 50. For example, it can also be applied to systems that infer the task of retrieving workpieces 50 from a flat state where multiple workpieces 50 do not overlap, or from a loaded state of box-shaped workpieces 50 (e.g., a stack of corrugated cardboard).

[0250] Furthermore, the data used for inference can be inferential image data, but it could also be inferential sound data or inferential character data, etc. If the image processing unit 122 is replaced with the feature analysis unit 131, it can be applied to... Figure 14 The learned model execution device 10a is shown.

[0251] Furthermore, the functions included in the learned model execution device 10 in the first embodiment, the learned model execution device 10a in the second embodiment, and the learned model execution device 10b in the third embodiment can be implemented respectively by hardware, software, or a combination thereof. Here, implementation by software means implementation by reading the program into a computer and executing it.

[0252] Programs can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disks), optical-magnetic recording media (e.g., optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash memory ROMs, and RAM). Additionally, programs can also be provided to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transient computer-readable media can provide programs to a computer via wired communication paths such as wires and optical fibers, or via wireless communication paths.

[0253] In addition, the steps of a program recorded in a recording medium include, of course, processing performed sequentially in time, as well as processing that is not necessarily performed sequentially but is executed in parallel or individually.

[0254] In other words, the inference calculation processing apparatus and inference calculation processing method disclosed herein can be adopted in various embodiments having the following structures.

[0255] (1) The learned model execution device 10, which is the inference calculation processing apparatus of this disclosure, is an inference calculation processing apparatus that inputs inference data into a learned model to perform inference calculation processing of the inference data. The learned model execution device 10 includes: an acquisition unit 110 that acquires inference data and a learned model; a preprocessing unit 120 that divides the inference data acquired by the acquisition unit 110 into multiple inference sub-data by batch processing; and an execution unit 130 that optimizes the inference calculation processing order of the multiple inference sub-data divided by the preprocessing unit 120 by batch processing, and performs inference calculation processing of the inference data based on each of the multiple inference sub-data and the learned model according to the optimized inference calculation processing order.

[0256] According to the learned model execution device 10, the robot 30 will not be idle for a long time and can perform inference calculation processing in a short time.

[0257] (2) In the learned model execution device 10 described in (1), the acquisition unit 110 can also acquire the training data used when generating the learned model in machine learning.

[0258] Thus, the learned model execution device 10 is able to segment the inference data based on the size of the training data.

[0259] (3) In the learned model execution device 10 described in (2), the preprocessing unit 120 may also divide the inference data into multiple inference sub-data by batch processing based on the scale of the training data.

[0260] Therefore, the learned model execution device 10 can prevent the segmented inference sub-image from being too small to contain the required image features, thus preventing the inference from failing to proceed smoothly in the inference calculation process using the learned model.

[0261] (4) In the learned model execution device 10 described in (2) or (3), the execution unit 130 may also perform matching processing of training data and each of multiple inference sub-data, assign an evaluation score corresponding to the degree of matching to each of the multiple inference sub-data, and optimize the inference calculation processing order of the multiple inference sub-data according to the priority order based on the assigned evaluation scores.

[0262] Therefore, for example, when retrieving multiple bulk workpieces 50 from container 60, the learned model execution device 10 can eliminate useless inference calculations, such as failing to find a candidate workpiece 50 to be retrieved even when performing inference calculations on inference sub-data with low matching degree, thus shortening the inference calculation processing time. Furthermore, by prioritizing inference calculations on inference sub-data with high matching degree, the learned model execution device 10 can quickly find a pre-specified number of candidate workpieces 50 and rapidly complete the inference calculation process, further shortening the inference calculation processing time.

[0263] (5) In any of the learned model execution devices 10 described in (1) to (4), the acquisition unit 110 may also acquire image data as inference data.

[0264] Thus, the learned model execution device 10 is able to identify the workpiece 50 that the robot 30 can remove.

[0265] (6) In the learned model execution device 10a described in (5), the preprocessing unit 120a can also perform image processing to extract the feature quantities of the image data obtained as inference data.

[0266] Therefore, the learned model execution device 10a is able to optimally segment the inference image data into multiple inference sub-image data even without obtaining training data.

[0267] (7) In the learned model execution device 10a described in (6), the preprocessing unit 120a may also divide the inference data into multiple inference sub-data by batch processing based on the result of image processing.

[0268] Therefore, the learned model execution device 10a is able to optimally segment the inference image data into multiple inference sub-image data even without obtaining training data.

[0269] (8) In the learned model execution device 10a described in (6) or (7), the execution unit 130a may also assign an evaluation score to each of the multiple inference sub-data based on the result of image processing, and optimize the inference calculation processing order of the multiple inference sub-data according to the priority order based on the assigned evaluation scores.

[0270] Therefore, the learned model execution device 10a can eliminate useless inference calculations, such as failing to find a candidate for the workpiece 50 to be retrieved even when performing inference calculations containing inference sub-data with a low probability of the workpiece 50 to be retrieved, thus shortening the inference calculation processing time. Furthermore, by prioritizing inference calculations containing inference sub-data with a high probability of the workpiece 50 to be retrieved, the learned model execution device 10a can quickly find a pre-specified number of candidate workpieces 50 and quickly end the inference calculation process, further shortening the inference calculation processing time.

[0271] (9) In the learned model execution device 10 described in (1) to (4), sound data can also be acquired as inference data.

[0272] Therefore, the learned model execution device 10 is able to find the prescribed conversation content from a large amount of audio data in a shorter inference calculation processing time.

[0273] (10) In the learned model execution device 10a described in (9), the preprocessing unit 120a can also perform feature analysis on the feature quantities of the sound data obtained as inference data.

[0274] Therefore, the learned model execution device 10a is able to optimally segment the inference sound data into multiple inference sub-sound data even without obtaining training data.

[0275] (11) In the learned model execution device 10a described in (10), the preprocessing unit 120a may also divide the inference data into multiple inference sub-data by batch processing based on the results of feature analysis.

[0276] Therefore, the learned model execution device 10a is able to optimally segment the inference sound data into multiple inference sub-sound data even without obtaining training data.

[0277] (12) In the learned model execution device 10a described in (10) or (11), the execution unit 130a may also assign an evaluation score to each of the multiple inference sub-data based on the result of feature analysis, and optimize the inference calculation processing order of the multiple inference sub-data according to the priority order based on the assigned evaluation scores.

[0278] Therefore, the learned model execution device 10a can eliminate useless inference calculations that fail to find the required session content even after inference calculations are performed, thus shortening the inference calculation processing time. Furthermore, by prioritizing inference calculations of inference sub-data with a high probability of containing the required session content, the learned model execution device 10a can quickly terminate the inference calculation process, thereby shortening the inference calculation processing time.

[0279] (13) In the learned model execution device 10 described in (1) to (4), character data can also be obtained as inference data.

[0280] Therefore, the learned model execution device 10 is able to find the specified character data from a large amount of character data in a shorter inference calculation processing time.

[0281] (14) In the learned model execution device 10a described in (13), the preprocessing unit 120a can also perform feature analysis on the feature quantity of the character data obtained as inference data.

[0282] Therefore, the learned model execution device 10a can optimally segment the inference character data into multiple inference sub-character data even without obtaining training data.

[0283] (15) In the learned model execution device 10a described in (14), the preprocessing unit 120a may also divide the inference data into multiple inference sub-data by batch processing based on the results of feature analysis.

[0284] Therefore, the learned model execution device 10a can optimally segment the inference character data into multiple inference sub-character data even without obtaining training data.

[0285] (16) In the learned model execution device 10a described in (14) or (15), the execution unit 130a may also assign an evaluation score to each of the multiple inference sub-data based on the result of feature analysis, and optimize the inference calculation processing order of the multiple inference sub-data according to the priority order based on the assigned evaluation scores.

[0286] Therefore, the learned model execution device 10a can eliminate useless inference calculations, such as when the required character data is not found even after inference calculations are performed, and can shorten the inference calculation processing time. In addition, by prioritizing inference calculations of inference sub-data that are highly likely to contain the required character data, the learned model execution device 10a can quickly end the inference calculation processing and shorten the inference calculation processing time.

[0287] (17) In the learned model execution device 10b described in (1) to (4), the acquisition unit 110b can also acquire three-dimensional measurement data.

[0288] Therefore, the learned model execution device 10b can optimally divide the inference data into multiple inference sub-data even without obtaining training data.

[0289] (18) In the learned model execution device 10b described in (17), the preprocessing unit 120b can also divide the inference data into multiple inference sub-data based on the three-dimensional measurement data through batch processing.

[0290] Therefore, the learned model execution device 10b can optimally divide the inference data into multiple inference sub-data even without obtaining training data.

[0291] (19) In the learned model execution device 10b described in (17) or (18), the execution unit 130b may also assign an evaluation score to each of the multiple inference sub-data based on the three-dimensional measurement data, and optimize the inference calculation processing order of the multiple inference sub-data according to the priority order based on the assigned evaluation scores.

[0292] Therefore, the learned model execution device 10b can eliminate useless inference calculations, such as failing to find a candidate for the workpiece 50 to be retrieved even when performing inference calculations containing inference sub-data with a low probability of the workpiece 50 to be retrieved, thus shortening the inference calculation processing time. Furthermore, by prioritizing inference calculations containing inference sub-data with a high probability of the workpiece 50 to be retrieved, the learned model execution device 10b can quickly find a pre-specified number of candidate workpieces 50 and quickly end the inference calculation process, further shortening the inference calculation processing time.

[0293] (20) The inference calculation processing method disclosed herein is a computer-implemented inference calculation processing method that performs inference calculation processing on inference data by inputting inference data into a learned model. It comprises: an acquisition step, which acquires inference data and a learned model; a preprocessing step, which divides the acquired inference data into multiple inference sub-data by batch processing; and an execution step, which optimizes the inference calculation processing order of the multiple inference sub-data and performs inference calculation processing on the inference data based on each of the multiple inference sub-data and the learned model according to the optimized inference calculation processing order.

[0294] Based on this inference, the calculation method can achieve the same effect as (1).

[0295] Symbol Explanation

[0296] 1. 1A Robot System

[0297] 10, 10a, and 10b serve as learned model execution devices for inference calculation processing.

[0298] 11, 11a, 11b Control Section

[0299] 110, 110a, and 110b obtained the part

[0300] 111 Data Storage Department

[0301] Pretreatment sections 120, 120a, and 120b

[0302] Batch processing departments 121, 121a, and 121b

[0303] 122 Image Processing Department

[0304] 122a, 131 Feature Analysis Department

[0305] 123 3D Processing Department

[0306] 130, 130a, 130b Execution Department

[0307] Evaluation Score Calculation Department for 132, 132a, and 132b

[0308] 133 Optimization Calculation Department

[0309] 134 Inference Calculation Processing Department

[0310] 135 Inference Results Preservation Department

[0311] 20 Robot Control Devices

[0312] 30 robots

[0313] 40 shooting devices

[0314] 45 Three-dimensional measuring instrument

[0315] 50 workpieces

[0316] 60 containers.

Claims

1. An inference calculation processing apparatus, which inputs inference data into a learned model to perform inference calculation processing on the inference data, characterized in that, The inference calculation processing device includes: The acquisition unit acquires image data and the learned model as data for the inference; The preprocessing unit divides the acquired inference data into multiple inference sub-image data through batch processing; as well as The execution unit performs inference computation processing on the inference data based on at least a portion of the multiple inference sub-image data and the learned model. The acquisition unit acquires the training image data used to generate the learned model. The execution unit performs matching processing between the training image data and multiple inference sub-image data, assigns evaluation scores corresponding to the degree of matching to the multiple inference sub-image data, and generates a processing order list representing the optimal inference calculation processing order of the multiple inference sub-image data according to the priority order of the assigned evaluation scores. According to the processing order list, the inference calculation processing of the inference data is performed based on at least a portion of each of the multiple inference sub-image data and the learned model.

2. The inference calculation processing device according to claim 1, characterized in that, The preprocessing unit performs batch processing of the inference data based on the training image data.

3. The inference calculation processing apparatus according to claim 1 or 2, characterized in that, The preprocessing unit performs image processing on the image data obtained as data for the inference.

4. The inference calculation processing device according to claim 3, characterized in that, The preprocessing unit performs batch processing of the inference data based on the results of the image processing.

5. The inference calculation processing apparatus according to claim 3, characterized in that, The execution unit assigns an evaluation score to each of the plurality of inference sub-image data based on the result of the image processing, and optimizes the inference calculation processing order of the plurality of inference sub-image data according to the priority order of the assigned evaluation scores.

6. The inference calculation processing apparatus according to claim 1 or 2, characterized in that, The acquisition unit acquires three-dimensional measurement data.

7. The inference calculation processing apparatus according to claim 6, characterized in that, The preprocessing unit performs batch processing of the inference data based on the three-dimensional measurement data.

8. The inference calculation processing apparatus according to claim 6, characterized in that, The execution unit assigns an evaluation score to each of the plurality of inference sub-image data based on the three-dimensional measurement data, and optimizes the inference calculation processing order of the plurality of inference sub-image data according to the priority order of the assigned evaluation scores.

9. An inference calculation processing apparatus, which inputs inference data into a learned model to perform inference calculation processing on the inference data, characterized in that, The inference calculation processing device includes: The acquisition unit acquires the sound data and the learned model as data for the inference; The preprocessing unit divides the acquired inference data into multiple inference sub-voice data through batch processing; The execution unit performs inference computation processing on the inference data based on at least a portion of the multiple inference sub-voice data and the learned model. The acquisition unit acquires the training audio data used to generate the learned model. The execution unit performs matching processing between the training sound data and multiple inference sub-sound data, assigns evaluation scores corresponding to the degree of matching to the multiple inference sub-sound data, and generates a processing order list representing the optimal inference calculation processing order of the multiple inference sub-sound data according to the priority order of the assigned evaluation scores. According to the processing order list, the inference calculation processing of the inference data is performed based on each of at least a portion of the multiple inference sub-sound data and the learned model.

10. The inference calculation processing apparatus according to claim 9, characterized in that, The preprocessing unit performs feature analysis on the sound data obtained as data for the inference.

11. The inference calculation processing apparatus according to claim 10, characterized in that, The preprocessing unit performs batch processing of the inference data based on the results of the feature analysis.

12. The inference calculation processing apparatus according to claim 10 or 11, characterized in that, The execution unit assigns an evaluation score to each of the plurality of inference sub-voice data based on the result of the feature analysis, and optimizes the inference calculation processing order of the plurality of inference sub-voice data according to the priority order of the assigned evaluation scores.

13. An inference calculation processing apparatus, which inputs inference data into a learned model to perform inference calculation processing on the inference data, characterized in that, The inference calculation processing device includes: The acquisition unit acquires character data and the learned model as inference data; The preprocessing unit divides the acquired inference data into multiple inference sub-character data through batch processing; as well as The execution unit performs inference computation processing on the inference data based on at least a portion of the multiple inference sub-character data and the learned model. The acquisition unit acquires the training character data used to generate the learned model. The execution unit performs matching processing between the training character data and multiple inference sub-character data, assigns evaluation scores corresponding to the degree of matching to the multiple inference sub-character data, and generates a processing order list representing the optimal inference calculation processing order of the multiple inference sub-character data according to the priority order of the assigned evaluation scores. According to the processing order list, the inference calculation processing of the inference data is performed based on at least a portion of each of the multiple inference sub-character data and the learned model.

14. The inference calculation processing apparatus according to claim 13, characterized in that, The preprocessing unit performs feature analysis on the character data obtained as data for the inference.

15. The inference calculation processing apparatus according to claim 14, characterized in that, The preprocessing unit performs batch processing of the inference data based on the results of the feature analysis.

16. The inference calculation processing apparatus according to claim 14 or 15, characterized in that, The execution unit assigns an evaluation score to each of the plurality of inference sub-character data based on the result of the feature analysis, and optimizes the inference calculation processing order of the plurality of inference sub-character data according to the priority order of the assigned evaluation scores.

17. A computer-implemented inference calculation processing method, wherein inference data is input into a learned model to perform inference calculation processing on the inference data, characterized in that, The inference calculation and processing method includes the following steps: The acquisition process involves acquiring image data and the learned model, which serve as the inference data. The preprocessing step involves batch processing to divide the obtained inference data into multiple inference sub-image data. as well as The execution process involves performing inference computation processing on the inference data based on at least a portion of the multiple inference sub-image data and the learned model. The acquisition step acquires the training image data used to generate the learned model. In the execution step, the training image data is matched with multiple inference sub-image data. An evaluation score corresponding to the degree of matching is assigned to each of the multiple inference sub-image data. Based on the priority order of the assigned evaluation scores, a processing order list representing the optimal inference calculation processing order of the multiple inference sub-image data is generated. In accordance with the processing order list, the inference calculation processing of the inference data is performed based on at least a portion of each of the multiple inference sub-image data and the learned model.

Images