Image processing device and image processing method
The image processing device reallocates surplus computational resources to correct answer data generation and accuracy evaluation tasks, addressing the inefficiency in resource usage and enhancing processing efficiency and model accuracy.
Patent Information
- Application Number
- PCT/JP2023/019962
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-29
- Publication Date
- 2024-12-05
AI Technical Summary
Existing image processing methods, such as the high-speed method P, lead to variable computational resource usage during inference processing, resulting in potential surplus unused resources, which are not effectively utilized.
An image processing device and method that allocates surplus computational resources to correct answer data generation and accuracy evaluation tasks, ensuring efficient use of resources by reallocating them from the inference unit to the correct data generation and accuracy evaluation units.
Effectively utilizes surplus computational resources, improving processing efficiency and model accuracy by generating correct answer data and evaluating model accuracy, thereby optimizing resource allocation and inference performance.
Smart Images

Figure JP2023019962_05122024_PF_FP_ABST
Abstract
Description
Image processing device and image processing method
[0001] The present invention relates to an image processing apparatus, an image processing method, and a computer-readable recording medium on which an image processing program is recorded.
[0002] The following methods are known as methods for speeding up inference for images (for example, inferring the type of object appearing in an image). That is, a method is known in which the processing efficiency of inference is increased and inference speed is increased by simplifying the processing for a predetermined portion of the image (for example, the background portion). Hereinafter, this method will be referred to as speed-up method P for convenience.
[0003] Patent Document 1 describes an inference device that uses a machine learning model to execute a predetermined inference process on input image data.
[0004] Furthermore, Patent Document 2 describes an information processing device that improves inference throughput.
[0005] JP 2021-144461 A JP 2022-142119 A
[0006] In the high-speed method P, the processing efficiency differs depending on the image. If the area of the predetermined part in the image (here, the predetermined part is the background part as an example) is large, the amount of processing that can be simplified increases, and processing efficiency improves. On the other hand, if the area of the predetermined part (background part) in the image is small, the amount of processing that can be simplified is small, and the degree of improvement in processing efficiency is also small.
[0007] Therefore, when the speed-up method P is adopted, the amount of computational resources used in inference on an image is not constant but increases and decreases. For example, when the speed-up method P is applied to an inference means and computational resources are allocated to the inference means, some of the allocated computational resources may or may not be surplus, which is unused computational resources. When unused computational resources allocated for inference processing are surplus, it is preferable to be able to effectively utilize the surplus computational resources. The computational resources may be, for example, the processing power of a processor. The computational resources may be the processing power of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). Alternatively, the computational resources may be, for example, the processing power of an ASIC (Application Specific Integrated Circuit).
[0008] Therefore, the present invention aims to provide an image processing device, an image processing method, and a computer-readable recording medium having an image processing program recorded thereon, which can effectively utilize surplus computational resources that are left unused among the computational resources allocated for inference processing.
[0009] The image processing device according to the present invention is characterized by comprising an inference means for sequentially performing a predetermined inference on input images and simplifying processing of a predetermined portion of the image when performing the predetermined inference, thereby increasing the efficiency of the inference processing; a correct answer data generation means for generating correct answer data for a portion of a plurality of input images, the correct answer data including the image and a correct answer label that is the correct answer for the predetermined inference for that image; and a computational resource allocation means for allocating unused computational resources to the inference means, and, when the inference means generates surplus computational resources that remain unused from the allocated computational resources, allocating the surplus computational resources to the correct answer data generation means.
[0010] The image processing method according to the present invention is characterized in that the computational resource allocation means allocates unused computational resources to the inference means, the inference means sequentially performs a predetermined inference on input images, and when performing the predetermined inference, simplifies the processing of a predetermined portion of the image, thereby increasing the efficiency of the inference processing, and when the inference means generates surplus computational resources that remain unused from the allocated computational resources, the computational resource allocation means allocates the surplus computational resources to the correct answer data generation means, and the correct answer data generation means generates correct answer data for some of the multiple input images, including the image and a correct answer label that is the correct answer for the predetermined inference for that image.
[0011] The computer-readable recording medium of the present invention is a computer-readable recording medium having recorded thereon an image processing program that causes a computer to function as an inference means that performs predetermined inferences on images that are input sequentially, and increases the efficiency of inference processing by simplifying processing on predetermined portions of the images when performing the predetermined inference; a correct answer data generation means that generates correct answer data for some of the multiple images that are input, including the image and a correct answer label that is the correct answer for the predetermined inference for that image; and a computational resource allocation means that allocates unused computational resources to the inference means, and when the inference means generates surplus computational resources that remain unused from the allocated computational resources, allocates the surplus computational resources to the correct answer data generation means.
[0012] According to the present invention, when surplus computing resources are generated that are left unused among the computing resources allocated for inference processing, the surplus computing resources can be effectively utilized.
[0013] FIG. 1 is a schematic diagram showing an image processing device according to an embodiment of the present invention, and a learning device connected to the image processing device. FIG. 2 is a block diagram showing an example of the configuration of the image processing device according to an embodiment of the present invention. FIG. 3 is a schematic diagram showing an example of a job entered into a job queue. FIG. 4 is a flowchart showing an example of the processing progress of inference processing by the image processing device. FIG. 5 is a flowchart showing an example of the processing progress of a supervised answer data generation job. FIG. 6 is a flowchart showing an example of the processing progress of an accuracy evaluation job. FIG. 7 is a flowchart showing an example of the processing progress of an accuracy evaluation job. FIG. 8 is a flowchart showing an example of the processing progress of a model update job. FIG. 9 is a flowchart showing an example of the processing progress of a process for allocating surplus computational resources. FIG. 10 is a block diagram showing a modified example of the image processing device according to an embodiment of the present invention. FIG. 11 is a schematic block diagram showing an example of the configuration of a computer relating to the image processing device of the present invention. FIG. 12 is a block diagram showing an overview of the image processing device of the present invention.
[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0015] 1 is a schematic diagram showing an image processing device according to an embodiment of the present invention and a learning device connected to the image processing device. A learning device 20 is connected to the image processing device 10, for example, via a communication network. The image processing device 10 is implemented, for example, by an edge terminal, and the learning device 20 is implemented, for example, by a server device. The following description will be given taking as an example a case where the image processing device 10 is an edge terminal and the learning device 20 is a server device.
[0016] The image processing device 10 periodically generates images by taking pictures using a camera or the like, and performs a predetermined inference on each image. At this time, an inference unit 2 (see FIG. 2 ) provided in the image processing device 10 (described later) performs the predetermined inference on each image using a model obtained by machine learning.
[0017] In the following example, the predetermined inference will be described as image recognition (classification). However, the predetermined inference is not limited to this and may be other inference processes (tasks).
[0018] When the learning device 20 receives the supervised data described below from the image processing device 10, the learning device 20 adds the supervised data to the existing learning data, and generates a new model by machine learning using the learning data including the supervised data.
[0019] The image processing device 10 acquires the new model from the learning device 20 and updates the model used by the inference unit 2 with the new model.
[0020] 2 is a block diagram showing an example of the configuration of an image processing device 10 according to an embodiment of the present invention. The image processing device 10 includes an image input unit 1, an inference unit 2, a computational resource allocation unit 3, a supervised data generation unit 4, and an accuracy evaluation unit 5.
[0021] The image input unit 1 generates images by capturing images of the vicinity of the image processing device 10 at a first fixed time interval (for example, every 100 ms), and sequentially inputs the generated images to the inference unit 2. The image input unit 1 is realized by, for example, a camera. Alternatively, a camera may be provided outside the image processing device 10, and the image input interface through which images are input from the camera may be the image input unit 1. In the following example, a case where the image input unit 1 is a camera will be described as an example. Note that there may be multiple image input units 1.
[0022] The inference unit 2 sequentially performs predetermined inference on the images input from the image input unit 1. Examples of predetermined inference include inference of the type of object (e.g., vehicle, person, etc.) shown in the image and inference of the pose of the object shown in the image. However, the predetermined inference is not limited to these examples. The inference unit 2 derives a class (in the above example, a class of object type or a class of object pose) as an inference result. Furthermore, the inference unit 2 may derive a confidence score of the result in addition to the class as an inference result.
[0023] Furthermore, when performing a predetermined inference, the inference unit 2 employs a speed-up technique P. That is, when performing a predetermined inference, the inference unit 2 increases the processing efficiency of the inference by simplifying the processing for a predetermined portion of the image (for example, the background portion).
[0024] The inference unit 2 is assigned unused computational resources by the computational resource allocation unit 3, and the inference unit 2 constantly performs a predetermined inference on images input at a first fixed time interval (e.g., 100 ms interval).
[0025] Since the inference unit 2 employs the high-speed method P, depending on the input image, surplus computational resources, which are computational resources that remain unused among the allocated computational resources, may or may not occur. In this embodiment, the computational resources that remain unused by the inference unit 2 among the computational resources allocated to the inference unit 2 by the computational resource allocation unit 3, are referred to as surplus computational resources. The surplus computational resources may include computational resources that are not allocated to any process. As described above, the computational resources are the processing power of a processor. The computational resources may also be the processing power of a CPU, GPU, or FPGA. Alternatively, the computational resources may be, for example, the processing power of an ASIC.
[0026] Furthermore, the inference unit 2 performs a predetermined inference on the image using a model obtained by machine learning (e.g., deep learning). This model is a model for deriving an inference result in the predetermined inference. This model may be a general-purpose model generated by machine learning using general training data, rather than training data corresponding to the surrounding environment of the edge terminal that is the image processing device 10. Therefore, if an image with characteristics different from those of the image included in the training data is input to the inference unit 2, the inference result derived by the inference unit 2 may be incorrect.
[0027] The computational resource allocation unit 3 reserves computational resources for its own use, while allocating unused computational resources to the inference unit 2. As a result, the inference unit 2 can always perform predetermined inference on images input at a first fixed time interval (for example, every 100 ms).
[0028] When the inference unit 2 has surplus computational resources that remain unused among the allocated computational resources, the computational resource allocation unit 3 allocates the surplus computational resources to the supervised answer data generation unit 4 and the accuracy evaluation unit 5. The supervised answer data generation unit 4 and the accuracy evaluation unit 5 are able to execute their respective processes by being allocated computational resources. When allocating the surplus computational resources to the accuracy evaluation unit 5, the computational resource allocation unit 3 may also allocate the surplus computational resources to the inference unit 2.
[0029] In this embodiment, the computational resource allocation unit 3 samples input images at a second fixed time interval (e.g., every 10 minutes), generates jobs for generating supervised data for the images (hereinafter referred to as supervised data generation jobs), and submits the supervised data generation jobs to a job queue. The job queue is a FIFO (First In First Out) job queue. FIG. 3 is a schematic diagram showing an example of jobs submitted to the job queue. The accuracy evaluation job and model update job shown in FIG. 3 will be described later.
[0030] The computational resource allocation unit 3 may arbitrarily allocate the surplus computational resources to the supervised data generation unit 4 or the accuracy evaluation unit 5. For example, the computational resource allocation unit 3 may allocate the surplus computational resources to the supervised data generation unit 4 or the accuracy evaluation unit 5 that executes the job at the top of the job queue.
[0031] The computational resource allocation unit 3 may allocate computational resources by time division or space division. In the case of time division, the computational resource allocation unit 3 divides time and allows processes to use the computational resources to which they are allocated. When a certain time interval (time slot) ends, the execution of the process allocated to that time interval is temporarily suspended, and the execution of the process allocated to the next time interval is started. The execution of the suspended process is resumed when the next allocated time interval begins. In the case of space division, the computational resource element groups that make up the computational resources are grouped, and the computational resource element groups are allocated to processes. The computational resource allocation unit 3 may use both time division and space division. The computational resource allocation unit 3 may use methods other than time division and space division.
[0032] The correct answer data generating unit 4 generates correct answer data for a sampled image, including the image and a correct answer label that is the correct answer for a predetermined inference for the image. The sampled image can be said to be a part of the multiple images input by the image input unit 1.
[0033] The supervised data generation unit 4 executes a supervised data generation job when the next job to be processed in the job queue is a supervised data generation job and computational resources are allocated by the computational resource allocation unit 3. That is, the supervised data generation unit 4 performs inference on the image (sampled image) indicated by the supervised data generation job, defines the inference result as a supervised label, and generates supervised data including the image and the supervised label. The supervised data generation unit 4 obtains the inference result, for example, by applying the image to a model generated by machine learning such as deep learning using training data specialized for the surrounding environment of the edge terminal, which is the image processing device 10. Therefore, the inference result by the supervised data generation unit 4 can be said to be a supervised label. The model used by the supervised data generation unit 4 may be large in scale, and it may take some time for the supervised data generation unit 4 to derive the supervised label.
[0034] The model used by the correct answer data generation unit 4 may be trained using more general-purpose training data rather than training data specialized for the environment surrounding the edge terminal. Furthermore, the correct answer data generation unit 4 may use any method for generating correct answer data, and may use a method other than the above-described method.
[0035] The correct data generating unit 4 stores correct data including images and correct labels as a database.
[0036] The accuracy evaluation unit 5 generates a job (hereinafter referred to as an accuracy evaluation job) for evaluating the accuracy of the model used by the inference unit 2 at a third fixed time interval (for example, one day interval) and submits the accuracy evaluation job to the job queue. The computational resources required to generate the accuracy evaluation job and submit it to the job queue are negligibly small. It is assumed that this negligible computational resource is allocated to the accuracy evaluation unit 5 by the computational resource allocation unit 3.
[0037] When the next job to be processed in the job queue is an accuracy evaluation job and computational resources have been allocated by the computational resource allocation unit 3, the accuracy evaluation unit 5 uses each piece of supervised data to evaluate the accuracy of the model used by the inference unit 2 (i.e., executes the accuracy evaluation job). Evaluating the accuracy of the model used by the inference unit 2 means, in other words, evaluating the accuracy of the inference result by the inference unit 2.
[0038] Specifically, the accuracy evaluation unit 5 evaluates the accuracy of the inference result by the inference unit 2 as follows: The accuracy evaluation unit 5 causes the inference unit 2 to perform a predetermined inference for each image included in each supervised data. In this case, the inference unit 2 is allocated surplus computational resources by the computational resource allocation unit 3, and performs the predetermined inference using those computational resources. The accuracy evaluation unit 5 then evaluates the accuracy of the inference result by the inference unit 2 by comparing each inference result (class) obtained by the inference unit 2 with each supervised label included in each of the supervised data.
[0039] At this time, the accuracy evaluation unit 5 derives an accuracy evaluation value, which is an evaluation value of the accuracy of the inference result by the inference unit 2. The higher the accuracy, the larger the accuracy evaluation value, and the lower the accuracy, the smaller the accuracy evaluation value. An example of the accuracy evaluation value is, for example, the rate at which the inference result (class) obtained by the inference unit 2 matches the correct label (match rate). However, the accuracy evaluation value is not limited to such a match rate, and may be an evaluation value obtained by other methods.
[0040] If the accuracy evaluation value is below a predetermined threshold, the accuracy evaluation unit 5 generates a job (hereinafter referred to as a model update job) to update the model used by the inference unit 2 and submits the model update job to a job queue.
[0041] When the next job to be processed in the job queue is a model update job and the computational resources have been allocated by the computational resource allocation unit 3, the accuracy evaluation unit 5 executes the model update job.
[0042] Specifically, the accuracy evaluation unit 5 sends correct answer data stored as a database to the learning device 20 (see FIG. 1 ), causing the learning device 20 to generate a new model using the correct answer data through machine learning such as deep learning. Upon receiving the correct answer data from the accuracy evaluation unit 5, the learning device 20 adds the correct answer data to existing learning data held by the learning device 20 and generates a new model through machine learning using the learning data including the correct answer data. The learning device 20 sends the new model to the accuracy evaluation unit 5 of the image processing device 10, and the accuracy evaluation unit 5 acquires the new model. The accuracy evaluation unit 5 then updates the model used by the inference unit 2 with the new model.
[0043] The learning device 20 may store learning evaluation data consisting of correct answer data for evaluating the generated model and use the data to evaluate the generated model. The learning device 20 may generate multiple models under different conditions, different sets of correct answer data, etc., and may select the model with the highest evaluation result for the learning evaluation data as the final model, for example. The learning device 20 may add a portion of the correct answer data received from the accuracy evaluation unit 5 to the existing learning data, and add the remainder to the existing learning evaluation data.
[0044] The new model is generated using training data including correct data related to images around the image processing device 10 that are spatially and temporally close to the image processing device 10. Therefore, it can be said that the accuracy of the new model is higher than the accuracy of the model before the update. Therefore, for example, if the accuracy of the model initially used by the inference unit 2 is low, the model can be updated to a model with higher accuracy. Furthermore, if the accuracy of the model used by the inference unit 2 decreases due to a change in the surrounding environment of the edge terminal that is the image processing device 10, the model can be updated to a model with higher accuracy.
[0045] The inference unit 2, computational resource allocation unit 3, supervised data generation unit 4, and accuracy evaluation unit 5 are realized, for example, by a CPU of a computer that operates according to an image processing program. For example, the CPU may read the image processing program from a program recording medium such as a program storage device of the computer, and operate as the inference unit 2, computational resource allocation unit 3, supervised data generation unit 4, and accuracy evaluation unit 5 according to the image processing program. Furthermore, a GPU, FPGA, ASIC, etc. may be used instead of or in addition to the CPU.
[0046] Next, the process will be described, and the description of the matters that have already been explained will be omitted as appropriate.
[0047] 4 is a flowchart showing an example of the progress of inference processing by the image processing device 10. It is assumed that the computational resource allocation unit 3 allocates unused computational resources to the inference unit 2. Steps A1 and A2 shown in FIG. 4 are repeatedly executed at a first fixed time interval (for example, every 100 ms). Note that the first fixed time interval is not limited to 100 ms.
[0048] In step A1 , the image input unit 1 generates an image by taking a photograph and inputs the image to the inference unit 2 .
[0049] In step A2, the inference unit 2 performs a predetermined inference on the image input in step A1. The inference unit 2 derives an inference result by applying the image to a model (the model used by the inference unit 2). The inference unit 2 also employs a high-speed method P to derive the inference result.
[0050] If the timing is every second fixed time interval (for example, every 10 minutes), the computational resource allocation unit 3 samples the input image, generates a supervised data generation job corresponding to the image, and submits the supervised data generation job to the job queue (step A3). Therefore, a supervised data generation job is created at the second fixed time interval, and the supervised data generation job is submitted to the job queue. Note that the second fixed time interval is not limited to every 10 minutes.
[0051] 5 is a flowchart showing an example of the processing progress of a supervised data generation job. When executing a supervised data generation job, the supervised data generation unit 4 performs inference on the image indicated by the supervised data generation job, sets the inference result as a supervised label, and generates supervised data including the image and the supervised label (step B1). The supervised data generation unit 4 performs inference using a model generated by machine learning using training data specialized for the surrounding environment of the edge terminal, which is the image processing device 10. Therefore, the inference result by the supervised data generation unit 4 can be said to be a supervised label.
[0052] The supervised data generating unit 4 stores the supervised data generated in step B1 (step B2), and then ends the supervised data generating job.
[0053] 6 is a flowchart showing an example of a process for generating an accuracy evaluation job. Step C1 shown in FIG. 6 is repeatedly executed at a third fixed time interval (for example, one day). Note that the third fixed time interval is not limited to one day.
[0054] In step C1, the accuracy evaluation unit 5 generates an accuracy evaluation job and submits the accuracy evaluation job to the job queue. The accuracy evaluation unit 5 performs this process at third fixed time intervals.
[0055] 7 is a flowchart showing an example of the processing progress of an accuracy evaluation job. When executing an accuracy evaluation job, the accuracy evaluation unit 5 causes the inference unit 2 to perform a predetermined inference for each image included in each supervised data stored in the supervised data generation unit 4 (step D1). At this time, the inference unit 2 is assigned surplus computational resources by the computational resource allocation unit 3, and performs the predetermined inference using the computational resources.
[0056] Next, the accuracy evaluation unit 5 calculates an accuracy evaluation value using each inference result obtained in step D1 and each correct answer data (step D2).
[0057] Next, the accuracy evaluation unit 5 determines whether the accuracy evaluation value is equal to or less than a predetermined threshold value (step D3).
[0058] If the accuracy evaluation value is greater than the predetermined threshold value (No in step D3), the accuracy evaluation job is terminated.
[0059] If the accuracy evaluation value is equal to or less than the predetermined threshold value (Yes in step D3), the accuracy evaluation unit 5 generates a model update job and submits the model update job to the job queue (step D4), and then ends the accuracy evaluation job.
[0060] 8 is a flowchart showing an example of the processing progress of a model update job. When executing a model update job, the accuracy evaluation unit 5 transmits the supervised data stored in the supervised data generation unit 4 to the learning device 20 (see FIG. 1) (step E1).
[0061] When the learning device 20 receives the correct answer data, it adds the correct answer data to existing learning data that it has stored in advance, and generates a new model through machine learning using the learning data that includes the correct answer data (step E2).
[0062] The learning device 20 transmits the new model to the image processing device 10, and the accuracy evaluation unit 5 of the image processing device 10 acquires the new model (step E3).
[0063] Next, the accuracy evaluation value 5 updates the model used by the inference unit 2 with the new model (step E4), and the model update job is then completed.
[0064] 9 is a flowchart showing an example of the progress of processing for allocating surplus computing resources. The computing resource allocation unit 3 determines whether there is a job in the job queue and whether there is a surplus computing resource (step F1).
[0065] If there are no jobs in the job queue or there are no surplus computing resources (No in step F1), the process of step F1 is repeated again.
[0066] If there is a job in the job queue and there is a surplus computing resource (Yes in step F1), the computing resource allocation unit 3 takes out one job from the job queue (step F2).
[0067] The computing resource allocation unit 3 then allocates the surplus computing resources to the element corresponding to the extracted job, and causes the element to execute the job (step F3).
[0068] For example, if the job extracted in step F2 is a supervised data generation job, the computational resource allocation unit 3 allocates the surplus computational resources to the supervised data generation unit 4 and causes the supervised data generation unit 4 to execute the supervised data generation job (see FIG. 5).
[0069] For example, if the job extracted in step F2 is an accuracy evaluation job, the computational resource allocation unit 3 allocates the surplus computational resources to the accuracy evaluation unit 5 and the inference unit 2, and causes the accuracy evaluation unit 5 and the inference unit 2 to execute the accuracy evaluation job (see FIG. 7 ).
[0070] For example, if the job extracted in step F2 is a model update job, the computational resource allocation unit 3 allocates the surplus computational resources to the accuracy evaluation unit 5 and causes the accuracy evaluation unit 5 to execute the model update job (see FIG. 8 ).
[0071] After step F3, the processes from step F1 onwards are repeated.
[0072] According to this embodiment, while the inference unit 2 is constantly performing a predetermined inference on images input at a first fixed time interval (for example, every 100 ms), if any surplus computational resources that are left unused among the computational resources allocated for the inference process occur, the surplus computational resources can be effectively utilized.
[0073] For example, the computational resource allocation unit 3 allocates surplus computational resources to the correct answer data generation unit 4, and the correct answer data generation unit 4 can evaluate the accuracy of the inference results by the inference unit 2 and generate correct answer data to be used in model training.
[0074] Furthermore, for example, the computational resource allocation unit 3 can allocate surplus computational resources to the accuracy evaluation unit 5 and the inference unit 2 , and the accuracy evaluation unit 5 can evaluate the accuracy of the inference results by the inference unit 2 .
[0075] Furthermore, for example, the computational resource allocation unit 3 can allocate surplus computational resources to the accuracy evaluation unit 5, and the accuracy evaluation unit 5 can update the model used by the inference unit 2 to a more accurate model.
[0076] Next, various variations of this embodiment will be described.
[0077] In the above description, the inference unit 2 performed inference for each image generated by the image input unit 1 (see FIG. 4). However, this embodiment is not limited to this. As another example, the inference unit 2 may perform inference on multiple images generated by the image input unit 1 at once. By processing multiple images at once, the improvement in processing efficiency by the high-speed method P may be greater, and excess computing resources may be made more available. Furthermore, if there are multiple image input units 1, the inference unit 2 may perform inference on images from multiple image input units 1 at once.
[0078] In the above description, the image processing device 10 performed image classification as the predetermined inference. However, this embodiment is not limited to this. As another example, the image processing device 10 may perform object detection. When the image processing device 10 performs object detection, the inference unit 2 performs inference using an object detection model. In this case, the inference unit 2 derives the class, accuracy, and position information (e.g., coordinates of the four vertices of a rectangle surrounding the object) for each detected object as an inference result. Furthermore, the correct answer data generation unit 4 generates correct answer data for object detection, and the learning device 20 generates a model for object detection. The accuracy evaluation unit 5 evaluates accuracy using, for example, precision, which is the proportion of correct answers in a group of detected objects; recall, which is the proportion of objects that were detected among correct answers (objects to be detected); average precision (AP), which is an average of multiple precisions derived under different conditions, and mean AP (mAP), or a combination thereof.
[0079] In the above description, an example has been shown in which the inference unit 2 performs a predetermined inference on an image using a model obtained by machine learning. The inference unit 2 may perform inference by selectively using multiple models or by combining the models. Furthermore, the inference unit 2 may use such multiple models in conjunction with the adoption of the speed-up method P.
[0080] The above description illustrates an example in which the inference unit 2 employs the speed-up method P. The inference unit 2 may employ multiple speed-up methods, such as the first speed-up method P1 and the second speed-up method P2, selectively or in combination. In this case, the inference unit 2 may initially retain and use only a model for the first speed-up method P1 for inference, and the supervised data generation unit 4 may generate supervised data using the surplus computing resources resulting from the use of the first speed-up method P1. Furthermore, in this case, the accuracy evaluation unit 5 may generate a model update job regardless of the accuracy evaluation value of the inference result by the inference unit 2. The accuracy evaluation unit 5 may generate a model update job, for example, when a predetermined amount of supervised data has been generated. The new model generated by the model update job may be a model for the second speed-up method P2. After the model for the second speed-up method P2 has been generated, the inference unit 2 may switch the method it employs from the first speed-up method P1 to the second speed-up method P2.
[0081] In the above description, an example has been shown in which the computational resource allocation unit 3 samples the input image at the second fixed time interval. However, this embodiment is not limited to this. As another example, the computational resource allocation unit 3 may sample the image using the following conditions: (a) When the accuracy of the inference result is above (or below) a predetermined threshold; (b) When the processing time for inference is above (or below) a predetermined threshold; (c) When the number of detected objects is above (or below) a predetermined threshold; (d) When the class of the detected object is a predetermined class (or not a predetermined class); (e) When the accuracy of the detected object is above (or below) a predetermined threshold; (f) When the detected object is contained in (or not contained in) a predetermined area; (g) When the area of the detected object is above (or below) a predetermined threshold; (h) When the area of a predetermined part of the image in speed-up method P is above (or below) a predetermined threshold; (i) When the degree of improvement in processing efficiency by speed-up method P is above (or below) a predetermined threshold. The computational resource allocation unit 3 may determine the image to sample using any of the above examples or a combination thereof. Furthermore, the computational resource allocation unit 3 may sample images using the results of inference, information obtained when using the speed-up method P during inference, the amount of computational resources consumed during inference, and time-related conditions.
[0082] In the above description, an example has been shown in which the accuracy evaluation unit 5 generates accuracy evaluation jobs at a third fixed time interval. However, this embodiment is not limited to this. As another example, the accuracy evaluation unit 5 may generate accuracy evaluation jobs based on conditions similar to those used by the computational resource allocation unit 3 described above to determine the images to sample. Furthermore, the accuracy evaluation unit 5 may also use conditions related to the amount of generated supervised data.
[0083] In the above description, an example has been shown in which the accuracy evaluation unit 5 generates a model update job when the accuracy evaluation value is equal to or less than a predetermined threshold. However, this embodiment is not limited to this. As another example, the accuracy evaluation unit 5 may generate a model update job based on conditions similar to those used by the computational resource allocation unit 3 described above to determine which images to sample.
[0084] In the above description, an example has been shown in which the accuracy evaluation unit 5 evaluates the accuracy of the model used by the inference unit 2 using the supervised data generated by the supervised data generation unit 4. However, this embodiment is not limited to this. As another example, the accuracy evaluation unit 5 may previously store evaluation data, which is supervised data for evaluation, and evaluate the accuracy of the model using the evaluation data. Furthermore, the accuracy evaluation unit 5 may evaluate the accuracy of the model by combining the evaluation data with the supervised data generated by the supervised data generation unit 4, or by using a portion of them. The accuracy evaluation unit 5 may exclude the supervised data used by the learning device 20 to train the model from the supervised data used for evaluation.
[0085] In the above description, an example has been shown in which the ratio of the number of image processing devices 10 to the number of learning devices 20 is 1:1. However, this embodiment is not limited to this. As another example, the ratio of the number of image processing devices 10 to the number of learning devices 20 may be n:m. That is, an image processing device 10 may be connected to m learning devices 20, and a learning device 20 may be connected to n image processing devices 10.
[0086] When the learning device 20 generates a new model, it may transmit the model to multiple image processing devices 10, and the accuracy evaluation unit 5 of each image processing device 10 may receive the model and update the model used by the inference unit 2 with the new model.
[0087] When generating a new model for a certain image processing device 10, the learning device 20 may also use correct answer data received from an image processing device other than the image processing device 10 to perform learning.
[0088] The learning device 20 may store and update the learning evaluation data for each image processing device 10 .
[0089] Next, a modified example of the image processing apparatus according to the embodiment of the present invention will be described.
[0090] Fig. 10 is a block diagram showing a modified example of the image processing device according to the embodiment of the present invention. Elements similar to those of the image processing device 10 according to the above embodiment (see Fig. 2) are designated by the same reference numerals as in Fig. 2, and description thereof will be omitted. The image processing device 10 shown in Fig. 10 includes a pre-processing unit 6 in addition to the elements shown in Fig. 2. The pre-processing unit 6 pre-processes an image input from the image input unit 1, and inputs the pre-processed image to the inference unit 2.
[0091] In this modification, the computational resource allocation unit 3 specifies the pre-processing to be performed by the pre-processing unit 6. The computational resource allocation unit 3 allocates the computational resources required to perform the specified pre-processing to the pre-processing unit 6. Examples of types of pre-processing include sharpening and gamma correction, but the pre-processing performed by the pre-processing unit 6 is not limited to these examples. Types of pre-processing also include "do nothing."
[0092] In addition, in this modified example, the computational resource allocation unit 3 generates a job (preprocessing selection job) that selects the preprocessing to be performed by the preprocessing unit 6 at a fourth fixed time interval (for example, one hour interval), and submits the preprocessing selection job to the job queue.
[0093] When the next job to be processed in the job queue is a pre-processing selection job and the computational resources have been allocated by the computational resource allocation unit 3, the accuracy evaluation unit 5 selects the pre-processing to be performed by the pre-processing unit 6.
[0094] Specifically, the accuracy evaluation unit 5 causes the preprocessing unit 6 and the inference unit 2 to perform preprocessing and predetermined inference for each image included in each supervised data. In this case, the preprocessing unit 6 and the inference unit 2 are allocated surplus computational resources by the computational resource allocation unit 3, and use the allocated computational resources to perform the predetermined inference.
[0095] The accuracy evaluation unit 5 may then evaluate the accuracy of the inference result by the inference unit 2 for each type of preprocessing and calculate an accuracy evaluation value. Furthermore, if a preprocessing whose accuracy evaluation value is equal to or greater than a predetermined second threshold is found, the computational resource allocation unit 3 selects that preprocessing. Note that the second threshold is different from and greater than the threshold used in step D3 (see FIG. 7 ).
[0096] Alternatively, the accuracy evaluation unit 5 performs a predetermined inference for each type of preprocessing and evaluates the accuracy of the inference result by the inference unit 2. The computational resource allocation unit 3 may, for example, select the preprocessing that has produced the most accurate result. If the amount of surplus computational resources is equal to or less than a predetermined threshold, the computational resource allocation unit 3 may select "do nothing" as the preprocessing. If there are multiple speed-up methods, a predetermined inference and accuracy evaluation may be performed for each speed-up method.
[0097] The computational resource allocation unit 3 sets the selected preprocessing as the preprocessing to be performed by the preprocessing unit 6. When executing the preprocessing selection job, the computational resource allocation unit 3 may use only a portion of the supervised data held by the supervised data generation unit 4. For example, the computational resource allocation unit 3 may use supervised data sampled during the same time period as when the preprocessing selection job is executed.
[0098] In the above description, an example has been shown in which the computational resource allocation unit 3 generates pre-processing selection jobs at the fourth fixed time interval. However, this modification is not limited to this. As another example, the computational resource allocation unit 3 may generate pre-processing selection jobs based on conditions similar to the conditions used by the computational resource allocation unit 3 to determine the images to sample.
[0099] In the above embodiment, the case where the learning device 20 generates a new model has been described. The image processing device 10 may be provided with a learning unit that adds correct answer data to existing learning data and generates a new model through machine learning using the learning data to which the correct answer data has been added. That is, the image processing device 10 may generate a new model.
[0100] 11 is a schematic block diagram showing an example of the configuration of a computer relating to the image processing apparatus of the present invention. The computer 2000 includes a CPU 2001, a main memory device 2002, an auxiliary memory device 2003, an interface 2004, a camera 2005, and a communication interface 2006. The communication interface 2006 is an interface with a communication network.
[0101] The image processing device 10 according to the embodiment and modification of the present invention is realized, for example, by a computer 2000. The operation of the image processing device 10 is stored in the form of a program (image processing program) in an auxiliary storage device 2003. A CPU 2001 reads the program from the auxiliary storage device 2003, loads the program into a main storage device 2002, and executes the processing described in the above embodiment and modification in accordance with the program.
[0102] The auxiliary storage device 2003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a semiconductor memory, all of which are connected via the interface 2004.
[0103] Next, an outline of the present invention will be described. Fig. 12 is a block diagram showing an outline of an image processing device of the present invention. The image processing device of the present invention comprises an inference means 72, a supervised data generation means 74, and a computational resource allocation means 73.
[0104] The inference means 72 (e.g., the inference unit 2) sequentially performs predetermined inferences on input images, and improves the processing efficiency of the inferences by simplifying the processing of predetermined parts of the images when performing the predetermined inferences.
[0105] The correct answer data generation means 74 (e.g., the correct answer data generation unit 4) generates correct answer data for some of the multiple input images, including the image and a correct answer label that is the correct answer for a specified inference for the image.
[0106] The computational resource allocation means 73 (e.g., the computational resource allocation unit 3) allocates unused computational resources to the inference means 72, and when the inference means 72 generates surplus computational resources that remain unused among the allocated computational resources, it allocates the surplus computational resources to the correct answer data generation means 74.
[0107] With such a configuration, when there is a surplus of computational resources that are left unused among the computational resources allocated for inference processing, the surplus computational resources can be effectively utilized.
[0108] The inference means 72 may be provided with an accuracy evaluation means (e.g., accuracy evaluation unit 5) that uses correct answer data to evaluate the accuracy of the inference result by the inference means 72, and when surplus computational resources, which are unused computational resources among the computational resources allocated by the inference means 72, occur, the computational resource allocation means 73 may allocate the surplus computational resources to the accuracy evaluation means.
[0109] The accuracy evaluation means may have the inference means 72 perform a predetermined inference on the image included in the correct answer data, and evaluate the accuracy of the inference result by the inference means 72 by comparing the inference result by the inference means 72 with the correct answer label included in the correct answer data.
[0110] The inference means 72 performs a predetermined inference on the image using a model obtained by machine learning, and when the accuracy evaluation value of the inference result by the inference means 72 falls below a predetermined threshold, the accuracy evaluation means sends correct answer data to a learning means (e.g., learning device 20) that learns the model, thereby generating a new model by machine learning using the correct answer data, obtaining the new model from the learning means, and updating the model used by the inference means 72 with the new model.
[0111] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. Possibility of industrial applicability
[0112] The present invention is suitably applied to an image processing device that performs predetermined inference on images that are input sequentially.
[0113] REFERENCE SIGNS LIST 1 Image input unit 2 Inference unit 3 Computational resource allocation unit 4 Correct answer data generation unit 5 Accuracy evaluation unit 6 Preprocessing unit 10 Image processing device
Claims
1. An image processing device comprising: an inference means for sequentially performing a predetermined inference on input images and simplifying processing on a predetermined portion of the image when performing the predetermined inference, thereby increasing the efficiency of the inference processing; a correct answer data generation means for generating correct answer data for a portion of a plurality of input images, the correct answer data including the image and a correct answer label which is the correct answer to the predetermined inference for that image; and a computational resource allocation means for allocating unused computational resources to the inference means, and when the inference means generates surplus computational resources that are unused among the allocated computational resources, the computational resource allocation means for allocating the surplus computational resources to the correct answer data generation means.
2. An image processing device as described in claim 1, further comprising an accuracy evaluation means for evaluating the accuracy of the inference result by the inference means using the correct answer data, and wherein the computational resource allocation means, when a surplus of computational resources remaining unused among the allocated computational resources by the inference means occurs, allocates the surplus computational resources to the accuracy evaluation means.
3. The image processing device according to claim 2, wherein the accuracy evaluation means causes the inference means to perform the specified inference on an image contained in the correct answer data, and evaluates the accuracy of the inference result by the inference means by comparing the inference result by the inference means with the correct answer label contained in the correct answer data.
4. The image processing device described in claim 2 or claim 3, wherein the inference means performs the specified inference on an image using a model obtained by machine learning, and the accuracy evaluation means, when an accuracy evaluation value of the inference result by the inference means becomes equal to or below a predetermined threshold, sends the correct answer data to a learning means that learns a model, thereby generating a new model by machine learning using the correct answer data, obtains the new model from the learning means, and updates the model used by the inference means with the new model.
5. An image processing method characterized in that a computational resource allocation means allocates unused computational resources to an inference means, the inference means sequentially performs a predetermined inference on input images and simplifies processing for a predetermined portion of the image when performing the predetermined inference, thereby increasing the efficiency of the inference processing, the computational resource allocation means, when the inference means generates surplus computational resources that remain unused among the allocated computational resources, allocates the surplus computational resources to a supervised data generation means, and the supervised data generation means generates supervised data for a portion of a plurality of input images, the supervised data including an image and a supervised label that is the correct answer for the predetermined inference for that image.
6. The image processing method according to claim 5, wherein the computational resource allocation means, when any surplus computational resources remaining unused among the computational resources allocated by the inference means, allocates the surplus computational resources to an accuracy evaluation means, and the accuracy evaluation means uses the correct answer data to evaluate the accuracy of the inference result by the inference means.
7. A computer-readable recording medium having recorded thereon an image processing program for causing a computer to function as: an inference means for sequentially performing a predetermined inference on input images and simplifying processing on a predetermined portion of the image when performing the predetermined inference, thereby increasing the efficiency of inference processing; a correct answer data generation means for generating, for a portion of a plurality of input images, correct answer data including the image and a correct answer label that is the correct answer for the predetermined inference for that image; and a computational resource allocation means for allocating unused computational resources to the inference means and, when a surplus of computational resources remains unused among the allocated computational resources, allocating the surplus computational resources to the correct answer data generation means.
Citation Information
Patent Citations
Processing device, processing device control method, and recording medium
WO2016136196A1
Recognition system, model processing device, model processing method, and recording medium
WO2021033288A1
Machine learning model determination system and machine learning model determination method
WO2021181605A1
