Image processing device, image processing method and program

Abstract

To appropriately perform machine learning of a machine learning model.SOLUTION: An image processing device comprises: a whole image processing unit which considers the whole of a reduced image obtained by reducing a photographed image as a detection object and performs detection processing of an object by using a learned machine learning model for whole images; an output unit which continuously outputs information showing a detection result; a partial image processing unit which considers each partial image obtained by dividing the photographed image as the detection object and performs the detection processing of the object in parallel with the detection processing by the whole image processing unit, by using the learned machine learning model for a partial image; a collation unit which collates a detection result of the whole image processing unit and a detection result of the partial image processing unit; and a model relearning unit which relearns the learned machine learning model for whole images by using a dataset including a pair of the corresponding reduced image and the detection result of the partial image processing unit as correct answer data, when the detection result of the whole image processing unit and the detection result of the partial image processing unit do not match each other. The learned machine learning model for whole images in operation is corrected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Patent Document 1 describes an image processing device that estimates the number of people in an image by combining two types of number-of-people estimation methods using image recognition. The image processing device described in Patent Document 1 estimates the number of people in an image using, for example, a trained machine learning model that has been trained using a large number of images with known numbers of people as a data set. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6995960 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to improve the recognition accuracy in image recognition processing using a trained machine learning model such as that described in Patent Document 1, there is a problem in that, for example, the machine learning model needs to be trained appropriately.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide an image processing device, an image processing method, and a program that can appropriately train a machine learning model used for image recognition. [Means for solving the problem]

[0006] In order to solve the above problem, the image processing device according to the present disclosure includes an image acquisition unit that continuously acquires captured images to be processed, an image reduction unit that reduces the captured images to create reduced images, an entire image processing unit that performs object detection processing using a trained machine learning model for entire images, with the entire reduced image as the detection target, an output unit that continuously outputs information representing the detection results by the entire image processing unit, an image division unit that divides the captured image into multiple partial images, a partial image processing unit that performs object detection processing using a trained machine learning model for partial images, with each partial image as the detection target, in parallel with the detection processing by the entire image processing unit, a comparison unit that compares the detection results of the entire image processing unit with the detection results of the partial image processing unit, and if the detection results of the entire image processing unit and the detection results of the partial image processing unit do not match as a result of the comparison, a model re-training unit that re-trains the trained machine learning model for entire images using a dataset that includes pairs of the corresponding reduced images and the detection results of the partial image processing unit as correct data, and the trained machine learning model for entire images that is currently in operation is corrected based on the results of the re-training.

[0007] The image processing method according to the present disclosure includes the steps of: continuously acquiring captured images to be processed; reducing the captured images to create reduced images; performing an object detection process on the entire reduced images as a detection target using a trained machine learning model for entire images; continuously outputting information representing a detection result of the detection process on the entire reduced images as a detection target; dividing the captured image into a plurality of partial images; performing an object detection process on each of the partial images as a detection target using a trained machine learning model for partial images in parallel with the detection process on the entire reduced images as a detection target; The method includes a step of comparing the detection result of the detection process in which the entire reduced image is the detection target with the detection result of the detection process in which each of the partial images is the detection target; if the comparison results in a mismatch between the detection result of the detection process in which the entire reduced image is the detection target and the detection result of the detection process in which each of the partial images is the detection target, a step of re-training the trained machine learning model for the entire image using a dataset that includes, as correct data, pairs of the corresponding reduced image and the detection result of the detection process in which each of the partial images is the detection target; and a step of correcting the trained machine learning model for the entire image that is currently in operation based on the results of the re-training.

[0008] The program according to the present disclosure includes the steps of: continuously acquiring captured images to be processed; reducing the captured images to create reduced images; performing an object detection process using a trained machine learning model for whole images with the entire reduced images as the detection target; continuously outputting information representing a detection result of the detection process using the entire reduced images as the detection target; dividing the captured image into a plurality of partial images; performing an object detection process using a trained machine learning model for partial images with each partial image as the detection target in parallel with the detection process using the entire reduced images as the detection target; The computer is caused to execute the steps of: comparing the detection results of the detection process that targets the body with the detection results of the detection process that targets each of the partial images; if the comparison results in a mismatch between the detection results of the detection process that targets the entire reduced image and the detection results of the detection process that targets each of the partial images; re-training the trained machine learning model for the entire image using a dataset that includes, as correct data, pairs of the corresponding reduced image and the detection results of the detection process that targets each of the partial images; and correcting the trained machine learning model for the entire image that is currently in operation based on the results of the re-training. [Effects of the Invention]

[0009] According to the image processing device, image processing method, and program of the present disclosure, it is possible to appropriately train a machine learning model used for image recognition. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an example configuration of an image processing device according to an embodiment of the present disclosure. [Figure 2] 10 is a flowchart illustrating an example of overall image processing according to an embodiment of the present disclosure. [Figure 3] 10 is a flowchart illustrating an example of partial image processing according to an embodiment of the present disclosure. [Figure 4] 10 is a flowchart illustrating an example of a relearning process according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a schematic diagram for explaining an example of the operation of an image processing device according to an embodiment of the present disclosure. [Figure 6] 10 is a timing chart for explaining an example of operation of an image processing device according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram for explaining an implementation example of an image processing device according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] An image processing device, an image processing method, and a program according to an embodiment of the present disclosure will be described below with reference to FIGS. 1 to 8. FIG. 1 is a block diagram showing an example configuration of an image processing device according to an embodiment of the present disclosure. FIG. 2 is a flowchart showing an example of overall image processing according to an embodiment of the present disclosure. FIG. 3 is a flowchart showing an example of partial image processing according to an embodiment of the present disclosure. FIG. 4 is a flowchart showing an example of relearning processing according to an embodiment of the present disclosure. FIG. 5 is a schematic diagram for explaining an example operation of an image processing device according to an embodiment of the present disclosure. FIG. 6 is a timing chart for explaining an example operation of an image processing device according to an embodiment of the present disclosure. FIG. 7 is a schematic diagram for explaining an implementation example of an image processing device according to an embodiment of the present disclosure. FIG. 8 is a schematic block diagram showing the configuration of a computer according to an embodiment of the present disclosure. Note that the same or corresponding components in each drawing are designated by the same reference numerals, and description thereof will be omitted as appropriate.

[0012] [Term definition] First, the definitions of some terms used in this embodiment will be explained. "Temporary correction": Correction of a trained machine learning model by relearning for a short period of time using a limited data set so that re-learning can be completed within a few minutes during operation. "Permanent correction": Correction of a trained machine learning model by relearning for a long period of time using a large data set including newly accumulated data sets. "DNN": Abbreviation for Deep Neural Networks. "Full image processing": This refers to detection processing in which a high-resolution image is reduced and the entire image is used as the detection target (high speed but low small object detection performance). "Partial image processing": This refers to detection processing on cut-out images obtained by dividing a high-resolution image (processing takes time as all the cut-out high-resolution images are processed, but small object detection performance is high). "GPU (Graphics Processing Unit)": A computing device specialized for image processing. It is used to implement DNNs (deep networks) because it can perform massive parallel calculations in a short period of time.

[0013] [Configuration of image processing device] FIG. 1 shows an example configuration of an image processing device 1 according to this embodiment. The image processing device 1 can be configured using one or more computers, such as a server equipped with a processor such as a GPU or a CPU (Central Processing Unit), and includes the following functional components, which are configured by a combination of hardware such as computers and peripheral devices and software such as programs executed by the computers. Specifically, the image processing device 1 includes, as its functional components, an image acquisition unit 10, an image reduction unit 11, a storage unit 12, an entire image processing unit 13, an output unit 14, an image segmentation unit 15, a partial image processing unit 16, a matching unit 17, a model re-learning unit 18, a model correction unit 19, and an accumulation unit 20. The entire image processing unit 13 includes a trained machine learning model 131 for the entire image. The partial image processing unit 16 includes a trained machine learning model 161 for the partial image.

[0014] The image acquisition unit 10 continuously acquires the captured images to be processed. Continuous acquisition means repeated periodic acquisition. The image acquisition unit 10 stores the acquired captured images in the storage unit 12. The image acquisition unit 10 may include a camera, or may not include a camera but may include an interface for importing image data captured by a camera. The camera that captures the captured images to be processed may be, for example, a camera mounted on a drone, or a fixed camera installed in a factory, port, river, event venue, etc. The camera may output image data representing the captured image as video data, or may output still image data.

[0015] The image reducing unit 11 reduces the captured image acquired by the image acquiring unit 10 to create a reduced image. The image reducing unit 11 stores the created reduced image, for example, in the storage unit 12. For example, the image reducing unit 11 reduces a captured image having a pixel count of 2,304 × 2,304 pixels to create a reduced image having a pixel count of 768 × 768 pixels.

[0016] The storage unit 12 temporarily stores the captured image, the reduced image, the detection results described later, etc. The storage unit 12 may include, for example, a plurality of distributed storage devices, or may be configured on a cloud or the like.

[0017] The entire image processing unit 13 performs object detection processing using the entire image trained machine learning model 131, with the entire reduced image as the detection target. The entire image trained machine learning model 131 is a trained machine learning model that has been previously trained under supervised machine learning using multiple reduced images, each labeled with information about the detected object contained in the reduced image, as a training dataset. The entire image trained machine learning model 131 inputs image data representing a reduced image and outputs information representing the type, number, position (e.g., information representing the pixel coordinate value of one vertex of a rectangle containing the object), and size (e.g., information specifying the size of two sides and the diagonal of the rectangle containing the object) of one or more objects contained in the reduced image. The entire image trained machine learning model 131 is a trained model that includes, for example, a neural network as an element, and the weighting coefficients between neurons in each layer of the neural network are optimized by machine learning so that the desired solution is output for a large amount of input data. The entire image trained machine learning model 131 is, for example, configured by a combination of a program that performs calculations from input to output and weighting coefficients (parameters) used in the calculations. The whole image processing unit 13 outputs information indicating the type, number, position, size, etc. of the detected objects for each reduced image as the detection result of the object detection process, and stores the information in the storage unit 12, for example.

[0018] The output unit 14 continuously outputs information representing the detection results by the entire image processing unit 13. The output unit 14 includes, for example, a display device, and displays a reduced image or a captured image on the screen of the display device, while superimposing and displaying information representing the detection results by the entire image processing unit 13. Based on the information output by the output unit 14, the observer can check information about the object contained in the captured image in almost real time.

[0019] The image dividing unit 15 divides the captured image into a plurality of partial images. The image dividing unit 15 stores the created partial images, for example, in the storage unit 12. For example, the image dividing unit 15 divides a captured image having a pixel count of 2,304 x 2,304 pixels to create nine partial images of 768 x 768 pixels.

[0020] The partial image processing unit 16 performs object detection processing using the partial image trained machine learning model 161 with each partial image as the detection target, in parallel with the detection processing by the entire image processing unit 13. The partial image trained machine learning model 161 is a trained machine learning model similar to the entire image trained machine learning model 131, and inputs image data representing a partial image and outputs information representing the type, number, position, and size of one or more objects contained in the partial image. The partial image processing unit 16 integrates information representing the type, number, position, size, etc. of the detected objects obtained for each partial image as the detection result of the object detection processing so that it corresponds to one captured image (or a reduced image thereof), outputs the information, and stores it in the memory unit 12, for example.

[0021] The matching unit 17 matches the detection results of the entire image processing unit 13 with the detection results of the partial image processing unit 16. The matching unit 17 compares the detection results of the entire image processing unit 13 with the corresponding detection results of the partial image processing unit 16 and determines whether the contents match. For example, the matching unit 17 compares the type, number, position, and size of the detected objects in each detection result, and determines that the detection results match if the type and number match and the position and size match within a predetermined error range.

[0022] If the result of the matching by the matching unit 17 is that the detection result of the entire image processing unit 13 and the detection result of the partial image processing unit 16 do not match, the model re-learning unit 18 re-learns the trained machine learning model for the entire image 131 using a dataset including, as correct answer data, a pair of the corresponding reduced image and the detection result of the partial image processing unit 16. Here, the corresponding reduced image is a reduced image obtained by reducing the captured image before dividing each partial image that was the detection target of the partial image processing unit 16. In addition, the dataset used for re-learning includes, for example, a dataset in which the detection result of the partial image processing unit 16 is labeled on the corresponding reduced image, and a partial dataset (e.g., a typical dataset) selected from the dataset used when training the trained machine learning model for the entire image 131. Note that the trained machine learning model for the entire image 131 to be re-trained is a model set to the same state as the model currently in operation.

[0023] The model re-learning unit 18 can repeatedly re-learn the trained machine learning model for entire images 131 using the same data set until a new detection result for a reduced image for which the matching result obtained by the detection process using the re-learned trained machine learning model for entire images 131 is inconsistent matches the detection result of the partial image processing unit 16. In this case, the trained machine learning model for entire images 131 will be able to output the same detection result as the detection result of the partial image processing unit 16 for at least the inconsistent reduced image.

[0024] The model correction unit 19 corrects the trained machine learning model for the entire image 131 currently in operation based on the results of re-learning by the model re-learning unit 18. This correction is called "provisional correction."

[0025] If the result of the comparison by the comparison unit 17 shows that the detection result of the entire image processing unit 13 and the detection result of the partial image processing unit 16 do not match, the storage unit 20 stores a pair of the corresponding reduced image and the detection result of the partial image processing unit 16. For example, at the discretion of an operator, the trained machine learning model for entire images 131 can be "permanently corrected" using the data set stored in the storage unit 20.

[0026] [Example of image processing device operation] The image processing device 1 described with reference to Fig. 1 executes the entire image processing shown in Fig. 2 and the partial image processing shown in Fig. 3 in parallel. Furthermore, if the detection results of the entire image processing and the partial image processing do not match, the image processing device 1 executes the re-learning processing shown in Fig. 4 in parallel with the entire image processing and the partial image processing. Figs. 5 and 6 schematically show examples of execution of the entire image processing, the partial image processing, and the re-learning processing. Note that the example image shown in Fig. 5 includes the sea, mountains, the sky, and two ships that are objects sailing on the sea.

[0027] [Whole image processing] FIG. 2 shows an example of whole image processing. In the whole image processing shown in FIG. 2, first, a camera included in the image acquisition unit 10 (or an external camera) captures a captured image (step S10). Next, the image acquisition unit 10 stores the captured image in the storage unit 12 (step S11). Next, the image reduction unit 11 reduces the captured image to create a reduced image, and stores the reduced image in, for example, the storage unit 12 (step S12). Next, the whole image processing unit 13 performs object detection processing on the reduced image using the trained machine learning model for whole image 131, and stores the detection result in, for example, the storage unit 12 (step S13). Next, the output unit 14 outputs the detection result and, for example, the reduced image to be processed (step S14). Thereafter, the processing of steps S10 to S14 is repeatedly executed. In the whole image processing, as shown in FIG. 5, the processing target at one time is the entire reduced image indicated by the black frame. However, the reduced image is an image whose resolution is reduced to, for example, 1 / 3 (total number of pixels is 1 / 9) of that of the captured image. In the example shown in FIG. 6, in the whole image processing, processing of one reduced image (I1) is executed in 60 milliseconds, and when the processing is completed, processing of the next reduced image (I2) begins.

[0028] [Partial image processing] Fig. 3 shows an example of partial image processing. In the partial image processing shown in Fig. 3, first, the image dividing unit 15 reads the latest captured image, for example, from the storage unit 12 (step S20), divides it into a plurality of partial images, and stores them, for example, in the storage unit 12 (step S21). In this case, it is assumed that the image dividing unit 15 divides one captured image into nine partial images (referred to as partial images 1 to 9).

[0029] Next, the partial image processing unit 16 sequentially executes detection processing on the divided partial images 1 to 9 (steps S22 to S30). Next, the partial image processing unit 16 integrates the detection results (step S31). Next, the matching unit 17 reads the detection results of the corresponding entire image processing from, for example, the storage unit 12 (step S32), and matches the detection results of the entire image processing with the detection results of the partial image processing (step S33).

[0030] The collation unit 17 determines whether the detection results of the entire image processing and the partial image processing are the same (step S34), and if they are the same (step S34: YES), returns the process to step S20. On the other hand, if they are not the same (step S34: NO), the collation unit 17 stores the result of the partial image processing, for example, in the storage unit 12 (step S35), and starts a re-learning loop (step S36). Note that in step S36, for example, a flag is simply set to start the re-learning process, and the re-learning process is not executed. Next, the collation unit 17 returns the process to step S20. Thereafter, the processes from step S20 onwards are repeatedly executed.

[0031] In the partial image processing, in the example shown in Fig. 5, the processing target at one time is one of the nine partial images indicated by black frames. Also, as shown in Fig. 6, in the partial image processing, processing for nine partial images (I1) corresponding to one reduced image (I1) is executed in 180 milliseconds, and when the processing is completed, processing for nine partial images (I4) corresponding to the next reduced image (I4) begins.

[0032] [Relearning process] FIG. 4 shows an example of the re-learning process. In the re-learning process shown in FIG. 4, the model re-learning unit 18 first repeatedly determines whether re-learning is necessary (step S40: NO is repeated). For example, if a flag indicating that the re-learning process should be started is set in step S36 of FIG. 3, the model re-learning unit 18 determines that re-learning is necessary (step S40: YES), reads the results of the partial image processing (step S41), and re-learns the trained machine learning model for the entire image 131 (step S42). Next, the model re-learning unit 18 inputs the reduced image for which the results were inconsistent into the re-learned trained machine learning model for the entire image 131, and obtains a detection result (step S43). Next, the model re-learning unit 18 compares the detection result of the partial image processing with the detection result output by the re-learned trained machine learning model for the entire image 131 (step S44). If they do not match (step S44: NO), the model re-learning unit 18 performs re-learning again (step S42). On the other hand, if they match (step S44: YES), the model correction unit 19 reflects the results of the re-learning in the trained machine learning model for the entire image 131 in operation (step S45). In addition, the results of the partial image processing are stored in the storage unit 20 as a data set for permanent correction (step S46).

[0033] As shown in Figures 5 and 6, when partial image processing for a partial image (I1) is completed (timing A), if there is an "object A that cannot be detected by full image processing but can be detected by partial image processing," re-learning of the trained machine learning model for full images is started using the partial image processing results. In the example shown in Figure 6, re-learning is completed in three minutes, and the trained machine learning model for full images 131 in operation is corrected (timing B). As shown in Figures 5 and 6, after re-learning is completed at timing B, object A can be detected by full image processing.

[0034] [Implementation example] Fig. 7 shows an example implementation of the image processing device 1 described with reference to Fig. 1. In the example shown in Fig. 7, an edge-side system 101 mounted on, for example, a drone includes a camera 102 and a GPU server 103 for overall image processing. Also, a base-side system 201 includes a GPU server 202 for partial image processing, a data storage server 203, a GPU server 204 for temporary correction of overall image processing, and a terminal 205 equipped with a monitor that serves as an output destination, all of which are shown below or to the right of the dashed line. The communication line 301 is a wired or wireless communication line or communication line.

[0035] 7, for example, the image acquisition unit 10, the image reduction unit 11, and the entire image processing unit 13 are included in the edge-side system 101, and the output unit 14, the image division unit 15, the partial image processing unit 16, the matching unit 17, and the model re-learning unit 18 are included in the base-side system 201 connected to the edge-side system 101 via wireless communication. Note that the data storage server 203 corresponds to, for example, the memory unit 12 and the accumulation unit 20.

[0036] [Actions and Effects of the Embodiments] Although the "detection process on reduced images" (whole image processing) targeting low-resolution images can be processed quickly, the detection accuracy is low and learning (improving accuracy) is difficult when used alone. However, by learning the detection results of the "detection process on cut-out images" (partial image processing) that is processed in parallel, it becomes possible to detect unknown objects quickly. Furthermore, since the results of the "detection process on cut-out images" are accumulated as a data set for permanent correction, it is possible to improve the performance of the system in the medium to long term. In other words, according to this embodiment, the machine learning model used for image recognition can be appropriately trained.

[0037] (Other embodiments) The above describes in detail the embodiments of the present disclosure with reference to the drawings, but the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope that does not deviate from the gist of the present disclosure.

[0038] [Computer Configuration] FIG. 8 shows a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91 , a main memory 92 , a storage 93 , and an interface 94 . The image processing device 1 described above is implemented in a computer 90. The operations of the above-described processing units are stored in the form of a program in a storage 93. A processor 91 reads the program from the storage 93, loads it into a main memory 92, and executes the above-described processing in accordance with the program. The processor 91 also allocates storage areas in the main memory 92 corresponding to the above-described storage units in accordance with the program.

[0039] The program may be for realizing some of the functions to be performed by the computer 90. For example, the program may be combined with other programs already stored in storage or other programs implemented in other devices to perform the functions. In other embodiments, the computer may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor may be realized by the integrated circuit.

[0040] Examples of storage 93 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of computer 90, or an external medium connected to computer 90 via interface 94 or a communication line. Furthermore, when this program is distributed to computer 90 via a communication line, computer 90 that receives the program may load the program into main memory 92 and execute the above-described processing. In at least one embodiment, storage 93 is a non-transitory tangible storage medium.

[0041] <Additional Notes> The image processing device 1 described in the embodiment can be understood, for example, as follows.

[0042] (1) The image processing device 1 according to the first aspect includes an image acquisition unit 10 that continuously acquires captured images to be processed, an image reduction unit 11 that reduces the captured images to create reduced images, an entire image processing unit 13 that performs object detection processing using a trained machine learning model for entire images 131 with the entire reduced images as the detection target, an output unit 14 that continuously outputs information representing the detection results by the entire image processing unit, an image division unit 15 that divides the captured image into a plurality of partial images, and a partial image processing unit 161 that performs the object detection processing using a trained machine learning model for partial images 161 with each of the partial images as the detection target. The image recognition system includes a partial image processing unit 16 that performs the detection process in parallel with the entire image processing unit, a matching unit 17 that compares the detection result of the entire image processing unit with the detection result of the partial image processing unit, and a model re-learning unit 18 that, if the comparison result shows a mismatch between the detection result of the entire image processing unit and the detection result of the partial image processing unit, re-learns the entire image trained machine learning model using a dataset that includes a pair of the corresponding reduced image and the detection result of the partial image processing unit as ground truth data, and the entire image trained machine learning model 131 in operation is corrected based on the results of the re-learning. According to this aspect and each of the following aspects, it is possible to appropriately train a machine learning model used for image recognition.

[0043] (2) The image processing device 1 according to a second aspect is the image processing device 1 of (1), further comprising a storage unit 20 that stores a pair of the corresponding reduced image and the detection result of the partial image processing unit when the result of the comparison indicates a mismatch between the detection result of the entire image processing unit and the detection result of the partial image processing unit. According to this aspect, the medium- to long-term performance of the system can be improved.

[0044] (3) The image processing device 1 according to the third aspect is the image processing device 1 of (1) or (2), in which, when the detection process for each of the partial images obtained by dividing one of the captured images is completed, the partial image processing unit 16 starts the detection process for each of the partial images obtained by dividing another of the captured images.

[0045] (4) The image processing device 1 according to the fourth aspect is the image processing device 1 of (1) to (3), wherein the model re-learning unit 18 repeatedly re-learns the trained machine learning model 131 for the entire image using the same dataset until a new detection result for the reduced image for which the matching result by the detection process using the re-learned trained machine learning model 131 for the entire image is inconsistent matches the detection result of the partial image processing unit 16.

[0046] (5) The image processing device 1 according to the fifth aspect is the image processing device 1 of (1) to (4), wherein the detection process is a process of detecting the type, number, position, and size of the object, and the matching unit 17 determines that the detection results match when the type and number match and the position and size match within a predetermined error range.

[0047] (6) The image processing device 1 according to the sixth aspect is the image processing device 1 of (1) to (5), in which at least the image acquisition unit 10, the image reduction unit 11, and the entire image processing unit 13 are included in an edge-side system 101, and the output unit 14, the image division unit 15, the partial image processing unit 16, the matching unit 17, and the model re-learning unit 18 are included in a base-side system 201 connected to the edge-side system 101 via wireless communication. [Explanation of symbols]

[0048] 1...Image processing device 10...Image acquisition unit 11...Image reduction section 12...Storage section 13... Overall image processing section 14...Output section 15...Image division unit 16...Part of the image processing section 17...Collation section 18...Model retraining section 19...Model correction section 20...Storage section 101...Edge system 131…Pre-trained machine learning model for the entire image 161...Pre-trained machine learning model for some images 201...Base system

Claims

1. an image acquisition unit that continuously acquires captured images to be processed; an image reduction unit that reduces the captured image to create a reduced image; an entire image processing unit that performs object detection processing using a trained machine learning model for an entire image with the entire reduced image as a detection target; an output unit that continuously outputs information representing the detection result by the entire image processing unit; an image dividing unit that divides the captured image into a plurality of partial images; a partial image processing unit that performs a detection process of the object using a trained machine learning model for partial images with each of the partial images as a detection target in parallel with the detection process by the entire image processing unit; a collation unit that compares the detection result of the entire image processing unit with the detection result of the partial image processing unit; a model re-learning unit that, when the result of the comparison shows that the detection result of the entire image processing unit and the detection result of the partial image processing unit do not match, re-learns the trained machine learning model for the entire image using a dataset that includes a pair of the corresponding reduced image and the detection result of the partial image processing unit as correct answer data; and Equipped with The trained machine learning model for the entire image currently in operation is corrected based on the results of the re-training. Image processing device.

2. If the result of the comparison shows that the detection result of the entire image processing unit and the detection result of the partial image processing unit do not match, a storage unit is provided for storing a pair of the corresponding reduced image and the detection result of the partial image processing unit. Further preparation The image processing device according to claim 1 .

3. When the detection process for each of the partial images obtained by dividing one of the captured images is completed, the partial image processing unit starts the detection process for each of the partial images obtained by dividing another of the captured images. The image processing device according to claim 2 .

4. The model re-learning unit repeatedly re-learns the trained machine learning model for the entire image using the same data set until a new detection result for the reduced image for which the matching result by the detection process using the re-learned trained machine learning model for the entire image has become inconsistent matches the detection result of the partial image processing unit. The image processing device according to claim 3 .

5. The detection process is a process for detecting the type, number, position, and size of the object, and the collation unit determines that the detection results match when the type and number match and the position and size match within a predetermined error range. The image processing device according to claim 4 .

6. At least the image acquisition unit, the image reduction unit, and the entire image processing unit are included in an edge-side system, The output unit, the image division unit, the partial image processing unit, the matching unit, and the model re-learning unit are included in a base-side system connected to the edge-side system via wireless communication. The image processing device according to claim 5 .

7. A step in which an image acquisition unit of an image processing device continuously acquires photographed images to be processed; an image reducing unit of the image processing device reducing the captured image to create a reduced image; an entire image processing unit of the image processing device performing an object detection process using a trained machine learning model for an entire image, with the entire reduced image as a detection target; an output unit of the image processing device continuously outputs information representing a detection result of the detection process targeting the entire reduced image as a detection target; an image dividing unit of the image processing device dividing the captured image into a plurality of partial images; a partial image processing unit of the image processing device performing a detection process of the object using a trained machine learning model for partial images with each of the partial images as a detection target in parallel with the detection process with the entire reduced image as a detection target; a step in which a comparison unit of the image processing device compares a detection result of the detection process in which the entire reduced image is the detection target with a detection result of the detection process in which each of the partial images is the detection target; a model re-learning unit of the image processing device, when the result of the comparison shows that the detection result of the detection process targeting the entire reduced image does not match the detection result of the detection process targeting each of the partial images, re-learns the trained machine learning model for the entire image using a dataset including, as correct answer data, a pair of the corresponding reduced image and the detection result of the detection process targeting each of the partial images; a step in which a model correction unit of the image processing device corrects the trained machine learning model for the entire image that is currently in operation based on the result of the re-learning; An image processing method comprising:

8. A step of continuously acquiring photographed images to be processed; reducing the captured image to create a reduced image; performing an object detection process using a trained machine learning model for an entire image, with the entire reduced image as a detection target; a step of continuously outputting information representing a detection result of the detection process that targets the entire reduced image; Dividing the captured image into a plurality of partial images; performing a process of detecting the object using a trained machine learning model for partial images with each of the partial images as a detection target in parallel with the process of detecting the object using the entire reduced image as a detection target; a step of comparing a detection result of the detection process in which the entire reduced image is the detection target with a detection result of the detection process in which each of the partial images is the detection target; If the result of the comparison shows that the detection result of the detection process targeting the entire reduced image does not match the detection result of the detection process targeting each of the partial images, retraining the trained machine learning model for the entire image using a dataset that includes, as correct answer data, a pair of the corresponding reduced image and the detection result of the detection process targeting each of the partial images; correcting the trained machine learning model for the entire image that is currently in operation based on the results of the re-training; A program that causes a computer to execute the following.

Images