Image processing device

Abstract

The purpose of the present invention is to prepare learning data for weak-point patterns at low cost to improve the detection precision of an object detection model at low cost. An image processing device 1 is provided with: a data expansion device 10 that, on the basis of image data 2 used for machine learning of an object detection model and correct value data 3 that discriminates objects appearing in the image data 2, generates a plurality of sets of weak-point image data that include weak-point patterns for which the object detection model is prone to erroneous detection; and a learning device 20 that selects, from among the plurality of sets of weak-point image data generated by the data expansion device 10, sets of weak-point image data to be used in additional learning, and by using the selected sets of weak-point image data and the correct value data 3 corresponding to the selected sets of weak-point image data, performs additional learning on the object detection model.

Description

Image Processing Device

[0001] The present invention relates to an image processing device.

[0002] Object detection technology is known that detects objects appearing in image data captured by an in-vehicle camera. This type of object detection technology uses a machine-learned object detection model. In machine learning of an object detection model, an annotated dataset that associates the image data with labels (ground truth values) that identify the types of objects appearing in the image data is often used as training data. In order to improve the detection accuracy of an object detection model, it is necessary to prepare an annotated dataset that includes various objects.

[0003] Patent Document 1 discloses a technology for generating label data to be used in object detection using three-dimensional point cloud data by clustering a three-dimensional point cloud using ground truth data assigned to a two-dimensional image.

[0004] International Publication No. 2021 / 181647

[0005] Object detection models tend to mistakenly detect shadows of people, reflections of objects in puddles or glass, etc., as parts of real people. Therefore, images containing these objects can be considered difficult patterns that object detection models have difficulty detecting with high accuracy. In order to improve the detection accuracy of object detection models, it is effective to perform machine learning using image data containing such difficult patterns as training data. Typically, preparing training data for difficult patterns requires photographing and annotating images containing the difficult patterns, which poses challenges in terms of labor hours and cost. Therefore, in order to improve the detection accuracy of object detection models at low cost, it is important to prepare training data for difficult patterns at low cost.

[0006] The technology disclosed in Patent Document 1 does not take into consideration the above-mentioned difficult patterns, and there is room for improvement in terms of improving object detection accuracy at low cost.

[0007] The present invention has been made in view of the above, and aims to prepare learning data for difficult patterns at low cost and improve the detection accuracy of an object detection model at low cost.

[0008] In order to solve the above problem, the image processing device of the present invention is an image processing device that performs additional learning on an object detection model that has been machine-learned to detect objects that appear in image data captured by an in-vehicle camera, and is characterized by comprising: a data extension device that generates a plurality of weak image data including weak patterns that are likely to be falsely detected by the object detection model based on the image data used in the machine learning of the object detection model and correct value data that identifies the objects that appear in the image data; and a learning device that selects the weak image data to be used for the additional learning from the plurality of weak image data generated by the data extension device, and performs the additional learning on the object detection model using the selected weak image data and the correct value data corresponding to the selected weak image data.

[0009] According to the present invention, it is possible to prepare learning data for difficult patterns at low cost and improve the detection accuracy of an object detection model at low cost. Problems, components, and effects other than those described above will become clear from the description of the following embodiments.

[0010] 9 is a block diagram showing the configuration of an image processing device according to this embodiment. It is a flowchart showing the processing performed by the data extension device shown in FIG. 1. It is a diagram explaining step S104 in FIG. 2. It is a diagram explaining step S105 in FIG. 2. It is a diagram explaining step S105 in FIG. 2. It is a diagram explaining step S107 in FIG. 2. It is a diagram explaining steps S108 and S109 in FIG. 2. It is a diagram explaining steps S108 and S109 in FIG. 2 using an example different from FIG. 7. It is a flowchart showing the processing performed by the learning device shown in FIG. 1. It is a diagram explaining steps S202 and S204 in FIG. 9.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that components with the same reference numerals in each embodiment have similar components in each embodiment unless otherwise specified, and description thereof will be omitted.

[0012] FIG. 1 is a block diagram showing the configuration of an image processing apparatus 1 according to this embodiment.

[0013] The image processing device 1 is configured by a computer. The image processing device 1 includes a processor such as a CPU and memories such as a ROM and a RAM, and realizes various functions by the processor executing programs stored in the memories.

[0014] The image processing device 1 is a device that performs additional learning on an object detection model that has undergone machine learning to detect objects appearing in image data 2 captured by an in-vehicle camera.

[0015] The image processing device 1 receives input of a data set in which image data 2 and correct answer data 3 assigned to each object appearing in the image data 2 are associated with each other. The image data 2 may be a still image captured by an on-board camera, or a moving image captured by an on-board camera. Objects appearing in the image data 2 include, for example, roads, people such as pedestrians, vehicles such as bicycles or automobiles, buildings, or other structures. The correct answer data 3 includes information identifying the position and size of an object appearing in the image data 2, and information identifying the type of object appearing in the image data 2. For example, the correct answer data 3 includes a bounding box that identifies the position and size of an object appearing in the image data 2 and surrounds the object, and a label that identifies the type of object appearing in the image data 2. The bounding box forms, for example, a rectangle along the horizontal and vertical directions of the image data 2 (the directions of the coordinate axes that are orthogonal to each other in the image coordinate system of the image data 2). The image data 2 and the correct answer data 3 are annotated data sets, and are learning data (e.g., training data) used in machine learning to build an object detection model. The object detection model is not particularly limited, but may be configured, for example, by a convolutional neural network model (CNN).

[0016] The image processing device 1 includes a data extension device 10 that generates a plurality of pieces of poor image data including poor patterns that are likely to be erroneously detected by an object detection model, and a learning device 20 that performs additional learning on the object detection model using the plurality of pieces of poor image data generated by the data extension device 10. The poor image data generated by the data extension device 10 is stored in an extended data storage device 4 in association with correct answer data 3 corresponding to the poor image data. The object detection model that has undergone additional learning by the learning device 20 is stored in a learning result storage device 5. Note that the extended data storage device 4, the learning result storage device 5, and the storage device that stores the image data 2 and the correct answer data 3 may be configured integrally with the image processing device 1 as part of the hardware of the image processing device 1.

[0017] A poorly detected pattern is an image pattern that is difficult to detect with high accuracy in an object detection model. A poorly detected pattern is an image pattern that is adjacent to a real object and has a shape similar to the real object. A poorly detected pattern is an image pattern that is likely to be erroneously detected as part of a real object, such as a shadow of an object, a puddle, or a reflection of an object on glass or a mirror.

[0018] The data expansion device 10 generates a plurality of poor image data based on image data 2 used in machine learning of the object detection model and correct answer data 3 that identifies objects appearing in the image data 2. The data expansion device 10 has, as its functions, an acquisition unit 101, an object detection unit 102, a target object determination unit 103, a mask image generation unit 104, a paste area calculation unit 105, and a synthesis unit 106.

[0019] The acquisition unit 101 acquires image data 2 and correct answer data 3 used in the machine learning of the object detection model. The object detection unit 102 detects objects appearing in the image data 2 acquired by the acquisition unit 101. The target object determination unit 103 compares the correct answer data 3 acquired by the acquisition unit 101 with the detection results of the object detection unit 102, and determines objects with high reliability from among the objects detected by the object detection unit 102. The mask image generation unit 104 distinguishes between areas where the object exists and areas where the object does not exist within the frame lines corresponding to the object determined to have high reliability by the target object determination unit 103, and generates a mask image extracting the area determined to contain the object.

[0020] The paste area calculation unit 105 calculates a paste area where the mask image generated by the mask image generation unit 104 is pasted onto the image data 2, based on the dimensions of the frame line corresponding to the object determined to have high reliability by the target object determination unit 103. The synthesis unit 106 transforms the mask image generated by the mask image generation unit 104 so that the mask image is pasted into the paste area calculated by the paste area calculation unit 105, and synthesizes the mask image with the image data 2, thereby generating poor image data. Details of the processes performed by each function of the data expansion device 10 will be described later using Figures 2 to 8.

[0021] The learning device 20 selects weak image data to be used for additional learning from the plurality of weak image data generated by the data expansion device 10, and performs additional learning on the object detection model using the selected weak image data and correct answer data corresponding to the selected weak image data. The learning device 20 has, as its functions, an inference unit 201, a learning data selection unit 202, and a learning unit 203.

[0022] The inference unit 201 inputs multiple pieces of poor image data generated by the data expansion device 10 into the object detection model before additional learning and infers objects appearing in each piece of poor image data. The training data selection unit 202 selects, from the multiple pieces of poor image data generated by the data expansion device 10, poor image data for which the inference result of the inference unit 201 has a large error (hereinafter also referred to as "inference error"), as training data to be used for additional learning, based on the correct answer data corresponding to the poor image data input into the object detection model before additional learning and the inference result of the inference unit 201. The learning unit 203 performs additional learning using the poor image data selected by the training data selection unit 202. Details of the processing performed by each function of the learning device 20 will be described later using Figures 9 and 10.

[0023] Fig. 2 is a flowchart showing the processing performed by the data expansion device 10 shown in Fig. 1. Fig. 3 is a diagram explaining step S104 in Fig. 2. Figs. 4 and 5 are diagrams explaining step S105 in Fig. 2. Fig. 6 is a diagram explaining step S107 in Fig. 2. Fig. 7 is a diagram explaining steps S108 and S109 in Fig. 2. Fig. 8 is a diagram explaining steps S108 and S109 in Fig. 2 using an example different from Fig. 7.

[0024] In step S101, the acquisition unit 101 acquires image data 2 and correct answer data 3 used in the machine learning of the object detection model. Furthermore, the acquisition unit 101 acquires setting information set in advance by a user. This setting information is information indicating the type of a difficult pattern (e.g., a shadow of an object) for which difficult image data is to be generated.

[0025] In step S102, the acquisition unit 101 identifies, based on the setting information acquired in step S101, the image data 2 and correct answer data 3 in which the target weak pattern type exists from the image data 2 and correct answer data 3 acquired in step S101.

[0026] In step S103, the object detection unit 102 detects an object appearing in the image data 2 identified in step S102. At this time, the object detection unit 102 performs object detection using a method different from the object detection model. Specifically, the object detection unit 102 performs object detection using a method with higher detection accuracy than the object detection model. For example, the object detection unit 102 performs object detection on a pixel-by-pixel basis in the image data 2, such as panoptic segmentation or instance segmentation. Because the object detection unit 102 can perform object detection on a pixel-by-pixel basis, it is possible to distinguish between real objects and difficult patterns as different objects.

[0027] In step S104, the target object determination unit 103 compares the detection result of the object detection unit 102 obtained in step S103 with the correct value data 3 identified in step S102. Then, the target object determination unit 103 determines an object (an object from which a weak pattern was generated) detected by the object detection unit 102 that matches the correct value data 3. Specifically, as shown in FIG. 3 , the target object determination unit 103 determines an object detected by the object detection unit 102 that matches the label of the correct value data 3 and is contained within the frame of the correct value data 3.

[0028] In step S105, the target object determination unit 103 calculates the reliability of the detection of the object determined in step S104. This reliability is calculated by at least one of the histogram processing shown in Fig. 4 and the labeling processing shown in Fig. 5.

[0029] The histogram processing shown in FIG. 4 will be described. The left side of FIG. 4 shows a case where the object detection unit 102 correctly detects the entire body of a person. The right side of FIG. 4 shows a case where the object detection unit 102 fails to correctly detect the entire body of a person, resulting in the torso being missing (undetected). The target object determination unit 103 counts the number of pixels in the region within the frame of the correct answer value data 3 where an object detected by the object detection unit 102 exists for each pixel row (x-direction bin) arranged in the horizontal direction (x-direction) to create a histogram (the upper histogram in FIG. 4). Similarly, the target object determination unit 103 counts the number of pixels in the region within the frame of the correct answer value data 3 where an object detected by the object detection unit 102 exists for each pixel row (y-direction bin) arranged in the vertical direction (y-direction) to create a histogram (the lower histogram in FIG. 4). The target object determination unit 103 then calculates the reliability using the following equation (1):

[0030] Confidence = (number of bins with a frequency of 1 or more) / (total number of bins in the x direction and y direction) (1)

[0031] That is, the target object determination unit 103 calculates the reliability as the ratio of the number of pixel rows that make up the area within the frame line of the correct value data 3 corresponding to the object detected by the object detection unit 102 to the total number of pixel rows that make up the area within the frame line in which the object exists.

[0032] The labeling process shown in FIG. 5 will be described. The left side of FIG. 5 shows a case where the object detection unit 102 correctly detects the entire body of a person. The right side of FIG. 5 shows a case where the object detection unit 102 fails to correctly detect the entire body of a person, resulting in the torso of the person being missing (undetected). The target object determination unit 103 labels each area within the frame of the correct answer value data 3 where an object detected by the object detection unit 102 exists, and determines the continuity of the area within the frame where the object exists. If the object detection unit 102 correctly detects an object, the same label number is assigned to the entire area within the frame where the object exists, and the number of assigned label numbers is 1. If the object detection unit 102 fails to correctly detect an object, the area within the frame where the object exists is often divided into multiple parts. In this case, multiple label numbers are assigned, and the number of assigned label numbers is 2 or more.

[0033] The target object determination unit 103 then calculates the area of ​​each labeled region. The table shown in Fig. 5 lists the relative area of ​​each labeled region when the area of ​​all regions in which objects exist within the frame is set to 100. The target object determination unit 103 then calculates the reliability using the following equation (2).

[0034] Confidence = (area of ​​the largest region among the labeled regions) / (area of ​​all regions in which an object exists within the frame) (2)

[0035] That is, the target object determination unit 103 calculates the reliability as the ratio of the area of ​​the largest area in which the object exists to the area of ​​all areas in which the object exists within the frame lines of the correct value data 3 corresponding to the object detected by the object detection unit 102.

[0036] The target object determination unit 103 may employ either the reliability calculated by the histogram processing shown in Fig. 4 or the reliability calculated by the labeling processing shown in Fig. 5. The target object determination unit 103 may employ the reliability calculated by the histogram processing shown in Fig. 4 in preference to the reliability calculated by the labeling processing shown in Fig. 5. The target object determination unit 103 may employ the multiplied value or average value of the reliability calculated by the histogram processing shown in Fig. 4 and the reliability calculated by the labeling processing shown in Fig. 5.

[0037] In step S106, the target object determination unit 103 determines whether the reliability calculated in step S105 is equal to or greater than a threshold. If the reliability calculated in step S105 is equal to or greater than the threshold, the target object determination unit 103 determines that an object with high reliability has been detected in the object detection unit 102, and the process proceeds to step S107. If the reliability calculated in step S105 is less than the threshold, the target object determination unit 103 determines that an object with high reliability has not been detected in the object detection unit 102, and the process shown in FIG. 2 ends.

[0038] In step S107, the mask image generation unit 104 distinguishes between areas within the frame of the correct answer data 3 corresponding to an object determined to have high reliability by the target object determination unit 103 and areas where the object exists. The mask image generation unit 104 then generates a mask image by extracting areas where the object is determined to exist. In the example of FIG. 6 , the mask image generation unit 104 sets the alpha value of all areas within the frame other than areas where the object determined to have high reliability exists (i.e., areas where the object does not exist) to zero. This allows the mask image generation unit 104 to generate a mask image by extracting only areas where the object determined to have high reliability exists.

[0039] In step S108, the paste area calculation unit 105 calculates, in accordance with the type of the target difficult pattern, a paste area where the mask image generated by the mask image generation unit 104 is pasted onto the image data 2. At this time, the paste area calculation unit 105 calculates the paste area of ​​the mask image using a value obtained by multiplying at least one of the horizontal dimension and the vertical dimension of the frame line of the correct answer value data 3 corresponding to the object determined by the target object determination unit 103 to have high reliability by a ratio specified in advance as a random number.

[0040] 7 shows an example of a paste area calculated when an object shadow is set as the type of difficult pattern, and a mask image pasted in the paste area. In the example shown in FIG. 7 , the paste area calculation unit 105 calculates the lower left and lower right vertices of the frame line of the correct answer value data 3 corresponding to the object determined to have high reliability by the target object determination unit 103 as the first and second coordinates. The paste area calculation unit 105 calculates the third coordinate as a point on an arc of a semicircle HC that is centered at the midpoint O between the first and second coordinates and has a radius of h', which is the vertical dimension h of the frame line multiplied by a ratio specified by a random number within a predetermined range. The paste area calculation unit 105 calculates the fourth coordinate as a point on the arc offset from the third coordinate by the value w', which is the horizontal dimension w of the frame line of the correct answer value data 3 multiplied by a ratio specified by a random number within a predetermined range. Then, the paste area calculation unit 105 calculates the area surrounded by a quadrangle having the first coordinate, the second coordinate, the third coordinate, and the fourth coordinate as vertices as the paste area.

[0041] 8 shows an example of a paste area calculated when a reflection of an object reflected on glass or the like is set as the type of difficult pattern, and an example of a mask image pasted onto the paste area. In the example shown in FIG. 8 , the paste area calculation unit 105 sets the center of gravity O' of the paste area within an area surrounded by a circle C whose center is the center of gravity O of an object determined to have high reliability by the target object determination unit 103 and whose radius is a value h' obtained by multiplying the vertical dimension h of the frame line of the correct answer value data 3 corresponding to the object by a ratio specified by a random number within a first predetermined range. The paste area calculation unit 105 calculates the value h' obtained by multiplying the vertical dimension h of the frame line by a ratio specified by a random number within a second predetermined range as the first dimension of the paste area. The paste area calculation unit 105 calculates the value w' obtained by multiplying the horizontal dimension w of the frame line by a ratio specified by a random number within a second predetermined range as the second dimension of the paste area. The paste area calculation unit 105 then calculates, as the paste area, an area surrounded by a rectangle having the set center of gravity O' and the calculated first and second dimensions h' and w'. Note that the first and second dimensions of the paste area may be dimensions in directions perpendicular to each other. The first and second dimensions of the paste area may be, for example, the vertical and horizontal dimensions of the paste area.

[0042] In step S109, the composition unit 106 deforms the mask image generated by the mask image generation unit 104 so that the mask image is pasted in the paste area calculated by the paste area calculation unit 105. Then, the composition unit 106 pastes the deformed mask image in the paste area and combines it with the image data 2. In the example shown in Fig. 7, the composition unit 106 deforms the mask image so that it fits into the paste area surrounded by a rectangle having first, second, third, and fourth coordinates as vertices. In the example shown in Fig. 8, the composition unit 106 deforms the mask image so that it fits into the paste area surrounded by a rectangle having a center of gravity O', a first dimension h', and a second dimension w'.

[0043] When synthesizing the deformed mask images, the synthesizing unit 106 may subtract the luminance value or RGB value of the deformed mask images or may perform a blurring process on the deformed mask images depending on the type of the target difficult pattern. For example, in the example shown in Fig. 7, the synthesizing unit 106 may subtract the luminance value of the deformed mask images, and in the example shown in Fig. 8, the synthesizing unit 106 may subtract the RGB value of the deformed mask images. This allows the synthesizing unit 106 to generate difficult image data that closely resembles the actual difficult pattern.

[0044] 8, the compositing unit 106 first cuts out the mask image from the original image (image data 2 before compositing) to express the context of the mask image, then pastes the transformed mask image into the paste area of ​​the original image, and then pastes the cut-out mask image back into its original position. At this time, the transformed mask image is composited using alpha blending to simulate a reflected image. In this way, the compositing unit 106 generates difficult image data containing the target difficult pattern.

[0045] In step S110, the data expansion device 10 outputs the generated poor image data and the correct answer value data 3 corresponding to the poor image data to the expansion data storage device 4. The poor image data and the correct answer value data 3 are associated with each other and stored in the expansion data storage device 4. Thereafter, the data expansion device 10 ends the process shown in FIG.

[0046] In step S108, the paste area calculation unit 105 can calculate multiple paste areas that are different from each other by changing the random number that specifies the ratio used in calculating the above-mentioned h' and w'. Then, in step S109, the composition unit 106 can transform the mask image into multiple variations and paste them into each of the multiple paste areas calculated by the paste area calculation unit 105.

[0047] That is, the paste area calculation unit 105 can calculate multiple paste areas using a value obtained by multiplying at least one of the horizontal dimension and the vertical dimension of the frame line of the correct value data 3 by multiple ratios specified in advance as random numbers. The synthesis unit 106 can generate multiple poor image data by transforming the mask image into multiple variations so that the mask image can be pasted into each of the multiple paste areas calculated by the paste area calculation unit 105.

[0048] Fig. 9 is a flowchart showing the processing performed by the learning device 20 shown in Fig. 1. Fig. 10 is a diagram for explaining steps S202 and S204 in Fig. 9.

[0049] In step S201, the inference unit 201 reads out a plurality of weak image data and its correct answer data 3 stored in the extended data storage device 4. The inference unit 201 reads out the object detection model before additional learning stored in the learning result storage device 5. The inference unit 201 inputs the read out plurality of weak image data into the object detection model before additional learning, and infers the objects appearing in each of the weak image data (performs object detection).

[0050] In step S202, the training data selection unit 202 compares the inference result of the inference unit 201 obtained in step S201 with the supervised value data 3 read out in step S201. For example, the training data selection unit 202 compares the reference points of the frame lines of the supervised value data 3 with the reference points of the frame lines included in the inference result of the inference unit 201 for the object from which the difficult pattern was generated. Then, the training data selection unit 202 calculates the inference error (Loss) of the inference unit 201. For example, the training data selection unit 202 calculates the inference error using the following equation (3).

[0051] Inference error = (reference point of the frame line of the correct answer data 3) - (reference point of the frame line included in the inference result of the inference unit 201) (3)

[0052] The learning data selection unit 202 calculates the inference errors of the inference unit 201 for a plurality of poor image data, and creates a histogram of the inference errors as shown in FIG.

[0053] In step S203, the learning data selection unit 202 selects, based on the histogram created in step S202, from among the multiple difficult image data, image data with a large inference error, together with the correct answer data 3 corresponding to the difficult image data, as learning data to be used for additional learning.

[0054] Furthermore, the learning data selection unit 202 selects, as the learning data to be used for additional learning, not only the poorly performing image data with a large inference error but also the image data 2 used in the machine learning of the object detection model, together with the correct answer data 3 corresponding to the image data 2. In other words, the learning data selection unit 202 selects, as the learning data to be used for additional learning, not only the poorly performing image data with a large inference error and the correct answer data 3, but also the existing image data 2 and the correct answer data 3.

[0055] In step S204, the learning unit 203 performs additional learning of the object detection model using the poor image data and image data 2 selected by the learning data selection unit 202 in step S203. The learning unit 203 outputs the object detection model after the additional learning to the learning result storage device 5. The object detection model after the additional learning is stored in the learning result storage device 5. Thereafter, the learning device 20 ends the processing shown in FIG.

[0056] 10, the histogram of inference errors after additional training has a peak position near zero and a smaller standard deviation than the histogram of inference errors before additional training. This shows that the detection accuracy of the object detection model additionally trained by the learning device 20 has clearly improved.

[0057] As described above, the image processing device 1 of this embodiment is an image processing device that performs additional learning on an object detection model that has undergone machine learning to detect objects appearing in image data 2 captured by an in-vehicle camera. The image processing device 1 includes a data expansion device 10 that generates a plurality of poor image data including poor patterns that are likely to be erroneously detected by the object detection model, based on the image data 2 used in the machine learning of the object detection model and correct answer data 3 that identifies objects appearing in the image data 2. The image processing device 1 includes a learning device 20 that selects poor image data to be used for additional learning from the plurality of poor image data generated by the data expansion device 10, and performs additional learning on the object detection model using the selected poor image data and the correct answer data 3 corresponding to the selected poor image data.

[0058] As a result, the image processing device 1 can generate difficult image data by expanding existing training data, eliminating the need for additional photographing and annotation work. Therefore, the image processing device 1 can easily generate a large amount of training data for difficult patterns. Furthermore, the image processing device 1 can perform additional training using selected difficult image data, rather than using all of the difficult image data generated by the data expansion device 10. Therefore, the image processing device 1 can efficiently train only the difficult patterns. Therefore, the image processing device 1 can prepare training data for difficult patterns at low cost, thereby improving the detection accuracy of the object detection model at low cost.

[0059] Furthermore, in the image processing device 1 of this embodiment, the correct value data 3 includes a frame line that identifies the position and size of an object appearing in the image data 2 and surrounds the object, and a label that identifies the type of object appearing in the image data 2. The data expansion device 10 includes an acquisition unit 101 that acquires the image data 2 and the correct value data 3 used in the machine learning of the object detection model, an object detection unit 102 that detects an object appearing in the image data 2 acquired by the acquisition unit 101, a target object determination unit 103 that compares the detection result of the object detection unit 102 with the correct value data 3 acquired by the acquisition unit 101 and determines an object with high reliability from the objects detected by the object detection unit 102, and a region within the frame line of the correct value data 3 corresponding to an object determined to have high reliability by the target object determination unit 103, where the object exists and where it does not exist, and The image processing device includes a mask image generating unit 104 that generates a mask image by extracting an area where an object is determined to exist, a paste area calculation unit 105 that calculates a paste area where the mask image generated by the mask image generating unit 104 is to be pasted onto image data 2 based on the dimensions of the frame line of the correct answer value data 3 corresponding to the object determined to have high reliability by the target object determination unit 103, and a synthesis unit 106 that generates poor quality image data by transforming the mask image generated by the mask image generating unit 104 so that the mask image can be pasted into the paste area calculated by the paste area calculation unit 105 and synthesizing the transformed mask image onto the image data 2.

[0060] As a result, the image processing device 1 can generate mask images only from the detection results of the highly reliable object detection unit 102, thereby generating high-quality mask images. Therefore, the image processing device 1 can generate high-quality difficult image data that accurately simulates difficult patterns. Furthermore, since the image processing device 1 calculates the attachment area using the dimensions of the frame lines of the existing correct answer data 3, it can easily determine the deformation mode and attachment position of the mask image, and can easily simulate various difficult patterns. Therefore, the image processing device 1 can easily generate a large amount of various difficult image data. Therefore, the image processing device 1 can prepare high-quality learning data for difficult patterns at even lower cost, and can further improve the detection accuracy of the object detection model at even lower cost.

[0061] Furthermore, in the image processing device 1 of this embodiment, the target object determination unit 103 calculates the reliability as the ratio of the number of pixel rows that make up the area within the frame line of the correct value data 3 corresponding to the object detected by the object detection unit 102 to the total number of pixel rows that make up the area within the frame line in which the object exists.

[0062] This allows the image processing device 1 to generate mask images only from detection results of the object detection unit 102 with high reliability, thereby preventing the generation of mask images in which, for example, parts of a person are missing. The image processing device 1 can reliably generate high-quality image data of difficult patterns that accurately simulates difficult patterns. Therefore, the image processing device 1 can reliably prepare high-quality training data of difficult patterns at low cost, and can reliably improve the detection accuracy of the object detection model at low cost.

[0063] Furthermore, in the image processing device 1 of this embodiment, the target object determination unit 103 calculates the reliability as the ratio of the area of ​​the largest area in which the object exists to the area of ​​the entire area in which the object exists within the frame lines of the correct value data 3 corresponding to the object detected by the object detection unit 102.

[0064] This allows the image processing device 1 to generate mask images only from detection results of the object detection unit 102 with high reliability, thereby preventing the generation of mask images in which, for example, parts of a person are missing. The image processing device 1 can reliably generate high-quality image data of difficult patterns that accurately simulates difficult patterns. Therefore, the image processing device 1 can reliably prepare high-quality training data of difficult patterns at low cost, and can reliably improve the detection accuracy of the object detection model at low cost.

[0065] Furthermore, in the image processing device 1 of this embodiment, the frame lines of the correct value data 3 form a rectangle along the horizontal and vertical directions of the image data. The paste area calculation unit 105 calculates multiple paste areas using a value obtained by multiplying at least one of the horizontal dimension and the vertical dimension of the frame line of the correct value data 3 corresponding to an object determined to have high reliability by the target object determination unit 103 by multiple ratios specified in advance using random numbers. The synthesis unit 106 transforms the mask image generated by the mask image generation unit 104 into multiple variations so that the mask image can be pasted into each of the multiple paste areas calculated by the paste area calculation unit 105, thereby generating multiple pieces of poor image data.

[0066] As a result, the image processing device 1 uses the dimensions of the borders of the existing correct answer data 3 and calculates the pasting area using a ratio specified by a random number, making it possible to easily simulate a wide variety of difficult patterns. Therefore, the image processing device 1 can easily generate a large amount of image data with a wide variety of difficult patterns. Therefore, the image processing device 1 can prepare learning data for difficult patterns at even lower cost, and can further improve the detection accuracy of the object detection model at lower cost.

[0067] Furthermore, in the image processing device 1 of this embodiment, the paste area calculation unit 105 calculates, as first and second coordinates, the lower left and right vertices of the frame line of the correct answer data 3 corresponding to an object determined to have high reliability by the target object determination unit 103. The paste area calculation unit 105 calculates, as third coordinates, a point on a semicircular arc extending below the first and second coordinates, with the center being the midpoint between the first and second coordinates and the radius being a value obtained by multiplying the vertical dimension of the frame line by a ratio specified by a random number within a predetermined range. The paste area calculation unit 105 calculates, as fourth coordinates, a point on the arc offset from the third coordinate by a value obtained by multiplying the horizontal dimension of the frame line by a ratio specified by a random number within a predetermined range. The paste area calculation unit 105 calculates, as the paste area, an area enclosed by a rectangle having vertices at the first, second, third, and fourth coordinates.

[0068] This allows the image processing device 1 to easily simulate difficult patterns, such as shadows extending below an object, which are prone to false detection by object detection models. Furthermore, because the image processing device 1 limits the range of random numbers used when calculating the attachment area, it is possible to prevent the generation of poor-quality difficult patterns, such as extremely long shadows. Therefore, the image processing device 1 can prepare high-quality learning data for difficult patterns at low cost, thereby further improving the detection accuracy of the object detection model at low cost.

[0069] Furthermore, in the image processing device 1 of this embodiment, the paste area calculation unit 105 sets the center of gravity of the paste area within an area surrounded by a circle whose center is the center of gravity of the object determined to have high reliability by the target object determination unit 103, and whose radius is a value obtained by multiplying the vertical dimension of the frame line of the correct value data 3 corresponding to the object by a ratio specified by a random number within a first predetermined range. The paste area calculation unit 105 calculates the value obtained by multiplying the vertical dimension of the frame line by a ratio specified by a random number within a second predetermined range as the first dimension of the paste area. The paste area calculation unit 105 calculates the value obtained by multiplying the horizontal dimension of the frame line by a ratio specified by a random number within the second predetermined range as the second dimension of the paste area. The paste area calculation unit 105 calculates the area surrounded by a rectangle having the set center of gravity and the calculated first and second dimensions as the paste area.

[0070] This allows the image processing device 1 to easily simulate difficult patterns, such as object reflections, that are prone to false detection by object detection models. Furthermore, because the image processing device 1 limits the range of random numbers used when calculating the attachment area, it is possible to prevent the generation of poor-quality difficult patterns, such as extremely large reflections. Therefore, the image processing device 1 can prepare high-quality learning data for difficult patterns at low cost, thereby further improving the detection accuracy of the object detection model at low cost.

[0071] Furthermore, in the image processing device 1 of this embodiment, the learning device 20 has an inference unit 201 that inputs multiple pieces of difficult image data generated by the data extension device 10 into the object detection model before additional learning and infers objects that appear in each piece of difficult image data, a learning data selection unit 202 that selects, from the multiple pieces of difficult image data generated by the data extension device 10, difficult image data with a large error in the inference result as learning data to be used for additional learning, based on the correct answer data 3 corresponding to the difficult image data input into the object detection model and the inference result of the inference unit 201, and a learning unit 203 that performs additional learning using the difficult image data selected by the learning data selection unit 202.

[0072] This allows the image processing device 1 to perform additional learning by focusing on image data containing difficult patterns that are particularly susceptible to false detection by the object detection model. Therefore, the image processing device 1 can efficiently learn only the particularly difficult patterns while suppressing the cost of additional learning. Therefore, the image processing device 1 can prepare learning data for difficult patterns that have a high learning effect at low cost, and can further improve the detection accuracy of the object detection model at low cost.

[0073] Furthermore, in the image processing device 1 of this embodiment, the training data selection unit 202 selects, as training data, not only the poor image data with a large inference error but also the image data 2 used in the machine learning of the object detection model. The training unit 203 performs additional training using the poor image data and the image data 2 selected by the training data selection unit 202.

[0074] This allows the image processing device 1 to prevent bias in the training data caused by using only the difficult image data, which would otherwise cause the generalization performance of the object detection model to decline. Therefore, the image processing device 1 can prepare training data for difficult patterns at low cost, thereby improving the detection accuracy of the object detection model at low cost and preventing a decline in the generalization performance of the object detection model.

[0075] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described components. Furthermore, some of the components of one embodiment can be replaced with components of another embodiment, and components of another embodiment can be added to components of one embodiment. Furthermore, some of the components of each embodiment can be added, deleted, or replaced with other components.

[0076] Furthermore, the above-described components, functions, processing units, or processing means may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described components or functions may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, or files that implement each function may be stored in a storage device such as a memory, a hard disk, or an SSD (solid state drive), or in a storage medium such as an IC card, an SD card, or a DVD.

[0077] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0078] 1...image processing device, 2...image data, 3...correct answer data, 4...extended data storage device, 5...learning result storage device, 10...data extension device, 101...acquisition unit, 102...object detection unit, 103...target object determination unit, 104...mask image generation unit, 105...paste area calculation unit, 106...synthesis unit, 20...learning device, 201...inference unit, 202...learning data selection unit, 203...learning unit

Claims

1. An image processing device that performs additional learning on an object detection model that has undergone machine learning to detect objects appearing in image data captured by an in-vehicle camera, comprising: a data expansion device that generates a plurality of pieces of poor image data including poor patterns that are likely to be falsely detected by the object detection model, based on the image data used in the machine learning of the object detection model and correct answer value data that identifies the objects appearing in the image data; and a learning device that selects the poor image data to be used for the additional learning from the plurality of pieces of poor image data generated by the data expansion device, and performs the additional learning on the object detection model using the selected poor image data and the correct answer value data corresponding to the selected poor image data.

2. The correct answer data includes a frame that identifies the position and size of the object appearing in the image data and surrounds the object, and a label that identifies the type of the object appearing in the image data, and the data expansion device comprises: an acquisition unit that acquires the image data and the correct answer data used in the machine learning of the object detection model; an object detection unit that detects the object appearing in the image data acquired by the acquisition unit; a target object determination unit that compares the detection result of the object detection unit with the correct answer data acquired by the acquisition unit and determines an object with a high reliability from among the objects detected by the object detection unit; a mask image generation unit that distinguishes between an area where the object exists and an area where the object does not exist within the area within the frame that corresponds to the object determined to have a high reliability by the target object determination unit, and generates a mask image that extracts the area where the object is determined to exist; and a paste area calculation unit that calculates a paste area in which the mask image generated by the mask image generation unit is pasted on the image data based on the dimensions of the frame that corresponds to the object determined to have a high reliability by the target object determination unit. and a synthesis unit that generates the poor quality image data by transforming the mask image generated by the mask image generation unit so that the mask image is pasted in the paste area calculated by the paste area calculation unit and synthesizing the mask image with the image data.

3. The image processing device described in claim 2, characterized in that the target object determination unit calculates the reliability as the ratio of the number of pixel rows that make up the area within the frame line in which the object exists to the total number of pixel rows that make up the area within the frame line corresponding to the object detected by the object detection unit.

4. The image processing device described in claim 3, characterized in that the target object determination unit calculates the reliability as the ratio of the area of ​​the largest area in which the object exists to the area of ​​the entire area in which the object exists within the frame line corresponding to the object detected by the object detection unit.

5. The image processing device described in claim 2, characterized in that the frame line forms a rectangle along the horizontal and vertical directions of the image data, the paste area calculation unit calculates the multiple paste areas using a value obtained by multiplying at least one of the horizontal dimension and the vertical dimension of the frame line corresponding to the object determined by the target object determination unit to have high reliability by multiple ratios specified in advance as random numbers, and the synthesis unit transforms the mask image generated by the mask image generation unit into multiple variations so that the mask image can be pasted into each of the multiple paste areas calculated by the paste area calculation unit, thereby generating the multiple difficult image data.

6. The image processing device according to claim 5, wherein the paste area calculation unit calculates, as first coordinates and second coordinates, the bottom left and bottom right vertices of the frame line corresponding to the object determined by the target object determination unit to have high reliability; calculates, as third coordinates, a point on a semicircular arc that is centered at the midpoint between the first coordinates and the second coordinates and that is expanded below the first coordinates and the second coordinates, with the radius being the value obtained by multiplying the vertical dimension of the frame line by the ratio specified by the random number within a predetermined range; calculates, as fourth coordinates, a point on the arc that is offset from the third coordinates by the value obtained by multiplying the horizontal dimension of the frame line by the ratio specified by the random number within the predetermined range; and calculates, as the paste area, an area surrounded by a rectangle whose vertices are the first coordinates, the second coordinates, the third coordinates, and the fourth coordinates.

7. The image processing device described in claim 5, characterized in that the paste area calculation unit sets the center of gravity of the object determined to have high reliability by the target object determination unit within an area surrounded by a circle whose center is the center of gravity of the object, and whose radius is the value obtained by multiplying the vertical dimension of the frame line corresponding to the object by the ratio specified by the random number within a first predetermined range, calculates the value obtained by multiplying the vertical dimension of the frame line by the ratio specified by the random number within a second predetermined range as the first dimension of the paste area, calculates the value obtained by multiplying the horizontal dimension of the frame line by the ratio specified by the random number within the second predetermined range as the second dimension of the paste area, and calculates the area surrounded by a rectangle having the set center of gravity, the calculated first dimension and second dimension as the paste area.

8. The image processing device according to claim 1, characterized in that the learning device comprises: an inference unit that inputs the plurality of pieces of difficult image data generated by the data expansion device into the object detection model before the additional learning and infers the objects that appear in each piece of difficult image data; a learning data selection unit that selects, from the plurality of pieces of difficult image data generated by the data expansion device, the piece of difficult image data that has a large error in the inference result as learning data to be used for the additional learning, based on the correct value data corresponding to the difficult image data input to the object detection model and the inference result of the inference unit; and a learning unit that performs the additional learning using the difficult image data selected by the learning data selection unit.

9. The image processing device described in claim 8, characterized in that the learning data selection unit selects as the learning data not only the difficult image data in which the error in the inference result is large, but also the image data used in the machine learning of the object detection model, and the learning unit performs the additional learning using the difficult image data and the image data selected by the learning data selection unit.

Images