Image processing device, image processing method, mobile device, and computer program

Abstract

To provide an image processing apparatus that achieves an image processing apparatus that can obtain an accurate area division result, while reducing a processing load.SOLUTION: In an image processing apparatus, an image processing unit is provided with an object detection unit 201 that detects objects from image data, an area setting unit 202 that sets target areas in which categories are applied to pixels according to a result of detection performed by the object detection unit, and a category application unit (semantic segmentation unit) 204 that applies categories to the pixels belonging to the categories of the target areas set by the area setting unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, a moving device, a computer program, and the like. [Background technology]

[0002] Technological developments are progressing for self-driving cars and self-propelled robots to understand their surrounding environments, and one example of this is distance estimation technology using machine learning. For example, to estimate the distance to a traffic light while a vehicle is moving, one technology estimates the distance to the traffic light by comparing the actual size of the traffic light's light-emitting part as defined by the standard with the number of pixels in the diameter of the light-emitting part and the area of ​​the light-emitting part on an image segmented by a machine learning model. One segmentation method is object detection, in which a machine learning model identifies (categorizes) the traffic light on the image and sets a rectangle (detection region) containing that object. Another method is semantic segmentation, which performs semantic region segmentation by categorizing the image pixel by pixel.

[0003] Non-Patent Document 1 discloses a technology for improving the accuracy of these region divisions by estimating the range in which an object exists from the object detection results, increasing the probability that this estimated range is an object, and then performing semantic segmentation.

[0004] Furthermore, Patent Document 1 discloses a technology for detecting an area where an object exists from an image, enlarging the area from the image to extract a first image, generating a second image from the object in the area, and performing machine learning using the first image and the second image. [Prior art documents] [Patent documents]

[0005] [Non-Patent Document 1] Lubor Ladicky, Paul Sturgess, Karteek Alahari, Chris Russell, and Philip HS Torr, What,Where & How Many? Combining Object Detectors and CRFs, ECCV2010 Summary of the Invention [Problem to be solved by the invention]

[0006] For object detection, several neural network configurations capable of high-speed processing have been proposed. On the other hand, semantic segmentation is known to be capable of segmenting regions with high accuracy, but tends to require a larger amount of calculation compared to object detection. The technology described in Non-Patent Document 1 implements each of these machine learning models, which poses a problem of excessive processing load for processing that requires real-time performance, such as in autonomous driving.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device or the like that can obtain highly accurate region segmentation results while suppressing the processing load. [Means for solving the problem]

[0008] In the image processing device, an object detection unit that detects an object from image data; a region setting unit that sets a target region in which a category is assigned to pixels according to a detection result by the object detection unit; a category assigning unit that assigns a category to pixels that belong to the category in the target area set by the area setting unit; an area displacement estimation unit that estimates a displacement of the target area based on the position of the object to which the category has been assigned by the category assignment unit, The region setting unit sets the target region by referring to the region displacement estimated by the region displacement estimation unit, and The area displacement estimation unit estimates the area displacement by comparing the position of the object to which the category has been assigned by the categorization unit with the position of a detection area in which the object detection unit has detected the object. It is characterized by: [Effects of the Invention]

[0009] According to the present invention, it is possible to obtain an image processing device or the like that can obtain highly accurate region segmentation results while suppressing the processing load. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram showing an example of the configuration of an image processing unit in the first embodiment. [Figure 3] 10 is a flowchart showing an example of the operation of the image processing unit in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of image data in the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating target region setting in the first embodiment. [Figure 6] 10A to 10C are diagrams illustrating region displacement estimation in the first embodiment. [Figure 7] FIG. 10 is a block diagram showing an example of the configuration of an image processing unit in the second embodiment. [Figure 8] 10 is a flowchart showing an example of the operation of an image processing unit in the second embodiment. [Figure 9] 9 is a flowchart showing a continuation of the operation example shown in FIG. 8. [Figure 10] 10A and 10B are diagrams illustrating a method for determining a magnification ratio in the second embodiment. [Figure 11] 10A to 10C are diagrams illustrating an image synthesis method according to a third embodiment. [Figure 12] 11 is a flowchart showing a method for creating training images used for model training in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0012] In the embodiments, an example in which the image processing device is applied to an in-vehicle camera will be described. However, the image processing device also includes electronic devices with an imaging function, such as digital still cameras, digital movie cameras, smartphones with cameras, tablet computers with cameras, network cameras, drone cameras, and cameras mounted on robots. In addition, in the embodiments, an image processing device mounted on a car as a mobile device is used as an example, but the mobile device is not limited to a car, and includes mobile devices such as airplanes, ships, robots, drones, etc. (Embodiment 1)

[0013] In a first embodiment of the present invention, object detection is performed on image data, and a category and its detection area are identified. Furthermore, an example will be described in which image data with an expanded detection area is extracted based on the degree of deviation between the detection area identified by the object detection and the predicted correct area, and the expanded image data is subjected to semantic segmentation. The categories of objects to be detected by the semantic segmentation of this embodiment are natural objects such as people, animals, automobiles, motorcycles, buildings, roads, traffic lights, and traffic signs, as well as artificial objects.

[0014] (Configuration of image processing device) 1 is a block diagram showing an example of the configuration of an image processing device according to this embodiment. The image processing device includes a CPU 100, a RAM 101, a ROM 102, an image capturing unit 103, an image processing unit 104, an I / F (interface) unit 105, and a bus 106.

[0015] The CPU 100 uses computer programs and data stored in the RAM 101 and ROM 102 to control the overall operation of the image processing apparatus.

[0016] The RAM 101 has a storage area for temporarily storing computer programs loaded via the I / F unit 105, image data for the most recent few frames, processing parameters used by the image processing unit 104, etc. It also has a storage area used by the CPU 100 when executing various processes, and a storage area used by the image processing unit 104 when performing image processing. Therefore, the RAM 101 can provide various storage areas as needed.

[0017] The ROM 102 stores the setting data and boot program of the device. The image capturing unit 103 is a camera including an optical lens, a CMOS image sensor, etc., and acquires video data of the subject. In this embodiment, the image data is subjected to general sensor correction and demosaic processing, converted into an RGB color image, and stored in the RAM 101.

[0018] The image processing unit 104 is realized as a processor or dedicated image processing circuit capable of executing a computer program stored in the ROM 102 or the like, and performs the image processing method of this embodiment. For example, upon receiving an instruction to execute image processing from the CPU 100, the image processing unit 104 processes image data stored in the RAM 101 and outputs a region segmentation map obtained by semantic segmentation of a region including a detection target. Furthermore, during image processing, the processing is performed while appropriately reading out processing parameters stored in the RAM 101. Note that the region segmentation map formed by semantic segmentation refers to a map in which category information is assigned to each pixel (for example, each pixel or each set of pixels).

[0019] The I / F unit 105 functions as an interface for connecting the device to an external storage device or the Internet. All of the above components are connected to a bus 106 . The configuration of the image processing apparatus of this embodiment is not limited to the configuration shown in FIG.

[0020] (Configuration of image processing unit) Next, FIG. 2 is a functional block diagram showing an example of the configuration of the image processing unit in the first embodiment. When the image processing unit 104 is realized as a dedicated image processing circuit, it may be configured using the functional blocks shown in FIG. In this embodiment, the image processing unit 104 is a processor capable of executing a computer program stored in a ROM 102 or the like, and realizes the functions of each block in FIG. 2 by executing processing according to the processing flowchart shown in FIG. 3.

[0021] 2, the image processing unit 104 includes an image acquisition unit 200, an object detection unit 201, a region setting unit 202, a target image acquisition unit 203, a semantic segmentation unit 204, and a region displacement estimation unit 205. Each of 200 to 205 is a functional block realized by a processor.

[0022] The image acquisition unit 200 acquires image data to be processed that is stored in the RAM 101 . The object detection unit 201 detects a plurality of objects from image data using a predetermined method, identifies the categories of the objects and the detection areas in which the objects are detected, and outputs the identified categories.

[0023] The region setting unit 202 sets a predetermined detection region, which is obtained by correcting the detection region in accordance with the deviation from the detection region and the region where the correct region may actually exist, as a target region for performing semantic segmentation. That is, the region setting unit sets a target region for assigning a category to pixels in accordance with the detection result by the object detection unit. The target image acquisition unit 203 acquires image data of the area set by the area setting unit 202 from the RAM 101 .

[0024] The semantic segmentation unit inputs image data of the target area set by the area setting unit 202 via the target image acquisition unit 203, and divides the object and background into areas according to categories. Here, the semantic segmentation unit functions as a category assignment unit that assigns a category to pixels belonging to a predetermined category in the image data of the target area.

[0025] When assigning a category to each pixel, a category may be assigned to each set of pixels rather than to each pixel. The region displacement estimation unit 205 compares the output results of the object detection unit 201 and the semantic segmentation unit 204 to estimate the amount of region displacement that occurs in the object detection unit 201 .

[0026] (Explanation of the operation of the image processing unit) 3 is a flowchart showing an example of the operation of the image processing unit in embodiment 1. The processor in the image processing unit 104 executes a computer program to realize the functions of the blocks shown in FIG. 2 and to perform the processing of each step in FIG. When the process starts, first, in step S300, the image acquisition unit 200 acquires image data from the RAM 101. FIG. 4 is a diagram for explaining an example of image data in the first embodiment, and the image data shown in FIG. 4(A) is an example in which people, cars, and traffic signs exist as target objects on a road.

[0027] In step S301, the object detection unit 201 detects an object in the image data, identifies and outputs object category information and the detection area in which the object is detected (object detection process). The object detection method may be the same as conventional methods. Known neural networks that can detect objects with a relatively low processing load and high robustness include R-CNN (Region-based CNN), YOLO (You Only Look Once), and SSD (Single Shot Detector).

[0028] In this embodiment, objects are detected using a model trained by machine learning on these general neural networks using a training image dataset linked to categories (people, cars, signs) and correct answer data for the detection area for each category. If the detection area overlaps with the correct answer area at a predetermined overlap rate or more, and the detected category is correct, the detection is deemed successful, and a detection rate indicating the detection accuracy in object detection is calculated. The overlap rate between the detection area and the correct answer area is measured using IoU (Intersection over Union), for example. Examples of the predetermined overlap rate include 50% and 75%.

[0029] FIG. 4B shows a schematic diagram of object detection performed in this embodiment. The neural network for object detection automatically defines the features required for object detection from training data, and gradually integrates these local features to obtain a global judgment. Therefore, for example, if the network is not large enough to express the features required for identifying the target object, or if there are characteristic differences between the training data and the image data used for actual inference, the neural network may not be able to fully demonstrate its object detection performance.

[0030] Therefore, when an object is detected in image data, the detected area output will differ from the detected area that is considered to be the correct answer. In this embodiment, the accuracy of semantic segmentation is improved by setting the area for semantic segmentation while taking this area misalignment into consideration.

[0031] In step S302, the process branches depending on whether or not there is an unprocessed object. If there is an unprocessed object, the process proceeds to step S303, and if all undetected and detected objects have been processed, the process ends.

[0032] In step S303, the region setting unit 202 corrects the detection region output by the object detection unit 201 according to the deviation from the detection region where the correct region may actually exist. This sets the target region for performing semantic segmentation so as to be optimized (region setting step). The region setting method will be described with reference to FIG. 5. 5A and 5B are diagrams for explaining the target area setting in embodiment 1, and Fig. 5A shows the detection area (framed with dashed lines in the figure) when a vehicle is detected, and the area assumed to be the correct answer (framed with solid lines in the figure). The area assumed to be the correct answer is assumed to be a rectangle circumscribing the target object.

[0033] As shown in the figure, there is a possibility that the detection area may deviate from the correct answer area. Therefore, in this embodiment, the detection area is corrected according to the deviation from the correct answer area to set the area for performing semantic segmentation. FIG. 5(B) shows an example of the relationship between the detection area and the correct answer area when IoU=50%. The two left sides of FIG. 5(B) show cases where the detection area is output so as to be included in the correct answer area, and the two right sides of FIG. 5(B) show cases where the detection area is output so as to include the correct answer area.

[0034] Assuming IoU=50% from Figure 5(B), we can see that the width w1 of the detection area falls within the range of w0 / 2 to w0*2, and the height h1 of the detection area falls within the range of h1 / 2 to h1*2. Therefore, for a detection area detected by an object detection model that obtains a detection rate based on IoU=50%, the actual correct answer area is expected to exist within the above range. Figure 5(C) shows the corrected area (outer rectangle in the figure) when the fluctuation rate I of the detection area is set to 0.5 (corresponding to IoU=50%). The detection area can be corrected to include the correct answer area by expanding the area by (1 / I-1)*w1 in the left-right direction and (1 / I-1)*h1 in the up-down direction, based on the relationship between the correct answer area and the detection area shown in Figure 5(B).

[0035] The corrected region is the detection region expanded three times in width and height, based on the assumption that the detection region overlaps with the correct region at IoU = 50% or more. Even if the detection region fluctuates, the object is contained within the corrected region, enabling reliable region segmentation. Meanwhile, by minimizing region expansion, the amount of processing required for subsequent semantic segmentation can be reduced.

[0036] In this embodiment, the detection area is corrected, for example by enlarging it, depending on the deviation from the detection area where the correct area may actually exist, and set as the target area for semantic segmentation. Note that the variation rate due to object detection may be different in the left-right direction and the up-down direction, or may be different depending on the category value. That is, the area setting unit may correct the detection area depending on the category and set the target area. Furthermore, when performing object detection such that the correct region is included within the detection region, semantic segmentation may be performed without expanding the detection region.

[0037] In step S304, the target image acquisition unit 203 acquires image data of the target area set in step S303 from the RAM 101. Because the image data acquired here is limited to the target area, it is possible to reduce the number of accesses to the RAM 101. Furthermore, in order to further reduce the number of accesses to the RAM 101, a cache configuration may be used in which a storage area for temporarily storing image data is secured in the image acquisition unit 200, and when image data of the target area exists in this storage area, the image data is read from the storage area.

[0038] In step S305, the semantic segmentation unit 204 assigns a category to each pixel in the image data of the target region that belongs to a predetermined category, thereby performing region segmentation for each pixel based on the category (category assignment step). The semantic segmentation method may be the same as conventional methods. SegNet and U-Net are known as neural networks that can perform region segmentation with high accuracy.

[0039] These networks are called Encoder-Decoder networks, and the first half of the Encoder performs pooling to globally summarize the features required for category classification.Then, the second half of the Decoder performs upsampling and gradually achieves region division (category classification) by shifting from region-based category judgment to pixel-based category judgment.

[0040] In this embodiment, semantic segmentation is performed on these general neural networks using a model trained by machine learning using a training image dataset linked to a category (people, cars, signs) determination for each pixel. Figure 4(C) shows the results of semantic segmentation on image data. In this embodiment, a target region (shown in a solid line frame) is set by expanding the detection region (shown in a dashed line frame) in object detection based on IoU, etc., and semantic segmentation is performed on this target region. Therefore, objects (people, cars, signs) and background areas can be segmented with high accuracy.

[0041] Next, in step S306, the area displacement estimation unit 205 estimates the amount of area displacement included in the detection area. In step S303 above, a method for estimating the amount of area displacement when IoU=50% is assumed has been described. This is an effective method for estimating the amount of area displacement when the detection accuracy of object detection is known in advance. However, the actual detection accuracy of object detection varies depending on the shooting environment and the diversity of target objects. Therefore, it is desirable to estimate the amount of area displacement by feedback in real time from the category of the object actually detected and the shooting environment, and this embodiment is configured in this manner.

[0042] A method of estimating the amount of region deviation for each category from the region detected by object detection and the result of semantic segmentation according to this embodiment will be described with reference to FIG. 6A and 6B are diagrams for explaining region displacement estimation in embodiment 1, showing examples of semantic segmentation of a person in Fig. 6A and a road sign in Fig. 6B. In each case, the dashed frame indicates the detection region obtained by object detection, with the hatched area in Fig. 6A representing a person and the black area in Fig. 6B representing an example of region division of a road sign based on categories using semantic segmentation.

[0043] First, the correct answer region is estimated for each object segmented based on its category using semantic segmentation. The correct answer region is a rectangular region circumscribing the segmented object. Specifically, the correct answer region is obtained by gradually narrowing a rectangular region that is sufficiently expanded in the vertical, horizontal, and lateral directions from the object detection region on the segmentation map, and stopping the narrowing at the edge that first touches the category value of the detection target (person, vehicle, sign) on each side. In this way, the deviation of the target region is estimated by calculating the difference between the position of the correct answer region (shown in a solid-line frame) of the categorized object and the position of the object detection region.

[0044] The difference on the left side of the detection area is Ex0, the difference on the right side is Ex1, the difference on the top side is Ey0, and the difference on the bottom side is Ey1. The amount of area deviation is estimated based on the ratio to the width or height of the detection area. Specifically, the amount of deviation on the left side between the detection area and the correct answer area is estimated as Ex0 / w1, the amount of deviation on the right side is Ex1 / w1, the amount of deviation on the top side is Ey0 / h1, and the amount of deviation on the bottom side is Ey1 / h1. Each is estimated as a positive or negative ratio, with a positive ratio indicating the direction of expanding the detection area and a negative ratio indicating the direction of shrinking the detection area.

[0045] The estimated region displacement amount is fed back to the region setting unit 202 and is referenced as the rate of increase or decrease in the up, down, left, and right directions when correcting the detection region during region setting. Note that object detection and semantic segmentation output results that include fluctuations depending on the image data. Therefore, in estimating the region displacement amount, it is possible to estimate the region displacement amount more accurately by smoothing multiple region displacement amounts in each direction and calculating the region displacement amount. Therefore, such a configuration may be adopted.

[0046] Furthermore, since the amount of area deviation may suddenly increase, a margin may be added in the direction of expanding the detection area relative to the estimated amount of area deviation. Furthermore, the tendency of the amount of area deviation varies depending on the category being detected. For example, the detection area for the category of people shown in Figure 6(A) tends to have a large amount of area deviation because it is subject to many variable factors such as the person's body shape, clothing, and posture. On the other hand, the size, color, and shape of the sign shown in Figure 6(B) are somewhat determined by traffic standards, etc., so the amount of deviation in the detection area is relatively small.

[0047] Therefore, when estimating the amount of region deviation, the amount of region deviation according to the category may be estimated, and the region setting unit 202 may correct the detected region using the amount of region deviation according to the category. The above process enables highly accurate region segmentation while reducing the processing load by performing semantic segmentation on the minimum necessary area for objects detected robustly by object detection.

[0048] (Image Processing Unit Variation 1) In the above example, a target region is set for each object detected by object detection, and semantic segmentation is performed. As a variation of this, a single target region that encompasses multiple detected objects may be set, and semantic segmentation may be performed.

[0049] FIG. 4(D) shows an example in which a modified detection region is calculated for each individual object, and then a region 400 that encompasses these modified detection regions is set as the target region for semantic segmentation, and semantic segmentation is performed. As the number of objects to be detected increases and the density increases, it is conceivable that the regions will overlap. In such cases, it may be possible to reduce the overall processing load by setting a comprehensive region such as region 400 as the target region and performing semantic segmentation.

[0050] (Image Processing Unit Variation 2) In the above, an example was described in which a model trained to be able to divide regions based on categories (people, cars, signs) was used as a machine learning model for semantic segmentation. On the other hand, an example was described in which a target region was set for each object detected by object detection, and semantic segmentation was performed. In other words, the category to be divided into regions for the target region to be subjected to semantic segmentation is known in advance. Therefore, the semantic segmentation method may be switched depending on the category.

[0051] For example, in the case of road signs, it may be possible to reduce the processing load by storing a standard sign image determined in advance as a template and performing area segmentation by matching with this template.In addition, when it comes to segmenting car areas, it is possible to improve the accuracy of area segmentation by first estimating the car model and then performing semantic segmentation using this car model information as additional information.

[0052] It is also possible to decide to perform semantic segmentation by switching between models that have been trained separately for each category in advance, or by switching between different parameter sets that define the training content. This makes it easy to improve the learning efficiency of segmentation for individual categories and to add categories to be segmented later. (Embodiment 2)

[0053] As a second embodiment of the present invention, an example will be described in which object detection is performed on image data, and the category and its detected area are identified. Furthermore, an example will be described in which the area containing the detected target object is resized to a size that is expected to maximize the division accuracy of semantic segmentation. The configuration of the image processing apparatus of this embodiment may be the same as that of the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0054] (Configuration of image processing unit) FIG. 7 is a block diagram showing an example of the configuration of an image processing unit in the second embodiment. 8 is a flowchart showing the operation of the image processing unit 104 in embodiment 2. Note that detailed explanations of components with the same names and symbols as those described above will be omitted, and only differences will be explained.

[0055] The image processing unit 104 of the second embodiment includes an image acquisition unit 200, an object detection unit 201, a region setting unit 202, a target image acquisition unit 203, and a semantic segmentation unit 204. It also includes a first magnification / reduction unit 206, an image synthesis unit 207, an image cropping unit 208, and a second magnification / reduction unit 209. Then, before supplying image data of the target region set by the region setting unit to the semantic segmentation unit, the image data is magnified at a first magnification / reduction rate by the first magnification / reduction unit.

[0056] Furthermore, the image data of the target region scaled by the first scaling unit is synthesized with image data of other target regions by an image synthesis unit 207, and then supplied to a semantic segmentation unit. Furthermore, the image data after the semantic segmentation unit assigns categories to pixels belonging to categories is scaled at a second scaling factor by a second scaling unit 209. Although not shown in Fig. 7, a region misalignment estimation unit 205 similar to that of the first embodiment may be provided after the second scaling unit 209, for example. In the second embodiment, the reference numerals 200 to 209 are functional blocks realized by processors.

[0057] The first scaling unit 206 scales the image data of the target region at a scaling factor such that the size of the detection target on the image becomes a size suitable for semantic segmentation. The image synthesis unit 207 synthesizes the image data of multiple target regions to generate image data for performing semantic segmentation.

[0058] The image cropping unit 208 crops out the target object region from the region division map that has undergone semantic segmentation. That is, the image data is cropped for each target region before the image data after the semantic segmentation unit assigns categories to pixels that belong to each category is supplied to the second zoom unit. The second scaling unit 209 enlarges or reduces the area division map at a scaling factor that is the reciprocal of the scaling factor in the first scaling unit 206, and obtains an area division map having the same resolution as the target object in the input image data.

[0059] (Explanation of the operation of the image processing unit) Fig. 8 is a flowchart showing an example of the operation of the image processing unit in embodiment 2, and Fig. 9 is a flowchart showing an example of the operation continued from Fig. 8. The processor in the image processing unit 104 executes a computer program to realize the functions of the blocks shown in Fig. 7 and to perform the processing of the steps in Figs. 8 and 9. Note that detailed explanations of steps with the same names and symbols as those described above will be omitted, and explanations will be added only where there are differences.

[0060] When processing starts, object detection is performed on the image data in steps S300 and S301, and object category information and detection areas are obtained. If there are any unprocessed objects in step S302, the detection areas of each object are adaptively corrected in step S303 using the same method as in embodiment 1, and a target area corresponding to the corrected detection area is set, and image data of the target area is obtained in step S304. If there are no unprocessed objects in step S302, processing ends.

[0061] In step S307, the scaling factor of the image data of the target region is determined so that the image size is suitable for the semantic segmentation unit in the subsequent stage. In the neural network structure using machine learning, multiple pooling layers are provided to increase the robustness of the detected object size in the image, and features are extracted at each resolution.

[0062] This enables object detection and region segmentation independent of the size of the image when the object contour is clear. However, since the amount of calculation required for inference and the amount of memory required to store the parameter set increase dramatically as the number of hidden layers increases, the number of hidden layers must be appropriately set according to the amount of variation in the size of the corresponding detected object.

[0063] FIG. 10 is a diagram for explaining a method for determining a magnification ratio in the second embodiment. In the second embodiment, a VGA image (640 pixels wide, 480 pixels high) is assumed as the input image size for semantic segmentation, and the optimal object size for region segmentation is set to approximately 100 pixels x 100 pixels. In this way, by determining the optimal object size for semantic segmentation to a certain size, it becomes possible to omit hidden layers in the network configuration and reduce the processing load.

[0064] Furthermore, as a criterion for reducing the number of hidden layers, it is preferable to scale the object size so that it falls within the range of N pixels x M pixels to N / 2 pixels x M / 2 pixels, where N and M are positive integers. This is because pooling often halves the amount of data vertically and horizontally, and effective features are switched depending on the number of pooling operations in response to changes in object size. Furthermore, even if the optimal object size for region segmentation is unknown for a model, the optimal object size can be determined in advance by actually processing object images with different sizes and checking the segmentation accuracy.

[0065] Taking into account this guideline for the optimal object size and the fact that the detection area in object detection will include variations, the VGA image is divided into small areas with a width of 160 pixels and a height of 120 pixels, by dividing it vertically and horizontally by 16. The magnification ratio is then set so that the detection area containing the object (the area enlarged to account for variations) fits into these small areas. If the width and height of the detected object are W pixels and H pixels, respectively, the magnification ratio obtained by magnifying the object to a size (width, height) that can be stored in the small areas can be selected from the magnification ratios of (W / 160) and (H / 120), and the first magnification ratio can be determined without changing the aspect ratio of the object.

[0066] In step S308, the first scaling unit 206 performs first scaling processing using the first scaling factor (first scaling step). The scaling method may be a known method. Note that the scaling method may be different depending on whether the scaling factor is used for reduction or enlargement. In area division using semantic segmentation, the clarity of the object's shape significantly affects the accuracy of area division. However, when reducing, a low-order interpolation method (bilinear interpolation) may be used because the shape of the original image data is less likely to deteriorate. On the other hand, when enlarging, the shape is more likely to deteriorate, so a high-order interpolation method (bicubic interpolation, Lanczos interpolation, etc.) may be used to balance the amount of calculation and the accuracy of area division.

[0067] In step S309, the image synthesis unit 207 generates synthesized image data to be input to the semantic segmentation unit 204 (image synthesis step). In this embodiment, as shown in Fig. 9, the image data is generated by sequentially synthesizing regions containing objects whose sizes have been adjusted using different scaling factors into each of the 16 small regions obtained by dividing the VGA image.

[0068] If the number of detected objects is less than 16, semantic segmentation may be performed in the presence of blank small areas. The blank small areas are not cut out in the image cutout unit 208 at the subsequent stage, so the presence of blank small areas will not have any effect. The synthesis in step S309 is performed to improve the processing efficiency of the semantic segmentation unit 204.

[0069] In step S310, it is determined whether the composite image is complete and the process branches. If the composite image is not complete, the process returns to step S303, where the processing region for the next object is composited. If the composite image is complete, the process proceeds to step S305 in FIG. 9, where semantic segmentation of the composite image is performed to obtain a region segmentation map for the composite image. In step S311, it is determined whether or not there are any small regions that have not been output in the region division map of the composite image, and the process branches accordingly. If there are any small regions that have not been output, the process proceeds to step S312, and if all small regions have been output, the process proceeds to step S302.

[0070] In step S312, the image cropping unit 208 crops out an area division map for one object from the area division map of the composite image. In this embodiment, the composite image is made up of 16 divided small areas, so the small areas are cropped in order. In step S313, a second magnification ratio is calculated. The second magnification ratio is a magnification ratio for returning the small area of ​​the composite image to the original resolution. Therefore, the second magnification ratio is calculated by calculating the reciprocal of the first magnification ratio.

[0071] In step S314, the second scaling unit 209 performs a second scaling process based on the second scaling factor (second scaling step). The second scaling process is performed on the area division map. Unlike normal image data, the area division map stores pixel values ​​with values ​​indicating categories added to them. Note that it is undesirable for values ​​different from category values ​​to appear as a result of the interpolation process, so in this embodiment, an interpolation method such as nearest neighbor interpolation is used as the second scaling process, in which only category values ​​appear in the map after interpolation.

[0072] The above process enables efficient and highly accurate region segmentation while reducing the processing load of semantic segmentation. Note that the composite image shown in this embodiment is intended to improve processing efficiency, and the effects of this embodiment can be achieved by using at least a first scaling process appropriate for region segmentation and a second scaling process for returning to the input resolution. (Embodiment 3)

[0073] In the third embodiment, a composite image is generated by placing image data of the target area at either a predetermined image position determined in advance according to the category detected by object detection or a predetermined position shared by multiple categories, and then semantic segmentation is performed on the composite image.

[0074] The configuration of the image processing device to which the third embodiment is applied may be the same as that of the first and second embodiments, and therefore a description thereof will be omitted. The configuration of the image processing unit 104 can also be realized by the configuration of the second embodiment, and therefore a description thereof will be omitted. In this embodiment, the learning efficiency of semantic segmentation is improved by the operations of the image synthesis unit 207 and the image cropping unit 208 shown in FIG. 11 is a diagram for explaining the image synthesis method according to the third embodiment, and shows a synthesized image 1000 synthesized by the image synthesis unit 207. The synthesized image 1000 is divided into small regions whose positions are fixed in advance according to the category, and small regions common to all categories.

[0075] In addition, the target area is enlarged based on the category and detection area of ​​the object detected by object detection, taking into account any deviation from the area where the correct area may actually exist, and the arrows indicate at which image position the area should be placed and synthesized depending on the category. By limiting the image area where an object exists in advance according to the object category, it becomes possible to omit learning global features in a model for realizing semantic segmentation. In particular, high learning efficiency can be achieved in a network configuration that has a hidden layer called a bottleneck layer that affects most of the image.

[0076] During model learning, it is generally not known where an object will appear in an image, so learning must be performed assuming various patterns for the combination of object category and object appearance position. However, by combining object detection as in embodiment 3, it is possible to fix the object category and the object appearance position, thereby limiting the patterns to be learned and improving learning efficiency. Furthermore, since there is always a boundary between small regions, the range of necessary features is limited between the small regions, making it possible to omit learning of the bottleneck layer.

[0077] Fig. 12 is a flowchart showing a method for creating training images used for model training in the third embodiment, and corresponds to the synthesis process in step S309 in Fig. 8. The processor in the image processing unit 104 executes a computer program to realize the functions of the blocks shown in Fig. 7 and to perform the processing of each step in Fig. 12. The method for creating training images in this embodiment is based on the premise that a general dataset (a set of an input image and an area division map divided into categories of objects present in the input image) is prepared (set) in advance.

[0078] Therefore, in step S1100, a detection area for each object (category value) in the image is set in the area division map. The detection area is set as a rectangular area circumscribing the object. Since the same objects are connected to each other by the same category value, the area division map can be set somewhat automatically by gradually narrowing the rectangular area from the top, bottom, left, and right directions of the object and fixing the sides that touch the category value.

[0079] In step S1101, the detection area obtained in step S1100 is expanded in consideration of fluctuations due to object detection. The method for expanding the detection area can be the same as that described in the first embodiment. In step S1102, the detection area enlarged in step S1101 is placed in the small area at a position corresponding to the category value. At this time, the enlarged detection area is resized to fit the size of the small area before being placed. In this step S1102, both the image data that will be used as input data during learning and an area division map linked to the image data are generated.

[0080] In step S1103, the detection area enlarged in step S1101 is placed in the small area at the shared position. At this time, scaling is performed to match the size of the small area at the shared position, but if the size differs from the small area size for each category in step S1102, scaling is performed using a different scaling factor. In this step S1103, both the image data that will be input data during learning and the area division map linked to the image data are generated, but the data generated in step S1102 may be overwritten when the map is generated.

[0081] In step S1104, a branch is made to determine whether all objects in the input images of the data set have been processed. In step S1105, a branch is made to determine whether all images in the data set have been processed. By the above process, the input image data for learning in this embodiment and the correct region segmentation map can be obtained. These data sets can be used to efficiently train a semantic segmentation model.

[0082] Furthermore, as in the third embodiment, by determining in advance the categories to be used for small regions, if there is a bias in the appearance of objects, for example, if there are a large number of objects in the car category, the efficiency of executing semantic segmentation may decrease. In contrast, in this embodiment, by setting shared small regions, it is possible to avoid a decrease in execution efficiency due to the frequency of object appearance. Note that setting shared small regions is in a trade-off relationship with learning efficiency.

[0083] Furthermore, by generating a composite image by arranging image data of the detection area in an area with a predetermined aspect ratio according to the category in which the small area is used, it is possible to generate a composite image more efficiently. For example, by setting a small area that is close to a square for the car category and a vertically long small area for the person category, it is possible to efficiently arrange the small areas and improve the efficiency of executing semantic segmentation. As described above, by synthesizing image data whose placement position is determined in advance according to the object category and then performing semantic segmentation on the synthesized image data, it is possible to efficiently achieve highly robust and highly accurate region segmentation.

[0084] Furthermore, when the imaging device or image processing device having the image processing device in the embodiments is mounted on a mobile device such as an automobile, the device may have the following configuration, for example: A control unit is provided that issues warnings about obstacles and controls the direction, speed, and other moving operations of the mobile device based on the image data after the semantic segmentation unit 204 assigns the categories to pixels that belong to the categories. The image processing device of this embodiment is highly robust and produces highly accurate region segmentation results while reducing the processing load, thereby achieving excellent effects such as being able to more quickly recognize distant vehicles and obstacles ahead and quickly avoid accidents.

[0085] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.

[0086] Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to an image processing device, a mobile device, or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing device, mobile device, or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0087] 100 CPU 101 RAM 102 ROM 103 Photography Department 104 Image processing section 105 I / F section 200 Image acquisition unit 201 Object detection unit 202 Area setting section 203 Target image acquisition unit 204 Semantic Segmentation Department 205 Area deviation estimation unit 206 First variable magnification unit 207 Image synthesis unit 208 Image cropping section 209 Second variable magnification unit

Claims

1. an object detection unit that detects an object from image data; a region setting unit that sets a target region in which a category is assigned to pixels according to a detection result by the object detection unit; a category assigning unit that assigns a category to pixels that belong to the category in the target area set by the area setting unit; an area displacement estimation unit that estimates a displacement of the target area based on the position of the object to which the category has been assigned by the category assignment unit, The region setting unit sets the target region by referring to the region displacement estimated by the region displacement estimation unit, and The area displacement estimation unit estimates the area displacement by comparing the position of the object to which the category has been assigned by the categorization unit with the position of a detection area in which the object detection unit has detected the object.

1. An image processing device comprising:

2. The image processing device according to claim 1 , wherein the region setting unit sets the target region by expanding a detection region in which the object detection unit detects the object.

3. The image processing device according to claim 2 , wherein the area setting unit sets the target area by modifying the detection area in accordance with the category.

4. The image processing device according to claim 1 , wherein the region displacement estimation unit estimates the region displacement amount according to the category.

5. 4. The image processing device according to claim 2, wherein the region setting unit corrects each of the plurality of detection regions detected by the object detection unit, and sets a region including the plurality of corrected detection regions as the target region.

6. 5. The image processing device according to claim 1, wherein the categorization unit assigns the category using a different parameter set depending on the category of the object detected by the object detection unit.

7. an object detection step of detecting an object from the image data; a region setting step of setting a target region in which a category is assigned to pixels according to a detection result of the object detection step; a category assignment step of assigning a category to pixels belonging to the category of the target area set in the area setting step; an area displacement estimating step of estimating a displacement of the target area based on the position of the object to which the category has been assigned in the categorizing step, The region setting step sets the target region by referring to the region displacement estimated by the region displacement estimation step, The area displacement estimation step estimates the area displacement by comparing the position of the object to which the category is assigned in the categorization step with the position of the detection area in which the object is detected in the object detection step. An image processing method comprising:

8. The image processing device according to any one of claims 1 to 6 is installed, a control unit that issues a warning or controls the movement of the mobile device based on the image data after the category assignment unit has assigned the category to the pixel that belongs to the category.

9. A computer program for controlling each part of the image processing device according to any one of claims 1 to 6 or the moving device according to claim 8 by a computer.

Images