Image processing device, image processing method, and program

Abstract

To detect a subject area intended by a user.SOLUTION: An inspection device includes the steps of: acquiring a captured image; receiving the instruction for the acquired captured image; acquiring a likelihood map representing the presence likelihood of an object in the captured image; estimating a region representing a position and a size of the object in the captured image; determining an object region corresponding to the instruction using one or more object region candidates selected on the basis of a position of the received instruction among the estimated area and a likelihood map; and correcting the object region based on the position of the received instruction and the likelihood map.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] This invention relates to an image processing technique for detecting a subject. [Background technology]

[0002] Object detection is a field of computer vision research that has been extensively studied. Computer vision is a technology that understands images input to a computer and recognizes various characteristics of those images. Within this field, object detection is the task of estimating the position and type of objects present in natural images. Non-patent document 1 uses a multilayer neural network to obtain a likelihood map indicating the center of an object, and detects the object's center position by extracting the peak points of the likelihood map. In addition, by inferring the offset amount corresponding to the center position and the object size, it is possible to obtain the frame of the object to be detected.

[0003] Object detection can be applied to the autofocus function of imaging devices. Patent Document 1 describes a system that receives user-specified coordinates and inputs them along with an image into a multilayer neural network to detect the main subject based on the user's intent, thereby realizing an autofocus function. In Patent Document 1, a position map is generated within the multilayer neural network based on a likelihood map as well as a two-dimensional Gaussian extending from the specified coordinates. Furthermore, the main subject is detected by integrating the position map and the likelihood map within the multilayer neural network. If there is a peak in the likelihood map near the specified coordinates, the contribution of the position map in the integration process is increased; otherwise, it is decreased. Patent Document 1 further adjusts the spread of the Gaussian when generating the position map from the specified coordinates using imaging information such as the electronic zoom rate and the amount of camera shake. For example, if the amount of camera shake is acquired as imaging information, it is considered that specifying the coordinates of the subject is difficult if the amount of camera shake is large, so the spread of the position map is increased. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-173678 [Non-patent literature]

[0005] [Non-Patent Document 1] Objects as Points, Xingyi Zhou et al., 2019 [Overview of the project] [Problems that the invention aims to solve]

[0006] As mentioned above, Patent Document 1 describes a method for detecting the main subject based on the user's intent. However, Patent Document 1 makes it difficult to autofocus on the subject as intended by the user when there are multiple subjects of the same category. For example, consider two subjects of the same category located one in front of the other. In this case, suppose the subject in the back is partially hidden by the subject in the front, overlapping it. If the user had specified the subject in the back in this case, the features of the subject in the back may not be extracted well, the reliability of the position map would be low, and there is a risk that the subject in the back will not be detected correctly. Alternatively, the reliability of the position map would be higher for the subject in the front, whose features can be extracted well, and there is a risk that the subject in the front would become the main subject. Furthermore, in Patent Document 1, if the main subject features include a dog's face and a dog's body, it is necessary to prepare a main subject detection unit that responds to each. There are countless objects that can be the main subject, and it is difficult to prepare a main subject detection unit in advance for all of them.

[0007] As described above, we consider applying Non-Patent Literature 1 to a case where subjects of the same category are located one behind the other, and using the detection result closest to the specified coordinates. It is difficult to separate and infer the likelihood map indicating the center for the foreground subject and the background subject, and there is a high possibility that a peak in the likelihood map will appear for the foreground subject.

[0008] This invention has been made in view of the above-mentioned problems, and aims to provide an image processing device capable of detecting a subject area intended by the user. It also aims to provide a method and program for this purpose. [Means for solving the problem]

[0009] The image processing apparatus according to the present invention comprises the following configuration: an image acquisition means for acquiring an image; an instruction receiving means for receiving instructions for the image acquired by the image acquisition means; and a likelihood map acquisition means for representing the likelihood of existence of an object in the image. Information representing the offset amount from the center position of the object at each location corresponding to the likelihood map, and the width and height of the object. Estimation means for estimating the and Based on the location of the instruction received by the instruction receiving means, the information obtained from the estimated information An image processing apparatus comprising: a determination means for determining an object region corresponding to the instruction using one or more candidate object regions and the likelihood map; and a correction means for correcting the object region determined by the determination means based on the position of the instruction received by the instruction receiving means and the likelihood map. [Effects of the Invention]

[0010] According to the present invention, which has the above configuration, it is possible to provide an image processing device that enables the detection of a subject area intended by the user. [Brief explanation of the drawing]

[0011] [Figure 1] Schematic diagram of the image processing device [Figure 2] Processing flow diagram of Embodiment 1 [Figure 3] Examples of captured images with two subjects and their likelihood maps. [Figure 4] Examples of candidate object regions [Figure 5] Detailed flow of object region candidate selection [Figure 6] Object Region Candidate Integration Example [Figure 7] Example of bilinear interpolation [Figure 8] Example of object region candidate selection using interpolation [Figure 9] Detailed flow of object region modification method [Figure 10] Example of object region modification [Figure 11] Configuration diagram of an image processing apparatus having a second likelihood map acquisition means. [Figure 12] Detailed flow of an image processing device having a second likelihood map acquisition means. [Modes for carrying out the invention]

[0012] The present invention will now be described in detail based on its preferred embodiments with reference to the attached drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0013] <Embodiment 1> Figure 1 shows a schematic configuration diagram of the image processing apparatus according to this embodiment. The configuration of this embodiment will be explained using Figure 1. Note that only an overview will be explained here, and details will be described later.

[0014] The imaging device 110 is composed of an optical system, an image sensor, etc., and captures images and outputs them to the image acquisition unit 101. For example, it is conceivable to use an imaging device such as a digital camera or a surveillance camera. The imaging device 110 also has an interface that receives input from the user and outputs the user input to the instruction receiving unit 104. For example, it is conceivable to have a touch display as the interface and output the results of touch operations from the user to the instruction receiving unit 104.

[0015] The image processing device 100 includes an image acquisition unit 101, a likelihood map acquisition unit 102, an estimation unit 103, an instruction reception unit 104, an object region candidate selection unit 105, and an object region candidate integration unit 106. The image acquisition unit 101 receives images from the imaging device 110. The images are input to the likelihood map acquisition unit 102 and the estimation unit 103, which acquire a likelihood map and object region candidates, respectively. The instruction reception unit 104 acquires user instructions from the imaging device 110. Specifically, it acquires one instruction coordinate entered by the user via touch operation. The object region candidate selection unit 105 selects one or more object region candidates to be integrated from among the object region candidates based on the likelihood map and the instruction coordinate. The object region candidate integration unit 106 integrates the selected object region candidates and acquires one object region. Some of the functional configurations within the image processing device 100 (for example, the likelihood map acquisition unit 102, the estimation unit 103, the object region candidate selection unit 105, and the object region candidate integration unit 106) can be provided by an image processing system on a network. The image processing device 100 is composed of hardware such as the CPU, ROM, RAM, and various interfaces shown in the attached diagram.

[0016] The result output unit 120 outputs an object region, which can be used, for example, in the camera's autofocus function. For instance, several focus points can be sampled from the object region and used for phase-detection autofocus. If the system can accurately detect the object region intended by the user, the autofocus accuracy will also improve.

[0017] Figure 2 shows a flowchart of the processing in the image processing apparatus 100 of this embodiment. Hereafter, the flowchart will be assumed to be realized by the CPU executing a control program. Only an overview is provided here; details will be described later.

[0018] In S200, the imaging device 110 starts capturing images. In S201, the image acquisition unit 101 of the image processing device 100 converts the captured images to a predetermined resolution. Based on the converted images, in S202, the likelihood map acquisition unit 102 acquires a likelihood map, and in S203, the estimation unit 103 acquires object region candidates. In S204, the instruction reception unit 104 receives coordinate instructions from the user, and in S205, it determines whether or not instructions were given. If no instructions were given, the process proceeds to S210 and "No object region" is output. If instructions were given, the process proceeds to S206. In S206, the received instruction coordinates are converted to correspond to the resolution converted in S201. Based on the converted instruction coordinates, in S207, the object region candidate selection unit 105 selects object region candidates. The detailed flow of S207 will be described later using Figure 5. As a result of S207, it is determined in S208 whether or not an object region candidate has been selected. If no candidate has been selected, the process proceeds to S210 and "No object region" is output. If selected, in S209 the object region candidate integration unit 106 integrates one or more selected object region candidates, and in S210 the result output unit 120 outputs one object region. In Figure 2, steps S207 to S209 were explained separately, but steps S207 to S209 are a series of steps that determine the object region.

[0019] <Image Conversion> The image transformation shown in S201 of Figure 2 will now be explained. In this embodiment, first, image capture is started in S200 and an image is acquired. The acquired image is, for example, an RGB image with a width of 6000 pixels and a height of 4000 pixels. In S201, the acquired image is transformed to a predetermined size to match the input format of the multilayer neural network that acquires likelihood maps and object region candidates. In this embodiment, the input size of these multilayer neural networks is an RGB image with a width of 500 pixels and a height of 400 pixels. In this embodiment, 500 pixels from the left and right of the acquired image are cropped, reducing it to one-tenth of its original size. Alternatively, a 400-pixel black image may be padded above and below the acquired image, reducing it to one-twelfth of its original size, or a region with a width of 500 pixels and a height of 400 pixels may be directly cropped from the acquired image. The transformed image has its origin at the top-left corner pixel, with coordinates (0,0). Coordinates (i,j) represent the coordinates of row j and column i, and the pixel at the bottom-right corner has coordinates (499,399). From now on, the coordinate system of the transformed image will be called the image coordinate system.

[0020] <Obtaining a Likelihood Map> The acquisition of the likelihood map shown in Figure 1, point 102 and Figure 2, point S202 will be explained using Figure 3. In this embodiment, the likelihood map is acquired by a multilayer neural network, as described in Non-Patent Literature 1. The input to the multilayer neural network is a resolution-transformed image, which is a 500-pixel wide, 400-pixel high, 3-channel RGB image. The output of the multilayer neural network is a 10-column, 8-row, 1-channel tensor (matrix). The acquired tensor (matrix) is called the likelihood map. The likelihood map (the first channel) has its origin at the top-left corner pixel, with coordinates (0,0). Coordinate (i,j) represents the coordinate at row j, column i, and the pixel at the bottom-right corner is at coordinate (9,7). Hereafter, the coordinate system of the likelihood map will be called the map coordinate system. The likelihood map can be obtained through calculations within the image processing device 100, or the likelihood map can be calculated outside the image processing device 100 and then obtained by the likelihood map acquisition unit 102 within the image processing device 100.

[0021] The multilayer neural network that acquires likelihood maps is pre-trained using a large amount of training data (pairs of images and likelihood maps). See Non-Patent Literature 1 for details. In this embodiment, the likelihood map is assumed to be a saliency map that responds to all objects, but it may also respond to only specific objects. A saliency map is a map that responds to parts that people tend to focus on.

[0022] Figure 3(a) shows an example of an image-transformed captured image, and Figure 3(b) shows an example of the corresponding likelihood map. The image-transformed captured image 300 contains two subjects: a subject 301 in the background and a subject 302 in the foreground. Each element of the likelihood map indicates the likelihood of existence of the object at the location corresponding to that element. The likelihood of existence takes values ​​from 0 to 255, with a higher value indicating a higher likelihood of existence. In Figure 3(b), each element is shown in white as the likelihood decreases and in black as the likelihood increases. The likelihood map 304 shows a particularly high likelihood for the foreground subject 302, with a maximum likelihood of 204 at (6,4).

[0023] <Acquiring candidate object regions> The estimation of object region candidates shown in Figure 1, point 103 and Figure 2, point S203 will be explained using Figure 4. Similar to the likelihood map, object region candidates are acquired using a multilayer neural network. The input to the multilayer neural network is a resolution-transformed image, similar to the likelihood map acquisition, consisting of a 500-pixel wide, 400-pixel high, 3-channel RGB image. The output of the multilayer neural network is a tensor with 10 columns, 8 rows, and 4 channels. The first channel of the tensor represents the x-direction offset from the element to the object center, and the second channel similarly represents the y-direction offset. The third channel of the tensor represents the width of the object represented by the element, and the fourth channel similarly represents the height. From the information of these four channels, the object's center position and size can be obtained. This tensor is called the object region candidate tensor. In this embodiment, each channel of the object region candidate tensor has the same number of rows and columns as the likelihood map, and their coordinate system is also the map coordinate system. The number of columns and rows of each channel in the object region candidate tensor may differ from those of the likelihood map, and if they differ, the number of rows and columns may be matched by interpolation (e.g., bilinear interpolation). Object region candidates can be obtained by calculations within the image processing device 100, or object region candidates can be calculated outside the image processing device 100 and then obtained by the estimation unit 103 within the image processing device 100.

[0024] The multilayer neural network that acquires candidate object region tensors is pre-trained using a large amount of training data (a set of image, offset amount, and width / height), similar to the acquisition of likelihood maps. In this embodiment, a multilayer neural network that outputs information from 4 channels simultaneously is used, but four multilayer neural networks that output one channel each may be prepared and their results combined.

[0025] Figure 4(a) shows an example of an image after image transformation. Figures 4(b) to (d) show the x-direction offset map relative to the object's center, the y-direction offset map relative to the object's center, the object's width map, and the object's height map, respectively. The numerical values ​​of the elements shown in Figures 4(b) to (d) are in pixels. The x-direction offset is positive to the right, and the y-direction offset is positive downwards.

[0026] Here, we focus on the coordinates (6,4) where the likelihood is maximized in Figure 3(b). The elements inverted in black and white in Figures 4(b) through (d) are the points of interest. In Figure 4(a), the point indicated as 401 corresponds to the location on the image of this point of interest. The x-direction offset at the point of interest is -3, and the y-direction offset is -2. In other words, the object's center is determined to be at point 402, which is located to the upper left of point 401, the point of interest.

[0027] The following equation 1 can be used to convert map coordinates to image coordinates.

[0028]

number

[0029] Here, I w , I h These represent the width and height of the image-converted captured image, respectively, M w M h These represent the width and height of the map, respectively. (I x ,I y ) represents a point in the image coordinate system, (M x M y ) represents a point in the map coordinate system. According to Equation 1, the map coordinate point (6,4) is converted to image coordinates (325,225). That is, the image coordinates of point 401 in Figure 4 are (325,225), and the result of adding an offset amount to this is (322,222), which is the image coordinate of point 402.

[0030] Also, from FIGS. 4(d) and (e), the width and height of the object at the attention point are 166 and 348 respectively. From the above, the object region candidate 400 at the attention point is represented as a rectangle centered at (322, 222) with a width of 166 and a height of 348 in the image coordinate system.

[0031] In this embodiment, the object region candidate is set as the amount of offset in the vertical and horizontal directions and the width and height of the object. However, it may be set as the distance to the left and right ends and the upper and lower ends, for example.

[0032] <Object Region Candidate Selection> The selection of the object region candidate shown in 105 of FIG. 1 and S207 of FIG. 2 will be described using the detailed flowchart of FIG. 5. First, in the preparation step S500, each variable is initialized. The variables are n and m as counters, N as the number of object region candidates to be selected, T as the likelihood threshold, D as the distance threshold, L ij as the likelihood, S ij as the object region candidate, and (P x , P y ) as the coordinates (instruction coordinates) obtained by the instruction receiving unit. The instruction coordinates (P x , P y ) are the instruction coordinates given in the image coordinate system converted to the map coordinate system based on Equation 1 and are two-dimensional real vectors. In S501, the map coordinate (u, v) that is the m-th closest to the instruction coordinates (P x , P y ) is selected from all the map coordinates. (u, v) is a two-dimensional positive integer vector. In S502, if (u, v) exists and the distance between (P x , P y ) and (u, v) is obtained, and if the distance is less than or equal to the threshold D, the process proceeds to the next step S503; otherwise, the process ends. In this embodiment, the Euclidean distance (Equation 2) is used as the distance function, but other distance functions may be used.

[0033] <00​​​​​​​​​​Extract. L in S504 uv We compare this with the likelihood threshold T, and L uv If it is T or greater, proceed to the next step S505. uv If it is less than T, proceed to S508 to increment m by 1 and return to S501 so that it is not selected as a candidate for the object region. In S505, the candidate for the object region S corresponding to the map coordinates (u,v) uv Extract the following. In S506, the current likelihood and the candidate object region are L respectively. n S n Save as. In S507, compare n and N, and if n is greater than or equal to N, terminate the process. If n is less than N, in S508, increment n by 1, in S509 increment m by 1, and return to S501. In this embodiment, a predetermined number N of object region candidates were selected, but the method for determining the number of object region candidates to select is not limited to this. For example, likelihood L n You may select candidate object regions such that the sum of these values ​​is greater than or equal to a predetermined value.

[0035] <Object region candidate integration> The integration of object region candidates shown at 106 in Figure 1 and S209 in Figure 2 will be explained using Figure 6. In this embodiment, we assume that the user selects a rear subject 301, but the process is the same even when selecting a front subject 302, only the indicated coordinates differ. If the user wants to select a rear subject 301, the user indicates the location 600 corresponding to the rear subject 301 as shown in Figure 6(a). The indicated coordinates are (235,245) in the image coordinate system, and when converted to the map coordinate system using Equation 1, they become (4.2,4.4). When the aforementioned object region candidate selection process is performed, the shaded area shown in 601, that is, the locations corresponding to the closest (4,4), the second closest (4,5), and the third closest (5,4) in the map coordinate system, are selected. Therefore, the likelihoods L1, L2, and L3 are L 44 , L 45 , L 54 However, in the candidate object regions S1, S2, and S3, there are S 44 S 45 S 54 The values ​​are substituted. Regions 602, 603, and 604 are S 44 S 45 S 54This diagram illustrates the corresponding candidate object regions. In Figure 6(b), 605 to 609 represent the likelihood, x-offset amount, y-offset amount, width, and height, respectively, corresponding to 601. Figure 6(c) shows the result of integrating the candidate object regions. 610 is the center position of the integrated candidate object region, and 611 is the candidate object region with width and height added to it.

[0036] The integration of candidate object regions will be explained using a specific calculation example. First, the center position of the candidate object region in the image coordinate system is calculated. From Figure 6(b), candidate object region S 44 The x-offset is -1 and the y-offset is -41. From Equation 1, the image coordinates corresponding to the map coordinate (4,4) are (225,225). The object region candidate S is then... 44 Add the offset amount of (602) to obtain the center position (224,184) of the candidate object region in the image coordinate system. Similarly, S 45 (603) Center position (224,172), S 54 The center position (276,223) of (604) can be obtained. A weighted average of likelihoods is used to integrate the candidate object regions. The weighted average of likelihoods can be calculated using the following equation 3.

[0037]

number

[0038] x n S is the value from which the weighted average is taken, and x is the result of the weighted average. For example, when finding the x-coordinate of the center position of the integrated object region, S n The x-coordinate of the center position of the corresponding object region candidate is x n You can substitute these values ​​into the equation. Similarly, by substituting the y-coordinate of the center, width, and height into equation 3, you can find the center, width, and height of the integrated object region.

[0039] Likelihood L nBy substituting 0 as the initial value, even if the number of candidate object regions exceeding the likelihood threshold T within the range of the distance threshold D is less than a predetermined number N, the integrated object regions can be obtained using Equation 3. Furthermore, all likelihood L n If the value is 0, the object region is considered to be empty.

[0040] The above embodiments are summarized. First, the captured image is transformed to enable likelihood map acquisition and object region candidate acquisition. Likelihood map acquisition and object region candidate acquisition are realized by a multilayer neural network. The object region candidate selection unit selects three object region candidates located near the user's indicated coordinates acquired by the instruction reception unit. The selected object region candidates are integrated into a single object region by the object region candidate integration unit, which calculates a weighted average using likelihood as the weight. As a result, even when the likelihood map strongly responds to the subject in front, as shown in Figure 3, an object region 611 corresponding to the subject in the background, as intended by the user, is output, as shown in Figure 6(c).

[0041] <Example 1> In Embodiment 1, the likelihood map value was used when integrating the object region candidates, but the distance between the coordinates (instruction coordinates) obtained by the instruction reception unit and the object region candidates may also be used. In Modification 1, the shorter the distance between the instruction coordinates and the object region candidates, the larger the weight used in the weighted average when integrating the object region candidates. Specifically, as shown in Equation 4 below, the instruction coordinates (P x ,P y The weights of the weighted average are calculated using the reciprocal of the distance between the object and the candidate object region.

[0042]

number

[0043] Here (u,v) n is the likelihood L n These are the corresponding map coordinates. The Euclidean distance in the map coordinate system used in Equation 2 is used to calculate the distance. The likelihood L in Equation 3. n Instead, the weight W obtained from Equation 4 nBy using this method to calculate a weighted average, it becomes possible to integrate candidate object regions while considering their distance from a specified position.

[0044] In the above explanation, the weights of the weighted average were recalculated in the step of integrating candidate object regions, but a likelihood map that takes into account the distance from the indicated coordinates may be calculated in advance. The likelihood map that takes into account the distance from the indicated coordinates is called the modified likelihood map. Modified likelihood map K ij It is calculated using the following formula 5.

[0045]

number

[0046] In Equation 5, the likelihood map L ij For all elements of the modified likelihood map K, a calculation is performed by dividing by the distance between the indicated coordinate and the element, and the modified likelihood map K is created. ij It is being substituted.

[0047] According to Modification 1, it is possible to integrate object region candidates by giving more emphasis to those closer to the specified coordinates.

[0048] <Modification 2> Modification 2 shows an example of extending the method for selecting object region candidates by interpolating each channel of the likelihood map and the object region candidate tensor.

[0049] First, interpolation will be explained using Figure 7. Figure 7 is an excerpt of the width map of the object region candidate shown in Figure 4(d) (the third channel of the object region candidate tensor mentioned above). Here, we show an example of applying bilinear interpolation to the width map 700, which is an excerpt covering map coordinates (4,4) to (5,5), when coordinate 701 is given. The interpolation method is not limited to bilinear interpolation; other interpolation methods such as nearest neighbor interpolation or bicubic interpolation may also be used. Furthermore, interpolation processing can be applied similarly to ranges other than map coordinates (4,4) to (5,5). The values ​​in parentheses shown for each element of the width map 700 in Figure 7 indicate the map coordinates, and the value to the right of the colon indicates the width of the object region candidate at that coordinate.

[0050] As shown in Figure 7, let x1, x2, y1, and y2 be the distances in the x and y directions between point 701 and each map coordinate. For example, if the map coordinates of point 701 are (4.2, 4.4), then x1=0.2, x2=0.8, y1=0.4, and y2=0.6. Bilinear interpolation is implemented using the distances in the x and y directions between point 701 and each map coordinate as shown in Equation 6 below.

[0051]

number

[0052] S ij is the value at the map coordinates (i,j) to be interpolated, and S is the interpolation result. In the example in Figure 7, S ij This represents the width of the candidate object region at map coordinates (i,j). The interpolated value for the height of the candidate object region can be calculated by performing a similar calculation. When interpolating the offset amount to the candidate object region, it is necessary to convert it to the center position of the candidate object region beforehand. For a method of converting the offset amount to the center position, please refer to <Integrating Candidate Object Regions> in Embodiment 1. The interpolated value for the likelihood map can also be calculated using a similar procedure.

[0053] By using interpolation, likelihoods and candidate object regions can be obtained for any coordinate position. Here, the selection of candidate object regions using interpolation is explained with reference to Figure 8. Given an image 800, and the user wanting to select a subject 802, consider a specified coordinate 801. Consider multiple concentric circles (e.g., 803) with different radii spreading from 801. Points obtained by dividing each concentric circle into a predetermined number of parts (e.g., 804) are called the neighboring point group. In this embodiment, the object region candidate selection unit selects a value obtained by interpolating the candidate object region between the specified coordinate 801 and the neighboring point group. However, points in the neighboring point group that are located outside the range of the map coordinates are not selected.

[0054] Let Nr be the number of concentric circles, dr be the difference in radii between concentric circles, and dq be the difference in the number of divisions between concentric circles. These values ​​are set in advance. For example, in Figure 8, Nr=3, dr=0.5, and dq=4 are set. Considering the nr-th concentric circle from the specified coordinate 801, its radius r nr Equation 7, partition number q nr This is calculated from Equation 8.

[0055]

number

[0056] In equations 7 and 8, nr=0 represents the specified coordinate 801, with radius r0=0 and division number q0=1. In Figure 8, 803 is the third concentric circle from the center, with radius r3=3 and division number q3=12.

[0057] The likelihood of an object at the specified coordinate 801 and in the neighboring point cloud is obtained by interpolating the likelihood map. Here, we define an index to represent a neighboring point. Consider the neighboring point cloud on the nr-th concentric circle. The neighboring point located upwards is designated as the 0th, and the points are numbered clockwise. The index of the q-th neighboring point is (nr,q). The index of the neighboring point located one position to the left of the 0th neighboring point is (nr,q). nr -1) is the result. 804 in Figure 8 is represented by the index (3,3).

[0058] By manipulating Equation 3 using the above index, the weighted average of the likelihood of the candidate object regions in this embodiment can be calculated (Equation 9).

[0059]

number

[0060] Here, L (nr,q) x is the interpolated likelihood value at the neighboring point represented by the index (nr,q). (nr,q) x is the value to be used for the weighted average of neighboring points represented by index (nr,q), and x is the result of the weighted average. For example, when finding the x-coordinate of the center position of the integrated object region, x is the interpolated value of the x-coordinate of the center position of the candidate object region corresponding to the neighboring point represented by index (nr,q). (nr,q) You can substitute these values ​​into the equation. Similarly, by substituting the y-coordinate of the center, width, and height into equation 3, you can find the center, width, and height of the integrated object region.

[0061] Radius r of concentric circles nr By using this method, the distance between the indicated coordinates and the candidate object region can be considered, as shown in Modification 1.

[0062]

number

[0063] r nr L is the radius of the nr-th concentric circle shown in Equation 7. (nr,q) W in equation 10 (nr,q) By replacing it with this, it becomes possible to integrate object region candidates while considering the distance between the indicated coordinates and the object region candidates.

[0064] When the multilayer neural network must be lightweight due to limitations in the circuitry installed in the imaging device, the resolution of its output map coordinates becomes low. Low map coordinate resolution results in a large discrepancy between the indicated coordinates and the map coordinates, even when selecting map coordinates near the indicated coordinates. In Modification 2, by interpolating each channel of the likelihood map and the object region candidate tensor, it is possible to achieve integration of object region candidates that is independent of the map coordinate resolution.

[0065] <Variation 3> The position of the object region determined in the object region candidate integration S209 of Embodiment 1 is calculated from the position of the object region candidate determined by the object region candidate selection. However, depending on the accuracy of the likelihood map, the object region obtained by the object region candidate integration S209 may differ from the object region intended by the user.

[0066] In the third modified example, the object region obtained as a result of the object region candidate integration S209 in Embodiment 1 is modified based on the coordinates (instruction coordinates) and likelihood map values ​​acquired by the instruction reception unit.

[0067] The specific correction method will be explained using Figure 10.

[0068] First, we will explain a method for obtaining and modifying only the likelihood related to the object region.

[0069] The object region likelihood, representing the likelihood of existence of object region 1001 obtained by object region candidate integration S209, is acquired. The position 1002(C) corresponds to the center of object region 1001 in the image coordinate system. x ,C y ) and each point (M) on the likelihood map x M y Point 1003(I x ,I y The Euclidean distance between (C) is calculated using the same distance function as in Equation 2. x ,C yThe object region likelihood is obtained using the likelihood map values ​​corresponding to one or more points with small Euclidean distances from ). The method for obtaining the object region likelihood may be, for example, the likelihood map value corresponding to the point with the closest Euclidean distance, or the average of the likelihood map values ​​corresponding to multiple points with close Euclidean distances. If the object region likelihood obtained in this way is below a certain value, it is considered to be an object region where the probability of the object's existence is estimated to be low. Therefore, it is possible that an object region 1001 with a center position 1002 different from the position indicated by the user using the indicated coordinate 1004 has been estimated. It is highly likely that the object region can be directly corrected to the region intended by the user by moving the center position 1002 of the object region in the direction of the indicated coordinate 1004. One way to move the object region in the direction of the indicated coordinate is to swap the center position 1002 of the object region with the indicated coordinate 1004. The object region is also shifted to match the center position. In addition, the object region likelihood L о Depending on the circumstances, a method for determining the amount of movement can also be considered. One example is the maximum value L output in the likelihood map. max and the object region likelihood L о Using the vector D1007 from the center position 1002 of the object region to the specified coordinates, each component V of the movement vector V1008 from the center position of the object region x、 V y This is calculated using equations 11 and 12. The position of the object region can be corrected by applying vector V1008 to the center position 1002 of the object region.

[0070]

number

[0071] Another possible method is to modify the object region according to the likelihood map values ​​near the indicated coordinates. First, obtain the object likelihood (indicated coordinate object likelihood) near the indicated coordinates (S902). Indicated coordinate 1004 (S x ,S y ) and each point 1003(I) of the likelihood map transformed into the image coordinate system x ,I yFor ), calculate the Euclidean distance and find one or more likelihood map points (I) that are close to the indicated coordinate 1004. x ,I y From this, the object likelihood of the indicated coordinates is obtained for the vicinity of the indicated coordinates. The object likelihood of the indicated coordinates can be obtained, for example, from the likelihood map value corresponding to the nearest single point, or from the average of the likelihood map values ​​corresponding to the nearest N points of the indicated coordinates. If the object likelihood of the indicated coordinates obtained in this way is high, it can be said that the probability of an object existing near the specified coordinates is estimated to be high. The center position 1002 of the object region can be moved in the direction of the indicated coordinates, similar to the correction based on the object region likelihood described above, to correct the object region to a position closer to the user's intended location.

[0072] Alternatively, one could obtain both the object region likelihood and the indicated coordinate object likelihood, and then use both to perform the correction. The flow of the correction method using both the object region likelihood and the indicated coordinate object likelihood will be explained according to the flowchart in Figure 9 and Figure 10.

[0073] First, the object region likelihood and the indicated coordinate object likelihood are compared (S903). If the indicated coordinate object likelihood is higher than the object region likelihood, a correction process is performed based on the object region position 1002 and indicated coordinate 1004 obtained by object region candidate integration S209 (S904). On the other hand, if the indicated coordinate object likelihood is less than or equal to the object region likelihood, the object region obtained by object region integration is output as is (S905).

[0074] The method of correction is explained using the example in Figure 10. When the object region is rectangle 1001, the width of the rectangle 、 The height of the rectangle is taken as the size of the object, and the estimated center of the rectangle is 1002(C x ,C y ) to indicated coordinates 1004(S x ,S y) and outputs the newly replaced object region 1005 as the corrected object region (S905). Additionally, a method of moving the object region in the direction where more objects are estimated to exist according to the object region likelihood of the object region 1001 and the value of the indicated coordinate object likelihood at the indicated coordinates 1004 can also be considered. Let the vector from the indicated coordinates 1004 to the center position 1002 of the object region be D1007, and the indicated coordinate object likelihood be L s , and the object region likelihood be L о . Using Equation 14 and Equation 15, each component V x、 V y of the vector V for moving the object region according to each likelihood is obtained. By setting the position obtained by applying the thus obtained vector V to the center position 1002 of the object region as the center position of the new object region, the object region 1001 can also be corrected in the direction with a higher likelihood.

[0075]

Number

[0076] By using the object region correction unit described in Modification Example 3, it is possible to correct to an object region closer to the object intended by the user than the object region output by the object region integration S209 described in Embodiment 1.

[0077] When realizing the likelihood map estimation unit using limited computing resources, the accuracy of the likelihood map and the object region candidates is limited, and there is a possibility that the position of the object cannot be captured as in the object region 1001 in FIG. 10. Even when the indicated coordinates 1004 accurately indicate the position of the object, the output object region may be estimated as an object region unintended by the user. In such a case, in Modification Example 3, it is possible to correct to the object region 1005 that more accurately captures the subject 1006 intended by the user using the indicated coordinates 1004 and the likelihood map.

[0078] <Modification Example 4> In the image processing apparatus of the above embodiment, the likelihood of objects included in the captured image was obtained using a single likelihood map acquisition unit. Therefore, the accuracy of the likelihood map acquisition unit directly affects the accuracy of the object region obtained by the object region candidate integration unit. To further improve the accuracy of the object region, a second likelihood map acquisition unit is introduced in this embodiment.

[0079] The configuration of the image processing apparatus in this embodiment will be explained in Figure 11.

[0080] The image processing device 1100 includes the same configuration as in Embodiment 1, and comprises a second likelihood map acquisition unit 1101 and an object region correction unit 1102. The configuration up to the point where the object region candidate integration unit 106 acquires one object region based on the captured image and instruction coordinates obtained by the image acquisition unit 101 and the instruction reception unit 104 is the same as in Embodiment 1. The image acquisition unit 101 receives the captured image and the second likelihood map acquisition unit 1101 and outputs a second likelihood map. The object region correction unit 1102 receives the object region, instruction coordinates, likelihood map and second likelihood map, and outputs one object region correction result by the result output unit 120.

[0081] Next, we will explain the specific processing flow using the flowchart in Figure 12.

[0082] First, the process from the start of shooting S200 to the acquisition of the likelihood map S202 is the same as in Embodiment 1. Next, the second likelihood map is acquired in the second likelihood map acquisition S1201. The second likelihood map acquisition S1201 outputs a likelihood map different from the likelihood map of Embodiment 1. For example, it may be a likelihood map acquisition unit that uses a color histogram or edge density, or a likelihood map acquisition unit that uses a multilayer neural network trained with a different learning method than in Embodiment 1. Following the second likelihood map acquisition S1201, the process from object region candidate acquisition S203 to object region candidate integration S209 is the same as in Embodiment 1. For the single object region integrated by object region candidate integration S209, modifications are made based on the indicated coordinates, the likelihood map obtained in likelihood map acquisition S202, and the second likelihood map obtained by the second likelihood map acquisition unit.

[0083] For example, the method of correction is as follows: First, a vector V1007 for correcting the object region is obtained by performing the same processing as the object correction unit described in Modification Example 3 using the likelihood map obtained in the instruction coordinate and likelihood map acquisition S202. Next, the distance between the point obtained by converting each point of the second likelihood map into the image coordinate system and the point obtained by converting the instruction coordinate into the image coordinate system is obtained, and the second likelihood map value corresponding to the second likelihood map coordinate closest to that point is set as the second instruction coordinate object likelihood Ls2. Also, the position (C x ,C y ) corresponding to the center of the object region and the second object region likelihood Lo2 is obtained using the second likelihood map values corresponding to one or more points whose distances from the points obtained by converting each point of the second likelihood map into the image coordinate system are close. The method for obtaining the second object region likelihood may be the second likelihood map value closest to (C x ,C y ), or may be the average of the second likelihood map values corresponding to a plurality of points whose distances up to (C x ,C y ) are close.

[0084] The second object region likelihood Lo thus obtained 2、 The second designated coordinate object likelihood Ls 2、 Using the vector D1007 from the center position of the object region to the instruction coordinate, each component of the second vector W for correcting the position of the object region is obtained by the following equations 16 and 17.

[0085]

Equation

[0086] A method of correction can be considered by applying the average vector of the vector V in Modification Example 3 and the second vector W thus obtained to the center position of the object region output by the object candidate integration unit.

[0087] One object region corrected by the object region correction unit of the fifth embodiment is output by the result output S210.

[0088] By using a second likelihood map acquisition unit in addition to the likelihood map acquisition unit in the above embodiment, it is possible to output a more plausible object region.

[0089] The second likelihood map acquisition unit acquires a likelihood map from the color histogram and edge density, allowing the validity of the object region output by the above embodiment to be determined using a method different from that of the likelihood map acquisition unit. Alternatively, the accuracy of the object region can be improved by making the second likelihood map acquisition unit a multilayer neural network trained with different training data than that of the likelihood map acquisition unit in Embodiment 1. For example, the second likelihood map acquisition unit could be trained to respond to specific objects that are prone to false detection, and the object region correction unit could then correct the object region based on that information. This makes it possible to perform corrections while considering the reliability of the likelihood map referenced during object region correction from multiple perspectives.

[0090] Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or CPU or MPU, etc.) of that system or device read and execute the program. [Explanation of symbols]

[0091] 100 Image Processing Devices 101 Image acquisition unit 102 Likelihood Map Acquisition Unit 103 Estimation part 104 Instruction Reception Department 105 Object Region Candidate Selection Unit 106 Object region candidate integration unit 110 Imaging device 120 Result Output Section

Claims

1. Image acquisition means for acquiring captured images and An instruction receiving means that receives instructions for the captured image acquired by the image acquisition means, A likelihood map acquisition means that represents the likelihood of the existence of an object in the aforementioned captured image, Estimation means for estimating information representing the offset amount to the center position of the object at each position corresponding to the likelihood map, and the width and height of the object. A determination means that determines an object region corresponding to the instruction based on the location of the instruction received by the instruction receiving means, using one or more candidate object regions obtained from the estimated information and the likelihood map. A modification means for modifying the object region determined by the determination means based on the position of the instruction received by the instruction receiving means and the likelihood map. An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, characterized in that the modification means modifies the object region by shifting the center position of the object region determined by the determination means to a position where the likelihood is higher, when the likelihood of the indicated position is higher than the likelihood of the object region determined by the determination means in the likelihood map.

3. The system further includes a second likelihood map acquisition means for acquiring a second likelihood map in a manner different from the method used to acquire the aforementioned likelihood map. The image processing apparatus according to claim 1 or 2, characterized in that the modification means modifies the object region determined by the determination means based on the position of the instruction received by the instruction receiving means, the likelihood map, and the second likelihood map.

4. The image processing apparatus according to claim 3, characterized in that the second likelihood map acquisition means acquires the second likelihood map using a color histogram or edge density.

5. The image processing apparatus according to claim 3, characterized in that the second likelihood map acquisition means acquires the second likelihood map using a neural network.

6. Image acquisition means for acquiring captured images and An instruction receiving means that receives instructions for the captured image acquired by the image acquisition means, A likelihood map acquisition means representing the likelihood of the existence of an object in the aforementioned captured image, Estimation means for estimating information representing the offset amount to the center position of the object at each position corresponding to the likelihood map, and the width and height of the object, A determination means that determines an object region corresponding to the instruction based on the location of the instruction received by the instruction receiving means, using one or more candidate object regions obtained from the estimated information and the likelihood map. An image processing system characterized by having a correction means for correcting an object region determined by the determination means based on the position of an instruction received by the instruction receiving means and the likelihood map.

7. An image processing method performed by one or more processors, Image acquisition process to acquire captured images and An instruction receiving step that receives instructions for the captured image acquired in the image acquisition step, and A process for obtaining a likelihood map representing the likelihood of the existence of an object in the captured image, An estimation step of estimating information representing the offset amount to the center position of the object at each position corresponding to the likelihood map, and the width and height of the object, A determination step in which, based on the location of the instruction received in the instruction receiving step, one or more candidate object regions obtained from the estimated information and the likelihood map are used to determine the object region corresponding to the instruction. An image processing method characterized by comprising a correction step for correcting the object region determined in the determination step based on the position of the instruction received in the instruction receiving step and the likelihood map.

8. Computers, Image acquisition means for acquiring captured images and An instruction receiving means that receives instructions for the captured image acquired by the image acquisition means, A likelihood map acquisition means representing the likelihood of the existence of an object in the aforementioned captured image, Estimation means for estimating information representing the offset amount to the center position of the object at each position corresponding to the likelihood map, and the width and height of the object, A determination means that determines an object region corresponding to the instruction based on the location of the instruction received by the instruction receiving means, using one or more candidate object regions obtained from the estimated information and the likelihood map. A program for causing a correction means to function as a correction means for correcting an object region determined by the determination means based on the location of the instruction received by the instruction receiving means and the likelihood map.

Images