Information processing device, information processing method, and program

Abstract

To set a focus target in an image by using a learning model that outputs a position subjectively desired by a user to be a focus target.SOLUTION: An information processing device acquires a captured image. A focus priority among a plurality of subjects in the captured image at the position of a subject in the captured image is output by inputting the captured image into a learning model that has been trained in advance so as to output the focus priority at the position of each subject in the input image.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Object detection, which detects any object in an image, is applied to the functions of digital cameras. Digital cameras can detect objects in the scene being imaged and focus on the detected object as the subject.

[0003] Patent Document 1 discloses a technology in which a priority is set for each subject, and shooting parameters including focal length are changed based on the priority changed according to the detected subject. Patent Document 2 discloses an invention in which a target subject for which shooting conditions should be set can be selected based on a preset priority, and the selection can be made according to the photographer's intention. Patent Document 3 further discloses a priority for determining the priority for each subject, calculated using parameters such as the size of the detected subject. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-57871 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-87572 [Patent Document 3] Japanese Patent Application Laid-Open No. 2010-141616 [Non-patent literature]

[0005] [Non-Patent Document 1] Xingyi Zhou et al., “Objects as Points”, [online], April 25, 2020, [Retrieved June 2, 2021], Internet<URL:https: / / arxiv.org / abs / 1904.07850> [Non-patent document 2] Alexy Bochkovskiy et al., “YOLOv4: Optimal Speed ​​and Accuracy of Object Detection”, [online], April 23, 2020, EESS, [searched on June 2, 2021], Internet<URL:https: / / arxiv.org / abs / 2004.10934> [Non-patent document 3] Chris Burges et al., “Learning to Rank using Gradient Descent”, [online], August 7, 2005, ICML, [Retrieved June 2, 2021], Internet<URL:https: / / icml.cc / 2015 / wp-content / uploads / 2015 / 06 / icml_ranking.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] In all of Patent Documents 1 to 3, the method of calculating the priority used for autofocusing is designed manually. That is, in Patent Documents 1 and 2, a priority is set in advance for the object to be detected, while in Patent Document 3 (and 2), a formula for calculating the priority is designed taking into account shooting conditions such as the position or size of the object to be detected.

[0007] However, when a user selects a focus target based on their own judgment, the selection becomes subjective and prone to fluctuation. Simply determining the AF priority in an image according to rules, as in Patent Documents 1 to 3, is not sufficient to express such subjective judgment.

[0008] The present invention aims to set a focus target in an image using a learning model that outputs a position that a user subjectively wants to set as a focus target. [Means for solving the problem]

[0009] To achieve the object of the present invention, for example, an information processing device according to one embodiment includes the following configuration: an acquisition unit that acquires a captured image, and an output unit that receives the captured image as an input and outputs a focus priority of a plurality of subjects in the captured image at a position of the subject, wherein the output unit outputs the focus priority by inputting the captured image to a learning model that has been trained in advance to output a focus priority at the position of each subject in the input image. [Effects of the Invention]

[0010] The focus target in the image is set using a learning model that outputs the position where the user subjectively wants to focus. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing the hardware configuration of an information processing apparatus according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing the functional configurations of the detection side and the learning side of the information processing device according to the first embodiment. [Figure 3] 4 is a flowchart showing a detection process and a learning process according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an NN used by the information processing device according to the first embodiment. [Figure 5] FIG. 2 is a diagram showing an input image used by the information processing device according to the first embodiment. [Figure 6] 5A and 5B are diagrams showing outputs of focusing adequacy estimated by the information processing device according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an output of a position estimated by the information processing device according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing an output of a magnitude estimated by the information processing device according to the first embodiment. [Figure 9] FIG. 3 is a view showing a display example of an object frame output by the information processing device according to the first embodiment. [Figure 10] 6 is a flowchart illustrating detailed detection processing performed by the information processing device according to the first embodiment. [Figure 11] FIG. 2 is a diagram showing training data used by the information processing device according to the first embodiment. [Figure 12] FIG. 10 is a diagram showing the functional configuration of an information processing device according to a second embodiment. [Figure 13] 10 is a flowchart showing processing for setting a focus target according to the second embodiment. [Figure 14] FIG. 10 is a diagram for explaining a screen displaying an object frame according to the second embodiment. [Figure 15] FIG. 10 is a diagram for explaining an enlarged screen on which a focus target is selected according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] [Embodiment 1] In this embodiment, a camera system is described that, when capturing an image with a digital camera, detects subjects to be candidates for focusing (focus candidates) from the captured image and outputs information indicating the focus priority at each position in the captured image.

[0014] FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device 100 according to this embodiment. The information processing device 100 includes a CPU 101, a memory 102, an input unit 103, a storage unit 104, a display unit 105, and a communication unit 106. The CPU 101 performs processing by each functional unit of the information processing device 100 shown in FIG. 2. The memory 102 is, for example, a ROM and a RAM, and stores data or programs used by the CPU 101. The input unit 103 is, for example, a touch panel, a button, a lever, a mouse, and a keyboard, and acquires user input. The storage unit 104 stores various data, such as results of processing by a learning device functional unit (described later) or images captured by an imaging device. The display unit 105 is, for example, a liquid crystal display (mounted on a camera), and displays captured images or results of processing by the CPU 101 to present to the user. The communication unit 106 communicates with an external device, and may acquire, for example, images captured by an imaging device or user input to an external device.

[0015] 2(a) is a block diagram showing an example of a functional configuration for detection processing of an information processing device according to this embodiment. The information processing device 100 includes an image acquisition unit 110, an inference unit 111, and a detection unit 112. In the following description, the information processing device 100 acquires captured images from an external imaging device 200 and performs processing on the acquired captured images using each functional unit. However, the information processing device 100 may also include an imaging unit (not shown) and perform detection processing (described later) on images captured by the imaging unit.

[0016] The image acquisition unit 110 acquires a captured image from the imaging device 200. Here, the captured image is assumed to be a single image, but multiple images, such as a group of temporally consecutive images (moving images), may be acquired and subsequent processing may be performed on one of them. The inference unit 111 estimates position information of a subject in the captured image. Here, the position information of the subject is assumed to be the position of the subject in the captured image and the size (width and height) of the subject. Furthermore, the inference unit 111 estimates the focusing adequacy (focusing priority) at each position in the image along with the position information of the subject.

[0017] The inference unit 111 estimates the focus adequacy in an image by a Learning to Rank method using a known machine learning method such as a neural network (NN) or SVM. The learning model used in this rank learning is learned by a learning unit 203 (to be described later), and each parameter is stored in the storage unit 104.

[0018] The focus accuracy according to the present embodiment is information indicating the priority of each position in a captured image as a position to be focused. The focus accuracy may be set for each partial region, for example, in the range of 0 to 100, or may be set in order of priority as a position to be focused, starting from 1, or may be set according to a degree of priority such as high, medium, or low, and the manner is not particularly limited. Furthermore, in the present embodiment, the focus is described as being adjusted by an autofocus (AF) function, but the focusing method is not limited to this and may be performed manually.

[0019] FIG. 4 is a diagram showing an example of each piece of data output by the inference unit 111 when a captured image is input. In FIG. 4, the inference unit 111 uses a multi-layer CNN to output three maps: a map indicating the position of each subject in the image (center map), a map indicating the size of each subject (size map), and a map indicating the focus adequacy at each position (adequacy map). Here, for example, the inference unit 111 can adopt a network structure used in known object detection techniques such as those shown in Non-Patent Document 1 or Non-Patent Document 2. That is, the inference unit 111 first inputs the image to a network called a backbone and outputs intermediate features. Next, the inference unit 111 inputs the intermediate features to separate networks for the tasks of estimating the position of the subject, estimating the size, and estimating the focus adequacy at each position, thereby obtaining the three maps described above.

[0020] Here, each map is a two-dimensional array, represented by a grid. Each map stores values ​​indicating image features in an array by repeatedly convolving or compressing the image. In the map output process, each map has a size that is smaller than the resolution of the input captured image.

[0021] 5 and 6 are diagrams illustrating the estimation process when a captured image is input to the inference unit 111. FIG. 5 shows an image 500 captured by the imaging device 200, which is input to the inference unit 111. When the image 500 is input, the inference unit 111 outputs an adequacy map 600 as shown in FIG. 6. The adequacy map 600 divides the input image into grid-like partial regions and is expressed as an array having the focus adequacy value of each grid as an element. Here, the adequacy map 600 expresses each element of the array using shading so that black areas have a higher focus adequacy than white areas. In the example of FIG. 6, the focus adequacy at position 601 is 10, the focus adequacy at position 602 is 80, the focus adequacy at position 603 is 50, and the focus adequacy at position 604 is 20, with the color at position 602 being the darkest black.

[0022] 7 is a diagram showing an example of a center map estimated by the inference unit 111. Here, the inference unit 111 uses the image 500 as input and infers the center map as an array having, as elements, the likelihood of each grid being the center position of the subject, which is divided into grids similar to the appropriateness map 600. In the center map 700, the position 701 of a chair, the position 702 of a person's face, the position 703 of the center of the car, the position 704 of the light, and the position 705 of the tire have high likelihoods, and the higher the likelihood, the darker the black is displayed.

[0023] FIG. 8 is a diagram illustrating an example of a size map estimated by the inference unit 111. In the example of FIG. 8, the inference unit 111 uses the image 500 as input and estimates a size map 800 representing the width of the subject and a size map 810 representing the height of the subject, which are displayed using a grid division similar to that of the appropriateness map 600. The size map 800 displays horizontal line segments centered on the subject's position and having a length equal to the subject's width on a grid, while the size map 810 displays vertical line segments centered on the subject's position and having a length equal to the subject's height on a grid. The size map 800 displays a chair width 801, a person's face width 802, a car width 803, a light width 804, and a tire width 805. The size map 810 displays a chair height 811, a person's face height 812, a car height 813, a light height 814, and a tire height 815.

[0024] The detection unit 112 generates an object frame (bounding box) as a focus candidate for the image based on the position information and focus adequacy of the subject estimated by the inference unit 111, and outputs it as a detection result together with the focus adequacy. The detection unit 112 can calculate the position and size of the subject by referring to the center map and size map inferred by the inference unit 111. In this example, the detection unit 112 may generate an object frame for each focus candidate from the position and size of the subject, and display it on the camera screen in association with the focus adequacy of the subject. For example, the detection unit 112 may display a numerical value indicating the focus adequacy of the subject position inside an object frame containing each detected subject, or may display the object frame in a color corresponding to the focus adequacy. Hereinafter, the focus adequacy of an object frame refers to the focus adequacy associated with the position of the subject included in the object frame.

[0025] The object frame having a color according to the focusing adequacy may be an object frame whose border or its interior has a color of varying shades according to the focusing adequacy. Furthermore, the detection unit 112 may display the inside or the frame of the object frame in a different color depending on whether the focusing adequacy of the object frame exceeds a threshold. For example, the detection unit 112 may display the color of the inside or the frame of the object frame in green when the focusing adequacy exceeds a first threshold, in yellow when the focusing adequacy is equal to or less than the first threshold and exceeds a second threshold, and in red when the focusing adequacy is equal to or less than the second threshold.

[0026] FIG. 9 is a diagram showing an example of a screen displaying the output result of the object frame generated by the detection unit 112. In the example of FIG. 9, an object frame 901 including a person and an object frame 902 including a horse are displayed on the screen. Here, the fact that the inference unit has inferred a higher degree of focus adequacy for the position of the person than for the horse is indicated by displaying the object frame 901 in a darker color. Note that the detection unit 112 may be configured to display only the object frame with the highest inferred degree of focus adequacy (here, 901) and not display the other object frames. In that case, the detection unit 112 may, for example, display a UI for selecting an object frame to be focused on, and when the user selects an object frame, visualize and display all object frames.

[0027] 2(b) is a block diagram showing an example of the functional configuration for learning processing of the information processing device according to this embodiment. The functional units of the information processing device 100 are the same as those in FIG. 2(a). The information processing device 100 transmits and receives information to and from a learning device 201 that generates a learning model, and acquires the learning model used in processing to output a focusing adequacy. In the following description, the information processing device 100 and the learning device 201 are described as separate devices, but each process performed by the learning device 201 may also be performed by the information processing device 100.

[0028] The learning device 201 acquires a group of training data including multiple training data each having position information indicating the position of a subject in an image and, in addition to the position information, information indicating the focusing adequacy corresponding to each position. The learning device 201 then uses the acquired group of training data as a correct answer to train a learning model that sets the focusing adequacy for the position of a subject detected in an input image. To this end, the learning device 201 includes an image database unit (DB unit) 210, an evaluation unit 211, a generation unit 212, and a training unit 213. The DB unit 210 stores multiple images to be used as training data for training the learning model.

[0029] The generation unit 212 generates multiple pieces of training data from images stored in the DB unit 210 to create a training data group. Here, the training data is data that includes the position of the subject in the image and information indicating the focusing adequacy at each position in the image (training adequacy), as described above. To achieve this, the learning device 201 transmits the images stored in the DB 210 to the information processing device 100, and detects and acquires the position of the subject in the image through processing by the inference unit 111 and the detection unit 112. Note that the position of the subject that the generation unit 212 includes in the training data may be detected by the learning device 201 itself, rather than by the information processing device 100.

[0030] The evaluation unit 211 sets a teacher suitability to be included in the teacher data. This teacher suitability may be calculated based on parameters in the image, or may be set according to user input, and the means for obtaining it is not limited.

[0031] An example of a method for setting teacher suitability will be described below. First, a case in which teacher suitability is calculated based on parameters in an image will be described. When a photograph is taken by a human, sharpness is generally high at the focused position. From this perspective, the evaluation unit 211 may calculate teacher suitability using sharpness as a parameter in the teacher image. Note that the image parameter used here is not limited to sharpness, as long as it is an image parameter that tends to be seen at the focused position. Here, the evaluation unit 211 calculates and sets teacher suitability for each object frame generated by the detection unit 112. For example, for each object frame, the evaluation unit 211 can divide the image within the object frame into small regions, calculate the variance of pixel values ​​within each small region, and then use the average of all the variance values ​​as the sharpness of that object frame. Note that the teacher suitability is set on a map of the same size as each map output by the inference unit 111.

[0032] Furthermore, in a photograph with a shallow depth of field, the difference in sharpness between the in-focus and out-of-focus areas of the image tends to be large. From this perspective, a training image captured with a depth of field shallower than a predetermined threshold depth of field may be used. The depth of field threshold used here can be set by the user as desired.

[0033] FIG. 11 is a diagram showing an example in which teacher data, which is a map with teacher suitability set and output by the generation unit 212, is superimposed (enlarged to a corresponding size) on a teacher image. Here, the generation unit 212 sets the teacher suitability calculated by the evaluation unit 211 based on sharpness at a position corresponding to the teacher image on a map of the same size as each map output by the inference unit 111. Here, the generation unit 212 can convert the coordinates of the subject on the teacher image to coordinates on the teacher suitability map by calculating the scale ratio between the teacher image and the map. Region 1101 corresponds to the eyes of a person in the teacher image, and the teacher suitability calculated from the sharpness is set to 40. Region 1102 corresponds to a person in the teacher image, and the teacher suitability calculated from the sharpness is set to 80. Region 1103 corresponds to a horse in the teacher image, and the teacher suitability calculated from the sharpness is set to 70. In the example of FIG. 11, region 1102 has the highest teacher suitability, indicating that this region is the most suitable as a focusing position.

[0034] Next, as described above, the teacher data may include image data whose teacher suitability is set according to user input. In this case, the evaluation unit 211 acquires a user's input of a focus suitability for the teacher data image (teacher image). Here, the user can, for example, specify a position in the teacher image where a detection target appears and set the focus suitability at that position. The focus suitability set here may be, for example, a focus priority among multiple types of detection targets (across multiple teacher images), or an evaluation value set for the subject. The evaluation value according to this embodiment may be, for example, a value set in the range of 0 to 100 (the higher the value, the higher the priority of focusing), or may be an evaluation such as high / medium / low priority. The focus suitability and evaluation value set here are values ​​input by the user, but may be set or corrected with reference to parameters in the image.

[0035] The evaluation unit 211 can set the teacher suitability in accordance with the user's input. Here, the evaluation unit 211 can set a position in the image not designated by the user as a non-focus position (e.g., a focus suitability of 0 or a low priority). The evaluation unit 211 may also set the teacher suitability so that it varies depending on the distance from the user-designated position, with the user-designated position as the center. That is, the evaluation unit 211 may set the teacher suitability of the user-designated position to the focus suitability value input by the user, and the teacher suitability value may decrease as the position moves away from the user-designated position. In this case, the evaluation unit 211 may subtract a value from the teacher suitability of a certain position in the teacher image depending on the distance from the user-designated position, or may classify the position into high, medium, or low priority depending on the magnitude relationship between the distance from the user-designated position and a threshold. The evaluation unit 211 may also detect a detection target from the teacher image and set the teacher suitability of the entire detection target area including the user-designated position as the focus suitability value input by the user.

[0036] By setting the teacher suitability in this way, it becomes possible to generate teacher data that reflects the focusing suitability based on the user's subjective opinion, and to carry out learning.

[0037] The learning unit 213 uses the training data generated by the generation unit 212 as a correct answer, an image as an input, and learns a learning model that outputs a focusing adequacy at each position in the image. In this embodiment, the learning unit 213 will be described as updating parameters for the inference unit 111 to output an adequacy map as the above-mentioned learning model. The method for learning the learning model by the learning unit 213 is not particularly limited as long as it can input an image and output the above-mentioned focusing adequacy, and can be performed by any known method.

[0038] The learning unit 213 may make the learning model learn the ranking between positions in the input image by rank learning using RankNet as described in Non-Patent Document 3. In this case, the learning unit 213 may learn the ranking between positions in the input image by learning the ranking between positions in the input image by using the value y of the i-th (1≦i≦N) element, where N is the number of elements in the map of the training data. iand the value of the jth (1≦j≦N) element, y j Here, we learn the order relationship between y i and y j The value of the element of the appropriateness map estimated by the inference unit 111 from certain training data corresponding to x i and x j Then, the error C ij is calculated using the following formula (1).

number

[0039] The learning unit 213 uses this C ij is calculated for all pairs (i, j), and the sum is calculated as the final error for the teacher data. Next, the learning unit 213 updates the parameters of the NN in the inference unit 111 using the error backpropagation method, and the updated parameters can be stored in the storage unit 104. By using the updated learning model in this step, the inference unit 111 can estimate the focusing adequacy using an image as input. Note that the learning unit 213 may perform learning, instead of the above-mentioned rank learning, so as to output, for example, a focusing adequacy value that is the same as the teacher adequacy value included in the teacher data.

[0040] 3A is a flowchart showing an example of a focus candidate detection process performed by the information processing device 100 according to this embodiment. In S301, the image acquisition unit 110 acquires an image from the imaging device 200. In S302, the inference unit 111 outputs a center map, a size map, and an adequacy map from the acquired image. In S303, the detection unit 112 generates an object frame as a focus candidate based on the maps output in S302, and outputs it to the imaging device 200 together with the focus adequacy.

[0041] 10 is a flowchart showing an example of an object frame detection process performed by detection unit 112. In S1001, detection unit 112 acquires each map output by inference unit 111. In S1002, detection unit 112 generates an object frame in the image using the center map and the size map. Here, detection unit 112 generates, as the object frame, a rectangular region whose center is centered on the position of each subject estimated in the center map and whose width and height correspond to the subject shown in the size map. In S1003, detection unit 112 outputs the generated object frame to imaging device 200 together with the focus adequacy corresponding to the position of the object frame on the adequacy map.

[0042] Here, imaging device 200 basically sets the object frame with the highest focus appropriateness as the focus target. However, the setting of the focus target actually used is not limited to this. The object frame to be the focus target may be selected by the user from among the displayed object frames, for example, or the object frame with the highest focus appropriateness may be set as the initial focus target and changed based on user input. A detailed description of the case where a user input specifying the focus target is given in embodiment 2.

[0043] 3(b) is a flowchart showing an example of the learning process of a learning model performed by the learning device 201 according to this embodiment. In S311, the evaluation unit 211 and the image acquisition unit 110 acquire a teacher image from the DB unit 210. In S312, the inference unit 111 outputs a center map, a size map, and an appropriateness map from the teacher image. The center map and size map are output to the detection unit 112, and the appropriateness map is output to the learning unit 213.

[0044] At S313, detection unit 112 generates an object frame by processing similar to that of S302. At S314, evaluation unit 211 sets teacher suitability from the teacher image. Here, evaluation unit 211 sets teacher suitability based on the object frame generated at S313 and the sharpness in the teacher image. At S315, generation unit 212 generates a map to serve as teacher data, including the teacher image, coordinate values ​​of the object frame, and the teacher suitability set at S313. At S316, learning unit 213 updates the parameters of the learning model of inference unit 111 based on the teacher data generated at S315 and the suitability map output at S312, and causes the learning model to learn.

[0045] With this configuration, it is possible to output the focus priority of multiple subjects at the position of the subject in the captured image using a learning model that has been trained in advance to output the focus priority at the position of each subject in the input image. Therefore, the focus target in the captured image can be determined using a learning model that outputs a focus position based on human subjectivity.

[0046] [Embodiment 2] In the first embodiment, the learning model infers the focus adequacy for each position in the image, and indicates the position that is most suitable as a focus position (having the highest focus adequacy). However, the subject at the position with the highest inferred focus adequacy is not necessarily the focus target desired by the user. From this perspective, the information processing device 100 according to this embodiment first inputs a captured image into a learning model trained in the same manner as in the first embodiment to set the focus adequacy. Next, the information processing device 100 presents the set focus adequacy to the user and acquires user input specifying the focus position (focus target). Hereinafter, the focus target refers to the subject present at the focus position or an object frame on the captured image that includes the subject.

[0047] 12, the information processing device 100 according to this embodiment has the same configuration as in embodiment 1 and performs the same processing, so duplicated explanations will be omitted. Furthermore, as in embodiment 1, each functional unit of the imaging device 200 described below may be included in a device external to the information processing device 100, or may be implemented in the same device as the information processing device 100. The imaging device 200 according to this embodiment includes an imaging unit 1201, a cropping unit 1202, an image generation unit 1203, a display unit 1204, a ranking unit 1205, a selection storage unit 1206, an operation unit 1207, and a switching unit 1208.

[0048] The imaging unit 1201 acquires a scene outside the imaging device 200 as a captured image (image data). The cropping unit 1202 crops a part of the captured image acquired by the imaging unit 1201 as a partial image. When the cropping unit 1202 acquires a user input specifying a focus position, the cropping unit 1202, which will be described in detail later with reference to FIG. 15 , can crop a part of the captured image based on the specified focus position to generate a partial image. Hereinafter, the information processing device 100 sets the focus adequacy using a learning model, using as input the captured image acquired by the imaging unit 1201 or the partial image cropped by the cropping unit 1202 (these will be referred to as "input image" without distinction between them).

[0049] The ranking unit 1205 ranks the object frames in the input image in descending order of the focusing suitability set by the information processing device 100. The ranking unit 1205 also updates the information on the object frames to be focused, stored in the selection storage unit 1206. Here, the ranking unit 1205 may determine whether any of the object frames set in the input image are identical to the object frames stored in the selection storage unit 1206. If these object frames are identical, the position (coordinate values) of the object frame stored in the selection storage unit 1206 is updated to the value set in the input image. If none of the object frames set in the input image are identical to the object frames stored in the selection storage unit 1206, the information on the object frames to be focused stored in the selection storage unit 1206 is deleted, and a new object frame is set. This determination of whether or not they are identical can be performed using a known technique for determining whether or not they are identical, such as calculating Intersection over Union (IoU). In this case, the ranking unit 1205 can determine whether the two object frames are the same object depending on whether the IoU of those object frames is equal to or greater than a preset threshold. Note that even if an object frame set as a focus target stored in the selection storage unit 1206 exists in the input image, the ranking unit 1205 may set the object frame in the input image with the highest focus adequacy as a focus target regardless of that.

[0050] The image generation unit 1203 generates an image in which the object frames are superimposed on the input image along with the rankings set by the ranking unit 1205. The format of the object frame display here is not particularly limited as long as the rankings of the object frames can be presented to the user. For example, the image generation unit 1203 may display each object frame differently based on the focusing accuracy, or may display the object frame with the highest ranking differently from the other object frames. Here, the image generation unit 1203 may display each object frame using a shading display according to the ranking, may display each object frame using a color corresponding to the ranking, or may display the ranking number together with each object frame. Furthermore, for example, the image generation unit 1203 may display the object frame with the highest ranking using a solid line (highlighted), and display the other object frames using dotted or dashed lines, or may display only the object frame with the highest ranking.

[0051] The display unit 1204 displays the image generated by the image generation unit 1203 and presents it to the user. Fig. 14 is a diagram showing an example of an image displayed by the display unit 1204 on the screen of the imaging device 200, which is a camera. Here, five cars are detected as subjects in the image, and the ranking of the focus appropriateness is displayed together with the corresponding object frame. Here, the object frame 1402 ranked first is displayed with a solid line, and the object frame 1401 ranked second and the other object frames are displayed with dashed lines. Here, as will be described in detail later, when the object frame 1402 is designated by the user (by a touch operation or the like), the area near the object frame 1402 is enlarged and displayed as shown in Fig. 15. Furthermore, when the object frame 1401 is designated by the user, the object to be focused is set to the object frame 1401.

[0052] Here, the display unit 1204 can acquire a user input specifying a focus position via the operation unit 1207. The focus position specification may be input, for example, by touching a corresponding area on a touch panel, or by operating a mechanical switch such as a lever or button, and is not particularly limited as long as it is a user input for selecting a focus position. The operation unit 1207 functions to acquire such user input. For example, the display unit 1204 can accept a focus position specification by a user via operation of the touch panel for each object frame (a candidate for a focus target) displayed on a touch panel mounted on the digital camera.

[0053] The switching unit 1208 switches the setting of the focus position. For example, when the object frame ranked first by the ranking unit 1205 is different from the object frame (set as the object frame of the focus position) stored in the selection storage unit 1206, the switching unit 1208 may switch the first-ranked object frame as the focus position. Furthermore, when a user input specifying a focus position is acquired, the switching unit 1208 may set the object frame specified by the user input as the focus position and store the setting in the selection storage unit 1206.

[0054] FIG. 15 is a diagram showing an example of a screen displaying a partial image cropped by cropping unit 1202 when object frame 1402 in FIG. 14 is selected by the user. Cropping unit 1202 crops out an area near object frame 1402 selected by the user from the screen in FIG. 14, enlarges it, and displays it on the screen. Here, a new object frame is set along with a focus adequacy for the small parts or person of the enlarged subject, and the person's head is given the first place ranking. Therefore, switching unit 1208 sets the focus position to the person's head. This processing makes it possible to set even smaller parts or people as the focus target in the area near the subject of interest.

[0055] 13 is a flowchart showing an example of processing for setting a focus target according to this embodiment. In S1301, the imaging unit 1201 acquires a captured image and outputs it to the cropping unit 1202. In S1302, the cropping unit 1202 sets an object frame for the input image using processing similar to that in the first embodiment.

[0056] In S1303, the ranking unit 1205 updates the information on the object frame to be focused, which is stored in the selection storage unit 1206. In S1304, the ranking unit 1205 ranks the object frames in the image in descending order of focus appropriateness. In S1305, the ranking unit 1205 determines whether or not the object frame that was set as the focus target exists. If it exists, the focus target is not changed and the process proceeds to S1307, and if not, the process proceeds to S1306.

[0057] In S1306, the switching unit 1208 sets the object frame with the highest rank set by the ranking unit 1205 as the focus target, and the process proceeds to S1307. In S1307, the image generation unit 1203 generates an image in which the object frame is superimposed on the input image, along with the rank set by the ranking unit 1205. In S1308, the display unit 1204 displays the image generated in S1307.

[0058] In S1309, the display unit 1204 acquires a user input specifying a focus position via the operation unit 1207. If no user input is acquired here, the display unit 1204 sets the object frame ranked first as the focus target and ends the processing. If a user input is acquired, the processing proceeds to S1310.

[0059] In S1310, the display unit 1204 determines whether the object frame specified by user input is the object frame to be focused stored in the selection storage unit 1206. If it is the object frame to be focused, the process proceeds to S1312; if not, the process proceeds to S1311. In S1311, the switching unit 1208 sets the specified object frame as the object frame to be focused, and the process returns to S1307.

[0060] In S1312, the cropping unit 1202 crops out an area near the object frame to be focused as a partial image, and returns the process to S1302 using the cropped image as an input image. Here, the area near the object frame may be an area obtained by adding a predetermined width in each of the height and width directions from the object frame, or may be the object frame itself.

[0061] This process allows the user to input a focus position in a captured image and change the focus position according to the user's input. Therefore, even if the subject with the highest focus accuracy inferred by the learning model is not the subject the user wants, the focus position can be adjusted to the desired subject.

[0062] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0063] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0064] 100: Information processing device, 110: Image acquisition unit, 111: Inference unit, 112: Detection unit, 104: Storage unit

Claims

1. an acquisition means for acquiring a captured image; an output unit that receives the captured image as an input and outputs a focus priority of the plurality of subjects in the captured image at a position of the subject in the captured image, the output means outputs the focus priority by inputting the captured image to a learning model that has been trained in advance to output a focus priority at the position of each subject in the input image; The information processing device is characterized in that the output means uses a partial image cut out from an area near an object frame including the subject in the captured image as input to the learning model, and further outputs a focus priority among multiple subjects in the partial image at the position of the subject in the partial image.

2. The information processing apparatus according to claim 1 , further comprising a determining unit for determining a focus position of the captured image in an imaging device that captures the captured image.

3. The information processing apparatus according to claim 2 , wherein the determining means determines a focus position in the captured image based on the focus priority output by the output means.

4. a second acquisition means for acquiring a user's designation of a focus position in the captured image; a change unit that changes the focus position in the captured image in accordance with a designation by the user; The information processing device according to claim 1 , further comprising:

5. 5. The information processing apparatus according to claim 4, further comprising: a presenting unit that presents the focusing priority output by the output unit to a user.

6. 6. The information processing device according to claim 5, wherein the presentation means presents the focusing priority output by the learning model to the user by performing a display according to the focusing priority output by the learning model for each object frame including a subject in the captured image.

7. 7. The information processing device according to claim 5, wherein the presentation means displays, by a first display, a first object frame including a subject at a position on the captured image that has the highest focusing priority output by the learning model, and displays, by a second display, a second object frame different from the first object frame.

8. 8. The information processing device according to claim 7, wherein the presentation means, when the second object frame is specified by the user specification acquired by the second acquisition means, changes the display of the second object frame to a first display and changes the display of the first object frame to a second display.

9. 9. The information processing device according to claim 7, wherein the first display and the second display are different in color of the object frame, color inside the frame, format of the frame, or shading of the frame.

10. a first acquisition means for acquiring a teacher data group including a plurality of teacher data having information indicating a focusing priority for each position in a teacher image; a learning means for performing learning of a learning model that outputs a focus priority of a plurality of subjects in a captured image in association with a position of the subject in the captured image when the group of teacher data is used as a correct answer and a captured image is input; Equipped with the first acquisition means calculates information indicating the focusing priority based on image parameters of a teacher image captured at a first depth of field; The information processing device, wherein the first depth of field is shallower than a second depth of field that serves as a threshold value.

11. 11. The information processing apparatus according to claim 10, wherein the image parameter is sharpness.

12. 12. The information processing device according to claim 10, wherein the output of the learning model indicates a focus priority of the object frame in association with each position within the object frame that includes the position of the subject in the training image.

13. The information processing apparatus according to claim 12, wherein the focus priority indicates the focus priority with respect to other object frames between the object frames.

14. The information processing device according to claim 1 , wherein the learning model is a neural network.

15. acquiring a captured image; and outputting, as an input of the captured image, focus priorities of the plurality of subjects in the captured image at positions of the subjects in the captured image, the focusing priority is output by inputting the captured image to a learning model that has been trained in advance to output a focusing priority at the position of each subject in a teacher image; An information processing method characterized in that a partial image cut out from an area near an object frame including the subject in the captured image is used as input to the learning model, and a focus priority of the multiple subjects in the partial image at the position of the subject in the partial image is further output.

16. acquiring a teacher data group including a plurality of teacher data having information indicating a focusing priority for each position in a teacher image; a step of performing learning of a learning model that sets a focus priority among a plurality of subjects in a captured image to a position of the subject in the captured image when the captured image is input using the group of teacher data as a correct answer; Equipped with information indicating the focusing priority is calculated based on image parameters of a teacher image captured at a first depth of field; An information processing method, wherein the first depth of field is shallower than a second depth of field that serves as a threshold value.

17. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 14.

Images