Information processing device, information processing method

Abstract

To provide technology for efficiently selecting learning models according to specific objectives. [Solution] The information processing device selects a representative image from the images captured by the imaging unit, which processes the captured images using a learning model, according to the user operations performed by the user during the capture. The information processing device then presents the user with the inference results of multiple learning models for the representative image, and changes the learning model used by the imaging unit to the learning model selected by the user in response to the presentation. In the above process, the device recognizes the subject in the captured image, focuses on the subject, and tracks it.

Description

Technical Field

[0001] The present invention relates to a technique for selecting a learning model.

Background Art

[0002] In recent years, computer vision (CV) tasks using machine learning methods have been utilized in various scenarios. As a conventional technique, there are services in which a learning model can be used by creating a machine-learned model (hereinafter referred to as a "learning model") according to the user's own purpose or selecting a learning model from a plurality of learning models publicly available and distributed on the service. For example, Patent Document 1 discloses a method for efficiently selecting a learning model by comparing learning models using the object detection results in a plurality of learning models.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, since the user does not have an index as to what kind of image to use to select a learning model suitable for their own purpose, it is conceivable that difficult-to-compare images or a large number of images will be compared, and there is a problem that the selection of the learning model may not be efficient. The present invention provides a technique for efficiently selecting a learning model according to the purpose.

Means for Solving the Problems

[0005] One aspect of the present invention includes a selection means for selecting a representative image from an image captured by an imaging unit that processes the captured image using a learning model, in accordance with user operations performed by the user during the capture, and a changing means for presenting to the user the inference results of a plurality of learning models for the representative image selected by the selection means, and changing the learning model used by the imaging unit to the learning model selected by the user in accordance with the presentation, wherein the processing is characterized by recognizing a subject in the captured image, focusing on the subject, and tracking it. [Effects of the Invention]

[0006] According to the present invention, a learning model can be efficiently selected according to the purpose. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram showing an example of the hardware configuration of the information processing device 1. [Figure 2] A block diagram showing an example of the functional configuration of a system to which the information processing device 1 is applied. [Figure 3] A flowchart of the system's operation. [Figure 4] A diagram showing the changes in the state of the imaging unit 210. [Figure 5] A diagram showing the changes in the state of the imaging unit 210. [Figure 6] A flowchart of the system's operation. [Figure 7] A block diagram showing an example of the system's functional configuration. [Figure 8] A diagram showing the changes in the state of the imaging unit 210. [Modes for carrying out the invention]

[0008] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0009] [First Embodiment] First, an example of the hardware configuration of the information processing device 1 according to this embodiment will be explained using the block diagram in Figure 1. The information processing device 1 according to this embodiment is a computer device such as a PC, tablet terminal, or smartphone.

[0010] The CPU (Central Processing Unit) 100 executes various processes using computer programs and data stored in the RAM 120. In doing so, the CPU 100 controls the operation of the entire information processing device 1 and executes or controls the various processes described as those performed by the information processing device 1.

[0011] The ROM (Read-Only Memory) 110 stores configuration data for the information processing device 1, computer programs and data related to the startup of the information processing device 1, computer programs and data related to the basic operation of the information processing device 1, and so on.

[0012] The RAM (Random Access Memory) 120 has an area for storing computer programs and data loaded from the ROM 110 and HDD (Hard Disk Drive) 130. Furthermore, the RAM 120 has an area for storing computer programs and data received from external devices via the communication unit 160. In addition, the RAM 120 has a work area used by the CPU 100 when executing various processes. In this way, the RAM 120 can provide various areas as appropriate.

[0013] The HDD 130 stores an operating system (OS), computer programs, data, etc. for causing the CPU 100 to execute or control various processes described as the processes performed by the information processing apparatus 1.

[0014] In addition to or instead of the HDD 130, an external storage device may be used. The external storage device can be realized, for example, by a medium (recording medium) and an external storage drive for accessing the medium. Examples of such a medium include a flexible disk (FD), CD-ROM, DVD, USB memory, MO, flash memory, etc. Also, the external storage device may be a server device or the like connected to the information processing apparatus 1 via a network.

[0015] The input unit 140 is a user interface such as a keyboard, mouse, touch panel, etc., and various instructions and information can be input to the information processing apparatus 1 by the user's operation.

[0016] The display unit 150 has a screen such as a liquid crystal screen or a touch panel, and displays the processing result by the CPU 100 as an image, characters, etc. Note that the display unit 150 may be a projection device such as a projector that projects images and characters.

[0017] The communication unit 160 performs data communication with an external device via a network such as a LAN or the Internet. For example, the information processing apparatus 1 may acquire various instructions and information input by the user operating an external device via the communication unit 160.

[0018] The CPU 100, ROM 110, RAM 120, HDD 130, input unit 140, display unit 150, and communication unit 160 are all connected to the system bus 170. Note that the hardware configuration applicable to the information processing apparatus 1 is not limited to the configuration shown in FIG. 1, and can be appropriately changed / modified.

[0019] Next, a functional configuration example of a system to which such an information processing apparatus 1 is applied is shown in the block diagram of FIG. 2. As shown in FIG. 2, the system includes an imaging unit 210 including an actually used learning model 215 and the information processing apparatus 1.

[0020] The actually used learning model 215 is a learning model selected by the information processing apparatus 1 from a group of candidate learning models 290 held in the information processing apparatus 1. The group of candidate learning models 290 is a set of learning models that have been learned to detect and track "a subject to be the target of tracking AF (Auto Focus)" from an image. Therefore, the imaging unit 210 detects "a subject to be the target of tracking AF" using the actually used learning model 215 from a captured image captured by the imaging unit 210, and automatically focuses and tracks based on the position where the subject is detected in the captured image, which is the "tracking AF" process.

[0021] The information processing apparatus 1 selects one or more captured images as selected images from a group of captured images captured by the imaging unit 210 in response to a user operation regarding tracking AF. Then, the information processing apparatus 1 evaluates the group of candidate learning models 290 using the selected images, and changes the actually used learning model 215 based on the result of the evaluation.

[0022] The operation of the system according to the present embodiment will be described according to the flowchart of FIG. 3. Hereinafter, a form in which the image storage unit 230 is implemented by the RAM 120 or the HDD 130 and each functional unit other than the image storage unit 230 is implemented by software (computer program) in the functional units of the information processing apparatus 1 shown in FIG. 2 will be described. Hereinafter, the functional units (excluding the image storage unit 230) of the information processing apparatus 1 shown in FIG. 2 will be described as the main body of the processing. Actually, the functions of the functional units are realized by the CPU 100 executing a computer program corresponding to the functional units. Note that one or more of the functional units other than the image storage unit 230 may be implemented by hardware.

[0023] In step S31, the selection unit 280 acquires the candidate learning model group 290 and stores the acquired candidate learning model group 290 in the HDD 130 or RAM 120. The method for acquiring the candidate learning model group 290 is not limited to a specific method. For example, the selection unit 280 may download the candidate learning model group 290 stored in an external device (server device, external storage device, cloud, etc.) to the RAM 120 or HDD 130 via the communication unit 160. Such a candidate learning model group 290 may be a learning model created by the user themselves, or it may be a learning model that is generally distributed.

[0024] In step S32, the selection unit 280 sets one of the candidate learning models 290 as the actual learning model 215 in the imaging unit 210. The actual learning model 215 may be a learning model that was set in the imaging unit 210 in step S31 or earlier.

[0025] The captured image taken by the imaging unit 210 is input to the information processing device 1 and stored in the image storage unit 230. The image acquisition unit 220 may acquire the captured image input from the imaging unit 210, or it may acquire the captured image stored in the image storage unit 230.

[0026] In step S33, the imaging unit 210 starts tracking AF on the captured image when a tracking AF execution instruction is input in response to user operation, and terminates (cancels) tracking AF when a tracking AF termination instruction is input in response to user operation.

[0027] For example, as shown in Figure 4(a), the imaging unit 210 displays the captured image on the display screen 4130. The imaging unit 210 performs tracking AF while the user is pressing the AF button 4110 with their finger, and terminates tracking AF when the user releases their finger from the AF button 4110. For example, the imaging unit 210 performs so-called "thumb AF," which continues tracking AF while the AF button 4110 is pressed with the thumb, or "shutter half-press AF," which continues tracking AF while the shutter button 4120 is half-pressed. Note that the input method for instructing the execution / termination of tracking AF is not limited to a specific input method.

[0028] Figure 4(a) shows a dog being photographed as the "subject targeted by tracking AF," and the display screen 4130 shows the captured image including the dog. Furthermore, the display screen 4130 also shows a bounding box (BB) surrounding the dog as a result of tracking AF.

[0029] In step S34-2, the acquisition unit 240 determines whether the elapsed time from the end of the previously executed tracking AF to the start of the currently executed tracking AF is less than or equal to a threshold (for example, within 0.5 seconds).

[0030] As a result of this judgment, if the elapsed time is below the threshold, that is, if the user inputs an instruction to end tracking AF due to a failure of tracking AF or other reasons during tracking AF, but immediately inputs an instruction to start tracking AF again, a so-called "tracking AF re-engagement" has occurred, the image acquisition unit 220 associates the "tracking AF release information" as "operation information representing user operation to the imaging unit 210" with the image of the current frame (the frame being shot) acquired from the imaging unit 210 or the image storage unit 230. Then the process proceeds to step S34-3. On the other hand, if the elapsed time is greater than the threshold, that is, if "tracking AF re-engagement" has not occurred, the process proceeds to step S35.

[0031] Figure 4(b) shows the state of the image sensor 210 performing tracking autofocus on a running dog as the "subject to be tracked by tracking AF". The leftmost image shows the state of the image sensor 210 at time t, the center image shows the state of the image sensor 210 at time (t+1), and the rightmost image shows the state of the image sensor 210 at time (t+2).

[0032] At time t, the user pressed the AF button 4110 with their finger, and as a result of tracking AF, the dog is in focus, and a bounding box is displayed at the dog's position.

[0033] At time (t+1), the user has pressed the AF button 4110 with their finger, but as a result of the tracking AF, the dog is not in focus, and the position and size of the bounding box do not match the position and size of the dog in the image. In this case, the user may judge that the tracking AF has failed and immediately release and restart the tracking AF during the shooting operation to regain tracking AF on the dog. In such a case, the image captured at time (t+1) is suitable as an "image where tracking AF failed" for evaluating each learning model in the candidate learning model group 290. Therefore, the image acquisition unit 220 associates "tracking AF release information" with the image captured at time (t+1).

[0034] At time (t+2), since the "tracking AF re-engagement" has occurred, the tracking AF has resulted in the dog being in focus, and a bounding box of the appropriate size and position is displayed.

[0035] In step S34-3, the acquisition unit 240 determines whether a threshold amount of time (e.g., 3.0 seconds or more) has elapsed since the start of the currently executing tracking AF. If, as a result of this determination, a threshold amount of time has elapsed since the start of the currently executing tracking AF, it is determined that the tracking AF has been executed accurately during the series of shooting operations, and the frame after a long period of continuous tracking AF is suitable as an "image with successful tracking AF" for evaluation of the candidate learning model group 290. Therefore, the image acquisition unit 220 associates the "tracking AF continuation information" as "operation information representing user operation to the imaging unit 210" with the image of the current frame acquired from the imaging unit 210 or the image storage unit 230. The process then proceeds to step S35. On the other hand, if a threshold amount of time has not elapsed since the start of the currently executing tracking AF, the process proceeds to step S36.

[0036] In step S35, the selection unit 260 selects from the captured images acquired by the image acquisition unit 220 as selected images (representative images) to be used for evaluating the candidate learning model group 290.

[0037] For example, the selection unit 260 may select as a representative image a shooting image acquired by the image acquisition unit 220 that has "tracking AF release information" associated with it as operation information. Alternatively, the selection unit 260 may select as a representative image a shooting image acquired by the image acquisition unit 220 that has "tracking AF continuation information" associated with it as operation information. The representative image group 265 is a collection of representative images selected by the selection unit 260. A representative image may be a single still image or a moving image containing multiple frames of captured images.

[0038] In step S36, the selection unit 260 determines whether the number of representative images included in the representative image group 265 is equal to or greater than a predetermined number set in advance as sufficient for evaluating the candidate learning model group 290.

[0039] As a result of this determination, if the number of representative images included in the representative image group 265 is equal to or greater than the specified number, the number of representative images is deemed sufficient, and the process proceeds to step S37. On the other hand, if the number of representative images included in the representative image group 265 is less than the specified number, the number of representative images is deemed insufficient, and the process proceeds to step S33.

[0040] In step S37, the processing execution unit 270 inputs the representative images included in the representative image group 265 into each of the learning models in the candidate learning model group 290 and performs calculations on the learning models to obtain the subject detection result by the learning model as the subject detection inference result.

[0041] In step S38, the processing execution unit 270 presents the inference results obtained in step S37 to the user. In step S39, the selection unit 280 determines the learning model to be set as the actual-use learning model 215 and changes the learning model currently in use as the actual-use learning model 215 to the selected learning model.

[0042] An example of the processing in steps S37-S39 will be described. For example, as shown in Figure 4(c), the processing execution unit 270 displays a list of thumbnails of each representative image in the representative image group 265 on the display screen of the imaging unit 210. When the user touches a thumbnail 4310 of one representative image in the representative image group 265 with their finger, the processing execution unit 270 inputs the thumbnail 4310 or the captured image corresponding to the thumbnail 4310 into each learning model in the candidate learning model group 290 and performs calculations on the learning model to obtain the inference result. The processing execution unit 270 then displays the inference results of each learning model in the candidate learning model group 290 in a list in the display area 4320. The display area 4320 displays the inference results (bounding boxes of the subject in the captured image corresponding to the thumbnail 4310) for each of the current model (actual use learning model 215), model 1, model 2, and model 3.

[0043] The user then touches the inference result that best suits their purpose from the inference results of each learning model displayed in the display area 4320. The processing execution unit 270 increments the counter corresponding to the inference result each time an inference result is touched and displays the percentage of the counter relative to the total number of votes. In Figure 4(c), the current model (actually used learning model 215), the current model's vote of 18% is displayed as the percentage of the current model's counter relative to the total number of counters for each of the models (current model, model 1, model 2, and model 3). Similarly, 70%, 10%, and 2% are displayed as the votes for model 1, model 2, and model 3, respectively.

[0044] This process is repeated each time the user selects and touches a thumbnail in the representative image group 265, and the votes corresponding to the learning model change. When the user selects all thumbnails in the representative image group 265, the processing execution unit 270 displays a dialog 4410 on the display screen of the imaging unit 210, as shown in Figure 4(d), to ask the user to confirm that Model 1, which has the most votes, should be set as the learning model 215 for actual use. When the user touches the button 4420 with their finger, the selection unit 280 sets Model 1 as the learning model 215 for actual use.

[0045] Alternatively, the selection unit 280 may set the learning model with the most votes as the actual-use learning model 215 without displaying the dialog box 4410. Furthermore, the processing execution unit 270 may calculate evaluation values ​​for each learning model in the candidate learning model group 290 for one or more representative images in the representative image group 265, and the selection unit 280 may set the learning model with the highest evaluation value as the actual-use learning model 215. For example, the likelihood of the subject being depicted may be used as the evaluation value.

[0046] Thus, according to this embodiment, it is possible to extract images suitable for evaluating the learning model used for tracking AF from a natural series of shooting operations, making it easy to evaluate the learning model.

[0047] In this embodiment, we have described a case in which tracking AF continuation information and tracking AF deactivation information are associated with the captured image. However, tracking AF continuation information and tracking AF deactivation information may also be associated with the metadata of the captured image.

[0048] Furthermore, the various operations described above are merely examples and are not limited to any specific method. For example, instead of, or in addition to, touch operations on the screen, users may also press buttons or other controls.

[0049] Furthermore, although this embodiment describes a case where the imaging unit 210 and the information processing device 1 are separate devices, the imaging unit 210 and the information processing device 1 may be integrated into a single information processing device. In this case, the information processing device operates to evaluate and modify the learning model used for tracking AF based on the captured images it has taken.

[0050] Furthermore, some or all of the processing described as being performed by the information processing device 1 may be performed by other devices (such as a smartphone or a server on the cloud) that are connected to the information processing device 1 via the communication unit 160.

[0051] Furthermore, in this embodiment, the learning model was a model that had learned tracking AF, but the processing that the learning model learns is not limited to a specific processing; it may be a learning model that has learned other processing.

[0052] <Modified form of the first embodiment> The method for collecting representative images using this modified example will be explained with reference to Figure 5. Figure 5 shows the state of tracking autofocus (AF) with a running dog as the "subject to be tracked AF". The leftmost image shows the state at time t, the center image shows the state at time (t+1), and the rightmost image shows the state at time (t+2).

[0053] In Figure 5(a), at all times t, (t+1), and (t+2), the user has pressed the AF button 4110 with their finger, and as a result of tracking AF, the dog is in focus and a bounding box is displayed at the dog's position.

[0054] Suppose the user determines that the inference result from the learning model for the image frame at time (t+2) is suitable for the user's purpose. At this time, the user inputs the result of this determination to the imaging unit 210 by voice. At this time, the selection unit 260 performs voice recognition on the voice spoken by the user, and if the result of the voice recognition is a word that indicates that "the inference result from the learning model is suitable for the user's purpose," such as an explicit word indicating success like "success" or "OK," or a word that implies success that the user unconsciously utters during the shooting operation, such as "nice" or "cute," then the image at time (t+2) is designated as the "image in which tracking AF was successful," and this image is selected as the representative image.

[0055] In Figure 5(b), the user presses the AF button with their finger at all three times: t, (t+1), and (t+2). As a result of tracking AF, at times t and (t+2), the dog is in focus, and a bounding box is displayed at the dog's position. At time (t+1), the user also presses the AF button with their finger, but as a result of tracking AF, the dog is not in focus, and the position and size of the bounding box do not match the position and size of the dog in the image.

[0056] Suppose the user determines that the inference result from the learning model for the image frame at time (t+1) is not suitable for the user's purpose. At this time, the user inputs the result of this determination to the imaging unit 210 by voice. At this time, the selection unit 260 performs voice recognition on the voice spoken by the user, and if the result of the voice recognition is a word that indicates that "the inference result from the learning model is not suitable for the user's purpose," such as an explicit word indicating failure, such as "failure" or "NG," or a word that suggests failure, such as "oh" or "no good," which the user may unconsciously utter during the shooting operation, the selection unit 260 designates the image at time (t+1) as "the image where tracking AF failed" and selects this image as the representative image.

[0057] [Second Embodiment] In this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, it will be assumed to be the same as the first embodiment. The operation of the system according to this embodiment will be described according to the flowchart in Figure 6. In Figure 6, processing steps similar to those shown in Figure 3 are given the same step numbers, and the explanation of these processing steps will be omitted.

[0058] In step S63, the imaging unit 210 outputs an image of a frame (captured image) captured by the imaging unit 210 in response to a shooting instruction input by the user by operating the imaging unit 210 or the information processing device 1, associating it with an image of a frame (related image) included in a specified period before and after the timing at which the shooting instruction was input. "An image of a frame included in a specified period before and after the timing at which the shooting instruction was input" refers, for example, to an image of a frame included in the period between 1.5 seconds before and after the timing at which the shooting instruction was input.

[0059] In step S64-1, the acquisition unit 240 determines whether the images of the frames captured by the imaging unit 210 so far are images that have been displayed on the imaging unit 210 and viewed by the user. If, as a result of this determination, the images of the frames captured by the imaging unit 210 so far are images that have been viewed, the process proceeds to step S64-2. On the other hand, if the images of the frames captured by the imaging unit 210 so far are not images that have been viewed, the process proceeds to step S63.

[0060] In step S64-2, the acquisition unit 240 determines whether the cumulative time spent viewing a viewed image is equal to or greater than a threshold (e.g., 5 minutes). If, as a result of this determination, the cumulative time spent viewing a viewed image is equal to or greater than the threshold, it is determined that the user has shown interest in the viewed image, and in that case, the viewed image is considered a "favorite image". Therefore, the image acquisition unit 220 associates the viewed image with "image viewing information" as "operation information representing user operations on an image obtained through capture". The process then proceeds to step S65-1. On the other hand, if the cumulative time spent viewing a viewed image is less than the threshold, the process proceeds to step S64-3.

[0061] In step S64-3, the acquisition unit 240 determines whether the image captured by the imaging unit 210 has been deleted. If, as a result of this determination, the image captured by the imaging unit 210 has been deleted, it is determined that the image is not the image intended by the user, and the image is considered a "failed capture image". Therefore, the image acquisition unit 220 associates "image deletion information" with the image as "operation information representing user operation on the image obtained through capture". The process then proceeds to step S65-1. On the other hand, if the image captured by the imaging unit 210 has not been deleted, the process proceeds to step S36.

[0062] In step S65-1, the acquisition unit 240 selects an image associated with "image viewing information" or "image capturing information" as the target image, and determines whether the target image is an image of a frame within a specified time period (e.g., 1 hour) from the captured image output in association with the target image. This "specified time" corresponds, for example, to the "specified period" mentioned above.

[0063] As a result of this determination, if the target image is an image of a frame within a specified time period from the captured image output in association with the target image, the process proceeds to step S65-3. On the other hand, if the target image is not an image of a frame within a specified time period from the captured image output in association with the target image, the process proceeds to step S65-2.

[0064] In step S65-2, the selection unit 260 selects the captured image output in association with the target image as the representative image. In step S65-3, the selection unit 260 selects the captured image output in association with the target image and the related images output in association with the said captured image (excluding deleted related images) as the representative images.

[0065] Thus, according to this embodiment, images suitable for evaluating the learning model used for tracking AF can be extracted from the natural operation of the imaging unit 210, making it easy to evaluate the learning model.

[0066] Furthermore, the method for determining whether an image is a favorite image is not limited to a specific method. For example, you could use the RATING function assigned after shooting and select images with high RATING scores as favorite images.

[0067] Furthermore, the method for determining whether an image is a failed shot is not limited to a specific method. For example, the RATING function assigned after shooting could be used to select images with a low RATING as failed shots.

[0068] <Modified form of the second embodiment> A block diagram of Figure 7 shows an example of the functional configuration of the system according to this embodiment. In Figure 7, functional units similar to those shown in Figure 2 are given the same reference numbers, and the explanation of these functional units is omitted.

[0069] In step S37, the acquisition unit 710 acquires correct information indicating the area of ​​the subject in each representative image in the representative image group 265. The method for acquiring correct information in the representative images is not limited to a specific method.

[0070] For example, as shown in Figure 8, the acquisition unit 710 displays a representative image on the display screen 820 of the imaging unit 210. The user specifies the area (bounding box) 810 of the subject in the representative image displayed on the display screen 820. The method of specifying the area is not limited to a specific method. For example, if the display screen of the imaging unit 210 is a touch panel screen, the user may specify a frame surrounding the area of ​​the subject by touching the touch panel screen. In that case, the acquisition unit 710 acquires information defining the frame (for example, the coordinates of the four corners of the frame, the coordinates of two opposing vertices, the coordinates of one vertex, and the height and width of the frame) as correct information.

[0071] Then, similar to the first embodiment, the processing execution unit 270 inputs the representative images included in the representative image group 265 into each of the learning models in the candidate learning model group 290 and performs calculations on the learning models to obtain the results of detecting the region of the subject by the learning models as the inference result of subject detection.

[0072] In step S38, the selection unit 280 calculates the Intersection over Union (IoU) for each learning model in the candidate learning model group 290 between the region of the subject detected by the learning model from each representative image in the representative image group 265 and the region represented by the ground truth information of that representative image. This allows the selection of the IoU for each representative image in the representative image group 265 for each learning model in the candidate learning model group 290.

[0073] The selection unit 280 then calculates the average IoU for each representative image in the representative image group 265 for each learning model in the candidate learning model group 290, and extracts a specified number of learning models from the candidate learning model group 290 in descending order of average value. Note that the method for extracting multiple learning models with higher average values ​​is not limited to a specific method.

[0074] Then, in step S38, the processing execution unit 270 presents the user with the inference results of the learning models extracted from the candidate learning model group 290 in step S37. The process thereafter is the same as in the first embodiment.

[0075] Furthermore, if the number of models extracted based on the average IoU obtained in step S38 is equal to the number of actual-use learning models 215, the selection unit 280 may change the actual-use learning models 215 by displaying a dialog box 4410 and accepting only input for the change button 4420, without accepting votes from the user, or it may change the actual-use learning models 215 automatically without displaying the dialog box 4410.

[0076] The numerical values, processing timing, processing order, processing entity, data (information) structure / acquisition method / destination / source / storage location, etc., used in the above embodiment are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0077] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0078] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0079] The inventions described herein include the following information processing devices, information processing methods, and computer programs. (Item 1) A selection means that selects a representative image from the captured image, according to the user operation performed by the user during the capture, using an imaging unit that processes the captured image using a learning model. The selection means evaluates multiple learning models using the representative image selected by the selection means, and the changing means modifies the learning model used by the imaging unit based on the results of the evaluation. An information processing device characterized by comprising: (Item 2) The information processing apparatus according to item 1, characterized in that the selection means selects an image of a frame being captured as a representative image if the elapsed time from the end of the previous processing to the start of the currently executing processing is less than or equal to a threshold. (Item 3) The information processing apparatus according to item 1 or 2, characterized in that the selection means selects the image of the frame being captured as a representative image if the elapsed time from the end of the previous processing to the start of the currently executing processing is less than or equal to a threshold, and if the time elapsed from the start of the currently executing processing is greater than or equal to a threshold. (Item 4) The information processing apparatus according to item 1, characterized in that the selection means selects an image of a frame being captured as a representative image in response to an audio input indicating the success or failure of the process. (Item 5) The modification means presents the user with the inference results of each of the multiple learning models for a representative image, and changes the learning model used by the imaging unit to the learning model selected by the user in response to the presentation. An information processing device according to any one of items 1 to 4, characterized by the features described in item 1 to 4. (Item 6) The modification means obtains evaluation values ​​for each of the multiple learning models for a representative image, and changes the learning model used by the imaging unit to the learning model that obtained the highest evaluation value. An information processing device according to any one of items 1 to 4, characterized by the features described in item 1 to 4. (Item 7) The information processing device according to item 1, characterized in that the selection means selects the captured image and the multiple frames as representative images if the cumulative viewing time of any of the multiple frames before and after the captured image is equal to or greater than a threshold, and the frame is within a specified time period from the captured image. (Item 8) The information processing device according to item 1 or 7, wherein the selection means selects the captured image as a representative image if, among multiple frames of images before and after the captured image, the cumulative viewing time of any image is greater than or equal to a threshold, and that image is not a frame within a specified time period from the captured image. (Item 9) The information processing device according to any one of items 1, 7, or 8, characterized in that the selection means selects the captured image and the images of the multiple frames as representative images if the deleted image is an image of a frame within a specified time period from the captured image. (Item 10) The information processing device according to any one of items 1, 7 to 9, characterized in that the selection means selects the captured image as a representative image if the deleted image is not an image from a frame within a specified time period from the captured image. (Item 11) The modification means extracts multiple learning models from each learning model based on the inference result for a representative image of each learning model and the correct information of the subject area in the representative image, evaluates the multiple learning models using the representative image, and modifies the learning model used by the imaging unit based on the results of the evaluation. The information processing device described in item 1, characterized by the features described herein. (Item 12) The information processing device according to any one of items 1 to 11, characterized in that the processing is a process of recognizing a subject in a captured image, focusing on the subject, and tracking it. (Item 13) The information processing device according to any one of items 1 to 12, characterized in that it has the imaging unit. (Item 14) An information processing method performed by an information processing device, The selection means of the information processing device includes a selection step of selecting a representative image from images captured by an imaging unit that processes captured images using a learning model, in accordance with user operations performed by the user during the capture process. The modification means for the information processing device includes a modification step in which a plurality of learning models are evaluated using the representative image selected in the selection step, and the learning model used by the imaging unit is modified based on the results of the evaluation. An information processing method characterized by comprising: (Item 15) A computer program that causes a computer to function as one of the means of an information processing device described in any one of items 1 through 13.

[0080] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0081] 1: Information processing device 210: Imaging unit 215: Actual-use learning model 220: Image acquisition unit 230: Image storage unit 240: Acquisition unit 260: Selection unit 265: Representative image group 270: Processing execution unit 280: Selection unit 290: Candidate learning model group

Claims

1. A selection means that selects a representative image from the captured image, according to the user operation performed by the user during the capture, using an imaging unit that processes the captured image using a learning model. The selection means presents the user with the inference results of each of the multiple learning models for the representative image selected by the selection means, and the changing means changes the learning model used by the imaging unit to the learning model selected by the user in response to the presentation. Equipped with, The aforementioned processing is characterized by recognizing a subject in a captured image, focusing on the subject, and tracking it.

2. The information processing apparatus according to claim 1, characterized in that the selection means selects an image of a frame being captured as a representative image if the elapsed time from the end of the previous processing to the start of the currently executing processing is less than or equal to a threshold.

3. The information processing apparatus according to claim 1, characterized in that the selection means selects an image of a frame being captured as a representative image if the elapsed time from the end of the previous processing to the start of the currently executing processing is less than or equal to a threshold, and if the time elapsed from the start of the currently executing processing is greater than or equal to a threshold.

4. The information processing apparatus according to claim 1, wherein the selection means selects an image of a frame being captured as a representative image in response to an audio input indicating the success or failure of the process.

5. The information processing apparatus according to claim 1, wherein the selection means selects the captured image and the multiple frames as representative images if the cumulative viewing time of any of the multiple frames before and after the captured image is equal to or greater than a threshold, and the frame is within a specified time period from the captured image.

6. The information processing apparatus according to claim 1, wherein the selection means selects the captured image as a representative image if, among multiple frames of images before and after the captured image, the cumulative viewing time of any image is greater than or equal to a threshold, and that image is not a frame within a specified time period from the captured image.

7. The information processing apparatus according to claim 1, wherein the selection means selects the captured image and the images of the multiple frames as representative images if the deleted image is an image of a frame within a specified time period from the captured image.

8. The information processing apparatus according to claim 1, wherein the selection means selects the captured image as a representative image if the deleted image is not a frame within a specified time period from the captured image among multiple frames of images before and after the captured image.

9. The information processing device according to claim 1, characterized in that the information processing device has the imaging unit.

10. An information processing method performed by one or more processors, A selection step involves selecting a representative image from the captured images, according to the user operations performed by the user during the capture, using an imaging unit that processes the captured images using a learning model. A modification step is to present the user with the inference results of each of the multiple learning models for the representative image selected in the selection step, and to change the learning model used by the imaging unit to the learning model selected by the user in response to the presentation. Equipped with, The aforementioned processing method is characterized by recognizing a subject in a captured image, focusing on the subject, and tracking it.

11. A computer program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 9.

Images