Information processing device and information processing method

Abstract

To provide a technology for efficiently selecting a learning model according to the intended purpose.SOLUTION: An information processing device selects a representative image from an image captured by an imaging unit, which processes the captured image using a learning model, based on the user operations performed during the capture. The information processing device evaluates multiple learning models using the selected representative image and, based on the evaluation results, changes the learning model used by the imaging unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for selecting a learning model. [Background technology]

[0002] In recent years, computer vision (CV) tasks using machine learning techniques have been utilized in a variety of situations. Conventional technologies include services that allow users to create machine-learned models (hereinafter referred to as "learned models") according to their own purposes, or select a learning model from multiple learning models published and distributed on the service. For example, Patent Document 1 discloses a method for efficiently selecting a learning model by comparing the learning models using the object detection results of multiple learning models. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-105923 Summary of the Invention [Problem to be solved by the invention]

[0004] However, since users do not have an indication of what kind of images they should use to select a learning model that suits their purpose, they may end up comparing images that are difficult to compare or a large number of images, which can result in inefficient selection of a learning model.The present invention provides a technology for efficiently selecting a learning model according to a purpose. [Means for solving the problem]

[0005] One aspect of the present invention is characterized by comprising a selection means for selecting a representative image from images captured by an imaging unit that processes the captured image using a learning model in accordance with a user operation performed by the user during the capture, and a modification means for evaluating multiple learning models using the representative image selected by the selection means and modifying the learning model used by the imaging unit based on the results of the evaluation. [Effects of the Invention]

[0006] According to the present invention, a learning model can be efficiently selected according to a purpose. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device 1. [Figure 2] 1 is a block diagram showing an example of the functional configuration of a system to which an information processing device 1 is applied. [Figure 3] 1 is a flowchart of the operation of the system. [Figure 4] 3A and 3B are diagrams showing changes in the state of the imaging unit 210. [Figure 5] 3A and 3B are diagrams showing changes in the state of the imaging unit 210. [Figure 6] 1 is a flowchart of the operation of the system. [Figure 7] FIG. 2 is a block diagram showing an example of the functional configuration of the system. [Figure 8] 3A and 3B are diagrams showing changes in the state of the imaging unit 210. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0009] [First embodiment] First, an example of the hardware configuration of an information processing device 1 according to this embodiment will be described using the block diagram of Fig. 1. The information processing device 1 according to this embodiment is a computer device such as a PC, a tablet terminal device, or a smartphone.

[0010] A CPU (Central Processing Unit) 100 executes various processes using computer programs and data stored in a RAM 120. As a result, the CPU 100 controls the overall operation of the information processing device 1 and executes or controls various processes described as processes performed by the information processing device 1.

[0011] ROM (Read-Only-Memory) 110 stores setting data for the information processing device 1, computer programs and data related to the startup of the information processing device 1, computer programs and data related to the basic operation of the information processing device 1, and the like.

[0012] The RAM (Random Access Memory) 120 has an area for storing computer programs and data loaded from the ROM 110 or the HDD (Hard Disk Drive) 130. The RAM 120 also has an area for storing computer programs and data received from an external device via the communication unit 160. The RAM 120 also has a work area used by the CPU 100 when executing various processes. In this way, the RAM 120 can provide various areas as needed.

[0013] The HDD 130 stores an OS (Operating System), computer programs and data for causing the CPU 100 to execute or control various processes described as processes performed by the information processing device 1, and the like.

[0014] An external storage device may be used in addition to or instead of the HDD 130. The external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Known examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories. The external storage device may also be a server device or the like connected to the information processing device 1 via a network.

[0015] The input unit 140 is a user interface such as a keyboard, a mouse, or a touch panel, and allows a user to input various instructions and information to the information processing device 1 by operating it.

[0016] Display unit 150 has a screen such as a liquid crystal display or a touch panel, and displays the results of processing by CPU 100 as images, text, etc. Display unit 150 may also be a projection device such as a projector that projects images and text.

[0017] The communication unit 160 performs data communication with an external device via a network such as a LAN or the Internet. For example, the information processing device 1 may acquire, via the communication unit 160, various instructions and information that a user inputs by operating an external device.

[0018] The CPU 100, ROM 110, RAM 120, HDD 130, input unit 140, display unit 150, and communication unit 160 are all connected to a system bus 170. Note that the hardware configuration applicable to the information processing device 1 is not limited to the configuration shown in FIG. 1 and can be changed / modified as appropriate.

[0019] 2 is a block diagram showing an example of the functional configuration of a system to which such an information processing device 1 is applied. As shown in FIG. 2, the system includes an imaging unit 210 including an actual use learning model 215, and the information processing device 1.

[0020] The actual use learning model 215 is a learning model selected by the information processing device 1 from a candidate learning model group 290 held in the information processing device 1. The candidate learning model group 290 is a collection of learning models that have been trained to detect and track a "subject that is a target of tracking AF (Auto Focus)" from an image. Therefore, the imaging unit 210 uses the actual use learning model 215 to detect a "subject that is a target of tracking AF" from a captured image captured by the imaging unit 210, and performs "tracking AF," which is a process of automatically focusing and tracking the subject based on the position where the subject was detected in the captured image.

[0021] The information processing device 1 selects one or more captured images as selected images from a group of captured images captured by the imaging unit 210 in response to a user operation related to tracking AF. Then, the information processing device 1 evaluates the candidate learning model group 290 using the selected images, and changes the actual use learning model 215 based on the results of the evaluation.

[0022] The operation of the system according to this embodiment will be described with reference to the flowchart in Fig. 3. In the following, a configuration will be described in which, among the functional units of the information processing device 1 shown in Fig. 2, the image storage unit 230 is implemented by the RAM 120 or the HDD 130, and each functional unit other than the image storage unit 230 is implemented by software (computer program). In the following, the functional units (excluding the image storage unit 230) of the information processing device 1 shown in Fig. 2 will be described as the main processing units, but in reality, the functions of these functional units are realized by the CPU 100 executing computer programs corresponding to these functional units. Note that one or more of the functional units other than the image storage unit 230 may be implemented by hardware.

[0023] In step S31, the selection unit 280 acquires a candidate learning model group 290 and stores the acquired candidate learning model group 290 in the HDD 130 or the RAM 120. The method for acquiring the candidate learning model group 290 is not limited to a specific method. For example, the selection unit 280 may download the candidate learning model group 290 stored in an external device (such as a server device, an external storage device, or a cloud) to the RAM 120 or the HDD 130 via the communication unit 160. Such a candidate learning model group 290 may be a learning model created by the user himself or may be a commonly distributed learning model.

[0024] In step S32, the selection unit 280 sets one of the candidate learning models 290 to the imaging unit 210 as the actual use learning model 215. Note that the actual use learning model 215 may be the learning model set in the imaging unit 210 before step S31.

[0025] The captured image captured by the imaging unit 210 is input to the information processing device 1 and stored in the image storage unit 230. The image acquisition unit 220 may acquire the captured image input from the imaging unit 210, or may acquire the captured image stored in the image storage unit 230.

[0026] In step S33, when an instruction to perform tracking AF is input in response to a user operation, the imaging unit 210 starts tracking AF for the captured image, and when an instruction to end tracking AF is input in response to a user operation, the imaging unit 210 ends (cancels) the tracking AF.

[0027] 4(a), the imaging unit 210 displays a captured image on a display screen 4130. The imaging unit 210 performs tracking AF while the user is pressing the AF button 4110 with their finger, and ends tracking AF when the user releases their finger from the AF button 4110. For example, the imaging unit 210 performs so-called "thumb AF," in which tracking AF continues while the AF button 4110 is pressed with the thumb, or "halfway shutter AF," in which tracking AF continues while the shutter button 4120 is halfway pressed. Note that the method for inputting an instruction to perform / end tracking AF is not limited to a specific input method.

[0028] 4(a) shows a situation in which a dog is being photographed as a "subject targeted for tracking AF," and a captured image including the dog is displayed on the display screen 4130. Furthermore, a bounding box (BB) surrounding the dog is displayed on the display screen 4130 as a result of the tracking AF.

[0029] In step S34-2, the acquisition unit 240 determines whether the elapsed time from the end timing of the previously performed tracking AF to the start timing of the currently performed tracking AF is equal to or less than a threshold (for example, within 0.5 seconds).

[0030] If the result of this determination is that the elapsed time is less than or equal to the threshold, that is, if the user inputs an instruction to end tracking AF due to a tracking AF failure or the like during tracking AF, but immediately inputs an instruction to start tracking AF again, i.e., if a so-called "re-engagement of tracking AF" has been performed, the image acquisition unit 220 associates "tracking AF release information" as "operation information representing a user operation on the imaging unit 210" with the image of the current frame (frame being captured) acquired from the imaging unit 210 or the image storage unit 230. Then, the process proceeds to step S34-3. On the other hand, if the elapsed time is greater than the threshold, that is, if a "re-engagement of tracking AF" has not been performed, the process proceeds to step S35.

[0031] 4(b) shows the state of the image capture unit 210 performing tracking AF on a running dog as the "subject targeted for tracking AF." The left side shows the state of the image capture unit 210 at time t, the center shows the state of the image capture unit 210 at time (t+1), and the right side shows the state of the image capture unit 210 at time (t+2).

[0032] At time t, the user presses the AF button 4110 with their finger, and as a result of tracking AF, the dog is in focus and a bounding box is displayed at the position of the dog.

[0033] At time (t+1), the user still presses the AF button 4110 with their finger, but as a result of the tracking AF, the dog is not in focus, and the position and size of the bounding box do not match the position and size of the dog in the image. In this case, the user may determine that the tracking AF has failed and immediately cancel and re-execute the tracking AF during the shooting operation to regain tracking AF on the dog. In such a case, the captured image of the frame at time (t+1) is suitable as an "image in which tracking AF has failed" for evaluating each learning model in the candidate learning model group 290. Therefore, the image acquisition unit 220 associates "tracking AF cancellation information" with the captured image of the frame at time (t+1).

[0034] At time (t+2), after "re-grabbing tracking AF," the dog is in focus as a result of tracking AF, and a bounding box of the appropriate size is displayed in the appropriate position.

[0035] In step S34-3, the acquisition unit 240 determines whether a time equal to or greater than a threshold (e.g., 3.0 seconds) has elapsed since the start of the currently-performing tracking AF. If this determination determines that a time equal to or greater than the threshold has elapsed since the start of the currently-performing tracking AF, tracking AF has been performed accurately during the series of shooting operations, and a frame obtained after long-term tracking AF is suitable as an "image in which tracking AF was successful" for evaluation of the candidate learning model group 290. Therefore, the image acquisition unit 220 associates "tracking AF continuation information" as "operation information representing a user operation on the imaging unit 210" with the image of the current frame acquired from the imaging unit 210 or the image storage unit 230. Then, the process proceeds to step S35. On the other hand, if a time equal to or greater than the threshold has not elapsed since the start of the currently-performing tracking AF, the process proceeds to step S36.

[0036] In step S35, the selection unit 260 selects, from the captured images acquired by the image acquisition unit 220, a captured image to be used for evaluation of the candidate learning model group 290 as a selected image (representative image).

[0037] For example, the selection unit 260 may select, as a representative image, a captured image associated with "tracking AF cancellation information" as operation information from among the captured images acquired by the image acquisition unit 220. Furthermore, for example, the selection unit 260 may select, as a representative image, a captured image associated with "tracking AF continuation information" as operation information from among the captured images acquired by the image acquisition unit 220. The representative image group 265 is a collection of representative images selected by the selection unit 260. The representative image may be a single still image or a moving image including multiple frames of captured images.

[0038] In step S36, the selection unit 260 determines whether the number of representative images included in the representative image group 265 is equal to or greater than a predetermined number that is sufficient for evaluating the candidate learning model group 290.

[0039] If the result of this determination is that the number of representative images included in representative image group 265 is equal to or greater than the specified number, it is determined that the number of representative images is sufficient, and the process proceeds to step S37. On the other hand, if the number of representative images included in representative image group 265 is less than the specified number, it is determined that the number of representative images is insufficient, and the process proceeds to step S33.

[0040] In step S37, the processing execution unit 270 inputs the representative images included in the representative image group 265 into each learning model in the candidate learning model group 290 and performs calculations on the learning models to obtain the subject detection results using the learning models as inference results for subject detection.

[0041] In step S38, the processing execution unit 270 presents the inference result obtained in step S37 to the user. In step S39, the selection unit 280 determines the learning model to be set as the actual use learning model 215, and changes the learning model currently being used as the actual use learning model 215 to the set learning model.

[0042] An example of the processing in steps S37 to S39 will be described. For example, as shown in FIG. 4(c), the processing execution unit 270 displays a list of representative images in the representative image group 265 as thumbnails on the display screen of the imaging unit 210. Here, when the user touches a thumbnail 4310 of one of the representative images in the representative image group 265 with their finger, the processing execution unit 270 inputs the thumbnail 4310 or the captured image corresponding to the thumbnail 4310 into each learning model in the candidate learning model group 290 and performs calculations on the learning model to obtain an inference result. Then, the processing execution unit 270 displays a list of the inference results of each learning model in the candidate learning model group 290 in the display area 4320. The display area 4320 displays the inference results (bounding boxes of the subjects in the captured images corresponding to the thumbnails 4310) for each of the current model (actually used learning model 215), model 1, model 2, and model 3.

[0043] The user then touches with their finger the inference result that best suits their purpose from the inference results of each learning model displayed in the display area 4320. Each time an inference result is touched, the processing execution unit 270 increments the counter corresponding to that inference result and displays the percentage of the counters relative to the total number of counters as the votes. In the case of FIG. 4(c), 18%, which is the percentage of the counters for the current model relative to the total number of counters for the current model (actual use learning model 215), Model 1, Model 2, and Model 3, is displayed as the votes for the current model. Similarly, 70%, 10%, and 2% are displayed as the votes for Model 1, Model 2, and Model 3, respectively.

[0044] This process is repeated each time the user selects and touches a thumbnail in the representative image group 265, and the votes corresponding to the learning model change. When the user selects all thumbnails in the representative image group 265, the process execution unit 270 displays a dialog 4410 on the display screen of the imaging unit 210, as shown in FIG. 4(d), to prompt the user to confirm that model 1 with the most votes is to be set as the actual use learning model 215. When the user touches button 4420 with their finger, the selection unit 280 sets model 1 as the actual use learning model 215.

[0045] Note that the selection unit 280 may set the learning model with the most votes as the actually used learning model 215 without displaying the dialog 4410. Furthermore, the processing execution unit 270 may calculate an evaluation value of each learning model in the candidate learning model group 290 for one or more representative images in the representative image group 265, and the selection unit 280 may set the learning model with the highest calculated evaluation value as the actually used learning model 215. For example, the certainty (likelihood) of the subject may be used as the evaluation value.

[0046] In this way, according to this embodiment, captured images suitable for evaluating a learning model used in tracking AF can be extracted from a natural series of shooting actions, making it possible to easily evaluate the learning model.

[0047] In this embodiment, the case where the tracking AF continuation information and tracking AF cancellation information are associated with the captured image has been described, but the tracking AF continuation information and tracking AF cancellation information may also be associated with metadata of the captured image.

[0048] The above-described various operation methods by the user are merely examples, and are not limited to specific operation methods. For example, pressing a button or the like may be performed instead of or in addition to a touch operation on the screen.

[0049] Furthermore, in the present embodiment, the case where the imaging unit 210 and the information processing device 1 are separate devices has been described, but the imaging unit 210 and the information processing device 1 may be integrated into one information processing device. In this case, the information processing device operates to evaluate and change the learning model used for tracking AF based on the captured image that it has captured.

[0050] In addition, some or all of the processing described as processing performed by information processing device 1 may be executed by another device (such as a smartphone or a server on the cloud) that is communicatively connected to information processing device 1 via communication unit 160.

[0051] Furthermore, in this embodiment, the learning model is a model that has learned tracking AF, but the process that the learning model learns is not limited to a specific process, and the learning model may be a model that has learned other processes.

[0052] <Modification of the first embodiment> A representative image collection method according to this modified example will be described with reference to Fig. 5. Fig. 5 shows the state in which tracking AF is being performed on a running dog as the "subject to be tracked by AF." The left side shows the state at time t, the center shows the state at time (t+1), and the right side shows the state at time (t+2).

[0053] In Figure 5(a), at times t, (t+1), and (t+2), the user presses the AF button 4110 with their finger, and as a result of the tracking AF, the dog is in focus and a bounding box is displayed at the position of the dog.

[0054] Here, suppose that the user determines that the inference result based on the learning model for the image of the frame at time (t+2) is an inference result that suits the user's purpose. At this time, the user inputs the result of this determination by voice to the imaging unit 210. At this time, the selection unit 260 performs voice recognition on the voice uttered by the user, and if the result of the voice recognition is a word that indicates that "the inference result based on the learning model is an inference result that suits the user's purpose," such as a word that explicitly indicates success, such as "success" or "OK," or a word that implies success that the user unconsciously utters during the shooting operation, such as "good" or "cute," the selection unit 260 determines the image at time (t+2) as an "image in which tracking AF was successful," and selects this image as the representative image.

[0055] In Figure 5(b), at times t, (t+1), and (t+2), the user presses the AF button with their finger. As a result of tracking AF, the dog is in focus at times t and (t+2), and a bounding box is displayed at the dog's position. At time (t+1), the user also presses the AF button with their finger, but as a result of tracking AF, the dog is not in focus, and the position and size of the bounding box do not match the position and size of the dog in the image.

[0056] Here, suppose that the user determines that the inference result based on the learning model for the image of the frame at time (t+1) is not suitable for the user's purpose. At this time, the user inputs the result of this determination by voice to the imaging unit 210. At this time, the selection unit 260 performs voice recognition on the voice uttered by the user, and if the result of the voice recognition is a word that explicitly indicates failure, such as "failure" or "NG," or a word that implies failure that the user unconsciously utters during the shooting operation, such as "ah" or "no good," which indicates that "the inference result based on the learning model is not suitable for the user's purpose," the selection unit 260 determines the image at time (t+1) as an "image for which tracking AF failed" and selects this image as the representative image.

[0057] [Second embodiment] In this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. The operation of the system according to this embodiment will be described with reference to the flowchart in Fig. 6. In Fig. 6, processing steps that are the same as those shown in Fig. 3 are assigned the same step numbers as those processing steps, and descriptions of those processing steps will be omitted.

[0058] In step S63, the imaging unit 210 outputs a frame image (captured image) captured by the imaging unit 210 in response to a capture instruction input by the user operating the imaging unit 210 or the information processing device 1, in association with a frame image (related image) included in a specified period before and after the timing at which the capture instruction was input. The "frame image included in a specified period before and after the timing at which the capture instruction was input" is, for example, a frame image included in a period of 1.5 seconds before and after the timing at which the capture instruction was input.

[0059] In step S64-1, the acquisition section 240 determines whether or not the frame images captured by the imaging section 210 so far have been displayed on the imaging section 210 and viewed by the user. If the result of this determination is that the frame images captured by the imaging section 210 so far have been viewed, the process proceeds to step S64-2. On the other hand, if the frame images captured by the imaging section 210 so far have not been viewed, the process proceeds to step S63.

[0060] In step S64-2, the acquisition unit 240 determines whether the cumulative time over which the viewed images have been viewed is equal to or greater than a threshold value (e.g., 5 minutes). If the result of this determination is that the cumulative time over which the viewed images have been viewed is equal to or greater than the threshold value, it is determined that the user is interested in the viewed images, and in this case, the viewed images are considered to be "favorite images." Therefore, the image acquisition unit 220 associates "image viewing information" with the viewed images as "operation information representing user operations on images obtained by shooting." Then, the process proceeds to step S65-1. On the other hand, if the cumulative time over which the viewed images have been viewed is less than the threshold value, the process proceeds to step S64-3.

[0061] In step S64-3, the acquisition unit 240 determines whether the image captured by the imaging unit 210 has been deleted. If this determination shows that the image captured by the imaging unit 210 has been deleted, it is determined that the image is not the image intended by the user, and the image is considered to be an "image that failed to be captured." Therefore, the image acquisition unit 220 associates "image deletion information" with the image as "operation information representing a user operation on an image obtained by capturing." Then, the process proceeds to step S65-1. On the other hand, if the image captured by the imaging unit 210 has not been deleted, the process proceeds to step S36.

[0062] In step S65-1, the acquisition unit 240 determines whether an image associated with "image viewing information" or "image capture information" is a target image, and determines whether the target image is a frame image within a specified time (e.g., one hour) from the captured image output in association with the target image. This "specified time" corresponds, for example, to the above-mentioned "specified period."

[0063] If the result of this determination is that the target image is a frame image within the specified time period from the captured image output in association with the target image, the process proceeds to step S65-3. On the other hand, if the target image is not a frame image within the specified time period from the captured image output in association with the target image, the process proceeds to step S65-2.

[0064] In step S65-2, the selection unit 260 selects the captured image output in association with the target image as the representative image. In step S65-3, the selection unit 260 selects the captured image output in association with the target image and the related images output in association with the captured image (however, deleted related images are not included) as the representative images.

[0065] As described above, according to this embodiment, images suitable for evaluating a learning model used in tracking AF can be extracted from natural operations of the imaging unit 210, making it possible to easily evaluate the learning model.

[0066] The method for determining whether an image is a favorite image is not limited to a specific method, and for example, an image with a high rating may be selected as a favorite image using a rating function that is assigned after shooting.

[0067] Furthermore, the method for determining whether an image is an image that has failed to be captured is not limited to a specific method, and for example, an image with a low rating may be selected as an image that has failed to be captured using a rating function that is assigned after capture.

[0068] <Modification of the second embodiment> An example of the functional configuration of the system according to this embodiment is shown in the block diagram of Fig. 7. In Fig. 7, functional units that are the same as those shown in Fig. 2 are given the same reference numerals as those functional units, and descriptions of those functional units will be omitted.

[0069] In step S37, the acquisition unit 710 acquires correct answer information indicating the area of ​​the subject in each representative image in the representative image group 265. The method for acquiring correct answer information in the representative image is not limited to a specific method.

[0070] For example, as shown in FIG. 8 , the acquisition unit 710 displays a representative image on a display screen 820 of the imaging unit 210. The user specifies a region (bounding box) 810 of the subject in the representative image displayed on the display screen 820. The method for specifying the region is not limited to a specific method. For example, if the display screen of the imaging unit 210 is a touch panel screen, the user may specify a frame surrounding the region of the subject by a touch operation on the touch panel screen. In this case, the acquisition unit 710 acquires information defining the frame (for example, the coordinates of the four corners of the frame, the coordinates of two opposing vertices, the coordinate of one vertex, and the height and width of the frame) as correct answer information.

[0071] Then, as in the first embodiment, the processing execution unit 270 inputs the representative images included in the representative image group 265 into each learning model in the candidate learning model group 290 and performs calculations on the learning models, thereby obtaining the detection results of the subject area using the learning models as the inference results for subject detection.

[0072] In step S38, the selection unit 280 calculates, for each learning model in the candidate learning model group 290, the IoU (Intersection over Union) between the area of ​​the subject detected by the learning model from each representative image in the representative image group 265 and the area represented by the correct answer information of the representative image. This makes it possible to calculate, for each learning model in the candidate learning model group 290, the IoU for each representative image in the representative image group 265.

[0073] Then, the selection unit 280 calculates the average value of IoU for each representative image in the representative image group 265 for each learning model in the candidate learning model group 290, and extracts a predetermined number of learning models in descending order of average value from the candidate learning model group 290. Note that the method for extracting multiple learning models with higher average values ​​is not limited to a specific method.

[0074] Then, in step S38, the processing execution unit 270 presents to the user the inference results of the learning model extracted in step S37 from the candidate learning model group 290. The subsequent steps are the same as those in the first embodiment.

[0075] In addition, when the model extracted based on the average IoU value calculated in step S38 is equal to the number of actual use learning models 215, the selection unit 280 may change the actual use learning model 215 without accepting votes from the user, by displaying a dialog 4410 and accepting only input to the change button 4420, or may change the actual use learning model 215 automatically without displaying the dialog 4410.

[0076] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in the above embodiment are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0077] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0078] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0079] The invention of this specification includes the following information processing device, information processing method, and computer program. (Item 1) a selection means for selecting a representative image from images captured by an imaging unit that processes the captured images using a learning model, in accordance with a user operation performed by a user during the capture; a change means for evaluating a plurality of learning models using the representative image selected by the selection means and changing the learning model used by the imaging unit based on the evaluation result; An information processing device comprising: (Item 2) The information processing device according to item 1, characterized in that the selection means selects the image of the frame being captured as the representative image if the elapsed time from the end timing of the previous processing to the start timing of the processing currently being executed is less than or equal to a threshold value. (Item 3) 3. The information processing device according to item 1 or 2, characterized in that the selection means selects an image of the frame being captured as a representative image if the elapsed time from the end timing of the previous processing to the start timing of the processing currently being executed is less than or equal to a threshold value and if the elapsed time from the start timing of the processing currently being executed is more than or equal to the threshold value. (Item 4) 2. The information processing device according to item 1, wherein the selection means selects an image of a frame being captured as a representative image in response to input of a voice indicating success or failure of the processing. (Item 5) The change means presents to a user inference results of the plurality of learning models for the representative image, and changes the learning model used by the imaging unit to a learning model selected by the user in response to the presentation. 5. The information processing device according to any one of items 1 to 4. (Item 6) The change means calculates an evaluation value for each of the plurality of learning models for the representative image, and changes the learning model used by the imaging unit to the learning model with the highest evaluation value. 5. The information processing device according to any one of items 1 to 4. (Item 7) The information processing device described in item 1 is characterized in that the selection means selects the captured image and the images of the multiple frames as representative images if the images of the multiple frames before and after the captured image have a cumulative viewing time that is greater than or equal to a threshold and are images of frames within a specified time period from the captured image. (Item 8) The information processing device described in item 1 or 7 is characterized in that the selection means selects the captured image as a representative image if an image of multiple frames before and after the captured image whose cumulative viewing time is greater than or equal to a threshold is not an image of a frame within a specified time period from the captured image. (Item 9) The information processing device described in any one of items 1, 7, and 8 is characterized in that, if the deleted image among multiple frame images before and after the captured image is a frame image within a specified time period from the captured image, the selected image and the multiple frame images are selected as representative images. (Item 10) The information processing device described in any one of items 1, 7 to 9 is characterized in that the selection means selects the captured image as a representative image if the deleted image among multiple frame images before and after the captured image is not an image of a frame within a specified time period from the captured image. (Item 11) The change means extracts a plurality of learning models from the respective learning models based on an inference result for the representative image of each learning model and correct answer information for the area of ​​the subject in the representative image, evaluates the plurality of learning models using the representative image, and changes the learning model used by the imaging unit based on the result of the evaluation. 2. The information processing device according to item 1, (Item 12) 12. The information processing device according to any one of items 1 to 11, wherein the processing is processing for recognizing a subject in a captured image, focusing on the subject, and tracking the subject. (Item 13) 13. The information processing device according to any one of items 1 to 12, wherein the information processing device has the imaging unit. (Item 14) An information processing method performed by an information processing device, a selection step in which a selection means of the information processing device selects a representative image from images captured by an imaging unit that performs processing on the captured images using a learning model, in accordance with a user operation performed by a user during the capture; a change step in which a change means of the information processing device evaluates a plurality of learning models using the representative image selected in the selection step, and changes the learning model used by the imaging unit based on the evaluation result; An information processing method comprising: (Item 15) A computer program for causing a computer to function as each means of the information processing device according to any one of items 1 to 13.

[0080] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0081] 1: Information processing device 210: Imaging unit 215: Actual use learning model 220: Image acquisition unit 230: Image storage unit 240: Acquisition unit 260: Selection unit 265: Representative image group 270: Processing execution unit 280: Selection unit 290: Candidate learning model group

Claims

1. a selection means for selecting a representative image from images captured by an imaging unit that processes the captured images using a learning model, in accordance with a user operation performed by a user during the capture; a change means for evaluating a plurality of learning models using the representative image selected by the selection means and changing the learning model used by the imaging unit based on the evaluation result; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the selection means selects an image of a frame being captured as a representative image if the elapsed time from the end timing of the previous process to the start timing of the process currently being executed is equal to or less than a threshold value.

3. The information processing device according to claim 1, characterized in that the selection means selects an image of the frame being captured as a representative image if the elapsed time from the end timing of the previous processing to the start timing of the processing currently being executed is less than a threshold value and if a time greater than or equal to the threshold value has elapsed since the start timing of the processing currently being executed.

4. 2. The information processing apparatus according to claim 1, wherein the selection means selects an image of a frame being photographed as a representative image in response to a voice input indicating success or failure of the processing.

5. The change means presents to a user inference results of the plurality of learning models for the representative image, and changes the learning model used by the imaging unit to a learning model selected by the user in response to the presentation.

2. The information processing apparatus according to claim 1, wherein:

6. The change means calculates an evaluation value for each of the plurality of learning models for the representative image, and changes the learning model used by the imaging unit to the learning model with the highest evaluation value.

2. The information processing apparatus according to claim 1, wherein:

7. The information processing device according to claim 1, characterized in that the selection means selects the captured image and the images of the multiple frames before and after the captured image as representative images if the images whose cumulative viewing time is greater than or equal to a threshold are images of frames within a specified time period from the captured image.

8. The information processing device according to claim 1, characterized in that the selection means selects the captured image as a representative image if an image among multiple frame images before and after the captured image whose cumulative viewing time is greater than or equal to a threshold is not an image of a frame within a specified time period from the captured image.

9. The information processing device according to claim 1, characterized in that the selection means selects the captured image and the images of the multiple frames as representative images if the deleted image is a frame image within a specified time period from the captured image among the images of multiple frames before and after the captured image.

10. 2. The information processing device according to claim 1, wherein the selection means selects the captured image as the representative image if the deleted image among the multiple frame images before and after the captured image is not a frame image within a specified time period from the captured image.

11. The change means extracts a plurality of learning models from the respective learning models based on an inference result for the representative image of each learning model and correct answer information for the area of ​​the subject in the representative image, evaluates the plurality of learning models using the representative image, and changes the learning model used by the imaging unit based on the result of the evaluation.

2. The information processing apparatus according to claim 1, wherein:

12. 2. The information processing apparatus according to claim 1, wherein the processing is processing for recognizing a subject in a photographed image, focusing on the subject, and tracking the subject.

13. The information processing apparatus according to claim 1 , further comprising the imaging unit.

14. An information processing method performed by an information processing device, a selection step in which a selection means of the information processing device selects a representative image from images captured by an imaging unit that performs processing on the captured images using a learning model, in accordance with a user operation performed by a user during the capture; a change step in which a change means of the information processing device evaluates a plurality of learning models using the representative image selected in the selection step, and changes the learning model used by the imaging unit based on the evaluation result; An information processing method comprising:

15. A computer program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 13.

Images