Information processing device, terminal, information processing method, and program

Abstract

Changing the detection target placed a significant burden on the user. [Solution] The information processing device is an information processing device that identifies a model from a plurality of machine learning models, and comprises: acquisition means for acquiring detection results of a target to be detected by the model; relation calculation means for calculating the relation between the plurality of models based on at least one of the target to be detected and the detection results; selection means for selecting a first model from the plurality of models; and identification means for identifying a second model related to the first model based on the relation.

Description

[[Technical Field]]

[0001] The present invention relates to an information processing device, a terminal, an information processing method, and a program. [[Background Art]]

[0002] In recent years, the utilization of AI (Artificial Intelligence) that assists or replaces human work has become widespread. AI operates via a machine learning model trained on a large amount of target data. Machine learning models are shared in environments prepared by companies or organizations, or in open environments such as shared websites on the web, and activities such as users downloading and using models have become common. Machine learning models are no longer limited to those produced by specific companies or organizations, and the number of cases where such models are created by general users has also increased.

[0003] However, with the increase in the number of machine learning models, it has become difficult for users to understand which model is suitable for their own purposes and to use an appropriate model. Accordingly, Patent Document 1 discloses a technique for proposing a model that matches the purpose acquired from a user from among a plurality of models. [[Prior Art Documents]] [[Patent Documents]]

[0004] [[Patent Document 1]] Japanese Patent No. 7068745 [[Summary of the Invention]] [[Problems to be Solved by the Invention]]

[0005] However, in the technology described in Patent Document 1, the user must search for and retrieve a model for each purpose. For example, consider a case where the model detects an object mounted on the camera. The user must search for and retrieve a model for each target object. Therefore, if the user wanted to detect a different object during shooting, they had to search for and retrieve a model again. This resulted in a significant burden on the user when performing object detection in an environment where multiple models could be used. [Means for solving the problem]

[0006] To solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is, An information processing device that identifies a model from multiple machine learning models, A means for obtaining detection results of the target to be detected by the model, A correlation calculation means for calculating the correlation between the plurality of models based on at least one of the detection target and the detection result, A selection means for selecting a first model from the aforementioned multiple models, A means for identifying a second model related to the first model based on the degree of relevance, It is equipped with. [Effects of the Invention]

[0007] The present invention provides a technology that can reduce the burden on users when performing object detection in an environment where multiple models can be used. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example of the overall configuration of the information processing system according to the first embodiment. [Figure 2] A diagram showing an example of the functional configuration of an information processing device and terminal according to the first embodiment. [Figure 3] A flowchart of the relevance calculation process performed by the information processing device in the first embodiment. [Figure 4]A diagram illustrating the specific method for calculating the degree of relevance by the degree of relevance calculation unit in the first embodiment. [Figure 5] This figure shows an example of a management table for the degree of relevance between models calculated by the degree of relevance calculation unit in the first embodiment. [Figure 6] A flowchart of the model identification process performed by the information processing device in the first embodiment. [Figure 7] A flowchart of the main subject determination process performed by the terminal in the first embodiment. [Figure 8] A diagram illustrating the calculation of the degree of relevance of the model in a modified example 2 of the first embodiment. [Figure 9] A diagram showing an example of the relationships between the names of the subjects used in Modification 3 of the First Embodiment. [Figure 10] A diagram illustrating a false detection in modification 5 of the first embodiment. [Figure 11] This figure shows the setting screen, detection screen, and decision screen for selecting the main subject in Modification 5 of the first embodiment. [Figure 12] A flowchart illustrating the subject determination process using a model executed by the terminal in Modification 5 of the First Embodiment. [Figure 13] A block diagram showing the functional configuration of the information processing device 100 of the modified example 6 of the first embodiment. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] <First Embodiment> [Example System Configuration] FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system according to an embodiment. As shown in FIG. 1, the information processing system includes an information processing apparatus 100 and a terminal 110. The information processing apparatus 100 and the terminal 110 are connected to be capable of transmitting and receiving information to and from each other via a network 120.

[0011] The information processing apparatus 100 may be a computer. The information processing apparatus 100 includes a CPU 101, a ROM 102, a RAM 103, an HDD 104, and a communication unit 105.

[0012] CPU 101 is an abbreviation for Central Processing Unit, and is a type of processor. The CPU 101 implements various functions and executes various processes by reading a computer program such as a control program (hereinafter also referred to as a program) stored in the ROM 102 and the HDD 104 and developing the program in the RAM 103.

[0013] ROM 102 is an abbreviation for Read Only Memory, and is a non-volatile memory. The ROM 102 stores, for example, a basic program such as a BIOS (Basic Input Output System).

[0014] RAM 103 is an abbreviation for Random Access Memory, and is a memory capable of reading and writing data at high speed. The RAM 103 is used as a main memory of the CPU 101 and a temporary storage area such as a work area.

[0015] HDD 104 is an abbreviation for Hard Disk Drive, and is a non-volatile storage device capable of storing large-capacity data. The HDD 104 stores various types of data, various programs, and the like. The information processing apparatus 100 may include another storage device such as an SSD (Solid State Drive) instead of or in addition to the HDD 104.

[0016] The communication unit 105 is configured to communicate with the terminal 110 via the network 120. The communication unit 105 may be a communication interface connected to a LAN (Local Area Network) or a WAN (Wide Area Network).

[0017] The information processing device 100 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of or in addition to the CPU 101. Furthermore, the information processing device 100 may have multiple processors of the same type, each performing a different function.

[0018] Some or all of the functions of the information processing device 100 may be implemented by one or more processors, including the CPU 101, reading a program stored in the ROM 102 or HDD 104, loading it into the RAM 103, and executing it. Alternatively, some or all of the functions of the information processing device 100 may be implemented by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0019] The program may be one that has been loaded into the RAM 103 from an external storage device connected to the information processing device 100.

[0020] Terminal 110 may be a computer. Terminal 110 has a display unit 111, a communication unit 112, an operation unit 113, a CPU 114, a ROM 115, a RAM 116, and an HDD 117.

[0021] The operation of the CPU 114, ROM 115, RAM 116, and HDD 117 is the same as that of the information processing device 100. The HDD 117 of the terminal 110 may be an external storage device that performs a similar role. Here, the external storage device can be, for example, a media (recording medium) and an external storage drive for accessing the media. The media may be, for example, a flexible disk (FD), CD-ROM, DVD, USB memory, MO, flash memory, etc. Alternatively, the external storage device may be a server device connected via the network 120.

[0022] The display unit 111 may be a display device that outputs images to a display screen such as a CRT display or a liquid crystal display. The display unit 111 may also be an external device connected to the terminal 110 by wire or wireless connection.

[0023] The operation unit 113 has input devices such as a touch-enabled touch panel and touchpad, a keyboard, and a mouse. The operation unit 113 accepts various operations from the user via the input devices.

[0024] The communication unit 112 may be a communication interface. The communication unit 112 performs wired or wireless bidirectional communication with other information processing devices, communication equipment, external storage devices, etc., using known communication technologies.

[0025] Here, an example of the information processing device 100 and terminal 110 in this embodiment will be described. The information processing device 100 may be a server connected to the network 120. The terminal 110 may be a camera, smartphone, tablet, PC (Personal Computer), etc. The information processing device 100 and the terminal 110 send and receive programs and data necessary for the application on the terminal 110 via the network 120. In this embodiment, the explanation will be given assuming that the terminal 110 is a camera.

[0026] The camera terminal 110 transmits data such as images, setting information, and operation history to the information processing device 100, which is a server, via the communication unit 112. Based on the information transmitted by the terminal 110, the information processing device 100 provides the data stored in the HDD 104 to the terminal 110 via the communication unit 105. The information processing device 100 manages the images provided by the terminal 110 and the machine learning model (hereinafter also referred to as the model or machine learning model) that detects subjects in the images in the HDD 104. The information processing device 100 provides the managed machine learning model to the terminal 110. The terminal 110 saves the machine learning model provided by the information processing device 100 to the HDD 117 and uses it to analyze the captured images. Note that the HDD 104 of the information processing device 100 may be cloud storage connected via the network 120, in addition to storage such as the HDD 104.

[0027] The information processing device 100 can acquire and aggregate information from multiple terminals 110. The information processing device 100 may manage the information received from the terminals 110, including the output of the machine learning model, in association with the machine learning model and manage it on the HDD 104.

[0028] Terminal 110 analyzes the captured image using the provided machine learning model. The analysis performed by the machine learning model of terminal 110 is, for example, detecting a subject from the image as a detection target. The detection target may be a concept that includes the image and the subject in the image. The subject may be, for example, the object to be focused on in autofocus mode. Therefore, the machine learning model may be a model that has been pre-programmed with machine learning to detect a subject from an image. Detection is also called recognition.

[0029] [Functional Block Configuration] Figure 2 is a diagram showing an example of the functional configuration of an information processing device and a terminal according to the first embodiment. Figure 2(a) is a block diagram showing an example of the functional configuration of the information processing device 100. Figure 2(b) is a block diagram showing the functional configuration of the terminal 110.

[0030] As shown in Figure 2(a), the information processing device 100 includes a management unit 201, an acquisition unit 202, a relevance calculation unit 203, a selection unit 204, a specification unit 205, and a presentation unit 206. Some or all of the acquisition unit 202, the relevance calculation unit 203, the selection unit 204, the specification unit 205, and the presentation unit 206 may be implemented as functions of one or more processors, including the CPU 101 that executes the program. Some or all of the acquisition unit 202, the relevance calculation unit 203, the selection unit 204, the specification unit 205, and the presentation unit 206 may be implemented by circuits such as ASICs.

[0031] The management unit 201 may be implemented by the HDD 104 shown in Figure 1. The management unit 201 stores and manages, for example, multiple machine learning models.

[0032] The acquisition unit 202 acquires detection results, which are the outputs of multiple machine learning models. The acquisition unit 202 may acquire the results of detecting subjects in the image as the detection result.

[0033] The correlation calculation unit 203 calculates the correlation between multiple machine learning models based on the detection results obtained by the acquisition unit 202. The specific method for calculating correlation will be described later. The correlation calculation unit 203 may have the management unit 201 manage the correlations calculated for the multiple machine learning models.

[0034] The selection unit 204 selects a model (hereinafter referred to as the first model) from a plurality of machine learning models. The selection unit 204 may accept the selection of the first model based on an operation received from the user by the operation unit 113 of the terminal 110. That is, the first model may be a model explicitly selected by the user of the terminal 110, such as a camera or smartphone, as the target to be used. The selection unit 204 may accept the selection of multiple first models. Furthermore, the selection unit 204 may receive a subject selected by the user in the terminal 110 and select a machine learning model for detecting that subject as the first model from the management unit 201.

[0035] The identification unit 205 identifies a second model that is related to the first model for detecting the subject selected by the user, based on the relevance calculated by the relevance calculation unit 203. The specific method for identifying the second model will be described later. The identification unit 205 may identify multiple second models for a single first model. The identification unit 205 may transmit the identified second models to the terminal 110 via the network 120.

[0036] The presentation unit 206 presents the second model identified by the identification unit 205 to the user. For example, the presentation unit 206 may present the second model by displaying information about the second model on the display unit 111 of the terminal 110. Alternatively, the presentation unit 206 may transmit the second model to the terminal 110 on behalf of the identification unit 205.

[0037] As shown in Figure 2(b), the terminal 110 includes a setting unit 211, an acquisition unit 212, a designation unit 209, and a determination unit 210. Some or all of the setting unit 211, acquisition unit 212, designation unit 209, and determination unit 210 may be implemented as functions of one or more processors, including the CPU 114 that executes the program. Some or all of the setting unit 211, acquisition unit 212, designation unit 209, and determination unit 210 may be implemented by circuits such as ASICs.

[0038] The setting unit 211 receives the user's setting for the subject to be detected with priority and sets the priority subject. The priority subject may be the object to be focused on. The setting unit 211 may receive the user's settings via the display unit 111 and the operation unit 113 in Figure 1.

[0039] The acquisition unit 212 acquires detection results from machine learning models installed in the terminal 110. For example, the acquisition unit 212 acquires detection results for subjects included in the captured image. The acquisition unit 212 may also acquire multiple related machine learning models from the information processing device 100 and acquire detection results from these multiple machine learning models.

[0040] The designation unit 209 receives a range specification from the user and sets a range (also called the designated range) for prioritizing the detection of the subject determined by the determination unit 210.

[0041] The determination unit 210 determines the subject that the terminal 110 will primarily detect (hereinafter also referred to as the primary subject). The primary subject can also be called the primary detection target. The specific method for determining the primary subject will be described later.

[0042] [Processing flow] Figures 3 and 6 are flowcharts showing the processing flow of the information processing device according to this embodiment. The correlation calculation process shown in Figure 3 and the model identification process shown in Figure 6 may be executed at different times in the information processing device 100. The process in step S603 of Figure 6 uses the information (correlation between models) generated in step S304 of Figure 3. Therefore, the execution order is the correlation calculation process shown in Figure 3 and the model identification process shown in Figure 6.

[0043] In the flowcharts from this point forward, the symbol S represents a processing step (processing step).

[0044] Figure 3 is a flowchart of the relevance calculation process performed by the information processing device 100 in the first embodiment. Figure 3 shows the operation of the acquisition unit 202 and the relevance calculation unit 203 of the information processing device 100.

[0045] In S301, the acquisition unit 202 acquires an image. The acquisition unit 202 may acquire an image taken by the terminal 110. For example, the terminal 110 may transmit the captured image to the information processing device 100 via the network 120.

[0046] In S302, the relevance calculation unit 203 detects subjects included in the acquired image using multiple machine learning models. The relevance calculation unit 203 may use known methods as a method for detecting subjects from an image using machine learning models.

[0047] In S303, the relevance calculation unit 203 calculates the relevance of the detected subjects based on the subject detection results.

[0048] In S304, the relevance calculation unit 203 calculates the relevance of the model that detects the subject based on the relevance of the subject.

[0049] Figure 4 illustrates the specific method for calculating relevance by the relevance calculation unit 203 in the first embodiment. Here, we will describe an example in which the relevance calculation unit 203 calculates relevance based on whether or not they were detected simultaneously within the same image.

[0050] The method for calculating the correlation between Model A, which detects Subject A, and Model B, which detects Subject B, by the correlation calculation unit 203 will be explained. It is also assumed that all subjects in the image have been successfully detected. In other words, each model is assumed to have been able to detect the subjects in the image exemplified in Figure 4 without any false positives or failures. Here, we consider the case where there are 3 images taken by a specific user X and 3 images taken by other users. The specific user X may be, for example, a user using the model on terminal 110. The probability that Subject A and Subject B are detected simultaneously in the 3 images taken by user X is 2 / 3. Similarly, the probability that Subject A and Subject B are detected simultaneously in the 3 images taken by other users is 1 / 3. In this case, the correlation CN between Subject A and Subject B is calculated by the following equation (1). CN = α × 2 / 3 + β × 1 / 3 ... (1)

[0051] Here, the coefficients α and β in equation (1) are weighting coefficients that take values ​​between 0 and 1.0, and may be coefficients associated with user X and other users. α and β can also be described as predetermined coefficients that determine which influence, user X's captured images or other users' captured images, is given more weight. When the correlation CN is calculated to match user X, coefficient α is set to be larger than coefficient β so that the influence of user X's captured images is greater. Here, as an example, α = 0.7 and β = 0.3. The correlation calculation unit 203 calculates the correlation CN between subject A and subject B as CN = 0.56. As a result, the correlation calculation unit 203 calculates the correlation CN between model A and model B as CN = 0.56.

[0052] The relevance calculation unit 203 calculates the relevance CN between subject B and subject C as 0.46. The relevance calculation unit 203 calculates the relevance CN between subject A and subject C as 0.43. The relevance calculation unit 203 determines the relevance CN between model B and model C as 0.46, and the relevance CN between model A and model C as 0.43.

[0053] The coefficients α and β may be set by the user. For example, the display unit 111 may display a message such as "Prioritize your own captured images more" on the screen. If the user sees this screen and chooses to prioritize, the relevance calculation unit 203 may increase the value of coefficient α.

[0054] Figure 5 shows an example of a management table for the correlation CN between models calculated by the correlation calculation unit 203 in the first embodiment. The management table is managed by the management unit 201. The management table shows the correlation CN between a specific model and other models.

[0055] In the above, the relevance calculation unit 203 calculated the relevance of the subject and the relevance of the model based on it using both the image taken by a specific user X and the images taken by other users. However, the method of calculating the relevance is not limited to this. The relevance calculation unit 203 may calculate the relevance using only one of the images.

[0056] Figure 6 shows a flowchart of the model identification process performed by the information processing device in the first embodiment. Figure 6 shows the operation of the selection unit 204, the identification unit 205, and the presentation unit 206 in Figure 2.

[0057] In S601, the selection unit 204 acquires information about user operations received by the operation unit 113 of the terminal 110. The user operates to obtain a new machine learning model for subject detection from the information processing device 100. The selection unit 204 may acquire information about the subject selected by the user and the model for detecting that subject as information about the operation.

[0058] In S602, the selection unit 204 selects a first model from a plurality of models based on the acquired operation and information about the user's operation. For example, the selection unit 204 may receive a subject selected by the user and select a model for detecting that subject as the first model from a plurality of models in the management unit 201. Alternatively, the selection unit 204 may receive information about the first model from the terminal 110 and select the first model from a plurality of models in the management unit 201.

[0059] In S603, the identification unit 205 identifies a second model related to the first model selected based on the degree of relevance between machine learning models calculated by the relevance calculation unit 203.

[0060] The identification unit 205 identifies the models selected in order of their relevance to the selected first model as the second model, based on the degree of relevance between the models. For example, the identification unit 205 may identify a predetermined number of models as the second model, in order of their relevance to the first model. In this case, the identification unit 205 may identify the second model from models whose relevance is above a predetermined threshold. The identification unit 205 may also identify the second model as a model that satisfies conditions set in consideration of the processing performance of the CPU 114 and the capacity of the HDD 117 of the terminal 110, and which has a high degree of relevance to the first model. The identification unit 205 may transmit the identified second model to the terminal 110.

[0061] In S604, the display unit 206 presents the second model to the terminal 110 along with the first model. The display unit 206 may transmit the second model to the terminal 110 instead of the identification unit 205. The display unit 111 of the terminal 110 may display the presented second model on the display unit 111.

[0062] The user of terminal 110 may select a model to use from one or more second models related to the first model they selected. Terminal 110 may accept the user's model selection with the second models displayed on the display unit 111. Terminal 110 saves the selected model to the HDD 117. If the user of terminal 110 changes the subject to another related subject, the acquisition unit 202 may select a second model from one or more second models that detects the changed subject and detect the changed subject.

[0063] Figure 7 is a flowchart of the main subject determination process performed by terminal 110 in the first embodiment. Figure 7 shows the operation of terminal 110 using the saved models. Here, it is assumed that terminal 110 has saved model A for detecting subject A and model B for detecting subject B.

[0064] In S701, the setting unit 211 determines whether a priority subject has been set. If the setting unit 211 determines that a priority subject has been set, it proceeds to S702. On the other hand, if the setting unit 211 determines that no priority subject has been set, it proceeds to S703.

[0065] In S702, the acquisition unit 212 uses a machine learning model to detect the designated priority subject and acquires the detection result. For example, if subject A is set as the priority subject, the acquisition unit 212 detects subject A using model A which detects subject A. If subject B is set as the priority subject, the acquisition unit 212 detects subject B using model B which detects subject B.

[0066] In S707, the determination unit 210 determines the main subject based on the detection results acquired by the acquisition unit 212.

[0067] In S703, the acquisition unit 212 detects the subject using multiple models stored in the terminal 110 and acquires the detection result. The multiple models here may be related models that the information processing device 100 has determined to be highly relevant. For example, the acquisition unit 212 detects the subject using both model A and model B and acquires the detection result for both.

[0068] In S704, the designation unit 209 determines whether a range for detecting the main subject (hereinafter also referred to as the designated range) has been specified. The designation unit 209 may receive the designation of the designated range from the user in advance via the operation unit 113. Specifically, the designated range may be the range specified by the user's touch on the touch panel. If the designation unit 209 has received a range designation, for example, it determines that a designated range has been specified and proceeds to S705. On the other hand, if the designation unit 209 has not received a range designation, it determines that a designated range has not been specified and proceeds to S706.

[0069] In S705, the determination unit 210 determines the main subject by prioritizing the detection results of subjects that exist within a specified range from one or more detection results from multiple models.

[0070] In S706, the determination unit 210 determines the main subject by prioritizing the detection result of a subject close to the center of the image from one or more detection results from multiple models. This is based on the assumption that a subject located in the center of the image is a subject of high interest to the user. However, the determination unit 210 may also determine the main subject based on its position within the image, rather than its position in the center. For example, the determination unit 210 may pre-determine a location with a high probability of containing the main subject based on the user's past photography, and then determine the main subject based on that location in the image.

[0071] As described above, the first embodiment calculates the degree of relevance of multiple models based on the detection results of multiple models and identifies a second model that is related to the first model based on the user's selection. As a result, the first embodiment can provide the second model related to the first model to the user's terminal 110. Consequently, even if the user changes the subject, the terminal 110 of the first embodiment can change the model to a related model without requiring any operation by the user to select a model. Therefore, the first embodiment can reduce the burden on the user when the subject is changed. Furthermore, by providing the second model to the terminal 110 in advance, the first embodiment can reduce the download time of the second model when the subject is changed.

[0072] Furthermore, as described above, the first embodiment stores models that detect subjects that are often photographed together and are related to each other in the camera terminal 110, thereby expanding the range of detectable subjects. As a result, even when a user wants to photograph subject B while photographing subject A, the model for detecting subject B can be acquired as a model related to subject A and used on the spot. This reduces the burden of having to acquire a different machine learning model in advance each time the subject changes, even when using various subject detection models tailored to different applications.

[0073] In the first embodiment, terminal 110 detects a preferred subject using a model for detecting that preferred subject, if the user has set a preferred subject. Here, since terminal 110 has acquired related models in advance, even if the preferred subject is changed to a related subject, the model can be changed without acquiring a new model. As a result, the first embodiment can reduce the burden of acquiring models and the burden of model selection by the user.

[0074] In the first embodiment, if a specified range for detection is provided by the user, the terminal 110 determines the subject detected within that range as the primary subject. As a result, the first embodiment can accurately determine the primary subject even when a preferred subject has not been selected by the user and subjects are detected by multiple models.

[0075] In the first embodiment, terminal 110 determines the main subject as the subject detected near the center of the screen where the user's desired main subject is most likely to be located, even if no specified range is given. As a result, the first embodiment can accurately determine the main subject even when a preferred subject has not been selected by the user and subjects have been detected by multiple models.

[0076] <Modification 1 of the first embodiment> Modification 1 of the first embodiment does not explicitly present the identified related model to the user in the identification unit 205. That is, Modification 1 omits the model presentation step in S604 of Figure 6. Specifically, even if the terminal 110 obtains the model (first model) selected by the user and the related model (second model) from the information processing device 100, it does not display them on the display unit 111, and adds the related model without the user's awareness.

[0077] For example, when terminal 110 accepts a selection of a first model for use by the user, the relevance calculation unit 203 calculates the model relevance in S304. The identification unit 205 may identify a second model based on the model relevance and the specifications of terminal 110, which is a camera that uses the model, and transmit it to terminal 110. Terminal 110 adds the received second model. The specifications of terminal 110 include hardware constraints such as the processing performance of the CPU 114 and the capacities of the ROM 115, RAM 116, and HDD 117.

[0078] <Modification 2 of the first embodiment> Figure 8 illustrates the calculation of the relevance of the model in Modification 2. Modification 2 of the first embodiment will be explained using Figure 8. In this embodiment, Exif (Exchangeable image file format) information recorded in the image is used to calculate the relevance between subjects for calculating the relevance of the model.

[0079] Figure 8(a) shows an image of subject A (a-1) and an image of subject B (a-2). The relevance calculation unit 203 analyzes the Exif shooting location information of multiple images (in this case, two images) and calculates the distance D between the shooting locations where the two images were taken. The shooting locations may be locations on a map. If the distance D is less than a predetermined distance threshold, the relevance calculation unit 203 may determine that subjects A and B are subjects that are likely to be photographed in close proximity. Similar to the first embodiment, the relevance calculation unit 203 may calculate the relevance between subjects A and subject B based on the number of images taken with a distance D less than the distance threshold relative to the total number of images. Specifically, the relevance calculation unit 203 may calculate the relevance between subjects based on the ratio of the number of images taken with a distance D less than the distance threshold relative to the total number of images. Note that the relevance calculation unit 203 may also determine whether the distance is less than or equal to a distance threshold, rather than less than the distance threshold.

[0080] Figure 8(b) shows an image of subject A (b-1) and an image of subject B (b-2). The relevance calculation unit 203 analyzes the Exif capture times of multiple images (in this case, two images) and calculates the difference in the time when the two images were taken (hereinafter referred to as the time difference T). If the time difference T is less than a predetermined time threshold, the relevance calculation unit 203 may determine that subjects A and B are subjects that are likely to be photographed at close times. The relevance calculation unit 203 may calculate the relevance between subjects A and subjects B based on the number of images taken with a time difference T less than the time threshold relative to the total number of images. Specifically, the relevance calculation unit 203 may calculate the relevance between subjects based on the ratio of the number of images taken with a time difference T less than the time threshold relative to the total number of images. Note that the relevance calculation unit 203 may also make the determination based on whether or not the time difference is less than the time threshold, rather than whether or not it is less than the time threshold.

[0081] The relevance calculation unit 203 may calculate the relevance based on at least one of the distance D and the time difference T. In other words, the relevance calculation unit 203 may calculate the relevance based on both the distance D and the time difference T.

[0082] Subsequently, the relevance calculation unit 203 may calculate the degree of relevance between model A, which detects subject A, and model B, which detects subject B, in the same manner as in the first embodiment. The relevance calculation unit 203 may also calculate the degree of relevance between models by including all the perspectives of relevance between subjects that have been described so far. Alternatively, the relevance calculation unit 203 may calculate the degree of relevance between models by using only some of the aforementioned perspectives.

[0083] Modification 2 calculates the degree of relevance based on distance D and time difference T, thereby reducing the processing load in the relevance calculation process. Furthermore, by combining Modification 2 with the relevance calculation of the first embodiment, the accuracy of calculating the relevance of the model can be improved.

[0084] <Modification 3 of the first embodiment> A third modification of the first embodiment will be explained using Figure 9. The correlation calculation unit 203 of the third modification calculates the correlation between subjects and the correlation between models based on the correlation of the names of the subjects detected by the model.

[0085] Figure 9 shows an example of the relationships between the names of subjects used in Modification 3 of the First Embodiment. Figure 9 shows an example of classifying the names of subjects (in this case, animals) based on the location where they mainly exist. Dogs and cats are often kept as pets by people or in homes. Elephants and giraffes are often found in zoos. Sea lions and penguins are often found in aquariums. As shown in the figure, the closer the animals are, the higher the relationship. The relationships between names may be managed using a graph structure (also called a tree structure) consisting of edges and nodes, as shown in Figure 9.

[0086] In the example shown in Figure 9, the relevance calculation unit 203 may calculate the relevance of the subjects as follows, for example. For example, the relevance calculation unit 203 may calculate the relevance between subjects based on the number of edges (also called distance) between nodes indicating the names of the subjects.

[0087] Specifically, when calculating the relevance between dogs and cats, the relevance calculation unit 203 starts with dogs and counts the number of edges along the path taken to reach cats. The relevance calculation unit 203 counts the number of edges as 2 because the path from dogs to cats goes through pets. In the case of dogs and elephants, the relevance calculation unit 203 counts the number of edges as 4 because the path from dogs to elephants goes through pets, animals, and zoos. The relevance calculation unit 203 may calculate the reciprocal of the number of edges as the relevance of the subjects. That is, the relevance calculation unit 203 may calculate the relevance between dogs and cats as 1 / 2 = 0.5. On the other hand, the relevance calculation unit 203 may calculate the relevance between dogs and elephants as 1 / 4 = 0.25.

[0088] In the example above, the relevance calculation unit 203 used the reciprocal of the number of edges (=distance) between nodes to calculate the relevance between nodes, but other methods are also acceptable. Here, let Dist be the distance between nodes. In the case of exponential decay, the relevance calculation unit 203 may calculate the relevance based on exp(-Dist). When using a logarithmic scale, the relevance calculation unit 203 may calculate the relevance based on 1 / log(Dist+1). When using the square root of the distance, the relevance calculation unit 203 may calculate the relevance based on 1 / sqrt(Dist).

[0089] The relevance calculation unit 203 may use the calculated relevance of the subjects as the relevance of the model that detects the target subject for each node. The relevance calculation unit 203 may calculate the relevance of the models in the same manner as in the first embodiment. The relevance calculation unit 203 may also calculate the relevance between models by including all the perspectives of relevance between subjects that have been shown so far. Alternatively, the relevance calculation unit 203 may calculate the relevance between models using only some of the perspectives.

[0090] <Modification 4 of the first embodiment> In the modified example 4, the relevance calculation unit 203 calculates the relevance between subjects and between models based on the search results obtained from a web search using a query that includes the name of the subject detected by the model.

[0091] As an example, let's explain how the relevance calculation unit 203 calculates the relevance between subjects such as dogs and cats. The relevance calculation unit 203 generates a search query such as "subjects often photographed together with dogs" and inputs it into a search engine. For example, the relevance calculation unit 203 calculates the relevance as the number of articles (hit rate) that contain the word "cat" among a predetermined number of top results in a web search.

[0092] The relevance calculation unit 203 may use the relevance to determine the relevance between models that detect subjects. The relevance calculation unit 203 may calculate the relevance of models in the same manner as in the first embodiment. The relevance calculation unit 203 may also calculate the relevance between models by including all the perspectives of subject relevance shown so far. Alternatively, the relevance calculation unit 203 may calculate the relevance between models by using only some of these perspectives.

[0093] <Modification 5 of the First Embodiment> A fifth modification of the first embodiment will be described using Figures 10, 11, and 12. In modification 5, the relevance calculation unit 203 calculates the relevance between subjects and the relevance between models based on the rate (or frequency) of false positives, in which the model mistakenly detects a different subject.

[0094] Figure 10 illustrates a false detection in Modification 5 of the first embodiment. The state of false detection will be explained using Figure 10. In Figure 10(a), Model A detects subject A and subject B. Model B detects subject B. In other words, subject B is detected by both Model A and Model B. In this state, the relevance calculation unit 203 may determine that Model A has falsely detected subject B. When multiple models detect the same subject in this way, the relevance calculation unit 203 may determine that one or both models that detect the subject have a high degree of relevance as models that falsely detect the subject.

[0095] Next, we will explain false detection when the subject has ground truth data (GT). Figure 10(b) shows a situation where subject B has been assigned the detection result GT, but model A has falsely detected subject B. The relevance calculation unit 203 may also determine that a false detection occurs when the detection result is incorrect, even when ground truth data has been assigned to the image. In this case, when model A, which detects different subjects, detects subject B with ground truth data, model A is determined to have a high degree of relevance to model B, which detects subject B with ground truth data, as a model that falsely detects.

[0096] The method for calculating the model relevance in Modification 5 will now be explained. The calculation flow is the same as the flow shown in Figure 3 of the first embodiment. In the calculation of the model relevance in S304, the subject relevance was used, but in Modification 5, the relevance calculation unit 203 calculates the model relevance based on the false detection rate of the subject.

[0097] The calculation of the false detection rate of a subject by the relevance calculation unit 203 will be explained. Taking model A, which detects subject A, and model B, which detects subject B, as an example, the relevance calculation unit 203 may determine false detection based on whether or not detection results are obtained from both models for the same subject. The relevance calculation unit 203 may determine the presence or absence of false detection based on the overlap rate (also called the duplication rate) between the detection range of the detection result of model A and the detection range of the detection result of model B. The overlap rate may be calculated using known techniques, such as IoU (Intersection over Union).

[0098] As mentioned above, if a GT (Grade Point) is assigned to the subject as a detection result, the relevance calculation unit 203 may determine whether or not a false detection has occurred based on the overlap rate between the range of the GT frame and the detection range.

[0099] The relevance calculation unit 203 may calculate the false positive rate as the ratio of the number of images determined to be false positives to the total number of images. The relevance calculation unit 203 may use the false positive rate as the degree of relevance between model A and model B.

[0100] The relevance calculation unit 203 may calculate the relevance between models by including all the perspectives of relevance between subjects that have been shown so far. Alternatively, the relevance calculation unit 203 may calculate the relevance between models by using only some of the perspectives.

[0101] Figures 11 and 12 illustrate an example of operation when using the models installed on terminal 110. Figure 11 shows the setting screen, detection screen, and decision screen for selecting the main subject in Modification 5 of the first embodiment. The models referred to here may be models associated based on the false detection rate. Model A, which detects subject A, and Model B, which detects subject B acquired as an associated model of Model A, are installed on terminal 110. The user of terminal 110 has selected subject A as the subject to be detected with priority, as shown in Figure 11(a).

[0102] Figure 12 is a flowchart illustrating the subject determination process using a model executed by terminal 110 in Modification 5 of the first embodiment. In the first embodiment, the procedure for prioritizing the models to be used was explained using Figure 7. In the explanation of Modification 6, it is assumed that Model A is prioritized as the model to be used for detection.

[0103] In S1201, the acquisition unit 212 acquires the subject detection result by Model A.

[0104] In S1202, the acquisition unit 212 acquires the subject detection result by Model B.

[0105] The acquisition unit 212 may select detection results that have a score above a predetermined threshold in the detection of each model. Therefore, the acquisition unit 212 may acquire multiple detection results in each detection. Note that the order of S1201 and S1202 is not limited to this order and may be performed in reverse order or in parallel.

[0106] Figure 11(b) shows the internal state including the detection results of Model A and Model B. In Modification 6, Model A falsely detects Subject B, and therefore has a high degree of association with Model B for detecting Subject B. Thus, Model A is associated with Model B. Because Model A falsely detects Subject B, it may detect both Subject A and Subject B.

[0107] In S1203, the determination unit 210 determines, for each detection result of Model A, whether the detection range of Model A and the detection range of Model B overlap. The determination unit 210 may determine the overlap using known techniques such as IoU, as described above. If the determination unit 210 determines that the detection ranges overlap, it proceeds to S1204. On the other hand, if the determination unit 210 determines that the detection ranges do not overlap, it skips S1204.

[0108] In S1204, the determination unit 210 excludes the detection result of Model A, which has been found to be duplicated, from the detection candidates. This is because if the detection range of Model A and the detection range of Model B overlap, the detection of Model A is likely to be a false positive.

[0109] Subsequently, terminal 110 repeats the processes of S1203 and S1204 for all of the detection results from Model A. After terminal 110 has performed the processes of S1203 and S1204 for all of the detection results from Model A, it proceeds to S1205.

[0110] In S1205, the determination unit 210 selects the detection result with the highest detection score from the remaining detection results of Model A as the AF target. As shown in Figure 11(c), the determination unit 210 may display an AF (Auto Focus) frame around the main subject (in this case, subject A) of the detection result selected as the AF target.

[0111] In addition to the above, a detection frame may be displayed for detection results from Model A that were excluded as detection candidates based on the detection results of Model B. In Figure 11(c), the detection result of Model A, which was falsely detected as subject B, is superimposed. This makes it possible to distinguish from the display that subject B was a detection candidate as a result of Model A's detection, but was suppressed due to the effect of false detection detection.

[0112] As described above, Modification 5 calculates relevance based on a false positive rate that may be higher for subjects that are similar to each other. This allows Modification 5 to improve the accuracy of relevance.

[0113] <Modification 6 of the First Embodiment> Modification 6 of the first embodiment modifies the relevance calculated by the relevance calculation unit 203 based on the usage history of the machine learning model. Modification 6 modifies the relevance based on the user's usage history of the model.

[0114] Figure 13 is a block diagram showing the functional configuration of the information processing device 100 of the modified example 6 of the first embodiment. The information processing device 100 of the modified example 6 includes a history acquisition unit 1401 and a relevance correction unit 1402 in addition to the configuration example in Figure 2(a).

[0115] The history acquisition unit 1401 acquires the history of how machine learning models have been used by users on the terminal 110 (hereinafter also referred to as usage history) via the network 120. The history acquisition unit 1401 may also acquire information as usage history indicating whether or not multiple models were used simultaneously in the same scene. From the perspective of whether a second model related to the first model was used, this refers to whether or not detection results were obtained for both models in the same image.

[0116] The relevance correction unit 1402 corrects the calculated relevance of the models based on the usage history. Here, CN is the relevance of the models before correction, and k is a coefficient. The relevance correction unit 1402 may correct the relevance based on the following equation (2). The coefficient k may be the ratio of the number of images in which detection results for both the first model and the second model were obtained for the same image, relative to the total number of images. CN = C × k ... (2)

[0117] The usage history may be based on criteria other than whether detection results from multiple models were obtained simultaneously in the same image. For example, as explained in Modification 2, it may use Exif information to determine whether detection results were obtained within a predetermined distance or within a predetermined time.

[0118] (Other examples) The embodiments and variations described above may be combined. When combined, the embodiments and variations may be configured so that the user can select any of the embodiments and variations.

[0119] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0120] The disclosures herein include the following information processing devices, terminals, information processing methods, and programs. (Item 1) An information processing device that identifies a model from multiple machine learning models, A means for obtaining detection results of the target to be detected by the model, A correlation calculation means for calculating the correlation between the plurality of models based on at least one of the detection target and the detection result, A selection means for selecting a first model from the aforementioned multiple models, A means for identifying a second model related to the first model based on the degree of relevance, An information processing device characterized by comprising: (Item 2) The information processing device according to item 1, characterized by comprising a presentation means for presenting the second model. (Item 3) The model is a model that detects a subject from an image as the target of detection, The correlation calculation means calculates the correlation based on at least one of the distance between the locations where the multiple images were taken, and the time difference between the times when the multiple images were taken. An information processing device according to item 1 or item 2, characterized in that it is an information processing device according to item 1 or item 2. (Item 4) The correlation calculation means calculates the correlation based on the correlation of the names of the detected targets. An information processing device according to any one of items 1 to 3, characterized by the features described in item 1 to 3. (Item 5) The relevance calculation means calculates the relevance based on the search results obtained from a web search using a query that includes the name of the target to be detected. An information processing device according to any one of items 1 to 4, characterized in that it is the same as described in item 1 to 4. (Item 6) The correlation calculation means calculates the correlation based on false detections in which the multiple models incorrectly detect the target. An information processing device according to any one of items 1 to 5, characterized by the features described in item 1 to 5. (Item 7) A history acquisition means for acquiring the usage history of the multiple models by the user, A relevance correction means for correcting the relevance based on the usage history, An information processing device according to any one of items 1 to 6, characterized by comprising: (Item 8) The model is a model that detects a subject from an image as the target of detection, The correlation calculation means calculates the correlation based on the probability of detecting a subject in an image taken by a specific user and the probability of detecting a subject in an image taken by another user. An information processing device according to any one of items 1 to 7, characterized by the features described in item 1 to 7. (Item 9) The correlation calculation means calculates the correlation based on the probability, along with weighting coefficients based on the specific user and the other users. The information processing device described in item 8, characterized by the features described herein. (Item 10) Acquisition means for acquiring multiple related models and obtaining detection results of the target to be detected by the multiple models, A determination means for determining the main detection target from the aforementioned detection results, A terminal characterized by being equipped with the following features. (Item 11) It includes a setting means for receiving and setting priority detection targets, The acquisition means acquires the detection result by the model for detecting the preferred detection target, The determination means determines the primary detection target based on the detection result. The terminal described in item 10, characterized by the features described herein. (Item 12) The model is a model that detects a subject from an image as the target of detection, The determination means determines the main detection target based on the position of the detected target within the image. A terminal as described in item 10 or item 11, characterized by the above. (Item 13) The model is a model that detects a subject from an image as the target of detection, It is equipped with a means for specifying a range within an image, The determination means determines the primary detection target based on the specified range. A terminal as described in any one of items 10 to 12, characterized by the features described in item 12. (Item 14) The determination means determines the primary detection target based on the overlap of detection ranges that enclose multiple detection targets. The terminal described in item 13, characterized by the features described herein. (Item 15) An information processing method for identifying a model from multiple machine learning models, The acquisition process involves obtaining the detection results of the target to be detected by the model, A correlation calculation step that calculates the correlation between the multiple models based on at least one of the detection target and the detection result, A selection step of selecting a first model from the aforementioned multiple models, A process for identifying a second model related to the first model based on the degree of relevance, An information processing method characterized by comprising: (Item 16) A program to cause a computer to function as one of the information processing devices described in any one of items 1 through 9.

[0121] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0122] 100...Information processing device, 110...Terminal, 202...Acquisition unit, 203...Relevance calculation unit, 204...Selection unit, 205...Specification unit, 206...Presentation unit, 211...Setting unit, 212...Acquisition unit, 209...Designation unit, 210...Decision unit, 1401...History acquisition unit, 1402...Relevance correction unit.

Claims

1. An information processing device that identifies a model from multiple machine learning models, A means for obtaining detection results of the target to be detected by the model, A correlation calculation means for calculating the correlation between the plurality of models based on at least one of the detection target and the detection result, A selection means for selecting a first model from the aforementioned plurality of models, A means for identifying a second model related to the first model based on the degree of relevance, An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, characterized by comprising a presentation means for presenting the second model.

3. The model is a model that detects a subject from an image as the target of detection, The correlation calculation means calculates the correlation based on at least one of the distance between the locations where the multiple images were taken, and the time difference between the times when the multiple images were taken. The information processing apparatus according to feature 1.

4. The correlation calculation means calculates the correlation based on the correlation of the names of the detected targets. The information processing apparatus according to feature 1.

5. The relevance calculation means calculates the relevance based on the search results obtained by performing a web search using a query that includes the name of the target to be detected. The information processing apparatus according to feature 1.

6. The correlation calculation means calculates the correlation based on false detections in which the multiple models incorrectly detect the target. The information processing apparatus according to feature 1.

7. A history acquisition means for acquiring the usage history of the multiple models by the user, A relevance correction means for correcting the relevance based on the usage history, The information processing apparatus according to claim 1, characterized by comprising:

8. The model is a model that detects a subject from an image as the target of detection, The correlation calculation means calculates the correlation based on the probability of detecting a subject in an image taken by a specific user and the probability of detecting a subject in an image taken by another user. The information processing apparatus according to feature 1.

9. The correlation calculation means calculates the correlation based on the probability, along with weighting coefficients based on the specific user and the other users. The information processing apparatus according to feature 8.

10. Acquisition means for acquiring multiple related models and obtaining detection results of the target to be detected by the multiple models, A determination means for determining the main detection target from the aforementioned detection results, A terminal characterized by being equipped with the following features.

11. It includes a setting means for receiving and setting priority detection targets, The acquisition means acquires the detection result by the model for detecting the preferred detection target, The determination means determines the primary detection target based on the detection result. The terminal according to feature 10.

12. The model is a model that detects a subject from an image as the target of detection, The determination means determines the main detection target based on the position of the detected target within the image. The terminal according to feature 10.

13. The model is a model that detects a subject from an image as the target of detection, It is equipped with a means for specifying a range within an image, The determination means determines the primary detection target based on the specified range. The terminal according to feature 10.

14. The determination means determines the primary detection target based on the overlap of detection ranges that enclose multiple detection targets. The terminal according to feature 13.

15. An information processing method for identifying a model from multiple machine learning models, The acquisition process involves obtaining the detection results of the target to be detected by the model, A correlation calculation step that calculates the correlation between the multiple models based on at least one of the detection target and the detection result, A selection step of selecting a first model from the aforementioned multiple models, A process for identifying a second model related to the first model based on the degree of relevance, An information processing method characterized by comprising:

16. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 9.

Images