Trained model selection device, image analysis system, image analysis device, trained model selection method, and trained model selection program

Abstract

Select a suitable pre-trained model through image analysis. [Solution] The trained model selection device includes a storage unit that stores a plurality of trained models that have been machine-learned using training images and information of a first object shown in the training images; an image acquisition unit that acquires an unknown image; an acquisition unit that acquires information of a second object shown in the unknown image; and a selection unit that selects and outputs at least one trained model from the plurality of trained models to analyze the unknown image based on the similarity between the information of the first object and the information of the second object.

Description

Technical Field

[0001] The present disclosure relates to a trained model selection device, an image analysis system, an image analysis device, a trained model selection method, and a trained model selection program.

Background Art

[0002] Patent Document 1 stores a machine-learned trained model obtained by analyzing known images, acquires an unknown image for which a trained model has not yet been created, and selects, from the stored trained models, a trained model of a known image in which the imaging conditions including the imaging position and imaging angle with respect to the acquired unknown image and the imaging target are similar. Then, using the selected trained model, the unknown image is analyzed, and an image analysis result providing system that provides the result of the image analysis is disclosed. The image analysis result providing system is at least composed of mathematical formulas and parameters used for image analysis by the trained model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 discloses a method of selecting a trained model for use in image analysis of an unknown image, in which a trained model with similar imaging conditions such as the imaging position and imaging angle of the imaging target by a camera is selected. However, the imaging target by a camera (for example, a person, a vehicle, etc.) may be different for each camera. In such a case, there is a possibility that the image analysis result providing system cannot select a trained model suitable for the imaging target to be analyzed by the image analysis only by the method of selecting a trained model based on the imaging conditions.

[0005] This disclosure aims to provide a trained model selection device, an image analysis system, an image analysis device, a trained model selection method, and a trained model selection program that have been devised in view of the conventional circumstances described above and that select a suitable trained model through image analysis. [Means for solving the problem]

[0006] This disclosure provides a trained model selection device comprising: a storage unit that stores a plurality of trained models trained using training images and information of a first object depicted in the training images; an image acquisition unit that acquires an unknown image; an acquisition unit that acquires information of a second object depicted in the unknown image; and a selection unit that selects and outputs at least one trained model from the plurality of trained models to analyze the unknown image based on the similarity between the information of the first object and the information of the second object.

[0007] Furthermore, this disclosure provides an image analysis system comprising: a trained model selection device as described in claim 1; an analysis device capable of communicating with the trained model selection device; and a display terminal capable of communicating with the analysis device, wherein the trained model selection device transmits the selected trained model to the analysis device; the analysis device analyzes the unknown image using the transmitted trained model; transmits the analysis result to the display terminal; and the display terminal displays the transmitted analysis result.

[0008] Furthermore, this disclosure provides an image analysis device comprising: a storage unit that stores a plurality of trained models that have been machine-learned using training images and information of a first object that appears in the training images; an image acquisition unit that acquires an unknown image; an acquisition unit that acquires information of a second object that appears in the unknown image; a selection unit that selects and outputs at least one trained model from the plurality of trained models to analyze the unknown image based on the similarity between the information of the first object and the information of the second object; and an analysis unit that analyzes the unknown image using the selected trained model and outputs the result.

[0009] Furthermore, this disclosure provides a method for selecting a trained model performed by one or more processors, which stores a plurality of trained models that have been machine-learned using training images and information of a first object depicted in the training images, acquires an unknown image, acquires information of a second object depicted in the unknown image, and selects and outputs at least one trained model from the plurality of trained models to analyze the unknown image based on the similarity between the information of the first object and the information of the second object.

[0010] Furthermore, this disclosure provides a trained model selection program that one or more processors perform, which includes the steps of: storing a plurality of trained models that have been machine-learned using training images and information of a first object depicted in the training images; acquiring an unknown image; acquiring information of a second object depicted in the unknown image; and selecting and outputting at least one trained model from the plurality of trained models to analyze the unknown image based on the similarity between the information of the first object and the information of the second object. [Effects of the Invention]

[0011] According to this disclosure, a suitable pre-trained model can be selected for image analysis. [Brief explanation of the drawing]

[0012] [Figure 1] Block diagram showing an example of the system configuration of the image analysis system according to Embodiment 1. [Figure 2] Sequence diagram showing an example of the operation procedure of the image analysis system according to Embodiment 1. [Figure 3] Sequence diagram showing example 1 of the procedure for selecting a trained model in the image analysis system according to Embodiment 1. [Figure 4] Sequence diagram showing example 2 of the procedure for selecting a trained model in the image analysis system according to Embodiment 1. [Figure 5]Sequence diagram showing an example of a procedure for generating a new trained model for the image analysis system according to Embodiment 1. [Figure 6] A diagram showing an example of the data structure of a trained model. [Figure 7] This figure shows an example of a screen displaying the results of a comparison of trained models using the selected trained model. [Figure 8] Block diagram showing an example of the system configuration of the image analysis system according to Embodiment 2. [Figure 9] Sequence diagram showing an example of the operation procedure of the image analysis system according to Embodiment 2. [Modes for carrying out the invention]

[0013] Hereinafter, with reference to the drawings as appropriate, each embodiment specifically disclosing the trained model selection device, image analysis system, image analysis device, trained model selection method, and trained model selection program relating to this disclosure will be described in detail. However, unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. The accompanying drawings and the following explanation are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0014] (Embodiment 1) Referring to Figure 1, an example of the system configuration of the image analysis system 100 according to this first embodiment will be described. Figure 1 is a diagram showing an example of the system configuration of the image analysis system 100 according to this first embodiment. Note that the configuration of the image analysis system 100 shown in Figure 1 is just an example and is not limited thereto. Also, the number of cameras installed at one location may be multiple.

[0015] The image analysis system 100 learns captured images (hereinafter referred to as "learning images") captured by cameras installed at a plurality of different bases, and thereby generates a learned model suitable for an imaging object (e.g., a person or a vehicle, etc.) or an imaging purpose (e.g., counting the number of people, calculating the congestion rate, or the moving speed of a vehicle, etc., hereinafter referred to as "target") captured by each camera. The image analysis system 100 stores and manages the generated learned model and the learning images used for generating the learned model in a server S1 that is communicably connected to devices installed at each base.

[0016] Here, at least one camera, a user terminal, and an analysis device are installed at each base. For example, in the example shown in FIG. 1, camera C1, user terminal U1, and analysis device P1 are installed at "Base A", and camera C2, user terminal U2, and analysis device P2 are installed at "Base B".

[0017] The image analysis system 100 is managed by the server S1, and selects a learned model for analyzing a captured image captured by a camera installed at a new base or a camera newly installed at an existing base among the learned models pre-generated at each of a plurality of different bases, and detecting a target. The image analysis system 100 includes at least one camera C1, C2,... installed at each of a plurality of bases (such as "Base A" and "Base B" shown in FIG. 1), user terminals U1, U2,..., and analysis devices P1, P2,..., the server S1, and the network NW.

[0018] In the present disclosure, for the sake of easy understanding, an example of selecting a learned model used for analyzing a captured image captured by a new camera C1 installed at "Base A", that is, an example of performing an initial setting in the operation of camera C1 will be described. In addition, it is assumed that the learned model as an initial setting has been selected for camera C2 installed at "Base B", and image analysis using the learned model is being performed.

[0019] Each of the cameras C1, C2, ... is installed at least once at each location to capture a predetermined imaging area. Cameras C1, C2, ... are connected to the user terminal U1 and analysis device P1 installed at each location to enable data communication and perform data transmission and reception.

[0020] Each user terminal U1, U2, ... accepts user operations to configure (select) or modify the trained model used for target detection in target detection using images captured by cameras C1, C2, ... installed at each location. User terminal U1 is connected to analysis devices P1, P2, ... installed at each location to enable data communication and to send and receive data. Note that user terminal U1 does not need to be installed at each location if data communication is possible with cameras C1, C2, ... installed at each location via the network NW.

[0021] Each of the analysis devices P1, P2, ... acquires images captured by cameras C1, C2, ... installed at each location, accepts settings or changes to pre-trained models used for target detection, and performs image analysis using the configured pre-trained models. Each of the analysis devices P1, P2, ... is connected to cameras C1, C2, ... installed at its own location, user terminals U1, U2, ... installed at its own location, and server S1 to enable data communication and perform data transmission and reception. Each of the analysis devices P1, P2, ... includes a communication unit 10, a processor 11, and a memory 12.

[0022] Since the analysis devices installed at each location have the same configuration and functionality, the following explanation of the analysis devices will only cover analysis device P1 installed at "Location A," and explanations of the other analysis devices will be omitted.

[0023] The communication unit 10 is connected to the camera C1 and user terminal U1 installed at its own site via a network NW, enabling wireless or wired communication between the server S1 and the camera C1 and the user terminal U1, respectively, and performs data transmission and reception. The communication unit 10 outputs various data or information transmitted from the camera C1 or server S1 to the processor 11. The communication unit 10 also transmits various data or information output from the processor 11 to the user terminal U1. The wireless communication referred to here is communication provided in accordance with wireless communication standards such as wireless Local Area Network (LAN), wireless Wide Area Network (WAN), 4G (fourth-generation mobile communication system), 5G (fifth-generation mobile communication system), or Wi-Fi (registered trademark).

[0024] The processor 11 is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU"), a Digital Signal Processor (hereinafter referred to as "DSP"), or a Field Programmable Gate Array (hereinafter referred to as "FPGA"), and controls the operation of each part of the processor 11. The processor 11 works in cooperation with the memory 12 to comprehensively perform various processing and control operations. Specifically, the processor 11 refers to the programs and data held in the memory 12 and executes those programs to realize the functions of each part in order to perform the analysis of captured images using the trained model. The parts referred to here include the analysis unit 111, etc.

[0025] After the selection (setting) of a trained model is complete, the analysis unit 111 uses the trained model selected by the server S1 to detect targets (e.g., people, wheelchairs, canes, strollers, or vehicles) from the images captured by the camera C1, and performs predetermined image analysis (e.g., calculation of the number of targets, direction of movement, speed of movement, or congestion rate, intrusion detection of targets, or motion detection of targets) based on the detected targets or the actions of the targets. The analysis unit 111 transmits the analysis results to the user terminal U1.

[0026] Memory 12 includes, for example, Random Access Memory (hereinafter referred to as "RAM") as work memory used when executing various processes performed by the processor 11, and Read Only Memory (hereinafter referred to as "ROM") which stores a program that defines the various processes performed by the processor 11 and the data used during the execution of that program. Data or information generated or acquired by the processor 11 is temporarily stored in RAM. The ROM contains a program that defines the various processes performed by the processor 11 and the data used during the execution of that program. Memory 12 stores a trained model for each camera used for image analysis of images captured by each camera installed at the site.

[0027] Server S1 selects a pre-trained model suitable for target detection and target image analysis using the images captured by each camera, based on the analysis results obtained from image analysis of the images captured by each camera installed at each site. Server S1 is connected to the analysis devices P1, P2, ... installed at each site via a network NW, enabling data communication. Server S1 may be an on-premise server or a cloud server. Server S1 includes a communication unit 20, a processor 21, and memory 22.

[0028] The communication unit 20 is connected to the analysis devices P1, P2, ... installed at each location via a network NW, enabling wireless or wired communication, and performs data transmission and reception. The communication unit 20 outputs various data or information transmitted from the analysis devices P1, P2, ... to the processor 21. The communication unit 20 also transmits various data or information output from the processor 21 to each analysis device P1, P2, ...

[0029] The processor 21 is configured using, for example, a CPU, DSP, or FPGA, and controls the operation of each part of the processor 21. The processor 21 works in cooperation with the memory 22 to comprehensively perform various processing and control operations. Specifically, the processor 21 refers to the programs and data held in the memory 22 and executes those programs to realize the functions of each part in order to select a trained model. The parts referred to here include the image acquisition unit 211, the pre-analysis unit 212, the selection unit 213, and the training unit 214, etc.

[0030] The image acquisition unit 211 acquires images captured by the camera C1 that will be used to select a trained model (hereinafter referred to as "unknown images"). The image acquisition unit 211 outputs the acquired unknown images to the pre-analysis unit 212.

[0031] The pre-analysis unit 212 performs image analysis on the unknown image and obtains its attributes. As an analysis result, the pre-analysis unit 212 outputs the attribute information obtained from the unknown image to the selection unit 213.

[0032] Attributes include image features contained in the entire unknown image, object composition information of the camera's target (e.g., people, vehicles, or objects or locations that appear in the unknown image such as intersections), detection frames or cropped images of objects where objects (e.g., people, vehicles, or intersections) appear in the unknown image were detected, imaging condition information when the unknown image was captured (e.g., information such as depression angle, installation height, illumination, or imaging distance), or object-related information (e.g., object movement speed, density (congestion rate), number, imaging time period, number of pixels in the detection area where the object was detected, object movement, items held by the object, etc.).

[0033] Furthermore, the "action" of an object in object-related information refers to specific actions such as a person falling, a person entering a designated area, or a vehicle moving in a designated direction (i.e., the direction of the vehicle's movement). The "object" in object-related information refers to specific objects such as a cane held by a person or a stroller.

[0034] Here, the object configuration information is the target of the camera's imaging, and therefore may be information entered (specified) by the user. Similarly, the imaging condition information may also be information entered by the user. In such cases, the user terminal U1 receives the object configuration information or imaging condition information from the user and transmits the entered object configuration information or imaging condition information to the analysis device P1.

[0035] The selection unit 213 selects a trained model from among the multiple trained models that have been generated in advance and stored in the trained model storage unit 221, based on the analysis results from the pre-analysis unit 212 (i.e., attributes obtained from the unknown image) and the attributes of the training images used to generate each of the multiple trained models stored in the trained model storage unit 221, to be used for target detection or analysis from the image captured by the camera C1. In this way, the selection unit 213 can select a trained model suitable for target detection or analysis from the unknown image, according to the attributes of the unknown image, that is, the environment in which the camera is installed and the object being captured by the camera C1 (object configuration information). The selection unit 213 transmits the selected trained model to the analysis device P1 at the site where the camera C1 is installed via the communication unit 20.

[0036] Furthermore, if the selection unit 213 determines, based on the attributes obtained from the unknown image and the attributes of the training image, that it is not possible to select a trained model to be used for target detection or analysis from the image captured by the camera C1, it requests the training unit 214 to generate a new trained model. In such a case, the selection unit 213 selects one or more new trained models generated by the training unit 214 as the trained model to be used for target detection or analysis from the image captured by the camera C1.

[0037] The learning unit 214 performs additional training using unknown images on one or more pre-trained models selected by the selection unit 213. The learning unit 214 outputs the newly generated pre-trained model to the selection unit 213.

[0038] Furthermore, if no pre-trained model selected by the selection unit 213 exists, the learning unit 214 performs training using one or more unknown images. The learning unit 214 outputs the newly generated pre-trained model to the selection unit 213.

[0039] Memory 22 includes, for example, RAM as work memory used when executing various processes performed by processor 21, and ROM which stores a program that defines the various processes performed by processor 21 and the data used during the execution of that program. Data or information generated or acquired by processor 21 is temporarily stored in RAM. Programs that define the various processes performed by processor 21 and the data used during the execution of that program are written to ROM.

[0040] The trained model storage unit 221 is a so-called storage device, and is configured using a storage medium such as flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The trained model storage unit 221 stores multiple trained models (such as "trained model A," "trained model B," and "trained model C" shown in Figure 1) that have been generated in advance using images captured by cameras installed at each of the multiple locations. The trained model storage unit 221 may be implemented as an external storage device separate from the server S1.

[0041] The network NW connects the analysis devices P1, P2, ... installed at each location to the server S1, enabling communication between them.

[0042] <Example of pre-trained model selection procedure 1> Next, examples of the operation procedure of the image analysis system 100 will be described with reference to Figures 2 and 3, respectively. Figure 2 is a sequence diagram showing an example of the operation procedure of the image analysis system 100 according to Embodiment 1. Figure 3 is a sequence diagram showing example 1 of the trained model selection procedure of the image analysis system 100 according to Embodiment 1.

[0043] Example 1 of the trained model selection procedure is a processing procedure in which steps St141 and St142 are repeatedly executed until a trained model that matches the object configuration information obtained from an unknown image is found.

[0044] For the sake of clarity, the following explanation will describe the procedure for selecting a pre-trained model used by camera C1 installed at "Location A" as shown in Figure 1. However, the procedure for selecting a pre-trained model is the same at other locations. Furthermore, the example of the operation procedure shown in Figure 2 is just one example and is not limited to it. For example, the processing in steps St12 and St13 may be performed by the analysis device P1 instead of the server S1.

[0045] The analysis device P1 is installed at the local site and acquires unknown images captured by camera C1, which is selected as a target for the trained model, and transmits them to server S1 (St11). Alternatively, the unknown images may be transmitted directly from camera C1 to server S1 without going through the analysis device P1.

[0046] Server S1 acquires the unknown image transmitted from the analysis device P1. Server S1 performs a preliminary analysis on the acquired unknown image (St12) and obtains attributes from the unknown image (St13). The attributes obtained here include object configuration information, object addition information, imaging condition information, and image features, and it is sufficient that at least object configuration information is included.

[0047] Server S1 compares the object configuration information obtained from the unknown image with the object configuration information of the training images used to generate each trained model stored in the trained model storage unit 221 (St14), and performs object configuration information matching. Note that the object configuration information shown in the unknown image may be specified in advance by the user.

[0048] Server S1 determines, based on the object configuration information matching results, whether or not there is a trained model generated using training images that have object configuration information that matches the object configuration information obtained from the unknown image (St141).

[0049] If server S1 determines that there is a trained model generated by training on training images that match the object configuration information obtained from the unknown image (St141, YES), it selects the one or more trained models that have the highest similarity between the object configuration information of the unknown image and the object configuration information that the trained model can analyze (St15). The number of trained models selected here may be set by the user. Server S1 may also select one or more trained models for each object in the unknown image.

[0050] On the other hand, if server S1 determines that there is no trained model generated by training images that match the object configuration information obtained from the unknown image (St141, NO), it compares (matches) other attributes other than the object configuration information (St142) and determines again whether there is a trained model that matches the other attributes obtained from the unknown image (St141). The matching process in step St142 may be a process that calculates and compares the similarity between other attributes in the unknown image and other attributes in the training images used to generate the trained model.

[0051] For example, server S1 may compare the imaging conditions of camera C1, which captured the unknown image, with the imaging conditions of the camera that captured the training image, or it may compare the image features extracted from the unknown image with the image features extracted from the training image. Alternatively, when performing an image feature comparison, server S1 may input the unknown image and the training image into a multimodal generation AI and calculate the similarity.

[0052] Server S1 transmits one or more selected trained models to the analysis device P1, which is installed at the same location as the corresponding camera C1 (St16).

[0053] The analysis device P1 performs image analysis of the unknown image using one or more trained models transmitted from the server S1 (St17). The analysis device P1 transmits the results of the analysis of the unknown image using the trained models to the user terminal U1 (St18).

[0054] User terminal U1 displays the analysis results transmitted from analysis device P1 (St19).

[0055] The selection process of the trained model in step St15 may be performed automatically by the server S1 or by user operation. If performed by user operation, the server S1 or analysis device P1 generates the trained model comparison result screen SC1 based on the result of the process in step St14 (i.e., the comparison process between the object configuration information of the trained image and the object configuration information of the unknown image).

[0056] The analysis device P1 transmits the trained model comparison result screen SC1 to the user terminal U1 for output (display), and accepts the user's operation to select a trained model to be used for detecting targets in unknown images captured by camera C1 via the user terminal U1. The analysis device P1 generates a control command requesting the transmission of the trained model selected by the user and sends it to server S1, and stores the trained model transmitted from server S1 in memory 12 as a trained model to be used for image analysis of unknown images captured by camera C1 at "site A".

[0057] Meanwhile, if the analysis device P1 receives information from the user terminal U1 indicating that there is no pre-trained model selected by the user, it sends this information to the server S1. If the server S1 receives information from the analysis device P1 indicating that there is no pre-trained model selected by the user, it generates a new pre-trained model (St20) to analyze the unknown image captured by the camera C1. The server S1 sends the newly generated pre-trained model to the analysis device P1. The analysis device P1 stores the new pre-trained model sent from the server S1 in memory 12 as a pre-trained model to be used for image analysis of the unknown image captured by the camera C1 at "site A".

[0058] Furthermore, after the analysis device P1 displays the analysis results transmitted from the analysis device P1 to the user terminal U1 (St19), if it receives information from the user terminal U1 indicating that the user wishes to perform the analysis with a new trained model, it also transmits this information to the server S1. When the server S1 receives information from the analysis device P1 indicating that the user wishes to perform the analysis with a new trained model, it generates a new trained model to analyze the unknown image captured by the camera C1 (St20). The server S1 transmits the generated new trained model to the analysis device P1. The analysis device P1 stores the new trained model transmitted from the server S1 in memory 12 as a trained model to be used for image analysis of the unknown image captured by the camera C1 at "site A".

[0059] Furthermore, information indicating that there is no pre-trained model selected by the user, or information indicating that the user wishes to perform analysis with a new pre-trained model, may be sent from the user terminal U1 to the server S1 without going through the analysis device P1.

[0060] Note that the process in step St20 (the process of generating a new trained model) is not mandatory and may be omitted. The process in step St20 will be described later.

[0061] As described above, the image analysis system 100 according to Embodiment 1 can select a pre-trained model from among a plurality of pre-trained models that have been generated in advance based on attributes (object configuration information, or other attributes other than object configuration information) obtained from an unknown image, using a training image that has similar attributes (for example, object configuration information, object addition information, imaging condition information, or image features) to the unknown image captured by the camera C1. This allows the system to select a pre-trained model that is more suitable for image analysis of the image captured by the camera C1.

[0062] <Example of pre-trained model selection procedure 2> Next, with reference to Figure 4, an example of the pre-trained model selection procedure 2 for the image analysis system 100 will be described. Figure 4 is a sequence diagram showing an example of the pre-trained model selection procedure 2 for the image analysis system 100 according to Embodiment 1.

[0063] Example 2 of the pre-trained model selection procedure is the procedure for when it is determined that there is no pre-trained model that matches the object configuration information obtained from an unknown image.

[0064] Note that the processing in Example 2 of the pre-trained model selection procedure includes the same processing as shown in Example 1 of the pre-trained model selection procedure. Therefore, processing that is the same as that in Example 1 of the pre-trained model selection procedure will be given the same descriptive numerals, and the explanation will be omitted.

[0065] Server S1 performs another comparison (matching) using attributes other than object configuration information (e.g., object addition information, imaging condition information, and image features) (St142), and determines again whether there is a trained model that matches the other attributes obtained from the unknown image (St143).

[0066] If server S1 determines that there is a trained model that matches other attributes obtained from the unknown image (St143, YES), it selects one or more trained models from among the one or more trained models that were determined to match, which have a high similarity between the object configuration information of the unknown image and the object configuration information that the trained model can analyze, i.e., the similarity is above a certain value (St15). The number of trained models selected here may be set by the user. Server S1 may also select one or more trained models for each object in the unknown image.

[0067] On the other hand, if server S1 determines that there is no trained model that matches the other attributes obtained from the unknown image (St143, NO), it selects one or more top trained models from the existing trained models stored in the trained model storage unit 221 that have a high similarity to the object configuration information that the trained model can analyze, that is, a similarity of a certain value or higher. Server S1 generates a new trained model X by further training the unknown image on one or more of the selected trained models. Server S1 selects the generated one or more new trained models X as trained models to analyze the captured image taken by camera C1 (St144). Note that the processing in step St144 may be replaced by the processing in step St20 described later. Also, the number of new trained models X selected here may be set by the user.

[0068] Server S1 generates trained model file IDs corresponding to one or more new trained models X generated by additional training. Server S1 associates one or more new trained models X, a training image folder containing unknown images used for training, the attributes of the new trained models X (e.g., object configuration information, object addition information, imaging condition information, and image features), and the trained model file ID, and stores them in the trained model storage unit 221 for each new trained model X.

[0069] As described above, the image analysis system 100 according to Embodiment 1 can suppress the use of unsuitable models for analysis by determining that there is no pre-trained model among the multiple pre-trained models that can analyze attributes similar to other attributes obtained from an unknown image, and that there is no pre-trained model suitable for image analysis of the image captured by camera C1.

[0070] Furthermore, the image analysis system 100 can select one or multiple high-ranking pre-trained models with the highest similarity to analyzable object configuration information for other attributes obtained from an unknown image, and perform additional training using this pre-trained model. This enables the generation and selection of a new pre-trained model X that is more suitable for image analysis of images captured by camera C1. Therefore, the image analysis system 100 can acquire and select a new pre-trained model X that can analyze images captured by camera C1 with higher accuracy, even with fewer training iterations.

[0071] Furthermore, the image analysis system 100 can increase the number of pre-trained models that can be selected at each of the multiple locations (location A, location B, ...) by generating new pre-trained models.

[0072] <Process for generating a new pre-trained model> Next, with reference to Figure 5, the process of generating a new learned model (step St20) shown in Figures 3 and 4 will be described. Figure 5 is a sequence diagram showing an example of the procedure for generating a new trained model of the image analysis system 100 according to Embodiment 1.

[0073] Server S1 determines whether there is one or more top-performing trained models among the multiple trained models stored in the trained model storage unit 221 that have a high similarity to other attributes obtained from the unknown image, i.e., a similarity of a certain value or higher (St21). The number of trained models selected here may be configurable by the user.

[0074] If Server S1 determines that there is one or more top-ranking trained models with high similarity to other attributes obtained from the unknown image (St21, YES), it further trains the unknown image on the one or more top-ranking trained models. Server S1 generates one or more new trained models X that have been further trained (St22). Server S1 generates trained model file IDs corresponding to the one or more new trained models X generated by the further training. Server S1 associates the one or more new trained models X, the training image folder containing the unknown images used for training, the attributes of the new trained models X (e.g., object configuration information, object addition information, imaging condition information, and image features), and the trained model file ID, and stores them in the trained model storage unit 221 for each new trained model X.

[0075] Server S1 selects one or more newly trained models X that have been generated as trained models to analyze the images captured by camera C1 (St23).

[0076] On the other hand, if server S1 determines that there is no top-ranking trained model with a high similarity to other attributes obtained from the unknown image (St21, NO), it generates a new trained model Y trained using the unknown image as training data (St24). Server S1 generates trained model file IDs corresponding to the one or more new trained models Y generated by the additional training. Server S1 associates the one or more new trained models Y, the training image folder containing the unknown images used for training, the attributes of the new trained models Y (e.g., object configuration information, object addition information, imaging condition information, and image features), and the trained model file ID, and stores them in the trained model storage unit 221 for each new trained model Y.

[0077] Server S1 selects the newly generated trained model Y as the trained model to analyze the images captured by camera C1 (St25). Note that there may be multiple unknown images used for training here.

[0078] As described above, if the image analysis system 100 according to Embodiment 1 determines that there is no pre-trained model among the multiple pre-trained models that can analyze attributes similar to other attributes obtained from an unknown image, it can perform additional training or training using the unknown image to generate and select new pre-trained models X and Y that are more suitable for image analysis of images captured by camera C1.

[0079] Next, with reference to Figure 6, an example of the data structure of a trained model stored in the trained model storage unit 221 will be described. Figure 6 is a diagram showing an example of the data structure of a trained model. Note that the configuration example of the data structure shown in Figure 6 is just one example and is not limited thereto. In addition, the attributes shown in Figure 6 may include information such as the size of the detection frame in which an object (for example, a person, a vehicle, or an intersection) is detected, or the number of pixels in which the object is captured.

[0080] The trained model data stored in the trained model storage unit 221 includes the trained model ID, attributes that the trained model can analyze, or attributes of the training images used to generate the trained model, the trained model file ID, and the training image folder. The attributes include object configuration information, object addition information, imaging condition information, and image feature information. Note that the attributes only need to include at least object configuration information.

[0081] The data "Trained Model ID" is identification information used to identify each of the multiple trained models stored in the trained model storage unit 221.

[0082] The data "attributes" are information about attributes obtained by image analysis of the training images used to generate the corresponding trained model. As an example, the "attributes" shown in Figure 6 include "object configuration information" visible in the training image, "object addition information" obtained from the training image, "imaging condition information" of the training image, and "image features" obtained from the training image.

[0083] The data "Trained Model File ID" is identification information used to identify the file in which the trained model is stored.

[0084] The data "Training Image File" is a link to a folder containing at least one training image to which the identification information (ID) shown in the data "Trained Model ID" has been assigned.

[0085] Next, we will explain the trained model comparison results screen SC1 with reference to Figure 7. Figure 7 shows an example of the trained model comparison results screen SC1 using the selected trained model. Note that the trained model comparison results screen SC1 shown in Figure 7 is just one example and is not limited to this.

[0086] The selection unit 213 of server S1 selects a predetermined number of top-performing trained models with high similarity as a result of comparing the object configuration information of the training images with the object configuration information of the unknown images. For each of the selected predetermined number of trained models, the selection unit 213 generates a trained model comparison result screen SC1 based on the trained model identification information, the similarity between the object configuration information of the training images and the object configuration information of the unknown images, and at least one training image stored in the training image folder. The selection unit 213 transmits the generated trained model comparison result screen SC1 to the analysis device P1 via the communication unit 20.

[0087] The trained model comparison results screen SC1 is transmitted from the analysis device P1 to the user terminal U1 and displayed, making it visible to the user. The generation of the trained model comparison results screen SC1 may also be performed by the analysis device P1.

[0088] The trained model comparison results screen SC1 shown in Figure 7 includes an unknown image IMG0, a table TB containing information about the top four trained models with high similarity in object configuration information or other attributes of the unknown image IMG0, and a selection button BT that accepts the selection operation of a trained model.

[0089] Unknown image IMG0 is an image captured by camera C1. Unknown image IMG0 is an image used for selecting a trained model, and it is an image in which the target person Tg0 is captured.

[0090] Table TB shows the selection results for multiple (in this case, the top four) pre-trained models, and selection fields CHK1, CHK2, CHK3, and CHK4 that accept the selection of a pre-trained model from among the multiple pre-trained models to analyze the images captured by camera C1. Table TB shown in Figure 7 stores the pre-trained models with pre-trained model IDs "0003", "0002", "0015", and "0010" in descending order of similarity.

[0091] For example, the trained model with trained model ID "0003" shown in Figure 7 shows that the similarity between the object configuration information of the training image IMG1 (here, person Tg1) and the object configuration information of the unknown image IMG0 (here, person Tg0) is "95". The trained model with trained model ID "0002" shows that the similarity between the object configuration information of the training image IMG2 (here, person Tg2) and the object configuration information of the unknown image IMG0 (person Tg0) is "80". The trained model with trained model ID "0015" shows that the similarity between the object configuration information of the training image IMG3 (here, person Tg3) and the object configuration information of the unknown image IMG0 (person Tg0) is "77". Furthermore, the trained model with trained model ID "0010" shows that the similarity between the object configuration information of the training image IMG4 (in this case, person Tg4) and the object configuration information of the unknown image IMG0 (person Tg0) is "60".

[0092] The selection unit 213 may also perform comparisons (matching) based on attributes other than object configuration information and calculate similarity. In such cases, the selection unit 213 calculates, for example, the similarity of camera imaging conditions (e.g., downward angle to the object, imaging distance, etc.) based on the sizes of the detection frames FR1, FR2, FR3, FR4 of the objects detected in each of the training images IMG1 to IMG4 (i.e., people Tg1 to Tg4) and the size of the detection frame (not shown) of the object (person Tg0) in the unknown image IMG0; the similarity of the number of pixels of the objects (person Tg1 to Tg4) in each of the training images IMG1 to IMG4 and the number of pixels of the object (person Tg0) in the unknown image IMG0; or the similarity of the number of pixels of the objects (person Tg1 to Tg4) in each of the training images IMG1 to IMG4 and the number of pixels of the object (person Tg0) in the unknown image IMG0. Furthermore, the similarity displayed on the trained model comparison results screen SC1 may be the sum or average of the similarity scores calculated for each attribute.

[0093] The user terminal U1 accepts user input on the selection fields CHK1 to CHK4 on the trained model comparison results screen SC1, and accepts the selection of one or more trained models. When the selection button BT is pressed (selected) with selection fields CHK1 to CHK4 checked by the user, the user terminal U1 transmits information on the trained models corresponding to the checked selection fields CHK1 to CHK4, i.e., selected by the user, to the analysis device P1.

[0094] In the example shown in Figure 7, the only object visible in the unknown image IMG0 is a person Tg0. If the unknown image contains multiple objects, such as a person and a vehicle, the user terminal U1 may accept the selection of one or more trained models corresponding to the object "person" and one or more trained models corresponding to the object "vehicle". In such cases, the user terminal U1 sends information about the one or more trained models selected for each object to the server S1.

[0095] The analysis device P1 sends information about the trained model selected by the user, along with a control command requesting the transmission of this trained model, to the server S1, thereby acquiring the trained model.

[0096] As described above, the image analysis system 100 can assist the user in selecting and approving a pre-trained model to be used for analyzing images captured by each camera when the user selects and approves a pre-trained model.

[0097] (Embodiment 2) The image analysis system 100 according to Embodiment 1 described above illustrates an example in which a server S1, which is connected to each analysis device P1, P2, ... via communication between the analysis devices P1, P2, ... and a pre-trained model for analyzing images captured by each camera C1, C2, ... is selected. The image analysis system 100A according to Embodiment 2 described below will explain an example in which the server S1A is capable of realizing the functions of the analysis devices P1, P2, ... and the server S1 in Embodiment 1, and the server S1A selects a pre-trained model for analyzing images captured by each camera C1, C2, ...

[0098] The image analysis system 100A according to Embodiment 2 has the same configuration and functions as the image analysis system 100 according to Embodiment 1. Therefore, in the following description of the image analysis system 100A according to Embodiment 2, the same reference numerals are used to represent the same components as in the image analysis system 100 according to Embodiment 1, and their descriptions are omitted.

[0099] Referring to Figure 8, the image analysis system 100A according to Embodiment 2 will be described. Figure 8 is a block diagram showing an example of the system configuration of the image analysis system 100A according to Embodiment 2.

[0100] The image analysis system 100A according to Embodiment 2 includes cameras C1, C2, ... and user terminals U1, U2, ... installed at each location, as well as a server S1A. The image analysis system 100A, via the server S1A, performs the selection of a trained model to be used for analyzing images captured by each camera C1, C2, ... as initial camera setup, and image analysis using the selected trained model as operational processing.

[0101] Server S1A is connected to each of the cameras C1, C2, ... and user terminals U1, U2, ... installed at each location, via a network NW, enabling data communication. Server S1A includes a communication unit 20A, a processor 21A, and memory 22.

[0102] The communication unit 20A is connected to cameras C1, C2, ... and user terminals U1, U2, ... installed at each location via the network NW, enabling wireless or wired communication, and performs data transmission and reception. The communication unit 20A outputs various data or information transmitted from cameras C1, C2, ... and user terminals U1, U2, ... to the processor 21A. The communication unit 20A also transmits various data or information output from the processor 21A to cameras C1, C2, ... and user terminals U1, U2, ....

[0103] The processor 21A is configured using, for example, a CPU, DSP, or FPGA, and controls the operation of each part of the processor 21A. The processor 21A works in cooperation with the memory 22 to comprehensively perform various processing and control. Specifically, the processor 21A refers to the programs and data held in the memory 22 and executes those programs to realize the functions of each part in order to select a trained model and analyze the captured images using the selected trained model. The parts referred to here are the image acquisition unit 211, the pre-analysis unit 212, the selection unit 213, the learning unit 214, and the analysis unit 215, etc.

[0104] The analysis unit 215 analyzes the captured images transmitted from the camera, which has completed the selection of a trained model, that is, the initial setup, using the selected trained model. The analysis unit 215 then transmits the analysis results to a user terminal installed at the same location as the camera.

[0105] Next, an example of the operation procedure of the image analysis system 100A will be described with reference to Figures 3 and 9, respectively. Figure 9 is a sequence diagram showing an example of the operation procedure of the image analysis system 100A according to Embodiment 2. Note that the processing of steps St12 to St15 shown in Figure 9 is the same as the processing of steps St12 to St15 shown in Figure 2, so the explanation is omitted here.

[0106] For the sake of clarity, the following explanation will describe the procedure for selecting a pre-trained model used in camera C1 installed at "Location A" as shown in Figure 8. However, the procedure for selecting a pre-trained model is the same at other locations.

[0107] User terminal U1 acquires unknown images captured by camera C1, which is installed at the user's site and is selected as a target for the trained model, and transmits them to server S1A (St11A). Alternatively, the unknown images may be transmitted directly from camera C1 to server S1A without going through user terminal U1.

[0108] Server S1A performs image analysis of the unknown image captured by camera C1 using the selected pre-trained model (St17A). Server S1A transmits the results of the analysis of the unknown image using the pre-trained model to user terminal U1, which is installed at the same location as camera C1 (St18A).

[0109] The user terminal U1 receives and displays the analysis results sent from the server S1A (St19A). Here, the user terminal U1 may accept user input based on the analysis results, indicating whether the trained model used for image analysis (i.e., the selected trained model) is suitable for analyzing the images captured by the camera C1, and send the user input to the server S1A.

[0110] If the user input indicates that the selected pre-trained model is not suitable for analyzing images captured by camera C1, server S1A proceeds to step St142 and performs a re-selection of the pre-trained model based on a comparison of other attributes. On the other hand, if the user input indicates that the selected pre-trained model is suitable for analyzing images captured by camera C1, server S1A stores the selected pre-trained model as the pre-trained model to be used for image analysis of camera C1.

[0111] As described above, the image analysis system 100A according to Embodiment 2 can select a pre-trained model from among several pre-generated trained models that is suitable for detecting a target using an image captured by a camera, based on attributes (object configuration information, or other attributes besides object configuration information) detected from an unknown image.

[0112] (Note) Based on the descriptions of each embodiment above, the following technologies are disclosed.

[0113] (Technology 1) A storage unit (memory 22) that stores multiple trained models that have been machine-learned using training images, and information (object configuration information) of a first object that appears in the training images, Image acquisition unit 211 that acquires unknown images, An acquisition unit (communication unit 20, 20A) acquires information (object configuration information) of a second object shown in the aforementioned unknown image, The system includes a selection unit 213 that selects and outputs at least one trained model from among the plurality of trained models for analyzing the unknown image, based on the similarity between the information of the first object (object configuration information) and the information of the second object (object configuration information). Pre-trained model selection device (servers S1, S1A). As a result, servers S1 and S1A in this disclosure can select a trained model more suitable for image analysis by selecting a trained model to be used for image analysis based on the similarity between the object configuration information obtained from an unknown image and the object configuration information of the training image used to generate the trained model.

[0114] (Technology 2) The memory unit (memory 22) stores first attribute information for each learning image, which includes information about the first object (object configuration information), information about the image features of the learning image, information about the first object appearing in the learning image (object addition information), and information about the imaging conditions of the learning image. The acquisition unit (communication unit 20, 20A) acquires second attribute information including information about the second object (object configuration information), information about the image features of the unknown image, information about the second object appearing in the unknown image (object addition information), and imaging condition information of the unknown image. The selection unit 213 selects and outputs the trained model for analyzing the unknown image based on the similarity of multiple pieces of information included in the first attribute information and the second attribute information. The trained model selection device (server S1, S1A) described in (Technology 1). As a result, servers S1 and S1A in this disclosure can select a pre-trained model to be used for image analysis based on the similarity between multiple pieces of information contained in the attributes obtained from the unknown image and multiple pieces of information contained in the attributes of the training image, thereby enabling them to select a pre-trained model that is more suitable for image analysis.

[0115] (Technology 3) If the selection unit 213 determines that it is not possible to select the trained model for analyzing the unknown image based on the similarity between the information of the first object (object configuration information) and the information of the second object (object configuration information), it selects and outputs the trained model for analyzing the unknown image based on the similarity between the multiple pieces of information included in the first attribute information and the second attribute information. A trained model selection device (server S1, S1A) as described in (Technology 1) or (Technology 2). As a result, servers S1 and S1A in this disclosure can select a pre-trained model to be used for image analysis based on the similarity between multiple pieces of information contained in the attributes obtained from the unknown image and multiple pieces of information contained in the attributes of the training image, thereby enabling them to select a pre-trained model that is more suitable for image analysis.

[0116] (Technology 4) The information of the second object (object configuration information) is information specified by the user. A trained model selection device (server S1, S1A) described in any one of (Technology 1) to (Technology 3). As a result, in this disclosure, servers S1 and S1A can select a pre-trained model that is more suitable for detecting (analyzing) the target image requested by the user during image analysis.

[0117] (Technology 5) The trained model selection device (server S1, S1A) described in (Technology 1), An analysis device P1 that can communicate with the aforementioned trained model selection device (server S1, S1A), An image analysis system 100, 100A comprising a display terminal (user terminal U1) capable of communicating with the analysis device P1, The aforementioned trained model selection device (server S1, S1A) The selected trained model is transmitted to the analysis device P1. The aforementioned analysis device P1 is The transmitted trained model is used to analyze the unknown image, and the analysis results are transmitted to the display terminal (user terminal U1). The aforementioned display terminal (user terminal U1) is Display the transmitted analysis results. Image analysis system 100, 100A. As a result, the image analysis systems 100 and 100A in this disclosure can select a pre-trained model more suitable for image analysis by selecting a pre-trained model to be used for image analysis based on the similarity between object configuration information obtained from an unknown image and object configuration information from a training image used to generate the pre-trained model. Furthermore, the image analysis systems 100 and 100A can visualize to the user whether the selected pre-trained model is capable of detecting (analyzing) the target requested by the user by performing analysis of the unknown image using the selected pre-trained model and outputting the analysis results.

[0118] (Technology 6) The aforementioned trained model selection device (server S1, S1A) further comprises a learning unit 214, The learning unit 214 performs additional training by having the previously trained model learn the unknown image, thereby generating a new trained model. The selection unit 213 selects and outputs the new trained model. A trained model selection device (server S1, S1A) according to claim 1. As a result, if the servers S1 and S1A according to Embodiment 1 determine that there is no pre-trained model among the multiple pre-generated pre-trained models capable of analyzing attributes similar to other attributes obtained from an unknown image, they can perform additional training using the unknown image to generate and select a new pre-trained model X that is more suitable for image analysis of the image captured by the camera C1.

[0119] (Technology 7) A storage unit (memory 22) that stores multiple trained models that have been machine-learned using training images, and information (object configuration information) of a first object that appears in the training images, Image acquisition unit 211 that acquires unknown images, An acquisition unit (communication unit 20A) acquires information (object configuration information) of a second object shown in the aforementioned unknown image, A selection unit 213 selects and outputs at least one trained model from among the plurality of trained models that analyzes the unknown image, based on the similarity between the information of the first object (object configuration information) and the information of the second object (object configuration information). The system includes an analysis unit 215 that analyzes and outputs the unknown image using the selected pre-trained model, Image analysis device (server S1A). As a result, the server S1A in this disclosure can select a trained model more suitable for image analysis by selecting a trained model to be used for image analysis based on the similarity between the object configuration information obtained from an unknown image and the object configuration information of the training image used to generate the trained model.

[0120] (Technology 8) A method for selecting a trained model performed by one or more processors 21, 21A, The system stores multiple trained models created using training images, and information about a first object (object configuration information) that appears in the training images. Obtain an unknown image, Information (object configuration information) of the second object shown in the aforementioned unknown image is obtained. Based on the similarity between the information of the first object (object configuration information) and the information of the second object (object configuration information), at least one trained model from among the multiple trained models that analyzes the unknown image is selected and output. Method for selecting a pre-trained model. As a result, the processors 21 and 21A in this disclosure can select a trained model more suitable for image analysis by selecting a trained model to be used for image analysis based on the similarity between the object configuration information obtained from an unknown image and the object configuration information of the training image used to generate the trained model.

[0121] (Technology 9) A trained model selection program performed by one or more processors 21, 21A, The steps include storing multiple trained models that have been machine-learned using training images, and information (object configuration information) of a first object shown in the training images, Steps to acquire unknown images, The steps include obtaining information (object configuration information) of a second object shown in the aforementioned unknown image, To perform the following steps: selecting and outputting at least one trained model from among the plurality of trained models that analyzes the unknown image, based on the similarity between the information of the first object (object configuration information) and the information of the second object (object configuration information), A program for selecting pre-trained models. As a result, the processors 21 and 21A in this disclosure can select a trained model more suitable for image analysis by selecting a trained model to be used for image analysis based on the similarity between the object configuration information obtained from an unknown image and the object configuration information of the training image used to generate the trained model.

[0122] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the various embodiments described above can be combined arbitrarily without departing from the spirit of the invention. [Industrial applicability]

[0123] This disclosure is useful as a trained model selection device, image analysis system, image analysis device, trained model selection method, and trained model selection program for selecting a suitable trained model through image analysis. [Explanation of Symbols]

[0124] 10,20,20A Communications Department 11,21,21A processor 12.22 memory 100,100A Image Analysis System 111 Analysis Department 211 Image acquisition unit 212 Pre-analysis Department 213 Selection Section 214 Learning Department 215 Analysis Department 221 Store of trained models C1, C2 Camera IMG0 Unknown image IMG1, IMG2, IMG3, IMG4 Training images NW Network P1,P2 analysis device S1, S1A Server SC1 Trained Model Comparison Results Screen U1, U2 User Terminals

Claims

1. A storage unit that stores multiple trained models that have been machine-learned using training images, and information about a first object that appears in the training images, An image acquisition unit that acquires unknown images, An acquisition unit that acquires information about a second object appearing in the aforementioned unknown image, The system includes a selection unit that, based on the similarity between the information of the first object and the information of the second object, selects and outputs at least one trained model from among the plurality of trained models that analyzes the unknown image. A pre-trained model selection device.

2. The storage unit stores first attribute information for each learning image, which includes information about the first object, information about the image features of the learning image, information about the first object appearing in the learning image, and information about the imaging conditions of the learning image. The acquisition unit acquires second attribute information which includes information about the second object, information about the image features of the unknown image, information about the second object appearing in the unknown image, and imaging condition information for the unknown image. The selection unit selects and outputs the trained model for analyzing the unknown image based on the similarity of multiple pieces of information included in the first attribute information and the second attribute information. A trained model selection device according to claim 1.

3. If the selection unit determines, based on the similarity between the information of the first object and the information of the second object, that it is not possible to select the trained model for analyzing the unknown image, it selects and outputs the trained model for analyzing the unknown image based on the similarity between the multiple pieces of information included in the first attribute information and the second attribute information. The trained model selection device according to claim 2.

4. The information of the second object is information specified by the user. A trained model selection device according to claim 1 or 2.

5. The learned model selection device according to claim 1, An analysis device capable of communicating with the aforementioned trained model selection device, An image analysis system comprising a display terminal capable of communicating with the aforementioned analysis device, The aforementioned trained model selection device is The selected trained model is transmitted to the analysis device. The aforementioned analysis device is The transmitted trained model is used to analyze the unknown image, and the analysis results are transmitted to the display terminal. The aforementioned display terminal is Display the transmitted analysis results. Image analysis system.

6. The aforementioned trained model selection device further comprises a learning unit, The learning unit performs additional training by having the previously trained model learn the unknown image, thereby generating a new trained model. The selection unit selects and outputs the new trained model. A trained model selection device according to claim 1.

7. A storage unit that stores multiple trained models that have been machine-learned using training images, and information about a first object that appears in the training images, An image acquisition unit that acquires unknown images, An acquisition unit that acquires information about a second object appearing in the aforementioned unknown image, A selection unit that, based on the similarity between the information of the first object and the information of the second object, selects and outputs at least one trained model from among the plurality of trained models that analyzes the unknown image, The system comprises an analysis unit that analyzes and outputs the unknown image using the selected pre-trained model, Image analysis device.

8. A method for selecting a trained model performed by one or more processors, The system stores multiple trained models created using training images, and information about a first object depicted in the training images. Obtain an unknown image, Information about the second object shown in the aforementioned unknown image is obtained, Based on the similarity between the information of the first object and the information of the second object, at least one trained model is selected from the plurality of trained models to analyze the unknown image and output. Method for selecting a pre-trained model.

9. A trained model selection program performed by one or more processors, A step of storing multiple trained models that have been machine-learned using training images, and information about a first object that appears in the training images, Steps to acquire unknown images, The steps include obtaining information about a second object visible in the aforementioned unknown image, To perform the steps of selecting and outputting at least one trained model from the plurality of trained models that analyzes the unknown image, based on the similarity between the information of the first object and the information of the second object, A program for selecting pre-trained models.

Images