Image processing system and method for controlling the image processing system

Abstract

To obtain dictionary data for subject detection that is appropriate to the user's intent. [Solution] The image processing system 10 includes an input data acquisition unit 111 that acquires an image captured of a specific subject and shooting information at the time the image was captured, an image data conversion unit 144 that performs image conversion on the image captured and generates a converted image, a dictionary data generation unit 115 that performs machine learning on the converted image and generates dictionary data which is a trained model for detecting a specific subject in subject detection, and a detection result output unit 116 that performs subject detection on multiple images using the dictionary data and outputs the detection results to display a detection result display screen for the user to view the detection results and select whether to register the dictionary data in the imaging device, and the image data conversion unit 114 sets the content of the image conversion according to the shooting information.

Description

Technical Field

[0001] The present invention relates to an image processing system and a method for controlling the image processing system.

Background Art

[0002] Computer vision is a technology that understands an image input to a computer and automatically recognizes various characteristics of the image. Among computer vision, object detection is a task of estimating the position and type of an object existing in a natural image. Object detection is applied to autofocus technology of an imaging device and the like. In recent years, an imaging device that detects an object by a machine learning method represented by a neural network or the like is known. Such an imaging device uses a learned model (dictionary data) corresponding to a specific object to detect the specific object and perform imaging control. The dictionary data installed in the imaging device is limited to types of objects that are in demand from many users in consideration of commerciality. On the other hand, since there are also users who require dictionary data for types of objects with low demand, in the future, a function for each user to generate a dictionary according to their own interests is expected. For example, when playing with other dogs in a dog park, there may be a demand for dictionary data dedicated to one's own pet that can detect only one's own pet as an object.

[0003] In the development of dictionary data, a pair of an image and a correct label is prepared as teacher data and the neural network is trained. And, the more this teacher data is, the higher the performance of the dictionary becomes. Therefore, as methods for increasing the pair of teacher data and the correct label, there are methods such as data augmentation and transfer learning. For example, Patent Document 1 discloses a method of using an image with an added correct label and automatically adding the correct label to an image that has been increased in number by image conversion or the like. When creating a dictionary requested by a user, it is necessary to confirm whether the created dictionary is made as intended by the user. As this confirmation method, for example, Patent Document 2 discloses a method of improving detection accuracy by using a general-purpose dictionary when the selected dictionary deviates from the purpose. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-009446 [Patent Document 2] Japanese Patent Publication No. 2020-008899 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, in order for a user to create the dictionary they desire, the user must prepare the training data themselves, resulting in the creation of a dictionary from a limited amount of training data. The technology in Patent Document 1 allows the use of a large amount of training data obtained through image conversion when creating a dictionary. While this improves the detection accuracy for the target category, it is likely to result in a dictionary that is general in nature and not specialized for the detection target desired by the user. For example, using images of one's own pet dog as training data and further using a large amount of training data obtained through image conversion improves the detection accuracy for the same dog breed, but it is not easy to create a dictionary specialized for one's own pet dog. Furthermore, the technology in Patent Document 2 does not allow for easy determination of whether the created dictionary is truly necessary, nor can it determine which dictionary is most effective when multiple dictionaries exist.

[0006] The present invention aims to provide dictionary data for subject detection that is suitable for the user's intent. [Means for solving the problem]

[0007] To solve the above problems, the image processing system of the present invention comprises: an image captured of a specific subject; acquisition means for acquiring shooting information at the time of capturing the image; image conversion means for performing image conversion on the image captured and generating a converted image; dictionary data generation means for performing machine learning on the converted image and generating dictionary data which is a trained model for detecting the specific subject in subject detection; and detection result output means for performing subject detection on multiple images using the dictionary data and outputting the detection results to display a detection result display screen for the user to view the detection results and select whether to register the dictionary data in the imaging device. The image conversion means sets the content of the image conversion according to the shooting information. [Effects of the Invention]

[0008] According to the present invention, it is possible to obtain dictionary data for subject detection that is suitable for the user's intent. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the configuration of the image processing system according to Example 1. [Figure 2] This figure shows an example configuration of the imaging device 100 in Example 1. [Figure 3] This figure shows the schematic configuration of the neural network processing unit 205 according to Example 1. [Figure 4] This diagram shows the hardware configuration of server 110. [Figure 5] This flowchart shows the image conversion selection process for Example 1. [Figure 6] This diagram illustrates the method for verifying dictionary data in Example 1. [Figure 7] This figure shows the configuration of the image processing system according to Example 2. [Figure 8] This diagram illustrates the data flow up to the selection of dictionary data in Example 2. [Figure 9] This diagram illustrates the method for verifying dictionary data in Example 2. [Modes for carrying out the invention]

[0010] (Example 1) Figure 1(A) shows the configuration of the image processing system according to Embodiment 1. The image processing system 10 of this embodiment generates a trained model (dictionary data) used to detect a specific subject based on images of a specific subject captured by the imaging device 100. The image processing system 10 performs subject detection based on a neural network and can also detect any subject specified by the user. A typical method for subject detection is called a convolutional neural network (hereinafter abbreviated as CNN). In a CNN, inference processing is performed based on the image signal and dictionary data which are processing parameters, and the dictionary data is generated in advance by machine learning training based on training data.

[0011] The image processing system 10 includes an imaging device 100 and a server 110 as an information processing server. The imaging device 100 and the server 110 are connected by a wireless communication network 120. The communication network 120 is, for example, wireless communication. The communication network 120 may also be implemented by wired communication, or by a combination of wireless and wired communication.

[0012] The configuration of the image processing system 10 is not limited to the configuration of the imaging device 100 and server 110 shown in Figure 1. For example, the system configuration may be such that all processing can be performed within the imaging device 100. Alternatively, the system configuration may be such that the input of training data is performed by another information processing device, such as a mobile terminal. Furthermore, in this embodiment, the case in which the server 110, which is the information processing server, is an image processing device is described as an example, but the system is not limited to this. The image processing device may be an electronic device with imaging capabilities, such as a digital still camera, a digital movie camera, a smartphone with a camera, a tablet with a camera, a network camera, an in-vehicle camera, a drone camera, or a camera mounted on a robot.

[0013] FIG. 1(B) is a diagram showing the software configuration of the image processing system 10. The software modules shown in FIG. 1(B) are the software configurations related to the generation and selection of dictionary data. The software configuration of the imaging device 100 shown in FIG. 1(B) is realized by causing a computer included in the imaging device 100 to execute a computer program stored in a memory as a storage medium. Similarly, the software configuration of the server 110 shown in FIG. 1(B) is realized by causing a computer included in the server 110 to execute a computer program stored in a memory as a storage medium.

[0014] In this embodiment, an image of a specific subject captured as image data is handled. The dictionary data generated in this embodiment is dictionary data for detecting a specific subject in subject detection. In the image processing system 10 of this embodiment, by executing a process of generating dictionary data and a process in which the user selects whether the dictionary data is valid, the user can obtain the desired dictionary data. First, the system configuration and process will be described from the perspective of the process of generating dictionary data.

[0015] The imaging device 100 includes an image data storage unit 101, a teacher data input unit 102, and a shooting information acquisition unit 103. The image data storage unit 101 stores the image data captured by the imaging device 100 and the shooting information (camera information) at the time of image data shooting acquired by the shooting information acquisition unit 103. In this embodiment, image data of a specific subject that the user desires to detect in subject detection and serves as teacher data in the generation of dictionary data is handled. The shooting information includes, for example, at least any one of shooting conditions at the time of shooting, lens information of the used lens, subject information, and shooting time. Then, the image data storage unit 101 transmits the stored image data and shooting information to the teacher data input unit 102.

[0016] The teacher data input unit 102 generates teacher data that sets the image data and the subject area information of the image data in which the target subject exists, based on the image data acquired from the image data storage unit 101. Then, the teacher data input unit 102 transmits the teacher data and the shooting information corresponding to the image data of the teacher data to the server 110. Note that the number of image data (captured images) output from the imaging device 100 and acquired by the server 110 is 1 or more, and is a smaller number compared to the number of images required as teacher data in normal machine learning.

[0017] The server 110 includes an input data acquisition unit 111, a shooting information extraction unit 112, an image conversion information setting unit 113, an image data conversion unit 114, and a dictionary data generation unit 115. The input data acquisition unit 111 acquires teacher data (image data and subject area information) and shooting information from the teacher data input unit 102 of the imaging device 100. Then, the input data acquisition unit 111 outputs the acquired teacher data and shooting information to the shooting information extraction unit 112. Also, the input data acquisition unit 111 outputs the acquired teacher data (image data and subject area information) to the image data conversion unit 114.

[0018] The shooting information extraction unit 112 extracts the shooting information from the data (teacher data and shooting information) acquired from the input data acquisition unit 111. Then, the shooting information extraction unit 112 outputs the extracted shooting information to the image conversion information setting unit 113. The image conversion information setting unit 113 sets the content of the image conversion to be performed by the image data conversion unit 114 according to the shooting information acquired from the shooting information extraction unit 112. Details of the processing in the image conversion information setting unit 113 will be described later. The image conversion information setting unit 113 outputs the set image conversion to the image data conversion unit 114.

[0019] The image data conversion unit 114 performs a predetermined image conversion on the image data of the training data, as set by the image conversion information setting unit 113. In this way, the image conversion information setting unit 113 and the image data conversion unit 114 function as image conversion means. The image data conversion unit 114 outputs the result of the image conversion on the image data of the training data and the subject area information to the dictionary data generation unit 115. The dictionary data generation unit 115 performs machine learning (learning process) using the image conversion result and the subject area information to generate dictionary data, which is a trained model for detecting a specific subject in subject detection. With the above configuration, the image processing system 10 performs a learning process using image data of a specific subject captured by the imaging device 100 as training data and generates dictionary data for detecting a specific subject.

[0020] Next, the system configuration and processing will be explained from the perspective of the process of allowing the user to select whether the generated dictionary data is valid. In this embodiment, in order to obtain dictionary data that can detect a specific subject suitable for the user's intent, subject detection is performed on multiple images using the generated dictionary data, the results are displayed for the user to confirm, and the user's judgment on the dictionary is accepted. Then, only the dictionary data approved by the user is set in the imaging device 100.

[0021] The server 110 has a detection result output unit 116. To verify the performance of the generated dictionary data, the detection result output unit 116 performs subject detection on a large amount of image data stored in the server 110 using the dictionary data and outputs the detection results to the imaging device 100. In this embodiment, as a performance verification of the dictionary data, the user is allowed to view the detection results of subject detection using the dictionary data for multiple images and select whether to register the dictionary data in the imaging device 100, that is, whether to use the dictionary data for subject detection in the imaging device 100. Therefore, the detection result output unit 116 performs subject detection using the dictionary data for multiple images and outputs the detection results in order to display a detection result display screen for the user to check the subject detection results using the dictionary data and make a selection. The output content here includes information indicating whether a specific subject can be detected and the reliability of subject detection for images that can be detected. The dictionary data created by the dictionary data generation unit 115 of the server 110 and the output of the detection result output unit 116 are transmitted to the imaging device 100.

[0022] The imaging device 100 includes a dictionary data and detection result acquisition unit 104, a detection result display unit 105, a dictionary data setting unit 106, and a subject detection unit 107. The dictionary data and detection result acquisition unit 104 communicates with the server 110 to acquire the dictionary data created by the dictionary data generation unit 115 and the detection results output by the detection result output unit 116.

[0023] The detection result display unit 105 displays the detection results from the server 110's detection result output unit 116. By checking the detection results displayed on the detection result display unit 105, including the detectable image and its reliability information, the user can determine whether the created dictionary performance is satisfactory. The detection result display unit 105 functions as a display control means that controls the display of the detection result display screen on a display unit such as a screen. Details of the detection result display screen will be described later with reference to Figure 6. The detection result display unit 105 also accepts the user's selection for the dictionary on the displayed detection result display screen. If the detection result display unit 105 receives a user's determination that the dictionary performance is satisfactory (for example, selection of the OK icon), it sends the dictionary data to the dictionary data setting unit 106. On the other hand, if the detection result display unit 105 receives a user's determination that the dictionary performance is not satisfactory (for example, selection of the NG icon), it communicates with the server 110 and notifies the server 110's image conversion information setting unit 113. Upon receiving notification from the detection result display unit 105, the server 110 can also perform image data conversion to generate a new dictionary.

[0024] If the dictionary data setting unit 106 confirms that the dictionary performance is satisfactory based on the detection result display unit 105, it sets the dictionary data created by the dictionary data generation unit 115 into the imaging device 100. When the imaging device 100 takes a picture, the subject detection unit 107 uses the dictionary data set by the dictionary data setting unit 106 to detect the subject. With this configuration, the user can confirm the subject detection performance of the generated dictionary data and set the dictionary data that satisfies the user as the dictionary to be used for subject detection in the imaging device 100.

[0025] Figure 2 shows the configuration of the imaging device 100. The imaging device 100 includes a CPU 201, memory 202, non-volatile memory 203, operation unit 204, neural network processing unit 205, imaging lens 211, imaging unit 212, image processing unit 213, and encoding processing unit 214. Furthermore, the imaging device 100 includes a display control unit 215, a display unit 216 as a display means, a communication control unit 217, a communication unit 218, a recording medium control unit 219, and an internal bus 230. In addition to the configuration shown in Figure 2, the imaging device 100 also has the configuration related to dictionary data shown in Figure 1(B). The CPU 201 to the neural network processing unit 205, the imaging unit 212 to the display control unit 215, the communication control unit 217, and the recording medium control unit 219 are all able to access each other via the internal bus 230.

[0026] The CPU (Central Processing Unit) 201 controls the operation of each part (each functional block) of the imaging device 100 via the internal bus 230 by executing computer programs stored in the non-volatile memory 203. Memory 202 is a rewritable volatile memory. Memory 202 temporarily stores computer programs for controlling the operation of each part of the imaging device 100, information such as parameters related to the operation of each part of the imaging device 100, and information received by the communication control unit 217. Memory 202 also temporarily stores images acquired by the imaging unit 212, images and information processed by the image processing unit 213, encoding processing unit 214, etc. Memory 202 has sufficient storage capacity to temporarily store these.

[0027] The non-volatile memory 203 is a memory that can be electrically erased and recorded. Examples of non-volatile memory 203 include EEPROM and hard disks. The non-volatile memory 203 stores computer programs that control the operation of each part of the imaging device 100, as well as information such as parameters related to the operation of each part of the imaging device 100. These computer programs enable the various operations performed by the imaging device 100. Furthermore, the non-volatile memory 203 stores computer programs describing the processing content of the neural network used by the neural network processing unit 205, as well as learned coefficient parameters such as weight coefficients and bias values. Weight coefficients are values ​​that indicate the strength of connections between nodes in a neural network, and bias is a value that provides an offset to the product of the weight coefficients and the input data. The non-volatile memory 203 can hold multiple sets of learned coefficient parameters and computer programs describing the processing of the neural network. Furthermore, the computer program describing the neural network processing used in the neural network processing unit 205 and the trained coefficient parameters may be temporarily stored in memory 202 instead of non-volatile memory 203. The computer program describing the neural network processing and the trained coefficient parameters correspond to the dictionary data for subject detection.

[0028] The control unit 204 provides a user interface for operating the imaging device 100. The control unit 204 includes various buttons such as a power button, a menu button, a shutter release button for shooting, a video recording button, and a cancel button, and these buttons are composed of switches, touch panels, etc. The CPU 201 controls the imaging device 100 according to user instructions input via the control unit 204. Although this explanation uses the case where the CPU 201 controls the imaging device 100 based on operations input via the control unit 204 as an example, it is not limited to this. For example, the CPU 201 may control the imaging device 100 based on requests input via the communication unit 218 from a remote controller (not shown) or a mobile terminal.

[0029] The neural network processing unit 205 performs inference processing for subject detection based on dictionary data for the image signal acquired by imaging. The neural network processing unit 205 performs the inference processing among the subject detection processing performed by the subject detection unit 107. In this embodiment, subject detection is performed using a CNN (Convolutional Neural Network). In the CNN, inference processing is performed on the image signal based on dictionary data, which is the processing parameter. Details of the processing by the neural network processing unit 205 will be described later with reference to Figure 3.

[0030] The imaging device 100 uses a photographic lens 211 (lens unit) to form an optical image of the subject onto the pixel array of the imaging unit 212. The photographic lens 211 may be integrated with the body (housing, main body) of the imaging device 100, or it may be detachable from the body. The photographic lens 211 includes a lens group including a zoom lens and a focus lens, a lens control unit, an aperture, etc. The photographic lens 211 can function as a zoom means to change the angle of view. The lens control unit of the photographic lens 211 controls the zoom, adjusts the focus, and controls the aperture value (F number) based on control signals transmitted from the CPU 201.

[0031] The imaging unit 212 functions as an acquisition means for sequentially acquiring multiple images. The images acquired by the imaging unit 212 include still images and moving images. For example, the imaging unit 212 may be a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide-semiconductor) image sensor. The imaging unit 212 has a pixel array (not shown) in which photoelectric conversion units (pixels) that convert the optical image of a subject into an electrical signal are arranged in a matrix, i.e., two-dimensionally. The optical image of the subject is formed on this pixel array by the imaging lens 211. The imaging unit 212 outputs the acquired images to the image processing unit 213 and the memory 202.

[0032] The image processing unit 213 performs predetermined image processing on image data output from the imaging unit 212 or image data read from the memory 202. Examples of image processing include dynamic range conversion, interpolation, reduction (resizing), and color conversion. The image processing unit 213 also performs predetermined calculations for exposure control, distance measurement control, etc., using the image data acquired by the imaging unit 212. Based on the calculation results obtained by the image processing unit 213, exposure control, distance measurement control, etc., are performed by the CPU 201. Specifically, AE (Auto Exposure) processing, AWB (Auto White Balance) processing, AF (Auto Focus) processing, etc., are performed by the CPU 201. These shooting controls are performed by referring to the subject detection results by the neural network processing unit 205. The encoding processing unit 214 compresses the size of the image data by performing intra-frame predictive coding (in-screen predictive coding), inter-frame predictive coding (inter-screen predictive coding), etc., on the image data from the image processing unit 213.

[0033] The display control unit 215 controls the display on the display unit 216. The display control unit 215 generates an image that can be displayed on the display screen of the display unit 216 and outputs the image, i.e., the image signal, to the display unit 216. In addition to outputting image data to the display unit 216, the display control unit 215 can also output image data to external devices such as monitors via the communication control unit 217. The display unit 216 displays the captured image on its display screen based on the image signal sent from the display control unit 215, and also displays data related to the image and detection results sent from the server 110.

[0034] The display unit 216 displays the image signal sent from the display control unit 215. The display unit 216 is, for example, a liquid crystal display, an organic EL display, etc. The display unit 216 may also include a touch panel. If the display unit 216 includes a touch panel, it can also function as an operation unit 204. In addition, the display control unit 215 and the display unit 216 also function as a detection result display unit 105.

[0035] Furthermore, the display unit 216 is equipped with an OSD (On Screen Display) function, which allows for the display of setting screens such as menus on the display screen. The display control unit 215 can superimpose an OSD image onto the image signal and output an image signal to the display unit 216. For example, the display control unit 215 can generate a subject frame as an object region based on the subject detection result from the neural network processing unit 205, superimpose it onto the image signal, and display it on the display unit 216.

[0036] The communication control unit 217 is controlled by the CPU 201 and controls communication between the imaging device 100 and external devices. The communication control unit 217 generates a modulated signal conforming to wireless communication standards such as IEEE 802.11 and outputs the modulated signal to the communication unit 218. The communication control unit 217 also receives modulated signals from external devices via the communication unit 218. The communication control unit 217 can also send and receive control signals for video signals. For example, it may control the communication unit 218 to send video signals conforming to communication standards such as HDMI (registered trademark) or SDI (Serial Digital Interface).

[0037] The communication unit 218 communicates with external devices such as the server 110. For example, the communication unit 218 converts video signals and control signals into physical electrical signals and sends and receives them with external devices. In this embodiment, the communication unit 218 communicates with the server 110 via the communication network 120. The communication unit 218 transmits training data to the server 110. The communication unit 218 receives dictionary data for subject detection in the neural network processing unit 205 from the server 110.

[0038] The recording medium control unit 219 controls the writing and reading of data to the recording medium 220. Based on a request from the CPU 201, the recording medium control unit 219 outputs control signals to the recording medium 220 for controlling the recording medium 220. For example, non-volatile memory or magnetic disks can be used as the recording medium 220. The recording medium 220 may or may not be detachable from the imaging device 100. The recording medium 220 stores encoded image data, etc., as files in a format compatible with the file system of the recording medium 220.

[0039] Note that some of the functional blocks shown in Figures 1(B) and 2 are implemented by having the CPU 201, which acts as a computer included in the imaging device 100, execute computer programs stored in a storage medium such as the non-volatile memory 203. However, some or all of these may be implemented in hardware. As hardware, dedicated circuits (ASICs) or processors (reconfigurable processors, DSPs) can be used. Furthermore, functions implemented in hardware can also be implemented, for example, by generating circuits based on data read by an FPGA (Field Programmable Gate Array) from the non-volatile memory 203. Alternatively, a gate array circuit can be formed in a similar manner to an FPGA and implemented in hardware, or by using an ASIC (Application Specific Integrated Circuit).

[0040] Figure 3 shows the configuration of the neural network processing unit 205. The neural network processing unit 205 includes a CPU 301, a multiply-accumulate circuit 302, a DMA (Dynamic Memory Access) 303, and internal memory 304 within the neural core 300. The CPU 301 obtains a computer program describing the processing content of the neural network from a recording medium (e.g., memory 202, non-volatile memory 203, internal memory 304) via the internal bus 230 and executes it. The CPU 301 also controls the multiply-accumulate circuit 302 and the DMA 303.

[0041] The multiply-accumulate (MLP) circuit 302 is a circuit that performs MPL operations in a neural network. The MPL 302 has multiple MPL units, which can perform MPL operations in parallel. The MPL 302 also outputs intermediate data calculated during MPL operations performed in parallel by multiple MPL units to the internal memory 304 via the DMA 303.

[0042] DMA303 is a circuit specialized in data transfer that does not involve the CPU301, and performs data transfer between memory 202 or non-volatile memory 203 and internal memory 304 via the internal bus 230. DMA303 also performs data transfer between the multiply-accumulate circuit 302 and internal memory 304. The data transferred by DMA303 includes computer programs describing the processing content of the neural network, learned coefficient parameters, and intermediate data calculated by the multiply-accumulate circuit 302.

[0043] The internal memory 304 stores computer programs describing the processing details of the neural network, learned coefficient parameters, intermediate data calculated by the multiply-accumulate circuit 302, and the like. The internal memory 304 may also have multiple banks, and the banks may be switched dynamically.

[0044] Figure 4 shows the configuration of server 110. Server 110 includes a CPU 401, memory 402, operation unit 403, recording unit 404, communication unit 405, and neural network processing unit 406. Some of the functional blocks shown in Figure 4 are implemented by having the CPU 401, which acts as a computer included in server 110, execute computer programs stored in the recording unit 404, which acts as a storage medium. However, some or all of these may be implemented in hardware. As hardware, dedicated circuits (ASICs) or processors (reconfigurable processors, DSPs) can be used.

[0045] The CPU 401 controls all processing blocks that make up the server 110 by executing computer programs stored in the recording unit 404. The memory 402 is mainly used as the work area for the CPU 401 and as a temporary data buffer area. The internal bus 407 is a bus for interconnecting each processing block within the server 110.

[0046] The control unit 403 receives user operation instructions. The control unit 403 may be, for example, a keyboard, mouse, buttons, touch panel, or remote control. Operation information input from the control unit 403 is transmitted to the CPU 401, which then executes control of each processing block based on the operation information.

[0047] The recording unit 404 is a recording medium that stores and reads various data based on instructions from the CPU 401. The recording medium can be, for example, an EEPROM, built-in flash memory, a built-in hard disk, or a removable memory card. In addition to computer programs, the recording unit 404 stores input data and training data, dictionary data, and a large amount of image data for verifying dictionary performance, which are used as training data for the neural network processing unit 406.

[0048] The communication unit 405 communicates with external devices. The communication unit 405 has a hardware configuration for performing wireless LAN and wired LAN communication, for example. In the case of wireless LAN, it performs processing using the IEEE802.11n / a / g / b method, for example. The communication unit 405 connects to an external access point via wireless LAN and performs wireless LAN communication with other wireless communication devices via the access point. In the case of wired LAN, the communication unit 405 communicates via an external router or switching hub using an Ethernet cable, etc. The communication unit 405 communicates with external devices, including the imaging device 100, and exchanges information such as training data and dictionary data. In this embodiment, the communication unit 405 communicates with the imaging device 100 via the communication network 120. The communication unit 405 receives training data from the imaging device 100. The communication unit 405 also transmits dictionary data for subject detection in the neural network processing unit 205 of the imaging device 100 to the imaging device 100.

[0049] The neural network processing unit 406 performs training on the neural network using training data acquired via the communication unit 405. Specifically, the neural network processing unit 406 performs training to construct dictionary data corresponding to each of different classes of subjects using the training data. The neural network processing unit 406 includes a GPU (Graphic Processing Unit) and a DSP (Digital Signal Processor), etc. The dictionary data, which is the result of the training process performed by the neural network processing unit 406, is stored in the recording unit 404.

[0050] Figure 5 is a flowchart showing the image conversion process in the dictionary data generation process performed by the server 110. The processes shown in Figure 5 are those performed by the shooting information extraction unit 112, the image conversion information setting unit 113, and the image data conversion unit 114, which are part of the dictionary data generation process. Each of the processes shown in Figure 5 is realized by having the computer included in the server 110 execute a computer program stored in the memory, which serves as a storage medium.

[0051] In S510, the shooting information extraction unit 112 extracts shooting information from the training data and shooting information, and verifies the extracted shooting information. In S520, the image conversion information setting unit 113 determines the content of the image conversion to be performed by the image data conversion unit 114 based on the shooting information. The dictionary data in this embodiment is a dictionary for detecting any object desired by the user. Therefore, since the user also provides the training data for training the dictionary, it is assumed that the amount of training data needed to generate the dictionary data is small. In order to create a better dictionary with a small amount of training data, necessary image conversions are performed based on the shooting information. In this embodiment, an example is described in which the content of the image conversion is determined by the conditions in S521 to S526, but there are many conditions for performing image conversion, and this is not the only example.

[0052] In S521, the image conversion information setting unit 113 determines whether the subject is small or not. Whether the subject is small or not can be determined by the subject area information attached to the training data. The image conversion information setting unit 113 determines that the subject is small if the size of the subject area based on the subject area information is smaller than a predetermined size. On the other hand, the image conversion information setting unit 113 determines that the subject is not small if the size of the subject area is greater than or equal to the predetermined size. If the subject is small, the image conversion information setting unit 113 performs the process in S522. On the other hand, if the subject is not small, the image conversion information setting unit 113 performs the process in S526.

[0053] In S522, the image conversion information setting unit 113 determines whether the subject distance is far or not. Subject distance information is obtained by the calculation processing for distance measurement control performed by the image processing unit 213. If the subject distance is longer than a predetermined distance, the image conversion information setting unit 113 determines that the subject distance is far. On the other hand, if the subject distance is less than or equal to the predetermined distance, the image conversion information setting unit 113 determines that the subject distance is close. If the subject distance is far, the image conversion information setting unit 113 performs the process in S523. On the other hand, if the subject is close, the image conversion information setting unit 113 performs the process in S526.

[0054] In S523, the image conversion information setting unit 113 determines whether the focal length is on the telephoto side. Whether the focal length is on the telephoto side can be determined from the setting information of the shooting lens at the time of shooting. If the focal length at the time of shooting is on the telephoto side of the shooting lens, it is determined to be on the telephoto side. On the other hand, if the focal length at the time of shooting is on the wide-angle side of the shooting lens, it is determined not to be on the telephoto side. If the focal length is on the telephoto side, the image conversion information setting unit 113 performs the process in S524. On the other hand, if the focal length is not on the telephoto side, the image conversion information setting unit 113 performs the process in S526.

[0055] Based on the shooting information confirmed in S521 to S523, if the subject is small, the distance to the subject is far, and the focal length is on the telephoto side, that is, if the subject is far away and cannot be photographed up close, the process in S524 is performed. In S524, the image conversion information setting unit 113 sets scaling conversion as the process to be executed in image data conversion. This is because if the subject is far away and cannot be photographed up close, it is necessary to perform an enlargement process in order to improve dictionary performance. In S525, the image conversion information setting unit 113 sets the image enlargement ratio in the scaling conversion process (enlargement conversion process) based on the size of the subject in the training data.

[0056] If the conditions for performing the scaling conversion process in S524 are not met, in S526, the image conversion information setting unit 113 determines whether to perform other image conversion processes. For example, if the training data consists only of dark images taken at night, brightness conversion processing is set. Also, if the camera is fixed on a tripod in a shaded area during macro photography, color conversion processing is set.

[0057] In S530, the image data conversion unit 114 performs the image conversion processing set by the image conversion information setting unit 113 in S520 on all the training data (group of images). Through these operations, necessary image conversions are performed on the images captured by the user and created as training data, making it possible to bring the final created dictionary data closer to what the user desired.

[0058] In this embodiment, an example was described in which the image conversion information setting unit 113 determines the content of the image conversion in S520 based on three pieces of shooting information: subject size, subject distance, and focal length. However, the embodiment is not limited to this. The shooting information that serves as the criteria for determining the content of the image conversion includes, for example, at least one of the shooting conditions at the time of shooting, lens information of the lens used, subject information, and shooting time. Furthermore, the shooting information may also include subject attributes, shooting location, shooting time, shooting period, detection area, background (periphery) condition, camera status, or combinations thereof. The lens used is, for example, information on the type of lens, such as wide-angle lens, telephoto lens, super telephoto lens, macro lens, or fisheye lens. The shooting location is, for example, information such as indoors, outdoors, under sunlight, or under fluorescent light. The camera status is, for example, information such as tripod, handheld, panning, or subject tracking. The subject attributes are, for example, categories of subjects such as people, types of animals (e.g., dogs, cats, birds), vehicles, flowers, insects, buildings, food, or landscapes. The shooting conditions at the time of shooting include data at the time of image capture, such as the lens focal length, aperture value, and subject distance.

[0059] Furthermore, while an example has been described in which scaling is performed as an image data conversion process set by the image conversion information setting unit 113 and executed by the image data conversion unit 114, the process is not limited to this. Image data conversion processes may include, for example, brightness conversion, color conversion, rotation, shearing, deformation, contrast adjustment, gamma conversion, smoothing, noise addition, inversion, or combinations thereof.

[0060] Next, the images displayed on the detection result display unit 105 will be described. It should be assumed that the detection result display unit 105 also has a touch panel function. Figure 6 shows an example of the screen displayed on the detection result display unit 105. The detection result display screen 600 displays the results of subject detection performed on a large amount of image data stored in the server 110 using the created dictionary data, as well as icons for the user to input their evaluation of the dictionary's performance.

[0061] The dictionary data generation unit 115, having obtained the image transformation results and subject area information from the image data transformation unit 114 performed on the training data through the process shown in Figure 5, uses these to perform a learning process and generate dictionary data. Then, the detection result output unit 116 uses the dictionary data generated by the dictionary data generation unit 115 to perform subject detection on multiple image data stored in the server 110. The subject detection results are classified according to the confidence level of the detection result, and the detection result display screen 600 displays the multiple images on which subject detection was performed classified into multiple confidence levels. For example, the subject detection results are classified into images with a high confidence level where the desired subject was detected, images with a low confidence level where the desired subject was detected, and images where the desired subject could not be detected (excluded images). On the detection result display screen 600, as information indicating the confidence level, images with a high confidence level are displayed as high confidence level 604, images with a low confidence level as low confidence level 605, and images where the desired subject could not be detected as cluttered 607. Arrows 602 and 603 indicate that you can scroll through the subject detection results screen to view images of other subject detection results. In addition to the detected image and confidence level, you may also display a frame indicating the area of ​​the detected subject, or display the detected subject itself.

[0062] The user can view the results of subject detection using the created dictionary data on the detection result display screen 600 to check the dictionary's performance (usefulness), thereby confirming whether the generated dictionary data for subject detection meets the user's expectations. The detection result display screen 600 also displays a display (for example, an OK / NG icon 601) for the user to input the result of their selection regarding the detection performance of the created dictionary, and the user inputs the result of checking the dictionary performance. In other words, the detection result display screen 600 displays icons to accept the user's selection regarding the dictionary data after viewing the detection results. If the user is satisfied that the generated dictionary data can perform the subject detection they desire after reviewing the image on the detection result display screen 600, they select the OK icon. On the other hand, if they are not satisfied with the subject detection performance of the generated dictionary data, they select the NG button. When the dictionary data setting unit 106 receives the user's selection of the OK icon, it sets the dictionary data as the dictionary data to be used for subject detection in the imaging device 100, making the dictionary data available for use during subject detection in the imaging device 100. While this example demonstrates how to evaluate the performance of newly generated dictionary data, it is not limited to this method; similar methods can also be used to check the performance of existing dictionary data.

[0063] In this embodiment, an example was described in which the detection result output unit 116 performs subject detection on multiple image data uploaded to the server 110 using dictionary data. However, the multiple image data may be obtained from the cloud. Furthermore, although an example was described in which the dictionary data generation process in the image processing system is implemented on the server 110, it may also be implemented using a virtual machine (cloud service) that utilizes resources provided by a data center including the server.

[0064] As described above, according to this embodiment, for images of a specific subject, image transformation is performed according to the shooting information at the time of shooting, and machine learning is performed using this as training data to generate dictionary data for detecting the specific subject. Furthermore, the results of subject detection performed on multiple image data using the generated dictionary data are displayed, allowing the user to check whether the dictionary data is in line with the user's intentions. The system then accepts the user's judgment on the usefulness of the dictionary data and sets the system to use only the dictionary data that has been determined to be in line with the user's intentions for subject detection in the imaging device. This makes it possible to obtain dictionary data for subject detection that is suitable for the user's intentions.

[0065] (Example 2) Example 2 describes an example in which multiple training data are generated by image transformation from a single image or a number of images smaller than the number usually required as training data, multiple dictionary data are generated, and the optimal dictionary is selected from among them. Therefore, in this example, multiple training data are secured by performing multiple different image transformations on a single image data. In addition, in order to allow the user to select the dictionary data to be set in the imaging device 700 while comparing the detection results from multiple dictionary data, multiple images used for subject detection in this case are carefully selected in advance according to the category of the subject. In the following, the same reference numerals are used for components that are the same as in Example 1 and their explanation is omitted, and only the parts that differ from Example 1 are described.

[0066] Figure 7 shows the configuration of the image processing system according to Embodiment 2. The image processing system 70 of Embodiment 2 includes an imaging device 700, a server 710 as an information processing server, and a display device 720. The imaging device 700 and the server 710, and the imaging device 700 and the display device 720 are connected by a communication network. The server 710 and the display device 720 may also be connected by a communication network. The communication network is, for example, wireless communication, but is not limited to this. The display device 720 may also be provided by the imaging device 700.

[0067] The software configuration of the imaging device 700 shown in Figure 7 is realized by having the computer included in the imaging device 700 execute a computer program stored in memory, which serves as a storage medium. Similarly, the software configuration of the server 710 shown in Figure 7 is realized by having the computer included in the server 710 execute a computer program stored in memory, which serves as a storage medium. The software configuration of the display device 720 shown in Figure 7 is realized by having the computer included in the display device 720 execute a computer program stored in memory, which serves as a storage medium.

[0068] The server 710 includes an input data acquisition unit 111, a shooting information extraction unit 112, an image conversion information setting unit 113, an image data conversion unit 711, a dictionary data generation unit 712, an image category determination unit 713, a confirmation image selection unit 714, and a detection result determination unit 715. The server 710 acquires multiple sets of training data and shooting information corresponding to the image data of the training data from the imaging device 700. In this embodiment, the number of image data (captured images) output from the imaging device 700 and acquired by the server 710 is one or more, which is a small number compared to the number of images required as training data in normal machine learning. Therefore, in this embodiment, in order to obtain multiple training data from captured images, multiple different image conversions are performed on the captured images.

[0069] The image category determination unit 713 determines the category of the input image. Categories are pre-set based on, for example, attribute classifications. By utilizing multiple dictionary data already present in the server 710, it is possible to roughly determine what category the subject of the input image belongs to. For example, the image category determination unit 713 can determine whether the subject of the input image is an animal or a vehicle. In addition, subject categories (attributes) include people, types of animals (e.g., dogs, cats, birds, etc.), vehicles, flowers, insects, buildings, food, landscapes, etc. The image category determination unit 713 outputs the determined image category result to the confirmation image selection unit 714.

[0070] The confirmation image selection unit 714 selects multiple images as confirmation images that include subjects belonging to the same category as a specific subject targeted for a dictionary created from a large number of images in the server 710, based on the image category determination unit 713. By selecting confirmation images based on category, the target images for subject detection testing using the created dictionary can be identified to some extent, thereby shortening the detection test time. Alternatively, instead of category determination by the image category determination unit 713, it is also possible to accept the user's specification of image categories from an operation unit (not shown).

[0071] The image conversion information setting unit 113 determines the processing details of multiple image conversions for a single training data set. For example, when performing scaling conversion, multiple different scaling ratios are set. Alternatively, multiple image conversion processes such as scaling conversion and color conversion may be combined. The image data conversion unit 711 creates multiple converted image sets from a single training data set by performing multiple different image conversions on that single training data set. The dictionary data generation unit 712 creates multiple dictionaries from the multiple converted image sets. The detection result determination unit 715 then performs subject detection on multiple target images using each of the multiple dictionaries created by the dictionary data generation unit 712, and outputs multiple detection results for each of the multiple dictionaries. The detection result here is the result of subject detection of a specific subject using the dictionary data, and the determination result is whether or not a specific subject was detected and the confidence level of that subject detection. In Example 2, since it is necessary to check the detection results of all multiple dictionaries, selecting and limiting the target images using the confirmation image selection unit 714 is effective in shortening the confirmation time. The detection result determination unit 715 notifies the detection result and determination result of the detection result and determination result of the detection result and determination result of the detection result and determination result of the detection result determination unit 724 in the display device 720 via the detection result display control unit 721.

[0072] The imaging device 700 includes an image data storage unit 101, a training data input unit 102, and a shooting information acquisition unit 103. The imaging device 700 also includes a dictionary data acquisition unit 706, a detection result and judgment result acquisition unit 704, a dictionary data selection result acquisition unit 705, a dictionary data setting unit 707, and a subject detection unit 107. The display device 720 includes a detection result display control unit 721 and a detection result display unit 722. The detection result display unit 722 displays the detection results and judgment results for each of the multiple dictionaries determined by the detection result judgment unit 715. The user can verify the performance of the multiple dictionaries created by checking the images detected using the dictionaries and their reliability information. The user then selects a dictionary to register with the imaging device 700 from among the multiple dictionaries. The display device 720 notifies the imaging device 700 of the user's selection result, i.e., the selected dictionary. The screen displayed by the detection result display unit 722 in this embodiment will be described later with reference to Figure 9.

[0073] The dictionary data selection result acquisition unit 705 acquires the result of the dictionary selected by the user on the screen displayed on the detection result display unit 722 and notifies the dictionary data acquisition unit 706. The dictionary data acquisition unit 706 acquires dictionary data corresponding to the dictionary acquired by the dictionary data selection result acquisition unit 705 from the server 710. In other words, the dictionary data acquisition unit 706 acquires the dictionary data selected by the user from the server 710. Then, the dictionary data setting unit 707 sets the dictionary data acquired from the server 710 in the imaging device 700. The subject detection unit 107 performs subject detection using the set dictionary. This generates multiple dictionary data from a small amount of training data prepared by the user, and the user can check the detection results from the multiple dictionary data and select the dictionary to set in the imaging device 700 from the multiple dictionaries, thereby performing the subject detection desired by the user. If the user selects two or more dictionary data from among the multiple dictionary data, the dictionary data acquisition unit 706 may select one of them and acquire the dictionary data from the server 710. For example, the dictionary data acquisition unit 706 selects the dictionary with the most reliable detection results from among the dictionaries selected by the user.

[0074] In this embodiment, an example was described in which the confirmation image selection unit 714 selects a target image from multiple image data uploaded to the server 110. However, the multiple image data may be obtained from the cloud. Furthermore, although an example was described in which the dictionary data generation process in the image processing system is implemented on the server 710, this may also be implemented using a virtual machine (cloud service) that utilizes resources provided by a data center including the server.

[0075] Figure 8 illustrates the data flow from the user-inputted training data to the selection of the required dictionary. Training data 800 is user-set training data acquired by the input data acquisition unit 111. The image data conversion unit 711 performs multiple different image conversions 801 ("image conversion 1 to 3") set by the image conversion information setting unit 113 on training data 800, which is "training data 1," thereby generating multiple image-converted training data ("training data 2 to 4"). Training data 800 and the image sets of multiple training data obtained by image conversion of training data 800 are treated as multiple training data 802.

[0076] The dictionary data generation unit 712 performs a learning process on each of the multiple training data 802 and generates multiple dictionary data 803. The learning timing 811 indicates the timing of learning by the dictionary data generation unit 712. "Dictionary data 1 to 4" are generated from "training data 1 to 4" respectively. The detection result determination unit 715 uses each of the generated dictionary data 803 to perform subject detection in the image and outputs a detection result 804. The dictionary performance confirmation timing 812 indicates the timing of the detection result determination unit 715 to determine the detection result using the dictionary. By performing subject detection using "dictionary data 1 to 4", "detection results 1 to 4" are obtained respectively. In addition to the detection result, the detection determination result is also output.

[0077] The detection results 804, which are the results of subject detection using multiple dictionaries, are displayed on the display device 720, and the user selects the dictionary to use from among the multiple dictionaries. The dictionary selection timing 813 indicates the timing when the user selects a dictionary and the imaging device 700 acquires the selection result. The dictionary data acquisition unit 706 of the imaging device 700 acquires (downloads) the selected dictionary data 805 from the server 710. Then, the dictionary data setting unit 707 sets the selected dictionary data 805 in the imaging device 700. When the imaging device 700 detects a subject, the set dictionary data 805 is used.

[0078] Next, the screen displayed on the detection result display unit 722 will be described. Here, it is assumed that the detection result display unit 722 also has a touch panel function. Figure 9 is a diagram showing an example of the screen displayed on the detection result display unit 722. Figure 9(A) and Figure 9(B) show different field sizes (screen sizes) of the detection result display unit 722 that displays the detection result display screen. Figure 9(A) shows an example of the detection result display screen when the detection result display unit 722 is displayed on a large screen such as a monitor. The detection result display screen 920 displays the results of subject detection performed using multiple dictionary data created for multiple image data selected as target images by the server 710, and buttons for the user to input the results of their evaluation of the dictionary performance.

[0079] Image 900, displayed as a result of subject detection, overlays the confidence level 903 of the detection result, which was obtained by performing subject detection on the target image using dictionary 4, onto the target image. Images in which the subject targeted by the dictionary data could not be detected are displayed as "NG," as shown in image 904. The up and down arrows 901 indicate that the image can be scrolled. By scrolling up and down, the user can view all images whose detection results were checked using the dictionary. In addition to the target image and confidence level, a frame indicating the area of ​​the detected subject or the detected subject itself may also be displayed.

[0080] The user can view the results of subject detection using multiple created dictionary data on the detection result display screen 920 to check the performance of each dictionary and confirm whether dictionary data for subject detection that meets the user's expectations has been generated. The detection result display screen 920 has a display (for example, OK / NG buttons 902) for each dictionary to input the result of the user's judgment on the detection performance of the created dictionary, and the user inputs the result of checking the dictionary performance. After checking the image on the detection result display screen 920, the user selects the OK button if they are satisfied that the generated dictionary data is able to perform the subject detection they desire. On the other hand, if they are not satisfied with the subject detection performance of the generated dictionary data, they select the NG button. When the dictionary data acquisition unit 706 receives the user's selection of the OK button, it acquires the corresponding dictionary data from the server 710, and the dictionary data setting unit 707 sets the acquired dictionary data in the imaging device 700 so that the dictionary data can be used in the imaging device 700. The user may be allowed to make judgments for each dictionary individually, or they may be allowed to select only the most suitable dictionary (OK).

[0081] Figure 9(B) shows an example of a detection result display screen where the detection result display unit 722 is displayed on a small screen, such as an LCD screen attached to the imaging device 700. The detection result display screen 930 displays some of the results of subject detection performed on multiple image data selected as target images by the server 710 using multiple dictionary data created, as well as buttons for the user to input the results of their evaluation of the dictionary's performance. While the detection result display screen 920, displayed on a large screen, could display the detection results from dictionaries 1 to 4 side by side, the detection result display screen 930, displayed on a small screen, displays only some of dictionaries 1 to 4, for example, the detection results from dictionaries 2 and 3. The up and down arrows 913 indicate that the image can be scrolled up and down. The left and right arrows 912 indicate that the image can be scrolled left and right. If the LCD screen is small, displaying all the detection results would make the image too small, so the left and right arrows 912 allow the user to select which dictionaries to display. The user can view all images whose detection results have been checked using the dictionaries by scrolling up, down, left, and right.

[0082] The image 910 displayed as a result of subject detection has the confidence level 914 of the detection result, obtained by performing subject detection on the target image using dictionary 2, superimposed on the target image. The detection result display screen 930 then displays a display (e.g., OK / NG button 911) for each dictionary to input the result of the user's judgment on the detection performance of the created dictionary, and the user inputs the result of the dictionary performance judgment. If an NG judgment is entered, the detection result display control unit 721 hides that dictionary on the detection result display screen 930. This makes it possible to select the optimal dictionary while comparing the detection results of multiple dictionaries even on a small screen.

[0083] While an example has been described of changing the number of detection results displayed at once on the detection result display screen 930 according to the field of view of the detection result display unit 722, this is not the only example. For example, the display of detection results may be changed according to the resolution of the detection result display unit 722. In addition to the number of detection results displayed, the display content (e.g., whether or not the confidence level is displayed, the subject detection frame) may also be changed to match the detection result display unit 722. For example, when the resolution is low, the display content and the number of displayed results can be reduced compared to when the resolution is high, making it easier to view the detection results. In this case as well, by scrolling the screen to switch the display content, other detection results can be displayed, and the user can check all results by switching screens. Furthermore, although this was described as a means of selecting the optimal dictionary from among multiple dictionaries that have been created, it is also an effective means when trying to select a specific dictionary from among multiple dictionaries that already exist.

[0084] As explained above, in Example 2, multiple dictionaries can be generated from a single training data set, and the best one can be selected from among them. Furthermore, by making it easy to compare the detection results from the multiple dictionaries created, it becomes possible to create dictionaries that better match the user's intent, even when the training data is limited. In addition, although we have described an example of generating multiple dictionaries from a single training data set, it is also possible to generate multiple dictionaries from a small amount of training data and select the best one from among them.

[0085] This embodiment includes the following configuration of an image processing system. (Composition 1) An image captured of a specific subject, and an acquisition means for acquiring shooting information at the time the image was captured, Image conversion means that performs image conversion on the captured image and generates a converted image, A dictionary data generation means that performs machine learning on the converted image and generates dictionary data which is a trained model for detecting a specific subject in subject detection, The system includes a detection result output means that outputs the detection results in order to perform subject detection on multiple images using the dictionary data and to display a detection result display screen for the user to view the detection results and select whether to register the dictionary data in the imaging device, The image processing system is characterized in that the image conversion means sets the content of the image conversion according to the captured information. (Configuration 2) The image conversion means performs multiple different image conversions on a single captured image to generate multiple converted images. The dictionary data generation means performs machine learning on each of the captured image and the plurality of transformed images to generate a plurality of dictionary data. The image processing system according to configuration 1, characterized in that the detection result output means outputs the detection result of subject detection for each of the multiple dictionary data for multiple images. (Composition 3) The image processing system according to configuration 1 or 2, characterized in that the aforementioned shooting information includes at least one of the shooting conditions at the time of shooting, lens information, subject information, and shooting time. (Composition 4) The image processing system according to any one of configurations 1 to 3, characterized in that the number of captured images is small compared to the number of images required as training data in normal machine learning. (Composition 5) The detection results include information indicating the reliability of the subject detection. The image processing system according to any one of configurations 1 to 4, characterized in that the detection result display screen displays the plurality of images that were the target of subject detection using the dictionary data or the subject detected by the subject detection, information indicating the reliability of the subject detection, and an icon for accepting selections from the user who viewed the detection results regarding the dictionary data. (Composition 6) The image processing system according to any one of configurations 1 to 5, further comprising dictionary data setting means for setting dictionary data selected by the user to be registered in the imaging device on the detection result display screen as dictionary data to be used for subject detection in the imaging device. (Composition 7) The image processing system according to configuration 2, characterized in that the plurality of images are images extracted from images uploaded to a server or cloud as images that include subjects belonging to the same category as the specific subject. (Composition 8) A detection result output means that performs subject detection on multiple images using dictionary data which is a trained model for detecting a specific subject in subject detection, and outputs the detection result, A display control means that displays a detection result display screen on a display unit for the user to select whether to register the dictionary data in the imaging device while viewing the detection result, based on the detection result, An image processing system characterized by having setting means for setting the dictionary data in the imaging device according to the result of the selection by the user. (Composition 9) The detection results include information indicating the reliability of the subject detection. The image processing system according to configuration 8, characterized in that the detection result display screen displays the multiple images that were the target of subject detection using the dictionary data or the subject detected by the subject detection, information indicating the reliability of the subject detection, and an icon for accepting selections from the user who viewed the detection results regarding the dictionary data. (Composition 10) A determination means for determining the category to which the particular subject belongs, The system includes a confirmation image selection means that selects multiple images from among those uploaded to a server or cloud, which include subjects belonging to the same category as the specific subject, as the confirmation target images. The image processing system according to configuration 8 or 9, characterized in that, if there are multiple dictionary data, the detection result output means performs subject detection on the target image using the multiple dictionary data and outputs a detection result for each of the multiple dictionary data. (Composition 11) The image processing system according to any one of configurations 8 to 10, characterized in that the number of detection results displayed on the detection result display screen and the content of the display are changed according to the size of the field of view or the resolution of the display unit.

[0086] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0087] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of Symbols]

[0088] 100 Imaging device 102 Teacher Data Input Section 103 Shooting Information Acquisition Unit 104 Dictionary data and detection result acquisition unit 105 Detection result display unit 106 Dictionary Data Setting Section 107 Subject detection unit 110 Servers 111 Input data acquisition unit 112 Shooting Information Extraction Unit 113 Image conversion information setting section 114 Image Data Conversion Unit 115 Dictionary Data Generation Unit 116 Detection result output section

Claims

1. An image captured of a specific subject, and acquisition means for acquiring shooting information at the time the image was captured, Image conversion means that performs image conversion on the captured image and generates a converted image, A dictionary data generation means that performs machine learning on the converted image and generates dictionary data which is a trained model for detecting a specific subject in subject detection, The system includes a detection result output means that outputs the detection results in order to perform subject detection on multiple images using the dictionary data and display a detection result display screen for the user to view the detection results and select whether to register the dictionary data in the imaging device, The image processing system is characterized in that the image conversion means sets the content of the image conversion according to the captured information.

2. The image conversion means performs multiple different image conversions on a single captured image to generate multiple converted images. The dictionary data generation means performs machine learning on each of the captured image and the plurality of transformed images to generate a plurality of dictionary data. The image processing system according to claim 1, characterized in that the detection result output means outputs the detection result of subject detection for each of the plurality of dictionary data for a plurality of images.

3. The image processing system according to claim 1, characterized in that the aforementioned shooting information includes at least one of the shooting conditions at the time of shooting, lens information, subject information, and shooting time.

4. The image processing system according to claim 1, characterized in that the number of captured images is small compared to the number of images required as training data in normal machine learning.

5. The detection results include information indicating the reliability of the subject detection. The image processing system according to claim 1, characterized in that the display screen of the detection results displays the plurality of images that were the target of subject detection using the dictionary data or the subject detected by the subject detection, information indicating the reliability of the subject detection, and an icon for accepting selections from the user who viewed the detection results regarding the dictionary data.

6. The image processing system according to claim 1, further comprising dictionary data setting means for setting the dictionary data selected by the user to be registered in the imaging device on the detection result display screen as dictionary data to be used for subject detection in the imaging device.

7. The image processing system according to claim 2, characterized in that the plurality of images are images extracted from images uploaded to a server or cloud as images that include a subject belonging to the same category as the specific subject.

8. A detection result output means that performs subject detection on multiple images using dictionary data which is a trained model for detecting a specific subject in subject detection, and outputs the detection result, A display control means that displays a detection result display screen on a display unit for the user to select whether to register the dictionary data in the imaging device while viewing the detection result, based on the detection result, An image processing system characterized by having setting means for setting the dictionary data in the imaging device according to the result of the selection by the user.

9. The detection results include information indicating the reliability of the subject detection. The image processing system according to claim 8, characterized in that the detection result display screen displays the plurality of images that were the target of subject detection using the dictionary data or the subject detected by the subject detection, information indicating the reliability of the subject detection, and an icon for accepting selections from the user who viewed the detection results regarding the dictionary data.

10. A determination means for determining the category to which the aforementioned specific subject belongs, The system includes a confirmation image selection means that selects multiple images from among those uploaded to a server or cloud, which include subjects belonging to the same category as the specific subject, as confirmation target images. The image processing system according to claim 8, characterized in that, if there are multiple dictionary data, the detection result output means performs subject detection on the target image using each of the multiple dictionary data and outputs the detection result for each of the multiple dictionary data.

11. The image processing system according to claim 8, characterized in that the number and content of detection results displayed on the detection result display screen are changed according to the size of the field of view or the resolution of the display unit.

12. A method for controlling an image processing system, A process of acquiring an image of a specific subject and the shooting information at the time the image was captured, The process involves performing image transformation on the captured image to generate a transformed image, The process involves performing machine learning on the converted image to generate dictionary data which is a trained model for detecting the specific subject in subject detection, The process includes a step of performing subject detection on multiple images using the dictionary data, and outputting the detection results to display a detection result display screen for the user to view the detection results and select whether to register the dictionary data in the imaging device, The control method for an image processing system is characterized in that the content of the image conversion is set according to the image capture information.

13. A method for controlling an image processing system, The process involves performing subject detection on multiple images that are to be reviewed, using a dictionary data which is a trained model for detecting specific subjects in subject detection, and outputting the detection results. A step of displaying a detection result screen on the display unit for the user to select whether to register the dictionary data in the imaging device while viewing the detection result, based on the detection result, A control method for an image processing system, characterized by comprising the step of setting the dictionary data in the imaging device according to the result of the selection by the user.

Images