Image processing apparatus, image pickup apparatus, and method for controlling the same

Abstract

An image processing apparatus is characterized by comprising: a first acquisition means for acquiring first teacher data including image data for a positive instance of a first object; a second acquisition means for acquiring second teacher data including image data for a negative instance of the first object; and a generation means for generating first dictionary data for detecting the first object on the basis of the teacher data acquired by the first acquisition means and the second acquisition means. The image processing apparatus is further characterized in that the learning means additionally learns second dictionary data to generate the first dictionary data, and the number of first teacher data acquired by the first acquisition means is smaller than the number of second teacher data acquired by the second acquisition means.

Description

Technical Field

[0001] The present invention relates to an image processing device, an imaging device, and a control method of these devices related to a trained model for detecting a specific object by using machine learning. Background Art

[0002] Object detection is a field of computer vision research that has been widely studied. Computer vision is a technology that understands an image that has been input to a computer and automatically recognizes various features of the image. In this technology, object detection is the task of estimating the location and type of objects present in an image. Object detection is used in automatic focusing technology of camera equipment, etc.

[0003] In recent years, image processing devices that detect subjects by machine learning schemes represented by neural networks and the like are known. Such image processing devices detect specific subjects to perform camera control by using a trained model (dictionary data) corresponding to the specific subject. As types of specific subjects, people, animals such as dogs or cats, or vehicles such as cars are representative, and these are subjects that have high demands on the auto focus (AF) function of the camera device.

[0004] Japanese Patent Laid-Open No. 2021-57672 discloses an image processing device that collects images from a user and detects an AF point suitable for the user through additional learning. However, since the images are automatically classified for learning, it is difficult to detect any new subject that the user desires to detect.

[0005] Citation List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2021-57672 Summary of the invention

[0008] Problem that the invention aims to solve

[0009] In addition, in order to generate dictionary data capable of detecting any new subject that the user desires to detect, the user needs to provide teacher data for the subject. However, in some cases, the amount of data that the user can prepare (teacher data of a positive example) is small, and the problem of not acquiring dictionary data with a sufficient level of detection performance occurs.

[0010] Solutions for solving problems

[0011] An object of the present invention is to provide an image processing apparatus capable of generating or using dictionary data having a certain level or higher of detection performance from a small amount of positive example data.

[0012] To achieve the above-mentioned purpose, the image processing device of the present invention includes: a first acquisition component, used to acquire first teacher data including image data of a positive instance for a first object; a second acquisition component, used to acquire second teacher data including image data of a negative instance for the first object; and a generation component, used to generate first dictionary data for detecting the first object based on the teacher data acquired by the first acquisition component and the second acquisition component, the image processing device being characterized in that the learning component performs additional learning on the second dictionary data to generate the first dictionary data, and the amount of first teacher data acquired by the first acquisition component is less than the amount of second teacher data acquired by the second acquisition component.

[0013] Effects of the Invention

[0014] According to the present invention, dictionary data having a certain level or higher of detection performance can be generated or used from a small amount of positive example data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a configuration diagram of an image processing system according to a first embodiment of the present invention.

[0016] Figure 2 is a block diagram showing a configuration example of the image pickup apparatus 100 of the first embodiment.

[0017] Figure 3 : is a block diagram showing a schematic configuration of the neural network processing unit 205 according to the first embodiment.

[0018] Figure 4 is a block diagram showing a hardware configuration example of the server 110 .

[0019] Figure 5 is a block diagram showing a hardware configuration example of the mobile terminal 120 .

[0020] Figure 6 is a flowchart illustrating processing of the image pickup apparatus according to the first embodiment.

[0021] Fig. 7A is a diagram for describing an example of subject detection based on dictionary data.

[0022] Figure 7B is a diagram for describing an example of subject detection based on dictionary data.

[0023] Figure 8 is a flowchart showing the processing of the server according to the first embodiment.

[0024] Fig. 9 is a flowchart for describing the processing flow of dictionary data generation according to the first embodiment.

[0025] Fig.10 is a flowchart showing an example of a flow of processing performed by the mobile terminal 120 according to the first embodiment.

[0026] Fig.11A : is a diagram for describing an input screen example of tutor data and a network structure of the display unit 504 of the mobile terminal according to the first embodiment.

[0027] Fig. 11B : is a diagram for describing an input screen example of tutor data and a network structure of the display unit 504 of the mobile terminal according to the first embodiment.

[0028] Fig. 11C : is a diagram for describing an input screen example of tutor data and a network structure of the display unit 504 of the mobile terminal according to the first embodiment. DETAILED DESCRIPTION

[0029] Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described using embodiments. Note that the same components or elements in the various drawings are assigned the same reference numerals, and repeated explanations will be omitted or simplified.

[0030] In addition, according to the present embodiment, an example of an information processing server applicable as an image processing device will be described. However, the image processing device includes an electronic device with a camera function such as a digital still camera, a digital movie camera, a smartphone with a camera, a tablet with a camera, a web camera, a car camera, a drone camera, or a camera mounted on a robot, etc.

[0031] <First Embodiment>

[0032] Hereinafter, an image processing system according to a first embodiment will be described in detail.

[0033] Figure 1 1 is a configuration diagram of an image processing system according to a first embodiment, and the image processing system is composed of an image pickup apparatus 100, a server 110 serving as an information processing server, and a mobile terminal 120 serving as an information processing terminal, etc. For example, the image pickup apparatus 100 and the server 110 are connected to each other via a wireless communication network. In addition, for example, the server 110 and the mobile terminal 120 are connected to each other via a wireless communication network.

[0034] Notice, Figure 1 The functional blocks in the server 110 and the mobile terminal 120 shown are respectively realized by causing computers respectively included in the server 110 and the mobile terminal 120 to execute computer programs stored in memories serving as storage media.

[0035] The image processing system according to the first embodiment performs subject detection based on a neural network and enables detection of any subject passing through a user. An example of a representative scheme for subject detection includes a scheme called a convolutional neural network (hereinafter, abbreviated as CNN). According to CNN, inference processing is performed based on an image signal and dictionary data as a processing parameter, and dictionary data is pre-generated through a learning process based on teacher data.

[0036] In the image processing system of the first embodiment, the mobile terminal 120 includes a teacher data input unit 121 serving as a teacher data input means for inputting teacher data for subject detection. In addition, the teacher data input unit 121 performs a teacher data input step of inputting teacher data for subject detection.

[0037] Furthermore, the teacher data input unit 121 may input a plurality of sets of teacher data while image data and subject area information of the image data where a target subject exists are prepared as one set of teacher data, and the plurality of sets of teacher data may be transmitted to the server 110 .

[0038] In the server 110, the input data acquisition unit 111 used as the first acquisition component acquires the positive instance teacher data and the negative instance teacher data sent from the mobile terminal 120, and the server-side data acquisition unit 112 used as the second acquisition component acquires a predetermined number of negative instance teacher data from the negative instance data group 113 prepared in advance in the server 110. At this time, the number of data including positive instance data and negative instance data input by the user using the teacher data input unit 121 is set to be equal to or less than 100. Here, it is desired that five or more positive instance teacher data are more preferably input from the user so that the subject desired by the user can be detected. However, when learning is performed by using only 100 or less data input by the user, a sufficient performance level cannot be achieved.

[0039] In view of the above, the server-internal data acquisition unit 112 acquires 101 or more teacher data for learning. Here, for example, the server-internal data acquisition unit 112 uses 10,000 or more negative instance teacher data, so that even if the number of positive instance teacher data from the user is small, learning utilized for compensation and improvement of the subject detection rate can be performed. As a result, sufficient learning can be performed while reducing the load of the data input operation on the user side.

[0040] The custom dictionary data is generated by additionally learning the pre-trained basic dictionary data based on the acquired data in the learning unit 114. After the judgment unit 115 judges whether to further acquire negative example teacher data, the generated custom dictionary data is sent to the camera 100. According to the first embodiment, the learning unit 114 is provided in the server 110 serving as an information processing server different from the camera.

[0041] In the imaging apparatus 100, the dictionary data transmitted from the server 110 is received, and the inference processing based on the neural network is performed by the subject detection unit 101 based on the received dictionary data. Subsequently, based on the inference result, imaging control such as autofocus is performed in the imaging control unit 102. That is, the imaging apparatus 100 performs subject detection based on the dictionary data, and performs predetermined imaging control (such as autofocus or exposure control) on the subject detected by the subject detection.

[0042] Figure 2 1 is a block diagram showing a configuration example of the imaging apparatus 100 of the first embodiment. Figure 2 As shown, the imaging device 100 includes a CPU 201, a memory 202, a nonvolatile memory 203, an operation unit 204, a neural network processing unit 205, an imaging unit 212, an image processing unit 213, and an encoding processing unit 214. The imaging device 100 also includes a display control unit 215, a display unit 216, a communication control unit 217, a communication unit 218, a recording medium control unit 219, and an internal bus 230.

[0043] The imaging apparatus 100 focuses an optical image of a subject on a pixel array of an imaging unit 212 by using a photographing lens 211, but the photographing lens 211 may be non-detachable or detachable from a main body (main body frame or main body) of the imaging apparatus 100. In addition, the imaging apparatus 100 writes and reads image data to and from a recording medium 220 via a recording medium control unit 219. The recording medium 220 may be detachable from the imaging apparatus 100 or non-detachable from the imaging apparatus 100.

[0044] The CPU 201 controls each unit (each functional block) in the imaging apparatus 100 via the internal bus 230 by executing a computer program stored in the nonvolatile memory 203 .

[0045] The memory 202 is a rewritable volatile memory. The memory 202 temporarily records a computer program for controlling the operation of each unit in the imaging apparatus 100, information such as parameters related to the operation of each unit in the imaging apparatus 100, and information received by the communication control unit 217. The memory 202 also temporarily records images acquired by the imaging unit 212 and images and information that have been processed by the image processing unit 213 or the encoding processing unit 214, etc. The memory 202 is provided with sufficient storage capacity to temporarily record these images and information.

[0046] The nonvolatile memory 203 is a memory that can be electrically deleted and recorded, and, for example, an EEPROM or a hard disk or the like is used. The nonvolatile memory 203 stores information such as a computer program for controlling the operation of each unit in the imaging apparatus 100 and parameters related to the operation of each unit in the imaging apparatus 100. According to such a computer program, various operations to be performed by the imaging apparatus 100 are realized. The nonvolatile memory 203 also stores a computer program that is to be used in the neural network processing unit 205 and describes the processing content of the neural network, and trained coefficient parameters such as a weight coefficient or a bias value.

[0047] Note that the weight coefficient is a value indicating the strength of connection between nodes in a neural network, and the bias is a value that provides an offset to the accumulated value of the weight coefficient and input data. The nonvolatile memory 203 can hold a plurality of trained coefficient parameters and a plurality of computer programs that describe the processing of the neural network.

[0048] Note that the memory 202 may temporarily store a plurality of computer programs describing the processing of the neural network and the plurality of trained coefficient parameters described above, which are to be used in the neural network processing unit 205, instead of the memory 203. Note that the computer programs describing the processing of the neural network and the trained coefficient parameters are equivalent to the dictionary data for subject detection.

[0049] The operation unit 204 provides a user interface for operating the imaging apparatus 100. The operation unit 204 includes various buttons such as a power button, a menu button, a release button for shooting, a movie recording button, and a cancel button. The various buttons are composed of switches and a touch panel, etc. The CPU 201 controls the imaging apparatus 100 according to instructions inputted via the operation unit 204 from the user.

[0050] Note that, in this document, the case where the CPU 201 controls the imaging apparatus 100 based on the operation input via the operation unit 204 has been described as an example, but it is not limited thereto. For example, the CPU 201 may control the imaging apparatus 100 based on a request input from the mobile terminal 120 or a remote controller not shown in the figure via the communication unit 218.

[0051] In the neural network processing unit 205, the inference processing of the subject detection unit 101 based on the dictionary data is performed. Figure 3 Describe its details.

[0052] The photographing lens (lens unit) 211 is composed of a lens group including a zoom lens and a focus lens, a lens control unit not shown in the figure, and a diaphragm not shown in the figure. The photographing lens 211 can be used as a zoom component for changing the angle of view. The lens control unit of the photographing lens 211 adjusts the focus and controls the aperture value (F value) based on a control signal sent from the CPU 201.

[0053] The imaging unit 212 can be used as an acquisition component for sequentially acquiring a plurality of images including moving images. For example, a CCD (charge coupled device) image sensor or a CMOS (complementary metal oxide semiconductor) image sensor or the like is used as the imaging unit 212. The imaging unit 212 has a pixel array not shown in the figure, in which photoelectric conversion units (pixels) that convert an optical image of a subject into an electrical signal are arranged in a matrix manner (i.e., in a two-dimensional manner). The shooting lens 211 focuses the optical image of the subject on the pixel array. The imaging unit 212 outputs the captured image to the image processing unit 213 or the memory 202. Note that the imaging unit 212 can also acquire a still image.

[0054] The image processing unit 213 performs predetermined image processing on the image data output from the imaging unit 212 or the image data read from the memory 202. Examples of the image processing include dynamic range conversion processing, interpolation processing, reduction processing (resizing processing), and color conversion processing, etc. The image processing unit 213 also performs predetermined calculation processing for exposure control, distance measurement control, etc. by using the image data acquired by the imaging unit 212.

[0055] Then, based on the calculation result obtained by the calculation processing of the image processing unit 213, exposure control, distance measurement control, etc. are performed by the CPU 201. Specifically, AE (automatic exposure) processing, AWB (automatic white balance) processing, AF (automatic focus) processing, etc. are performed by the CPU 201. The neural network processing unit 205 performs these shooting controls by referring to the subject detection result.

[0056] The encoding processing unit 214 compresses the size of the image data by performing intra prediction encoding (intra-picture prediction encoding) or inter prediction encoding (inter-picture prediction encoding) or the like on the image data from the image processing unit 213 .

[0057] The display control unit 215 controls the display unit 216. The display unit 216 includes a display screen not shown in the figure. The display control unit 215 generates an image that can be displayed on the display screen of the display unit 216, and outputs the image (i.e., image signal) to the display unit 216. In addition, the display control unit 215 can output image data not only to the display unit 216, but also to an external device via the communication control unit 217. The display unit 216 displays an image on the display screen based on the image signal sent from the display control unit 215.

[0058] The display unit 216 includes an OSD (On Screen Display) function, i.e., a function of displaying a setting screen such as a menu on a display screen. The display control unit 215 can superimpose an OSD image on an image signal and output the image signal to the display unit 216. The neural network processing unit 205 can also generate a subject frame based on the subject detection result and display the subject frame by superimposing the subject frame on the image signal. The display unit 216 is composed of a liquid crystal display or an organic EL display, etc., and displays the image signal sent from the display control unit 215. The display unit 216 may include, for example, a touch panel. In the case where the display unit 216 includes a touch panel, the display unit 216 can also be used as the operation unit 204.

[0059] The communication control unit 217 is controlled by the CPU 201. The communication control unit 217 generates a modulation signal conforming to a wireless communication standard such as IEEE802.11, outputs the modulation signal to the communication unit 218, and also receives the modulation signal from an external device via the communication unit 218. In addition, the communication control unit 217 can send and receive a control signal of a video signal. For example, by controlling the communication unit 218, the communication control unit 217 can send a video signal conforming to a communication standard such as HDMI (registered trademark) (High Definition Multimedia Interface) or SDI (Serial Digital Interface).

[0060] The communication unit 218 converts the video signal and the control signal into physical electrical signals for transmission to and reception from an external device. Note that the communication unit 218 not only transmits and receives the video signal and the control signal, but also receives the dictionary data used for subject detection in the neural network processing unit 205, etc.

[0061] The recording medium control unit 219 controls the recording medium 220. The recording medium control unit 219 outputs a control signal for controlling the recording medium 220 to the recording medium 220 based on a request from the CPU 201. For example, a nonvolatile memory or a magnetic disk or the like is used as the recording medium 220. The recording medium 220 may be detachable as described above, or may be non-detachable. The recording medium 220 saves the encoded image data or the like as a file in a format conforming to the file system of the recording medium 220.

[0062] The respective functional blocks 201 to 205 , 212 to 215 , 217 , and 219 can access each other via the internal bus 230 .

[0063] Note that the above operation is realized by causing the CPU 201 serving as a computer included in the image pickup apparatus 100 to execute a computer program stored in the nonvolatile memory 203 serving as a storage medium or the like. Figure 2 However, some or all of these functional blocks may be implemented by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor or DSP) or the like may be used as hardware.

[0064] like Figure 3 As shown, the neural network processing unit 205 includes a CPU 301, a product-sum operation circuit 302, a dynamic memory access (DMA) 303, and an internal memory 304 in the neural core 300.

[0065] The CPU 301 acquires a computer program describing the processing contents of the neural network from the memory 202 or the nonvolatile memory 203 or from the internal memory 304 via the internal bus 230 and executes the computer program. The CPU 301 also controls the product-sum operation circuit 302 and the DMA 303 .

[0066] The product-sum operation circuit 302 is a circuit for performing product-sum operation in a neural network. The product-sum operation circuit 302 includes a plurality of product-sum operation units, and these product-sum operation units can perform product-sum operation in parallel. In addition, the product-sum operation circuit 302 outputs intermediate data calculated when the product-sum operation is performed in parallel by the plurality of product-sum operation units to the internal memory 304 via the DMA 303.

[0067] The DMA 303 is a circuit dedicated to data transfer without the mediation of the CPU 301, and performs data transfer between the memory 202 or the nonvolatile memory 203 and the internal memory 304 via the internal bus 230. In addition, the DMA 303 performs data transfer between the product-sum operation circuit 302 and the internal memory 304. Examples of data to be transferred by the DMA 303 include a computer program describing the processing contents of a neural network, trained coefficient parameters, and intermediate data calculated by the product-sum operation circuit 302, and the like.

[0068] The internal memory 304 stores a computer program describing the processing contents of the neural network, trained coefficient parameters, and intermediate data calculated by the product-sum operation circuit 302, etc. In addition, the internal memory 304 may include a plurality of memory banks, and may dynamically switch between the memory banks.

[0069] Figure 4 is a block diagram showing a hardware configuration example of the server 110 .

[0070] like Figure 4 As shown, the server 110 includes a CPU 401, a memory 402, a display unit 403, an operation unit 405, a recording unit 406, a communication unit 407 and a neural network processing unit 408.

[0071] Note that the above is realized by causing the CPU 401 serving as a computer included in the server 110 to execute a computer program stored in the recording unit 406 serving as a storage medium or the like. Figure 4 However, part or all of these functional blocks may be implemented by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor or DSP) or the like may be used as hardware.

[0072] The CPU 401 controls all processing blocks constituting the server 110 by executing a computer program stored in the recording unit 406 .

[0073] The memory 402 is a memory mainly used as a work area of ​​the CPU 401 or a temporary buffer area of ​​data.

[0074] The display unit 403 is composed of a liquid crystal panel, an organic EL panel, or the like, and displays an operation screen or the like based on a command from the CPU 401 .

[0075] The internal bus 404 is a bus used to connect various processing blocks in the server 110 to each other.

[0076] The operation unit 405 is composed of a keyboard, a mouse, a button, a touch panel, or a remote controller, etc., and accepts operation instructions from the user. Operation information input from the operation unit 405 is sent to the CPU 401, and the CPU 401 performs control on each processing block based on the operation information.

[0077] The recording unit 406 is composed of a recording medium, and is a processing block that stores various data in the recording medium or reads various data from the recording medium based on the instruction of the CPU 401. For example, the recording medium is composed of an EEPROM, a built-in flash memory, a built-in hard disk, or a removable memory card, etc. In addition to the computer program, input data and teacher data as data for learning in the neural network processing unit 408, and dictionary data, etc. are stored in the recording unit 406.

[0078] The communication unit 407 includes hardware configured to perform communication of a wireless LAN and a wired LAN, etc. For example, in a wireless LAN, processing of the IEEE802.11n / a / g / b standard is performed. The communication unit 407 is connected to an external access point via a wireless LAN, and performs wireless LAN communication with other wireless communication devices via the access point. In addition, the communication unit 407 communicates in a wired LAN via an external router or a switching hub via an Ethernet cable, etc. The communication unit 407 communicates with an external device including the camera apparatus 100 to exchange information such as teacher data or dictionary data.

[0079] The neural network processing unit 408 performs a learning process of the neural network using the teacher data acquired via the communication unit 407. The neural network processing unit 408 is equivalent to Figure 1 The learning unit 114 in the embodiment of the present invention performs a learning process for developing dictionary data corresponding to respective subjects in different categories by using the teacher data.

[0080] The neural network processing unit 408 is constituted by a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor), etc. In addition, dictionary data as a result of the learning process performed by the neural network processing unit 408 is held in the recording unit 406 .

[0081] Figure 5 is a block diagram showing a hardware configuration example of the mobile terminal 120 .

[0082] like Figure 5As shown, the mobile terminal 120 includes a CPU 501, a memory 502, a camera unit 503, a display unit 504, an operation unit 505, a recording unit 506, a communication unit 507, and an internal bus 508. The CPU 501 used as a computer included in the mobile terminal 120 executes a computer program stored in the recording unit 506 used as a storage medium, etc. Figure 5 However, some or all of these functional blocks may be implemented by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor or DSP) or the like may be used as hardware.

[0083] The CPU 501 controls all processing blocks constituting the mobile terminal 120 by executing a computer program stored in the recording unit 506 .

[0084] The memory 502 is a memory mainly used as a work area or a temporary buffer area for data of the CPU 501. A program such as an OS (Operating System) or application software is loaded on the memory 502 and executed by the CPU 501.

[0085] The imaging unit 503 includes an optical lens, a CMOS sensor, a digital image processing unit, etc., and captures an optical image input via the optical lens to obtain captured image data by converting the optical image into digital data. The captured image data acquired by the imaging unit 503 is temporarily stored in the memory 502, and is processed based on the control of the CPU 501. For example, recording is performed in a recording medium by the recording unit 506 or transmission is performed to an external device by the communication unit 507. The imaging unit 503 also includes a lens control unit, and performs control such as zooming, focusing, or aperture adjustment based on a command from the CPU 501.

[0086] The display unit 504 is composed of a liquid crystal panel or an organic EL panel, etc., and performs display based on the instruction of the CPU 501. The image of the teacher data is selected from the captured image, and the operation screen or the captured image is also displayed to specify the network structure.

[0087] The operation unit 505 is composed of a keyboard, a mouse, a button, a four-way operation key, a touch panel or a remote controller, etc., and accepts user operation instructions. The operation information input from the operation unit 505 is sent to the CPU 501, and the CPU 501 performs control on each processing block based on the operation information.

[0088] The recording unit 506 is composed of a large-capacity recording medium and is a processing block that stores and reads various data in and from the recording medium based on instructions of the CPU 501. For example, the recording medium is composed of a built-in flash memory, a built-in hard disk, a removable memory card, or the like.

[0089] The communication unit 507 includes processing hardware configured to perform communication via an antenna, wireless LAN, or wired LAN, etc. For example, the communication unit 507 performs wireless LAN communication of the IEEE802.11n / a / g / b standard. The communication unit 507 is connected to an external access point via a wireless LAN, and performs wireless LAN communication with other wireless communication devices via the access point. Through the communication unit 507, teacher data or a network structure that has been input from a user via the operation unit 505 is sent to the server 110.

[0090] The internal bus 508 is a bus configured to connect the respective processing blocks in the mobile terminal SP to each other.

[0091] Figure 6 is a flowchart showing the processing of the imaging apparatus according to the first embodiment, and refers to Figure 6 , the flow of processing for receiving dictionary data to be executed in the imaging apparatus 100 and performing subject detection to execute imaging control in the first embodiment will be described. This operation is realized when a computer program stored in the nonvolatile memory 203 is loaded onto the memory 202 and the CPU 201 reads and executes the computer program in the memory 202 in a state where the power of the imaging apparatus 100 is turned on.

[0092] In step S601, the imaging apparatus 100 checks the server 110 via the communication unit 218 whether or not there is dictionary data which has not been received from the server 110. When the dictionary data which has not been received from the server 110 exists in the server 110 (in step S601, it is determined as "Yes"), in step S602, the dictionary data is acquired from the server 110 via the communication unit 218 to be stored in the nonvolatile memory 203. When there is no dictionary data which has not been received from the server 110 (in step S601, it is determined as "No"), the flow proceeds to step S603.

[0093] In step S603, subject detection is performed by the neural network processing unit 205 by using the dictionary data recorded in the nonvolatile memory 203. The dictionary data may be copied from the nonvolatile memory 203 to the memory 202 or the internal memory 304 of the neural network processing unit 205 to be used in subject detection. In addition, subject detection in step S603 is performed by using the image data acquired by the imaging unit 212 as input data.

[0094] In step S604, based on the subject detection result, imaging control such as auto focus is performed by the imaging unit 212. That is, the focus is on the detected subject, and imaging control such as auto focus and exposure control is also performed so that appropriate exposure is established. Here, steps S603 and S604 serve as imaging steps of performing subject detection based on dictionary data and performing predetermined imaging control on the subject detected by subject detection.

[0095] According to the present embodiment, a step for acquiring dictionary data from a server, subject detection based on the acquired dictionary data, and shooting control are performed in the same process. However, the configuration is not limited to this, and for example, a mode or timing for acquiring dictionary data by querying the server in advance during a non-shooting period may be provided. In addition, with respect to dictionary data for subject detection, it is not necessary to query the server to acquire dictionary data that has not yet been acquired and use the dictionary data as it is. For example, before using the dictionary data (for example, before step S604), as a step for determining the dictionary data, a step for accepting a user operation or a step for automatically determining the dictionary data may be provided.

[0096] Fig. 7A and Figure 7B is a diagram for describing an example of subject detection based on dictionary data.

[0097] The dictionary data of the first embodiment has, for each subject type, a computer program describing the processing contents for executing a subject detection task by the neural network processing unit 205 , or trained coefficient parameters.

[0098] For example, as the type of subject to which the dictionary data is registered, a person, an animal such as a dog or a cat, or a vehicle such as a car or a motorcycle is exemplified.

[0099] Fig. 7A and Figure 7B Reference numerals 701 and 705 in FIG. 2 denote an example of a menu screen of the display unit 216 on which a user sets a subject to be detected via the operation unit 204. Fig. 7A , “person” 702 is set as a subject to be detected. When “person” is set, subject detection is performed by using dictionary data of “person” pre-stored in the nonvolatile memory 203. Reference numeral 703 denotes a captured image displayed on the display unit 216, showing a state in which the face of the “person” is detected and a display frame 704 is superimposed.

[0100] exist Figure 7B, “Custom” 706 is set as a subject to be detected. In the case of “Custom”, subject detection is performed by, for example, using “fish” as dictionary data for customization received from the server 110. Reference numeral 707 denotes a captured image displayed on the display unit 216, showing a state in which the dictionary data of “Custom” is “fish” and a frame 708 is superimposed on the detected fish for display.

[0101] Figure 8 and Fig. 9 is a flowchart showing processing performed by the CPU 401 of the server 110 according to the first embodiment.

[0102] Note that, in a state where the power of the server 110 is turned on, when the computer program stored in the recording unit 406 is loaded onto the memory 402 and the CPU 401 reads and executes the computer program in the memory 402, the Figure 8 processing.

[0103] refer to Figure 8 , a process extracted from the process performed by the CPU 401 of the server 110 will be described, in which information related to teacher data and a network structure is acquired from the mobile terminal 120 to generate dictionary data, and the generated dictionary data is sent to the camera 100.

[0104] In step S801, the server 110 acquires positive example data and negative example data from the mobile terminal 120 via the communication unit 407. In step S802, the negative example data pre-stored in the storage unit 406 is acquired. Here, steps S801 and S802 serve as a teacher data acquisition component for acquiring teacher data for subject detection. In addition, in step S803, the basic dictionary data stored in the storage unit 406 is acquired.

[0105] Subsequently, in step S804, it is determined through steps S801 to S803 whether the data required for generating the dictionary data is prepared. When the data is prepared (in step S804, it is determined as "yes"), the process proceeds to step S805. When at least part of the data is not prepared (in step S804, it is determined as "no"), the process proceeds to step S806.

[0106] For example, when the teacher data exists but the basic dictionary data does not exist, it is determined as “No” in step S804 . When determined as “No”, in step S806 , the occurrence of an error is notified to the mobile terminal 120 via the communication unit 507 .

[0107] In step S805, the neural network processing unit 408 generates custom dictionary data. As the generation of dictionary data, a method of generating dictionary data by learning from teacher data is proposed (for example, Fig. 9 ). Step S805 serves as a learning means for generating custom dictionary data.

[0108] Fig. 9 : is a flowchart for describing the flow of processing for generating dictionary data according to step S805.

[0109] When learning is performed in a state where the initial value of the dictionary data is a random number, a large amount of teacher data is required. When a large amount of teacher data is required, it takes time for the user to input the teacher data, so a method of learning with a small amount of teacher data is desired.

[0110] In view of the above, in step S901, dictionary data obtained by learning a variety of subjects in advance is set as the basic dictionary data as the initial value. As the basic dictionary data, dictionary data obtained by learning a specific type of subject such as a dog, a cat, or a motorcycle can be used.

[0111] In step S902, teacher data for learning is read.

[0112] In step S903, it is determined whether the read teacher data is data acquired by the input data acquisition unit, that is, whether the read teacher data is data acquired from the mobile terminal 120. When the data is data acquired by the input data acquisition unit, the process proceeds to step S904. When the data is not data acquired by the input data acquisition unit, the process proceeds to step S905.

[0113] In step S904, the coefficient of the loss function of the read data is set to be greater than the coefficient of the loss function of the data acquired by the data acquisition unit 112 in the server. For example, the coefficient is set so that the loss calculated at the time of learning is doubled. This is because the teacher data acquired by the input data acquisition unit is considered as data that clearly reflects the user's intention with high subject detection accuracy.

[0114] In addition, in step S905, the coefficient of the loss function of the read data is set to be smaller than the coefficient of the loss function of the data acquired by the input data acquisition unit. When the negative instance data acquired by the in-server data acquisition unit 112 is not data provided from the user, learning is performed by considering the possibility that the positive instance data that the user desires to detect is included in a part of the negative instance data. In view of the above, by dividing the processing into steps S904 and S905, the following negative impact can be reduced: in the case where the positive instance data is mixed in the negative instance data, the loss is calculated to be large.

[0115] In step S906, learning is performed based on the teacher data. Since the initial value of the dictionary data is not a random number but a value obtained by learning the possibility of becoming a subject, so-called fine tuning is performed. Here, step S906 serves as a dictionary generation component for generating dictionary data by learning based on the teacher data.

[0116] Description Return Figure 8 Flowchart in . When the dictionary data is generated in step S805, it is determined in step S807 whether to additionally obtain negative instance data. For example, when the value of the learned loss function is equal to or less than a predetermined threshold, the generation of the dictionary data is determined to be successful. When the value of the loss function is greater than a predetermined threshold, the generation of the dictionary data is determined to be a failure. When the generation of the dictionary data fails, it is determined that negative instance data is to be additionally obtained. Here, step S807 is used as a judgment component, which is used to determine whether the in-server data acquisition unit 112 further obtains negative instance teacher data in addition to the data obtained by the in-server data acquisition unit 112.

[0117] When the generation of the dictionary data is successful (NO in step S807 ), in step S808 , the dictionary data is transmitted to the imaging apparatus 100 via the communication unit 407 .

[0118] When the generation of the dictionary data fails (in step S807, it is judged as "yes"), the flow proceeds to step S809, and the in-server data acquisition unit 112 additionally acquires negative example data. After the acquisition, the flow returns to step S804. When the generation failure of the dictionary data has continued for a predetermined number of times, it is assumed that the generation of the dictionary data will not succeed even when further negative example data is added, and it is judged as "no" in step S804, so that the notification of the error can be performed in step S808.

[0119] Fig.10 is a flowchart showing an example of a flow of processing performed by the mobile terminal 120 according to the first embodiment.

[0120] The extraction related to the processing in the mobile terminal 120 will be described, in which information related to the teacher data and the network structure is input to notify the server 110 of the start of learning in the mobile terminal 120. This operation is achieved when the computer program stored in the recording unit 506 is loaded onto the memory 502 and the CPU 501 reads and executes the computer program in the memory 502 in a state where the power of the mobile terminal 120 is turned on.

[0121] refer to Fig.11A , Fig. 11B and Fig. 11C , will describe Fig.10 The processing flow in the flowchart.

[0122] Fig.11A , Fig. 11B and Fig. 11C : is a diagram for describing an input screen example of tutor data of the display unit 504 of the mobile terminal according to the first embodiment.

[0123] exist Fig.10 In step S1001 of , the user selects an image to be used as teacher data from among the captured images stored in the recording unit 506 via the operation unit 505 . Fig.11A An example of an image selection screen of the display unit 504 is shown, and 12 captured images are displayed as indicated by reference numeral 1101. For example, the user selects two images as teacher data from the 12 captured images by touching the operation unit 505, etc. As indicated by reference numeral 1102, the captured image with a circle displayed in the upper left corner is set as the image of the selected teacher data.

[0124] In step S1002 , the user specifies, via the operation unit 505 , a subject region set as a target in the image for two images selected as teacher data. Fig. 11B An example of an input screen of a subject area of ​​the display unit 504 is shown, and a rectangular frame 1103 indicates a subject area input by a user. The subject area is set for each image selected as teacher data. As a method of setting the subject area, area selection can be performed directly from the displayed image via a touch panel that is part of the operation unit 505 and integrated with the display unit 504.

[0125] Alternatively, the subject region may be a characteristic subject region selected by selection or fine-tuning from a subject frame simply detected by the CPU 501 or the like based on a feature amount such as an edge or the like.

[0126] In step S1003 , the user determines that generation of dictionary data will start via the operation unit 505 . Fig. 11C An example of a confirmation window for starting to generate dictionary data using the display unit 504 is shown, in which "Yes" or "No" is input. When "Yes" indicated by reference numeral 1104 is selected, information on the type of teacher data and camera apparatus is transmitted to the server 110 via the communication unit 507, and dictionary data is generated in the server 110. Fig. 11C If "No" is selected in the process, the process ends.

[0127] Note that during the generation of the dictionary data in the server 110, the subject area in the image data of the teacher data is regarded as a positive example, and the other areas are regarded as negative examples. In the above description, an example of selection of an image in which a subject area exists has been shown, but an image in which a subject area does not exist may be selected. In this case, the information of the subject area is not input, and the entire image is regarded as a negative example.

[0128] As described above, according to the image processing system of the first embodiment, any dictionary data passed by the user can be generated from a small amount of positive example teacher data.

[0129] <Other embodiments>

[0130] The present invention has been described in detail above through the preferred embodiments, but the present invention is not limited to the above embodiments. Various modifications can be made based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.

[0131] The subject matter to which the present invention can be applied is not limited to the camera apparatus 100, the server 110, and the mobile terminal 120, etc. described in the above-mentioned embodiments. For example, even when the camera apparatus 100 is a system composed of a plurality of devices, functions similar to those of the above-mentioned embodiments can be realized. In addition, when part of the processing of the camera apparatus 100 is realized by an external device on a network, the function can be realized. In addition, the processing of the mobile terminal 120 can be realized by the camera apparatus 100.

[0132] Note that the computer program for realizing the functions of the above-described embodiments of the control of part or all of this embodiment can be supplied to the camera system, etc. via a network or various storage media. Then, the computer (or CPU or MPU, etc.) in the camera system, etc. can read and execute the program. In the above case, the program and the storage medium storing the program constitute the present invention.

[0133] The present invention is not limited to the above-described embodiments, and various modifications and changes may be made without departing from the spirit and scope of the present invention. Therefore, the claims are attached to disclose the scope of the present invention.

[0134] This application claims the benefit of Japanese Patent Application No. 2022-074365, filed on Apr. 28, 2022, which is hereby incorporated by reference herein in its entirety.

Claims

1. An image processing device, comprising: a first acquisition component for acquiring first teacher data including image data of a positive instance of a first object; a second acquisition component, configured to acquire second teacher data including image data of a negative instance of the first object; as well as a generating component for generating first dictionary data for detecting the first object based on the teacher data acquired by the first acquiring component and the second acquiring component, The image processing apparatus is characterized in that the learning section performs additional learning on the second dictionary data to generate the first dictionary data, and the amount of the first teacher data acquired by the first acquisition section is smaller than the amount of the second teacher data acquired by the second acquisition section.

2. The image processing device according to claim 1, characterized in that The location where the first teacher data is obtained is different from the location where the second teacher data is obtained.

3. The image processing device according to claim 1 or 2, characterized in that: At least one of the first teacher data is teacher data provided from a user.

4. The image processing device according to any one of claims 1 to 3, characterized in that: The second acquisition component acquires pre-prepared data.

5. The image processing device according to any one of claims 1 to 4, characterized in that: In addition to the acquired first teacher data, the first acquisition component also acquires the second teacher data.

6. The image processing device according to any one of claims 1 to 5, characterized in that: The second object is obtained by further restricting the first object.

7. The image processing device according to any one of claims 1 to 6, characterized in that: The first object is a characteristic subject area.

8. The image processing device according to any one of claims 1 to 6, characterized in that: The first object is a specific subject.

9. The image processing device according to any one of claims 1 to 8, characterized in that: The learning component includes a judgment component for judging whether second teacher data including image data of a negative example is further acquired by the second acquisition component in addition to the second teacher data acquired by the second acquisition component.

10. The image processing device according to any one of claims 1 to 9, characterized in that: The number of first teacher data acquired by the first acquisition component is 100 or less, and the number of data acquired by the second acquisition component is 101 or more.

11. The image processing device according to any one of claims 1 to 10, characterized in that: A control unit is included for performing subject detection based on the dictionary data generated by the learning unit to perform shooting control.

12. The image processing device according to any one of claims 1 to 11, characterized in that: The learning component sets a coefficient of a loss function when learning is performed using the data acquired by the first acquisition component to be larger than a coefficient of a loss function when learning is performed using the data acquired by the second acquisition component.

13. A camera device, comprising: a sending component for sending first teacher data including image data of a positive instance of a first object; a receiving means for receiving first dictionary data generated by using the first teacher data sent by the sending means and second teacher data including image data of a negative instance for the first object; as well as a control component, configured to detect a subject based on the dictionary data received by the receiving component, so as to perform shooting control, The imaging apparatus is characterized in that the first dictionary data is dictionary data generated by additionally learning second dictionary data using the first teacher data and second teacher data including image data of a negative example for the first object, and The amount of the first teacher data is smaller than the amount of the second teacher data.

14. A control method for an image processing device, the control method comprising: A first acquisition step for acquiring first teacher data including image data of a positive instance of a first object; A second acquisition step is used to acquire second teacher data including image data of a negative instance of the first object; as well as a generating step for generating first dictionary data for detecting the first object based on the teacher data acquired by the first acquiring component and the second acquiring component, The control method is characterized in that, in the learning step, the second dictionary data is additionally learned to generate the first dictionary data, and the amount of the first teacher data acquired in the first acquisition step is smaller than the amount of the second teacher data acquired in the second acquisition step.

15. A method for controlling a camera device, the method comprising: a sending step for sending first teacher data including image data of a positive instance of a first object; a receiving step for receiving first dictionary data generated by using the first teacher data sent in the sending step and second teacher data including image data of a negative instance for the first object; as well as a control step for performing object detection based on the first dictionary data received in the receiving step to perform shooting control, The control method is characterized in that the first dictionary data is dictionary data generated by additionally learning second dictionary data using the first teacher data and second teacher data including image data of a negative example for the first object, and The amount of the first teacher data is smaller than the amount of the second teacher data.

Images