Information processing device and method, imaging device, and imaging system
The imaging device enhances machine learning by using both still and live view images to efficiently generate training data, addressing the challenge of capturing object appearance changes and reducing manual labeling efforts, thereby improving object recognition accuracy.
Patent Information
- Application Number
- JP2020122345
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-07-16
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-07-16
AI Technical Summary
Existing machine learning-based object recognition systems face challenges in obtaining sufficient training data, particularly due to the difficulty in accurately capturing and labeling changes in object appearance across different angles and distances, and the labor-intensive nature of human visual judgment for final labeling.
An imaging device that captures both still images and live view images before the still image, automatically detects and associates subject areas in these images, and uses them as training data, while filtering out images with minimal differences to enhance the training dataset efficiently.
This approach significantly increases the amount of training data available for object recognition, improving the accuracy and efficiency of machine learning by incorporating diverse image variations and reducing the need for extensive manual labeling.
Smart Images

Figure 0007595430000001 
Figure 0007595430000002 
Figure 0007595430000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device and method, an imaging device, and an imaging system, and more particularly to an information processing device and method, an imaging device, and an imaging system that perform machine learning to detect a subject from an image obtained by capturing. [Background technology]
[0002] In recent years, with the advancement of deep learning technology, the development of image object recognition technology using machine learning has become more and more popular. For example, in digital cameras, such technology is applied to detect subjects from images captured by live view shooting and automatically focus on and track the subject (AF).
[0003] Machine learning is divided into the learning and inference phases, and inference is performed based on the dictionary data obtained through learning. In the learning phase for object recognition using machine learning, a large number of images with correct labels indicating the target object to be recognized are required.
[0004] Patent Document 1 discloses a technique for efficiently generating training data for machine learning using time-series image data and pairs of images and correct labels as training data. When there is an already labeled image in the time-series images, this technique estimates the subject position in the image recorded before that and labels it, thereby efficiently learning using the time-series images.
[0005] Furthermore, Patent Document 2 discloses a technique for increasing the amount of training data by performing image processing such as inversion, color change, enlargement / reduction, translation, distortion, and synthesis with other images on the original training data. By using this technique, it is possible to reduce the number of images required for learning. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2019-75130 A [Patent Document 2] Patent No. 6441980 Summary of the Invention [Problem to be solved by the invention]
[0007] In order to perform more accurate object recognition using machine learning, several thousand to tens of thousands of pieces of training data are required, but the examples shown in Patent Documents 1 and 2 do not provide sufficient training data.
[0008] 2 shows an example in which a user follows a subject, automobile 200, while viewing live view (LV) images 201-205, and captures a still image 206. As described in Patent Documents 1 and 2, when a user selects an image after capture and labels the automobile 200 as the subject for learning, what is recorded is the still image 206, and the LV images 201-205 are not recorded. In this series of images, if the subject, automobile 200, can be recognized at an early stage, such as in the LV image 201, it becomes possible to perform more appropriate capture, such as by adjusting the exposure, focus, and framing.
[0009] However, the subject automobile 200 changes its apparent shape and size due to changes in the shooting angle and distance. Therefore, as described in Patent Document 1, if training data is generated using still image 206 and the image recorded before it, it is considered that automobile 200, which has once again become like LV image 201, cannot be accurately recognized. Moreover, such changes in appearance are beyond the scope of training data that can be generated by the image processing of Patent Document 2.
[0010] Furthermore, as described in Patent Document 1, when the final labeling of images is performed based on visual judgment by humans, a great deal of effort is required, which is also an issue.
[0011] The present invention has been made in consideration of the above problems, and aims to efficiently increase training data that can be used in machine learning for object recognition. [Means for solving the problem]
[0012] In order to achieve the above object, an information processing device of the present invention includes a storage means for storing a first image acquired in response to an instruction to capture an image and a plurality of second images automatically acquired before the first image, an identification means for identifying a subject in the first image, a detection means for detecting the subject identified by the identification means from the plurality of second images acquired before the first image, and a control means for controlling the adoption of sets in which information regarding the area of the subject detected from each of the plurality of second images is associated with each of the plurality of second images as learning data. 、 Have The detection means does not detect the subject when the difference between the first image and the second image is smaller than a predetermined difference. . Effect of the Invention
[0013] According to the present invention, it is possible to efficiently increase training data that can be used in machine learning for object recognition. [Brief description of the drawings]
[0014] [Figure 1] FIG. 2 is a block diagram showing a functional configuration according to the embodiment of the present invention. [Diagram 2] 1A and 1B are diagrams showing an example of a still image and an LV image before the still image is captured. [Diagram 3] 4A to 4C are diagrams showing examples of still images with LV images in the embodiment. [Figure 4] FIG. 4 is a diagram showing the concept of training data in the embodiment. [Diagram 5] 6A and 6B are diagrams showing examples of displays when specifying a subject type to be learned in the embodiment. [Figure 6] 11A and 11B are diagrams showing examples of displays when selecting an image to be studied in the embodiment. [Figure 7] 6A to 6C are diagrams showing an example of an operation for designating an area of a subject to be learned in the embodiment. [Figure 8] 11A to 11C are diagrams showing an example of tracking a subject from a still image toward an LV image in an embodiment. [Figure 9] 5 is a flowchart illustrating a detailed flow of a learning process for object recognition in the embodiment. [Figure 10] 5 is a flowchart illustrating a detailed flow of a learning process for object recognition in the embodiment. [Figure 11] A figure showing an example in which a subject frame of a subject estimated using a trained inference model is superimposed on a live view image in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.
[0016] 1 is a block diagram showing the functional configuration of an image capturing apparatus 100 according to an embodiment of the present invention. The image capturing apparatus 100 has an image capturing unit 101, an image processing unit 102, a captured image storage unit 103, a recording and playback unit 104, a live view (LV) image storage unit 105, and a teacher data generating unit 106. It also has an operation and display unit 107, a teacher data storage unit 108, a learning unit 109, an inference model storage unit 110, an inference unit 111, and a subject position information storage unit 112.
[0017] The imaging unit 101 includes an optical system and an imaging element (not shown), and performs photoelectric conversion on the subject image incident through the optical system by the imaging element, converts it into video data, and outputs it. The image processing unit 102 performs appropriate image processing on the video data output by the imaging unit 101. The image processing unit 102 is implemented, for example, by a central processing unit (CPU). It may also be implemented by a graphics processing unit (GPU) that is good at image processing. A live view (LV) display can be realized by repeatedly performing a shooting operation by the imaging unit 101 at a predetermined cycle and sequentially displaying the video data image-processed by the image processing unit 102 on the operation and display unit 107. The user can decide the composition and shooting settings while viewing the LV display and shoot a still image. The operation and display unit 107 is implemented by a display equipped with a touch panel. In the following description, the display image acquired for LV display is called an LV image.
[0018] The captured image storage unit 103 is a non-volatile memory for storing still images and LV images, and the live view (LV) image storage unit 105 is a memory for temporarily storing display images for outputting LV images to the operation and display unit 107 while the LV images are being captured. The recording and playback unit 104 controls the recording and playback of video data of still images and LV images processed by the image processing unit 102 into the captured image storage unit 103 and the LV image storage unit 105. The recording and playback unit 104 is implemented by, for example, a central processing unit (CPU).
[0019] The operation / display unit 107 is a display device (not shown) equipped with a touch panel, which displays the LV images temporarily stored in the LV image storage unit 105 and a user interface (hereinafter referred to as UI), and also accepts operations from the user.
[0020] The imaging device 100 has a learning function and an inference function for detecting a subject from a captured still image or an LV image, and the teacher data generation unit 106 generates teacher data for detecting the subject. The teacher data generation unit 106 has a subject type designation unit 106A, an image selection unit 106B, a subject area designation unit 106C, and a subject tracking unit 106D. The teacher data generation unit 106 is implemented, for example, by a central processing unit (CPU).
[0021] The teacher data storage unit 108 stores the teacher data generated by the teacher data generation unit 106, and transfers it to the learning unit 109. The learning unit 109 performs learning based on the teacher data and generates an inference model. The inference model storage unit 110 stores the inference model. The inference unit 111 performs subject detection (inference) from the LV image based on the inference model. The subject position information storage unit 112 stores the position of the subject detected by the inference unit 111. The teacher data storage unit 108 and the inference model storage unit 110 are implemented, for example, in a non-volatile memory. The learning unit 109 and the inference unit 111 are implemented, for example, in a central processing unit (CPU). They may also be implemented in a GPU that is more adept at matrix operations. Alternatively, the CPU and the GPU may be implemented in cooperation with each other. Specific algorithms used in the inference model include the nearest neighbor method, the naive Bayes method, a decision tree, and a support vector machine. In addition, deep learning, which uses a neural network to generate features and connection weighting coefficients for learning, can also be used. Any of the above algorithms can be used as appropriate and applied to this embodiment. In the following description, an example will be described in which an inference model is generated using deep learning.
[0022] In this embodiment, among still images and LV images, images used for learning of subject detection are called "learning images."
[0023] (Imaging function) The following describes the imaging function of the imaging device 100. When the user sets the imaging mode by issuing an instruction to the operation / display unit 107, a live view operation starts.
[0024] During live view operation, LV images periodically captured by the imaging unit 101 are subjected to appropriate image processing by the image processing unit 102, and then sequentially displayed in LV format on the display device of the operation / display unit 107 via the LV image storage unit 105. The LV image storage unit 105 buffers a predetermined number of the most recent LV images.
[0025] When the user sets the composition and shooting conditions while viewing the LV display and issues an instruction to shoot a still image, the following process is carried out. First, a still image captured by the imaging unit 101 is subjected to appropriate image processing by the image processing unit 102, and then recorded in the captured image storage unit 103 via the recording / playback unit 104. Furthermore, a predetermined number of the most recent LV images buffered in the LV image storage unit 105 are associated with the still image and recorded in order to use them as learning images. Hereinafter, this associated data set will be referred to as a "still image with LV image."
[0026] In the shooting example shown in Fig. 2, the LV image-added still image 300 is composed of a still image 206 and a plurality of immediately adjacent LV images 201 to 205, as shown in Fig. 3. The LV image-added still image 300 serves both as recorded data of the still image 206 captured by the user and as a learning image for subject detection.
[0027] In addition, if the captured image does not move, even if the LV image is used as a learning image, it will not become additional training data. Also, if there is little difference between the frames of the LV images, it is not efficient to use all the LV images for learning. Therefore, an inter-image difference detection unit (not shown) may be further provided, and if the difference between the LV images is below a predetermined threshold, the corresponding LV image may not be recorded.
[0028] (About learning) Next, a description will be given of a learning function of the imaging device 100. This function is a function that, after shooting, when a user specifies a subject appearing in a captured still image, the imaging device 100 learns to be able to detect the subject.
[0029] In this embodiment, the training data for learning object detection is a set of a learning image, an object type, and position information of the object in the image. The concept of the learning function will be described with reference to Fig. 4. In this example, the training data consists of an image 400, an object type, and information on a rectangular area 401 that exactly contains the object (car) to be trained in this image. The data format is the image file name, object type name, and coordinate values of two diagonal points 402, 403 of the rectangular area 401, as shown in 404.
[0030] The teacher data is generated by the teacher data generating unit 106 based on the still image with LV image and an instruction from the user. Specifically, the type of the subject to be learned is specified by the subject type specifying unit 106A, and the still image to be learned is selected from the still image with LV image by the image selecting unit 106B. Then, the subject area in the selected still image is specified by the subject area specifying unit 106C to obtain the teacher data. Furthermore, the subject area in the LV image associated with the still image is estimated by the subject tracking unit 106D, and these are also used as additional teacher data. The obtained teacher data is stored in the teacher data storage unit 108, and the learning unit 109 performs learning by referring to this teacher data.
[0031] Hereinafter, the learning process for object recognition will be described in detail with reference to the flowchart in FIG.
[0032] In S901, subject type designation unit 106A performs control to receive a subject type from the user via operation / display unit 107. An example of the display on operation / display unit 107 at this time is shown in Fig. 5. When a user wishes to add a new car as a subject, the user inputs the subject type as shown in 501 and taps confirmation button 502 to confirm the input. When the subject type has already been registered, the user selects it from registered list 503.
[0033] Next, in S902, the image selection unit 106B performs control for selecting an image to be learned. Still images stored in the captured image storage unit 103 are read out via the recording and playback unit 104, and a list is displayed on the operation and display unit 107. An example of the display is shown in Fig. 6. A list of images is displayed as thumbnails in an area 601 in a window 600, and the user can select a still image by tapping a desired image and pressing an end button 602.
[0034] Next, in S903, a learning process for object recognition is performed using a still image with an LV image corresponding to the selected still image. Here, the learning process performed in S903 will be described with reference to the flowchart in FIG.
[0035] In S1001, subject area designation unit 106C performs control to obtain position information of the subject in the selected still image from the user. An example of operations on operation / display unit 107 at that time is shown in Fig. 7. In this example, in order to learn about an automobile, first, a selected still image 701 is displayed on operation / display unit 107. Then, by dragging from position A to B on touch panel 700 of operation / display unit 107, rectangle 702 with diagonals A and B that exactly includes the area of the automobile is designated. A set of the still image selected in this way, the subject type, and rectangle information representing the subject area becomes training data (S1002).
[0036] In this embodiment, the user himself specifies the rectangular area of the subject, but this is not limited to this. For example, the user may tap on a part of the subject and the subject area may be extracted using a known contour extraction technique.
[0037] In S1003, the subject tracking unit 106D uses the LV image-attached still image of the selected still image as a learning target, and judges whether there is an LV image taken one image before the still image. If there is not, the process ends. On the other hand, if there is an LV image taken one image before, the process proceeds to S1004, and for the LV image-attached still image to be learned, the LV image is tracked back in chronological order based on the subject on the still image, and a control is performed to estimate the subject area. Here, in the case of the LV image-attached still image 300 shown in FIG. 3, a specific example of tracking performed at this time is shown in FIG. 8. As described above, the LV image-attached still image 300 consists of the still image 206 and LV images 201 to 205. In terms of the time order of taking the images, the LV image 201 is the earliest, followed by the LV images 202, 203, 204, 205, and still image 206.
[0038] When a rectangular area 807 of an automobile, which is a subject, is specified in a still image 206, the subject tracking unit 106D detects the automobile, which is a subject, from the LV image 205 immediately preceding the still image 206. If subject tracking is successful and the subject can be detected (YES in S1005), rectangular information indicating the subject area detected in the LV image 205 is stored as teacher data in the teacher data storage unit 108 (S1006). Then, the process returns to S1003, and the above-described process is repeated for the LV image 204 immediately preceding the still image 206.
[0039] In this way, the subject is tracked while going back in time in the LV images 205-201, and rectangular areas 808-812 in the LV images 205-201 are estimated. In this way, training data is also generated for the LV images 205-201.
[0040] On the other hand, if subject tracking has failed and the subject cannot be found (NO in S1005), the process proceeds to S1007, where the subject is deemed lost and the process ends.
[0041] The tracking process of the subject tracking unit 106D can be realized using a known technique such as motion vector detection.
[0042] In addition, if the captured image does not move, the LV image will not become additional training data even if it is used as a learning image. Also, if there is little difference between the frames of the LV images, it is not efficient to use all the LV images for learning. Therefore, an inter-image difference detection unit (not shown) may be provided, and when generating training data from the LV images, the LV images may be thinned out without being used as training data while the difference between the image or subject area from the still image or LV image used as training data last time is below a predetermined threshold.
[0043] When the learning process shown in Fig. 10 is completed, the process proceeds to S904 in Fig. 9. In S904, it is determined whether or not to add a still image to be registered as training data, and if so, the process returns to S902 and repeats the above process, and if not, the process ends.
[0044] In addition, in this embodiment, the captured image storage unit 103, the teacher data generation unit 106, the teacher data storage unit 108, and the learning unit 109 are arranged in the imaging device 100 for processing, but the present invention is not limited to this. Some or all of these may be cloud-based and configured to process on the cloud via a communication unit (not shown) in the imaging device 100, or configured to process by connecting to an information processing device such as a PC, thereby forming an imaging system together with the imaging device 100.
[0045] As described above, by using not only a still image taken by the user but also a live view image taken immediately before that, it is possible to efficiently increase the amount of training data that can be used for machine learning to detect subjects.
[0046] Then, the learning unit 109 stores the training data in the teacher data storage unit 108 as described above. To Learning is performed using the stored training data, an inference model is generated, and stored in the inference model memory unit 110.
[0047] (Inference function) Finally, we will explain the inference function of the imaging device 100. The inference function is a function that detects a subject from a captured image based on an inference model obtained by the above-mentioned learning function. Here, we will explain an example in which the function is used to detect a subject during live view shooting and assist in shooting.
[0048] The inference unit 111 acquires an LV image from the LV image storage unit 105, and performs inference (subject detection) based on the trained inference model recorded in the inference model storage unit 110. As a result of the inference, information on the subject type and a rectangular area that exactly contains the subject is obtained. This information is hereinafter referred to as "subject position information." The subject position information is recorded in the subject position information storage unit 112. The operation / display unit 107 uses the subject position information to superimpose a subject frame on the displayed live view image. This allows the user to know the situation in which the camera is recognizing the subject. FIG. 11 is a diagram showing an example in which a subject frame of a subject estimated using a trained inference model is superimposed on a live view image.
[0049] Furthermore, the imaging unit 101 and the image processing unit 102 can also use the subject position information to adjust the focus and exposure to the subject, and perform optimal image processing according to the type of subject.
[0050] Furthermore, if the system is configured so that the user can select which of the learned subject types to detect, it is possible to have the camera react only to subjects that the user desires depending on the shooting conditions.
[0051] <Other embodiments> The present invention can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.
[0052] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).
[0053] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]
[0054] 100: imaging device, 101: imaging section, 102: image processing section, 103: captured image storage section, 104: recording and playback section, 105: LV image storage section, 106: teacher data generation section, 106A: subject type designation section, 106B: image selection section, 106C: subject area designation section, 106D: subject tracking section, 107: operation and display section, 108: teacher data storage section, 109: learning section, 110: inference model storage section, 111: inference section, 112: subject position information storage section
Claims
1. a storage means for storing a first image acquired in response to an instruction to capture an image and a plurality of second images automatically acquired before the first image; an identification means for identifying a subject in the first image; a detection means for detecting the subject identified by the identification means from the plurality of second images acquired before the first image; a control means for controlling the adoption of a set of information relating to the area of the subject detected from each of the plurality of second images and the plurality of second images as learning data, and when a difference between the first image and the second image is smaller than a predetermined difference, the detection means does not detect the subject.
2. 2 . The information processing apparatus according to claim 1 , wherein the specifying unit specifies the subject in the first image by a user's designation.
3. 2. The information processing apparatus according to claim 1, wherein the specifying means detects the area of the subject in the first image by contour extraction.
4. 4. The information processing device according to claim 1, wherein the detection means detects the subject from the plurality of second images using a tracking process that tracks the subject in the first image identified by the identification means.
5. 5. The information processing apparatus according to claim 4, wherein the detection means performs tracking processing by going back in time in the plurality of second images.
6. 6. The information processing apparatus according to claim 4, wherein the tracking process detects a motion vector.
7. An imaging means; An information processing device according to any one of claims 1 to 6, a learning means for generating an inference model for inferring a subject based on information on the area of the subject detected by the detection means; an inference means for inferring an area of the subject from an image captured by the imaging means, using the inference model generated by the learning means; An imaging device comprising:
8. 8. The imaging apparatus according to claim 7, further comprising a display unit that displays a display indicating the area of the subject estimated by the estimation unit in a superimposed manner on the image captured by the imaging unit.
9. An information processing device according to any one of claims 1 to 6, An imaging device; The imaging device, An imaging means; A communication means for communicating with the information processing device; a learning means for generating an inference model for inferring a subject based on information on the area of the subject detected by the detection means of the information processing device; an inference means for inferring an area of the subject from an image captured by the imaging means, using the inference model generated by the learning means; 1. An imaging system comprising:
10. a storage step of storing in a storage means a first image acquired in response to an instruction to capture an image and a plurality of second images automatically acquired before the first image; an identification step for identifying a subject in the first image; a detection step of detecting the subject identified by the identification step from the plurality of second images acquired before the first image; a control step of controlling so as to adopt, as learning data, a set in which information on the area of the subject detected from each of the plurality of second images and the plurality of second images are associated with each other; The information processing method, wherein in the detection step, when a difference between the first image and the second image is smaller than a predetermined difference, detection of the subject is not performed.
11. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 6.
12. A computer-readable storage medium storing the program according to claim 11.
Citation Information
Patent Citations
Bar code displaying device
JP1989041980A
Device and method for picture processing and recording medium
JP2000048212A
Image processing device, image processing method
JP2012088787A
Imaging apparatus and control method of the same
JP2018129682A
Information processing unit, control method, program
JP2019075130A
