Image processing device, control method thereof, and program
The image processing device enhances behavior recognition accuracy by selecting an optimal video range for input to an inference model, using 3D-CNN, and adding metadata to still images, addressing the challenge of determining the video input range in existing technologies.
Patent Information
- Application Number
- JP2020020890
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-02-10
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2040-02-10
AI Technical Summary
Existing image recognition technologies using inference models do not effectively determine the optimal range of video input for accurately inferring the behavior of a subject, leading to inaccurate behavior recognition.
An image processing device that selects a predetermined period of live view images before and after the shooting instruction to input to an inference model, using a 3D-CNN for behavior recognition, and adds metadata to the still image based on the inferred behavior.
Enables accurate inference of subject behavior by determining an appropriate video range for input, improving the accuracy of behavior recognition in still images.
Smart Images

Figure 0007583526000001 
Figure 0007583526000002 
Figure 0007583526000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing apparatus, a control method thereof, and a program. [Background technology]
[0002] In recent years, image recognition technology using machine learning has become known, specifically, a technology that recognizes a specific subject by inputting an image captured by an imaging device into an inference model trained using training data that includes the specific subject.
[0003] Also, a technique is known for applying an inference model related to image recognition to video, not limited to still images. Specifically, a technique is known in which a video is input to an inference model trained using teacher data including a video of a subject's specific behavior, and the specific behavior contained in the video is recognized. Patent Document 1 proposes a technique for inputting a video captured by a surveillance camera into an inference model using a multi-layer neural network, and automatically generating a description of the video. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2018-101317 A Summary of the Invention [Problem to be solved by the invention]
[0005] Incidentally, in order to accurately obtain the behavior of a subject in a video using an inference model, it is necessary to input an appropriate range of the video including the target behavior scene to the inference model. In this regard, although Patent Document 1 analyzes the video using a multi-layer neural network, it does not consider determining the optimal range of the video to be input to the neural network.
[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that can determine an appropriate range of video for inferring the behavior of a subject, and more accurately infer the behavior of the subject being filmed. [Means for solving the problem]
[0007] In order to solve this problem, for example, the image processing device of the present invention has the following configuration. Live View An acquisition means for acquiring image data; Multiple In response to receiving a shooting instruction to take a still image, A means for taking continuous shots and acquiring data of a plurality of still images; The plurality of Live View Among the image data, a plurality of images during a predetermined period determined based on the timing of receiving the image capture instruction Live View A means for identifying image data as image data to be processed, and a means for identifying the image data based on a behavior of a subject inferred using the image data to be processed. Multiple A control means for adding information indicating the action of the subject to data of the still image. The specifying means specifies, as the image data to be processed, a plurality of live view image data in the predetermined period before and after a central timing of a continuous shooting section of the continuous shooting among the plurality of live view image data acquired by the acquiring means, and the predetermined period includes a period that is not a continuous shooting section. It is characterized by: Effect of the Invention
[0008] According to the present invention, it is possible to determine an appropriate range of video for inferring the behavior of a subject, and to more accurately infer the behavior of the subject being filmed. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional arrangement of a digital camera as an example of an image processing device according to a first embodiment. [Diagram 2] FIG. 1 is a diagram for explaining the flow of image data in a digital camera according to a first embodiment. [Diagram 3] FIG. 1 is a diagram for explaining an outline of processing in an inference unit according to the first embodiment. [Figure 4] 1 is a flowchart showing a series of operations in a still image shooting process according to the first embodiment. [Diagram 5]FIG. 1 is a diagram for explaining an LV image for inference according to the first embodiment. [Figure 6] FIG. 1 is a diagram showing an example of a stored inference result according to the first embodiment. [Figure 7] 11 is a flowchart showing a series of operations in a still image shooting process according to a second embodiment. [Figure 8] 11 is a flowchart showing a series of operations in an inference process according to the third embodiment. [Figure 9] 11 is a flowchart showing a series of operations in a still image shooting process according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] (Embodiment 1) Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.
[0011] In the following, an example of an image processing device will be described using a digital camera capable of applying an inference model to a moving image. However, this embodiment is not limited to a digital camera, and can also be applied to other devices capable of applying an inference model to a moving image. These devices may include, for example, personal computers, mobile phones including smartphones, game consoles, tablet terminals, watch-type and eyeglass-type information terminals, medical devices, and devices for monitoring systems and in-vehicle systems.
[0012] (Digital Camera Configuration) Fig. 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an example of an image processing device of this embodiment. One or more of the functional blocks shown in Fig. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or an MPU executing software. They may also be realized by a combination of software and hardware. Therefore, even when different functional blocks are described as operating subjects in the following description, they may be realized by the same hardware as the subject.
[0013] The control unit 101 includes one or more processors, such as a CPU, an MPU, or a GPU, and controls the overall operation of the digital camera by executing programs stored in a ROM 102.
[0014] The ROM 102 includes, for example, a non-volatile semiconductor memory, and stores the control program executed by the control unit 101. The RAM 103 includes, for example, a volatile semiconductor memory, and is used as a work memory for the control unit 101 to execute the program, a temporary storage area for various data, and the like.
[0015] The imaging unit 104 includes an imaging lens, an aperture, a shutter, an imaging element such as a CMOS sensor, and an A / D converter, and generates an analog image signal from light imaged on the imaging element through the imaging lens, and further converts it into a digital image signal to output image data. The image data output from the imaging unit 104 is expanded in the RAM 103. The imaging unit 104 can output a plurality of image data (LV images) that are continuously output, and a still image for recording that is one piece of image data. The LV image is a moving image output from the imaging unit 104 during a shooting operation, and is image data in a YUV format that is displayed on the display unit 106 in real time. The LV image is also used as data for inference input to the inference unit 107. The still image data for recording is high-resolution image data generated by a user's shooting instruction, and is recorded in a storage medium 109 via the recording unit 108. Since the still image data for recording is high-resolution, the imaging unit 104 may output it in a compressed JPEG format, for example.
[0016] The operation unit 105 accepts operations by the user and notifies the control unit 101 of the input information. The operation unit 105 includes, for example, a touch panel, button switches, a cross key, etc. The display unit 106 includes display members such as a liquid crystal panel and an LED, and displays to the user information regarding the state of the digital camera 100, a live view image generated by the imaging unit 104, image data recorded in a storage medium 109, etc.
[0017] The inference unit 107 includes, for example, a GPU or a dedicated circuit, and analyzes the video data input from the imaging unit 104 to infer the behavior of the subject included in the video data. The inference unit 107 outputs an activity label and a recognition rate for the activity label as an inference result. The configuration of the inference unit 107 will be described later.
[0018] The recording unit 108 controls reading and writing of data from and to the storage medium 109. The recording unit 108 controls initialization of the storage medium 109 and data transfer between the storage medium 109 and the RAM 103. The storage medium 109 is a device having a large capacity storage area, including, for example, a memory card or a flash memory. The internal bus 110 is a bus that connects control signals exchanged between the above-mentioned processing units.
[0019] (Flow of processing image data) Next, the flow of processing image data by digital camera 100 will be described with reference to Fig. 2. In this process, an inference process (behavior recognition process) is applied to a video associated with the shooting of a still image, and metadata is added to the still image. More specifically, when a still image is shot with digital camera 100, images (i.e., LV images) that are shot consecutively in the time direction before and after the timing of shooting the still image are input to an inference model to recognize the behavior of the subject. Then, metadata based on the recognition result is added to the shot still image.
[0020] For example, when filming figure skating, if a still image of an athlete performing is taken, metadata such as "quadruple jump" is automatically added to the captured image. This "quadruple jump" information can be obtained by performing behavior recognition on video around (before and after) the timing of taking the still image (it cannot be obtained by analyzing the captured still image itself).
[0021] First, the imaging unit 104 generates an LV image (constituting video data) and still image data for recording. The LV images are output sequentially at regular time intervals while the imaging unit 104 is operating, and transmitted to the display unit 106 and the LV image buffer 201. In this embodiment, for example, image data in YUV format with a resolution of 800×600 at a speed of 20 frames per second is generated and output as an LV image. The display unit 106 displays the transmitted LV image in real time.
[0022] The LV image buffer 201 is an area for temporarily storing a predetermined number of frames of LV images, and is configured as a part of the RAM 103. When the number of input LV images exceeds the upper limit, the oldest LV image is deleted in order. The LV images stored in the LV image buffer 201 are used as images to be input to the inference unit 107. For this reason, the number of buffer stages (the maximum number of LV images that can be stored) is determined in advance according to the time of the action scene to be inferred. In this embodiment, as an example, the number of stages is configured assuming that the type of jump performance in figure skating is inferred. For example, assuming that the time from when the athlete starts the jumping movement to when he lands on the ice is about 2 seconds, the number of stages is set to 3 seconds including a margin. Since the speed of generation is 20 frames per second as described above, the number of frames is 60. The LV images temporarily stored in the LV image buffer 201 are input to the inference unit 107 by the control unit 101 at the timing when inference is performed.
[0023] Still image data is generated by the imaging unit 104 in response to a shooting instruction issued by a user pressing the shooting button on the operation unit 105, and is stored in the still image buffer 202. In this embodiment, for example, image data in JPEG format with a resolution of 4000×3000 is generated. The still image data stored in the still image buffer 202 is then converted by the control unit 101 into a predetermined file format, and is recorded via the recording unit 108 in a large-capacity storage medium such as a memory card.
[0024] (Overview of processing in the inference section) Next, an overview of the processing in the inference unit 107 will be described with reference to Fig. 3. The inference unit 107 is configured with a learning model based on 3D-CNN (Convolutional Neural Network), which is known as a deep learning technology for video recognition. 3D-CNN is a neural network that has been expanded to perform analysis on three-dimensional data with an additional time direction, in contrast to a general CNN that processes two-dimensional information (still images) that has been known in the past.
[0025] The learning model used by the inference unit 107 is a model that has undergone learning processing using teacher data in advance in accordance with the object to be inferred. In this embodiment, for example, in order to infer jumping performances in figure skating, a model that has undergone learning processing regarding jumping performances in advance is used. In the learning processing, for example, teacher data consisting of a pair of a video of various jumping performances in figure skating and a correct answer label is repeatedly input to the learning model to optimize parameters of the learning model such as a weighting factor to be described later. The correct answer label of the teacher data includes an action label corresponding to the jumping performance in the video, such as "single jump", "double jump", and "triple jump".
[0026] In the example of the inference unit 107 of this embodiment, a learning model for inferring which jump performance in figure skating corresponds to has been described as an example, but the inference target may be something else. The inference unit 107 may infer performances other than jump performances in figure skating, or may infer which sport the video is taken of. Also, it may infer which technique or skill in another sport the video is taken of.
[0027] For example, when the learning model infers an action other than a jump in figure skating, learning data in which a video of figure skating including an action other than a jump is given an action label such as "xx step" that indicates the corresponding action may be used. Also, by learning using learning data in which a video of various sports is given an action label that indicates the corresponding sport such as "figure skating", "soccer", "rugby", etc., it is possible to infer which sport the action included in the video is. Of course, it is also possible to infer which sport and which action, etc., is by giving which action label which is which sport. Also, other actions may be used as long as the action of the subject is inferred from the video before and after the timing of shooting, without being limited to the above example. For example, in the case of a soccer shot or a rugby kick, it is assumed that a still image is taken at the moment or immediately after the ball is kicked. Therefore, an application is conceivable in which LV images of a predetermined period before and after the shooting of the still image are input to infer whether the shot or kick went into the goal or not.
[0028] Image data 301 input to the inference unit 107 is a plurality of image data that are continuous in the time direction. This image data 301 corresponds to the LV image selected by the control unit 101 for inference.
[0029] The convolution / pooling processor 302 is a processor that repeats convolution processing and pooling processing on input image data. In the convolution processing, feature points of the image are extracted using a filter that can be applied to the two-dimensional space of the image and a filter that can be applied in the time direction. In the pooling processing, after the convolution processing, the amount of information of the image data is reduced while maintaining information of important feature points (for example, by processing such as max pooling that takes the maximum value of the data to be processed).
[0030] Feature maps 303 are generated as a result of repeating the convolution process and pooling process a predetermined number of times. The size of each feature map and the total number of feature maps generated here depend on the specifications of the filter used in the convolution process (e.g., the filter size and stride) and the number of times the convolution is repeated.
[0031] The fully connected layer 304 is a fully connected layer composed of multi-layer perceptrons, and performs inference processing on the extracted feature map according to weight coefficients between neurons determined in advance by learning. The fully connected layer 304 is composed of multiple layers, and the final layer is the output layer. The round shapes shown in each layer represent neurons, which are units that model nerve cells. Information on each pixel of the feature map 303 is input to each unit in the first layer of the fully connected layer, multiplied by a weight coefficient optimized by a prior learning process, and output to all nodes in the next layer. This process is repeated through multiple layers, and finally, the final result is output in each unit of the output layer. The final result is, for example, a numerical value corresponding to the recognition rate for the action label. The numerical value of the recognition rate is output as a value normalized by, for example, a softmax function so that the sum of the outputs of all units is 1.0 (100%).
[0032] The number of units in the output layer corresponds to the type of action label to be inferred. For example, as shown in FIG. 3, if the output layer is composed of four units, each unit corresponds to one of the four types of action labels. That is, each unit in the output layer corresponds to the action labels of "single jump", "double jump", "triple jump", and "quadruple jump". When an LV video is input and calculations are performed in the model, a signal value indicating a recognition rate is output from each of the four units constituting the output layer, each of which indicates the probability that the LV video is recognized as each jump. If, for example, the value of recognition rate 3 is high among the outputs (recognition rates 1 to 4) of each unit in the output layer, it can be determined that there is a high possibility that the input video included a "triple jump" performance.
[0033] When inferring various types of performances including other types of performances besides figure skating, the output layer is configured to output the recognition rates for other performances as well, such as "single jump", "double jump", ... and "xx step". When inferring which sport an action included in a video belongs to, the output layer is configured to output the recognition rates for various sports, such as "figure skating", "soccer", "rugby", ...
[0034] In addition, although the configuration of the convolution / pooling processing unit 302 is not explicitly shown in the diagram shown in FIG. 3, it may be configured as a multi-layer perceptron including a convolution layer and a pooling layer.
[0035] (A series of operations related to still image shooting processing) Next, a series of operations related to the still image shooting process will be described with reference to Fig. 4. This process is realized by the control unit 101 expanding a program stored in the ROM 102 in a working area of the RAM 103, executing it, and operating the imaging unit 104, the inference unit 107, etc. This series of processes is started when the digital camera 100 transitions to a shooting mode. Here, the shooting mode is one of the operation modes of the digital camera 100, and means an operation mode in which a user can shoot a still image at any time by pressing a shooting button. For example, an operation mode in which the imaging function is not operated and a still image recorded on a memory card is reproduced and displayed on a liquid crystal monitor does not fall under this category.
[0036] In S401, upon transition to the shooting mode, the control unit 101 controls the imaging unit 104 to start the generation process of the LV image. The generated LV image is transmitted to the display unit 106 and the LV image buffer 201 as described above.
[0037] In S402, the control unit 101 determines whether a shooting instruction for still image shooting has been received from the user. If the control unit 101 determines that a shooting instruction for still image shooting has been received based on, for example, a signal from the operation unit 105, the process proceeds to S403. On the other hand, if the control unit 101 determines that a shooting instruction for still image shooting has not been received, the process proceeds to S410. At this time, if it is determined in S410 that the operation mode will not be changed to another operation mode, the process returns to S402 again and continues to wait for a shooting instruction.
[0038] In S403, the control unit 101 determines whether to execute inference of behavior recognition together with still image shooting. If a subject (person or animal) to be a target of behavior recognition is not included in the still image shot by the user, the subsequent inference process becomes redundant, so the control unit 101 determines the necessity of executing inference of behavior recognition in this step. For example, the control unit 101 determines whether a person or an animal can be detected from an LV image generated at the time when a shooting instruction is received. This detection process can be realized by applying a known face detection technique or object detection process, so the details of the process are omitted. If the control unit 101 determines that a subject to be a target of behavior recognition is detected as a result of the detection process and therefore that an inference of behavior recognition is to be executed, the control unit 101 proceeds to S404, and if not, the control unit 101 proceeds to S410 (then returns to S402 again to wait for a shooting instruction for still image shooting).
[0039] In S404, the control unit 101 determines LV images to be input to the inference unit 107 from the LV images temporarily stored in the LV image buffer 201. Specifically, the control unit 101 determines the first and last LV images from a plurality of LV images stored in the LV image buffer 201 and consecutive in the time direction.
[0040] In order for the inference unit 107 to perform highly accurate behavior recognition processing, the first LV image is preferably an LV image taken before the subject starts the target behavior, and the last LV image is preferably an LV image taken after the subject has finished the behavior to be inferred. For this reason, the control unit 101 determines the range so as to include LV images both before and after the timing of the shooting instruction. In other words, the control unit 101 determines, as the LV images to be processed, from among the multiple LV images stored in the LV image buffer 201, multiple LV images taken during a period including the timing of the shooting instruction. In this embodiment, for example, assuming that the total time of a jump performance is about 2 seconds, a total of 2 seconds of LV images centered on the shooting timing are selected. For example, as shown in FIG. 5(a), at the shooting timing at time S, LV images generated at the same time are f n For example, when LV images are generated at a speed of 20 frames per second, the interval between adjacent LV images is 50 milliseconds. When selecting LV images for a total of 2 seconds centered on the shooting timing, the first LV image is the LV image generated 1 second before the shooting timing, and f n-20 On the other hand, the last LV image was generated 1 second after the shooting timing, and f n+20 In this way, the control unit 101 can, for example, n-20 From f n+20 The LV images up to are determined as the LV images to be input to the inference unit 107.
[0041] In S405, the control unit 101 inputs the LV image determined as the processing target in S404 to the inference unit 107, causes the inference unit 107 to execute the inference process, and obtains information on the recognition rate corresponding to each activity label as the inference result. In addition, the control unit 101 performs a process of counting the number of times the inference process is executed, and a process of recording the inference execution history in the RAM 103. This execution history information is used in the process of determining the input image when the inference process is executed again.
[0042] Here, Fig. 6 shows an example of the execution history information. In this example of the execution history information, for example, the number of executions, a timestamp, an LV image number, and a recognition rate are recorded. The timestamp is information for identifying the timing at which the inference was executed, and is recorded in the format of "hours:minutes:seconds.milliseconds", for example. The LV image number indicates the identifier of the LV image input to the inference unit 107 when the inference was executed. The recognition rate is information indicating the result of the inference execution, and indicates the recognition rate for each behavior label associated with the output of the inference unit 107.
[0043] In the example shown in FIG. 6, among the execution counts identified in the execution count column, the recognition rates for the first and second execution results show that all action labels are below 30%. In other words, it is indicated that a high recognition rate was not obtained in the first and second execution results. On the other hand, the third execution result shows that the recognition rate for the third action label in the recognition rate column is 90%. In other words, it is indicated that the third execution result shows that there is a 90% chance that the performance of a "triple jump" is captured.
[0044] In S406, the control unit 101 determines whether or not metadata can be generated based on the inference result acquired in S405. If an inference result with a high recognition rate exceeding a predetermined threshold is obtained, the control unit 101 determines that metadata can be generated and proceeds to S409, otherwise proceeds to S407.
[0045] In S407, the control unit 101 determines whether the number of times the inference has been executed has reached a predetermined upper limit. If the control unit 101 determines that the number of times the inference has been executed has reached the upper limit, it resets the count value of the number of times the inference process has been executed to zero, deletes the execution history information, and proceeds to S410. On the other hand, if the control unit 101 determines that the number of times the inference has been executed has not reached the upper limit, it proceeds to S408.
[0046] In S408, the control unit 101 executes the LV image determination process again to re-execute inference. At this time, the control unit 101 utilizes the execution history information described above to determine the range of the LV image so that the range is different from the range of the LV image input when a low recognition rate is output from the inference unit 107. For example, the new range of the LV image is determined by widening or narrowing the width of the range of the LV image, or by shifting it forward or backward by several frames without changing the width.
[0047] In S409, the control unit 101 associates information on the behavior label that has acquired a high recognition rate with the still image data generated in response to the shooting instruction in S402. Specifically, the information is additionally recorded as metadata in the header information of the still image data file recorded in the storage medium 109. The metadata is utilized in processes such as searching and classifying the still image data files. In this step, the control unit 101 also resets the count value of the number of times the above-mentioned inference process has been executed to zero, and deletes the execution history information.
[0048] In S410, the control unit 101 determines whether the operation mode of the digital camera 100 has been changed from the shooting mode to another mode by the user. If the control unit 101 determines that the operation mode of the digital camera 100 has been changed to another mode, this series of operations ends. On the other hand, if the control unit 101 determines that the operation mode of the digital camera 100 is still in the shooting mode, the process returns to S402 and waits for a shooting instruction.
[0049] Although the above-described still image shooting process is based on the assumption that a single image is shot, it can also be applied to continuous shooting in which multiple still images are shot. The following provides a supplementary explanation of the processes in S404 and S409 that are applied when continuous shooting is performed.
[0050] In the case of continuous shooting, in S404, the control unit 101 selects LV images for a total of 2 seconds centered on the central shooting timing from among multiple shooting timings in the continuous shooting section as LV images for inference. For example, FIG. 5(b) shows an example in which three still images are shot by continuous shooting at a continuous shooting speed of 10 frames per second. 1 , S 2 , S 3 is the shooting timing during the continuous shooting period, and the LV image generated at each timing is f n , f n+2 , f n+4 For example, if the continuous shooting speed is 10 frames per second, the interval between each shooting timing is 0.1 seconds. 2 It is. 2 When selecting LV images for a total of 2 seconds centered on the 2 The LV image was generated 1 second before f n-18 On the other hand, the LV image at the end is S 2 is the LV image generated 1 second after f n+22 That is, the control unit 101 corresponds to f n-18 From f n+22 The LV images up to are determined as the LV images to be input to the inference unit 107.
[0051] In the case of continuous shooting, in S409, the control unit 101 associates the information of the behavior label having a high recognition rate with all still image data generated by continuous shooting. In other words, the control unit 101 associates metadata of the same content with multiple still image data.
[0052] As described above, in this embodiment, when taking still images of a subject performing a specific behavior, the period before and after a shooting instruction is determined, and the video of that period is input into a learning model to infer the behavior of the subject. Furthermore, information indicating the inferred behavior of the subject is generated as additional information for the still image shooting. In this way, it is possible to determine a video of an appropriate range for inferring the behavior of the subject, and to more accurately infer the behavior of the subject to be shot. In addition, it is possible to add data that appropriately represents the behavior of the subject to be shot.
[0053] In the present embodiment, an example has been described in which the LV image generated by the imaging unit 104 is used for display and for inference, but an image for display and an image for inference may be generated separately from the imaging unit 104. Also, in the above embodiment, an example has been described in which the LV image generated by the imaging unit 104 is input directly to the inference unit 107. However, before being input to the inference unit 107, the LV image may be converted into a data format (resolution, encoding method, etc.) that is convenient for processing by the inference unit 107.
[0054] In the above embodiment, the method of determining the LV images to be input to the inference unit 107 in S404 is a method of determining the beginning and end of the LV images stored in the LV image buffer 201, but this is not limiting. In order to reduce the amount of data to be input to the inference unit 107, the LV images in the middle of the determined range may be thinned out, or the LV images to be input may be individually selected.
[0055] In addition, in S404, the method of determining the LV image to be input to the inference unit 107 has been described as a method of determining the top and bottom of the LV image so as to center on the shooting timing, but this is not limiting. A mechanism for detecting a specific state in which it is estimated that a still image is likely to be shot may be provided separately, and the top LV image may be determined according to the detection timing. This detection timing may be, for example, the timing at which a sensor such as a gyro mechanism detects that the digital camera has stopped while waiting for a shooting instruction. Alternatively, it may be the timing at which the user performs focus control on the subject, or the timing at which tracking control on the subject is performed.
[0056] In addition, the method of determining the trailing LV image may be the LV image corresponding to the timing when a predetermined time has elapsed since still image shooting was performed, or the timing when the subject moves out of the LV image after still image shooting was performed.
[0057] Furthermore, in the above embodiment, the inference process in S405 is executed inside the digital camera 100, but the inference process may be executed in an external device such as a cloud server. In this case, the digital camera 100 does not include the inference unit 107 shown in Figs. 1 and 2, but instead includes a communication unit for communicating with the external device. The digital camera 100 provides the LV image to the cloud server via this communication unit, requests the inference process, and then obtains the inference result provided by the external device. In this configuration, communication occurs between the digital camera 100 and the external device when performing the inference, but there is an advantage that the abundant computational resources of the cloud server can be used for the inference process (hence, an inference algorithm with a higher accuracy and a larger amount of computation can be used).
[0058] (Embodiment 2) Next, a second embodiment will be described. In the first embodiment, if an inference result with a high recognition rate is not obtained as a result of performing inference, the process of determining the LV image to be input and the inference process are repeated. In this case, the number of times the inference process is repeated varies depending on the LV image input to the inference unit 107, so the time from when the shooting instruction is received until the metadata is generated in S409 is not constant. In other words, if an inference result with a high recognition rate is obtained in the first inference, the metadata is generated immediately, whereas if the inference is repeated many times, the time until the metadata is generated becomes longer. Therefore, in the second embodiment, the configuration described below ensures that the metadata is always generated after a certain time has elapsed since the shooting instruction is received. Note that the configuration of the digital camera 100 in this embodiment is the same or substantially the same as the above-mentioned embodiment, and the operation related to the still image shooting process is different. For this reason, the description of the same or substantially the same configuration is omitted, and the description will focus on the differences.
[0059] (A series of operations related to still image shooting processing) The still image shooting process according to the second embodiment will be described with reference to Fig. 7. This process is realized by the control unit 101 expanding a program stored in the ROM 102 into a working area of the RAM 103, executing the program, and operating the imaging unit 104, the inference unit 107, etc.
[0060] First, the control unit 101 performs the processes of S401 to S403 in the same manner as in the first embodiment, and proceeds to S701 if it is determined that inference is to be performed.
[0061] In S701, the control unit 101 uses the LV images temporarily stored in the LV image buffer 201 to determine multiple types of LV image groups to be input to the inference unit 107. There are several possible methods for forming groups (of different types). For example, groups may be created with the same LV image at the center and different range widths (the number of LV images included). Also, groups may be created with the same number of LV images included in each group, with the positions of the first and last LV images shifted.
[0062] In this embodiment, for example, three types of groups are created with different numbers of LV images included in the range centered on the shooting timing. FIG. 5(c) shows the method of creating these groups. Group 1, group 2, and group 3 shown in FIG. 5(c) indicate the groups determined in this step. The first group is f n-21 From f n+21 The first group is LV images up to f n-20 From f n+20 The third group is the LV images up to f n-19 From f n+19 These are LV images up to now.
[0063] In S702, the control unit 101 sequentially inputs the groups of the multiple LV images determined in S701 to the inference unit 107, and acquires the inference results of each group. The details of the inference results are the same as those in the first embodiment.
[0064] In S703, the control unit 101 determines whether metadata can be generated based on the inference results acquired in S702. For example, the control unit 101 refers to the inference results acquired for each group, and if an inference result with a high recognition rate exceeding a predetermined threshold is obtained, the control unit 101 determines that metadata can be generated and proceeds to S409, and if not, proceeds to S410.
[0065] As described above, in this embodiment, the multiple LV images to be inferred are determined as multiple types of LV image groups each composed of multiple different LV images. In this way, it is possible to generate metadata after a certain period of time has elapsed since the image capture instruction was received.
[0066] (Embodiment 3) Next, a third embodiment will be described. In the first and second embodiments, an embodiment in which inference is executed in response to receiving an instruction to capture a still image has been described. This method is extremely efficient in terms of processing load because the inference process is executed at the required timing. However, it always takes time for the inference process to be executed from the time the image capture instruction is received until the metadata is generated, which is equal to the time it takes to execute the inference process.
[0067] In contrast, in the third embodiment, the execution time of the inference process is not included between when a shooting instruction is received and when metadata is generated. That is, inference process is executed for LV images input at any time to generate inference result information, and when a shooting instruction is received, metadata is generated from the generated inference result information (without executing the inference process). In this way, in this embodiment, the process of executing the inference and the process of shooting a still image are executed asynchronously.
[0068] The configuration of the digital camera in this embodiment is the same or substantially the same as that of the above-described embodiment, but the operation related to the still image shooting process is different. Therefore, a description of the same or substantially the same configuration will be omitted, and the description will focus on the differences.
[0069] (A series of operations related to inference processing) A series of operations of the inference process according to this embodiment will be described with reference to Fig. 8. As described above, the inference process according to this embodiment is a process of performing inference processing on an LV image at any time and storing inference result information. This process is realized by the control unit 101 expanding a program stored in the ROM 102 into a working area of the RAM 103, executing the program, and operating the imaging unit 104, the inference unit 107, etc. The series of operations related to this process are started when the digital camera 100 transitions to the shooting mode.
[0070] In S801, upon transition to the shooting mode, the control unit 101 controls the imaging unit 104 to start the generation process of the LV image. The generated LV image is transmitted to the display unit 106 and the LV image buffer 201, as in the above-described embodiment.
[0071] In S802, the control unit 101 determines whether a predetermined number of LV images have been buffered in the LV image buffer 201, and proceeds to S803 if it is determined that the images have been buffered, and otherwise waits until the images are buffered. The predetermined number is a number indicating whether the number is sufficient to perform inference, and is set according to the behavior scene to be inferred. As an example, when inferring the type of jump performance, the number of LV images corresponding to 2 seconds may be determined as a number sufficient to perform inference.
[0072] In S803, the control unit 101 inputs the LV images stored in the LV image buffer 201 to the inference unit 107 and acquires the inference result.
[0073] In S804, only when the recognition rate of the inference result acquired in S803 exceeds a predetermined threshold, the control unit 101 stores the inference result information in an area of the RAM 103. Details of the information stored here are the same as those shown in Fig. 6, and include the number of executions, the time stamp, the LV image number, and the recognition rate.
[0074] In S805, the control unit 101 determines whether the operation mode of the digital camera 100 has been changed from the shooting mode to another mode by the user. If the control unit 101 determines that the operation mode of the digital camera 100 is still in the shooting mode, the process returns to S802. On the other hand, if the control unit 101 determines that the operation mode of the digital camera 100 has been changed to another mode, the control unit 101 ends this series of operations.
[0075] (A series of operations for still image shooting processing) Next, a series of operations in the still image shooting process of the digital camera 100 according to the third embodiment will be described with reference to Fig. 9. This process is realized by the control unit 101 expanding a program stored in the ROM 102 into a working area of the RAM 103 and executing it. The series of operations in this process is started when an instruction to shoot a still image is received from the user.
[0076] In S901, the control unit 101 acquires the inference result information stored in S804. In S902, the control unit 101 determines whether metadata can be generated based on the inference result information acquired in S901. Specifically, the control unit 101 refers to the timestamp of the inference result information, and when it detects a timestamp that is the same as or close to the timing of receiving an instruction to shoot a still image, it determines that metadata can be generated and proceeds to S903. On the other hand, when the control unit 101 determines that metadata cannot be generated, it does not generate metadata and ends the series of operations of the still image shooting process.
[0077] In S903, the control unit 101 associates information on the behavior label that has a high recognition rate in the inference result corresponding to the timestamp detected in S902 with the generated still image data. Specifically, the control unit 101 additionally records the information as metadata in the header information of the still image data file recorded in the storage medium 109. After recording the metadata, the control unit 101 ends this series of processes.
[0078] As described above, in this embodiment, inference processing is performed on LV images input at any time to generate inference result information, and when a shooting instruction is received, metadata is generated from the generated inference result information (without executing the inference processing). In this way, it is possible to shorten the time required from receiving a shooting instruction to generating metadata. Also, even in this way, it is possible to determine a video of an appropriate range for inferring the behavior of the subject, and more accurately infer the behavior of the subject being shot.
[0079] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0080] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]
[0081] 101...control unit, 104...imaging unit, 106...display unit, 107...inference unit, 108...recording unit, 201...LV image buffer
Claims
1. An acquisition means for acquiring a plurality of live view image data sequentially outputted from the imaging means; a means for performing continuous shooting and acquiring data of the plurality of still images in response to receiving a shooting instruction for shooting the plurality of still images; a specifying unit for specifying, from among the plurality of live view image data, a plurality of live view image data during a predetermined period determined based on a timing at which the image capture instruction is received, as image data to be processed; a control means for adding information indicating a behavior of the subject to data of the plurality of still images based on a behavior of the subject inferred using the image data to be processed; The identification means identifies, among the multiple live view image data acquired by the acquisition means, multiple live view image data for the specified period before and after the central timing of the continuous shooting section of the continuous shooting as the image data to be processed, and the specified period includes a period that is not the continuous shooting section.
2. The method further comprises: inferring a behavior of the subject using the image data to be processed; 2. The image processing apparatus according to claim 1, wherein the control means adds information indicating the behavior of the subject based on the behavior of the subject inferred by the inference means.
3. The device further includes a communication means for communicating with an external device, The image processing device according to claim 1, characterized in that the image data to be processed is provided to the external device, and information indicating the behavior of the subject is added to the data of the multiple still images based on the behavior of the subject inferred using the image data to be processed provided from the external device.
4. the specifying means specifies, as image data to be processed, a plurality of live view image data for each of a plurality of different periods determined based on a timing at which the shooting instruction is accepted, from the plurality of live view image data acquired by the acquiring means; 4. The image processing device according to claim 1, wherein the control means adds information indicating the behavior of the subject to the data of the still images using an inference result having a recognition rate exceeding a predetermined threshold among the behavior of the subject inferred using a plurality of live view image data in each of the different periods.
5. 5. The image processing device according to claim 4, wherein when identifying a plurality of live view image data for each of the different periods, the identification means identifies the image data to be processed so that the timing at which the shooting instruction is received is the center of each of the different periods.
6. a storage unit for storing a behavior of the subject inferred using the image data to be processed; The image processing device according to any one of claims 1 to 5, characterized in that the control means adds information indicating the behavior of the subject to the data of the multiple still images based on the behavior of the subject stored in the memory means that corresponds to the timing of taking the multiple still images.
7. 7. The image processing apparatus according to claim 1, further comprising an image capturing means.
8. an acquisition step of acquiring a plurality of live view image data sequentially outputted from an imaging means; In response to receiving a shooting instruction to shoot a plurality of still images, performing continuous shooting to acquire data of the plurality of still images; a step of identifying, from among the plurality of live view image data, a plurality of live view image data during a predetermined period determined based on a timing at which the image capture instruction is received, as image data to be processed; and a control step of adding information indicating a behavior of the subject to data of the plurality of still images based on a behavior of the subject inferred using the image data to be processed, a control method for an image processing device, characterized in that in the identification process, among the multiple live view image data acquired in the acquisition process, multiple live view image data for the predetermined period before and after the central timing of the continuous shooting section of the continuous shooting are identified as the image data to be processed, and the predetermined period includes a period that is not the continuous shooting section.
9. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 7.
Citation Information
Patent Citations
Natural language processing device, natural language processing method and natural language processing program
JP2018063502A
Abnormality monitoring system
JP2018101317A
Method and system for estimating operation of work vehicle, method for manufacturing learned classification model, learning data, and method for manufacturing learning data
JP2019087200A
System and method for determining work by work vehicle, and learned model manufacturing method
JP2020004096A
Box packing apparatus
JP2020045132A
