Image processing apparatus, imaging apparatus, and image processing method

Abstract

In an image processing device that detects the behavior of a subject from an image, it is desirable to perform interpolation from past frames while minimizing the decrease in accuracy as much as possible. [Solution] The system comprises a first detection means that detects a first object from an image using a first learning model based on machine learning, a second detection means that detects a second object different from the first object from an image using a second learning model based on machine learning, and a behavior detection means that detects specific behavior of a subject based on the detection results by the first and second detection means in a moving image consisting of multiple input frames, wherein the first detection means is capable of referring to the detection results from up to n frames prior (n≧0) in the target frame in order to output the detection result of the first object in the target frame, and the second detection means is capable of referring to the detection results from up to m frames prior (m>n) in the target frame in order to output the detection result of the second object in the target frame.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, and an image processing method having a subject detection mechanism.

Background Art

[0002] Conventionally, in imaging apparatuses such as digital cameras, various techniques for detecting a subject to be controlled in order to perform imaging control such as Auto Focus (AF) have been proposed.

[0003] Patent Document 1 discloses a technique for recognizing lane lines on a road surface, and when the reliability of recognition is low, interpolation is performed from past images to improve accuracy.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Since the interpolated road does not deform dynamically, it seems that there is no problem with the disclosed method in the recognition of lane lines on the road surface described in Patent Document 1, but when applied to the recognition of a human body, it is not always possible to obtain high accuracy by interpolating from past images.

Means for Solving the Problems

[0006] To achieve the above objective, the image processing apparatus according to the present invention comprises: a first detection means for detecting a first object from an image using a first learning model based on machine learning; a second detection means for detecting a second object different from the first object from an image using a second learning model based on machine learning; and an action detection means for detecting a specific action of a subject in a video image consisting of multiple input frames based on the detection results of the first and second detection means, wherein the first detection means outputs the detection result of the first object in the target frame. , showing the same subject The detection results up to n frames prior (n≧0) of the target frame can be referenced, and the second detection means can reference detection results up to m frames prior (m>n) of the target frame in order to output the detection result of the second object in the target frame. the law of nature, The subject is a person, the first object is a joint of the person, and the second object is a ball. It is characterized by the following. [Effects of the Invention]

[0007] According to the present invention, when a part of the human body, or associated objects, belongings, or sports equipment used in a particular sport, is obscured or cannot be detected, past information can be used with reduced detection accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram of an interchangeable lens camera system [Figure 2] A flowchart illustrating the procedure for detecting human behavior. [Figure 3] Diagram showing the operation of the joint detection unit 251 and the object detection unit 252. [Figure 4] Diagram showing the operation of the joint detection unit 251 and the object detection unit 252. [Figure 5] A diagram showing examples of joint detection and object detection across multiple frames. [Modes for carrying out the invention]

[0009] The best mode for carrying out the present invention will be described in detail below with reference to the attached drawings. The embodiments described below are merely examples of means for realizing the present invention, and should be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions. The present invention is not limited to the embodiments described below.

[0010] [First Embodiment] In this embodiment, when photographing group sports, the system detects a specific action to be photographed, thereby determining which player from among multiple players will be the primary subject for autofocus (AF) and other shooting controls. In determining the primary subject, the system detects the joint positions of the people to identify specific actions. Depending on the sport, the system may also detect the ball and its relationship to the people's joint positions to detect specific actions.

[0011] However, in detecting joint positions and balls in this way, there is a possibility that the opposite arm or ball may be hidden by the human body or torso, preventing sufficient or continuous detection. When detecting movement based on a person's posture or the ball they are holding, obstruction of these elements hinders motion detection.

[0012] For example, if you keep track of the ball's coordinates before it hides, and the ball becomes hidden in the next frame, the ball's movement speed isn't very fast, so using its position from the previous frame can still provide a reasonable hint.

[0013] However, if we use the previous position because the opposite arm, hidden by the torso, cannot be detected, the human body will move while changing its posture, resulting in an unnatural posture. When trying to estimate movement from a person's posture, an unnatural posture hinders detection.

[0014] Therefore, in this embodiment, if joint points or balls cannot be detected in the target frame, interpolation is performed by referencing past frames. In this case, the configuration is such that the more stable ball (second subject) can reference earlier frames than the more unstable joint (first subject).

[0015] The camera 20 of this embodiment detects and tracks specific actions of a person and supports shooting.

[0016] FIG. 1 is a block diagram showing a functional configuration example of an interchangeable-lens camera as an example of an imaging device according to a first embodiment of the present invention.

[0017] The imaging device of this embodiment is composed of an interchangeable lens unit 10 and a camera body 20. A lens control unit 106 that comprehensively controls the operation of the entire lens and a camera control unit 250 that comprehensively controls the operation of the entire camera system including the lens unit 10 can communicate with each other through terminals provided on the lens mount. The camera body 20 is a digital still camera, a video camera, or the like that captures a subject and records moving image or still image data on various media such as a tape, a solid-state memory, an optical disk, or a magnetic disk, but is not limited thereto. For example, a mobile phone (smartphone), a personal computer (laptop type, desktop type, tablet type, etc.), a game machine, an in-vehicle sensor, FA (Factory Automation) equipment, a drone, a medical device, etc. It is applicable to any device that incorporates or externally connects an imaging device. Therefore, the "imaging device" in this specification is intended to include any electronic device having an imaging function. Also, the "image processing device" in this specification is intended to include any electronic device that determines a main subject based on an image captured by an imaging device.

[0018] First, the configuration of the lens unit 10 will be described. The fixed lens 101, the aperture 102, and the focus lens 103 constitute an imaging optical system. The aperture 102 is driven by an aperture drive unit 104 to control the amount of incident light to an image sensor 201 described later. The focus lens 103 is driven by a focus lens drive unit 105, and the combined focal length of the imaging optical system changes according to the position of the focus lens 103. The aperture drive unit 104 and the focus lens drive unit 105 are controlled by the lens control unit 106 to determine the aperture amount of the aperture 102 and the position of the focus lens 103.

[0019] The lens operation unit 107 is a group of input devices for the user to perform settings related to the operation of the lens unit 10. Specific operations of the lens unit 10 include switching between AF (auto focus) / MF (manual focus) modes, adjusting the position of the focus lens by MF, setting the operating range of the focus lens, setting the shake correction mode, etc. When the lens operation unit 107 is operated, the lens control unit 106 performs control according to the operation.

[0020] The lens control unit 106 controls the aperture drive unit 104 and the focus lens drive unit 105 according to control commands and control information received from the camera control unit 250 described later, and also transmits lens control information to the camera control unit 250.

[0021] Next, the configuration of the camera body 20 will be described. The camera body 20 is configured to be able to acquire an imaging signal from the light beam that has passed through the imaging optical system of the lens unit 10.

[0022] The imaging element 201 is composed of a CCD or a CMOS sensor. The light beam incident from the imaging optical system of the lens unit 10 forms an image on the light receiving surface of the imaging element 201, and is converted into signal charges corresponding to the incident light amount by the photodiodes provided in the pixels arranged in the imaging element 201. The signal charges accumulated in each photodiode are sequentially read out from the imaging element 201 as voltage signals corresponding to the signal charges by the drive pulses output by the timing generator 214 according to the commands of the camera control unit 250.

[0023] The CDS / AGC / AD converter 202 performs correlated double sampling, gain adjustment, and AD conversion on the imaging signal read out from the imaging element 201 to remove reset noise. The converter 202 outputs the imaging signal and the AF signal on which these processes have been performed to the image input controller 203 and the AF signal processing unit 204, respectively.

[0024] The image input controller 203 stores the imaging signal output from the converter 202 as an image signal in the SDRAM 209 via the bus 21. The image signal stored in the SDRAM 209 is read out by the display control unit 205 via the bus 21 and displayed on the display unit 206. In recording mode, where the image signal is recorded, the image signal stored in the SDRAM 209 is recorded in a recording medium 208 such as a semiconductor memory by the recording medium control unit 207.

[0025] ROM 210 stores control programs and processing programs executed by the camera control unit 250, as well as various data necessary for their execution. Flash ROM 211 stores various setting information related to the operation of the camera 20, as set by the user.

[0026] Within the camera control unit 250, the joint detection unit 251 detects the coordinates of each joint of the target person in the imaging signal input from the image input controller 203, as shown in Figure 4(a), and estimates the connection relationships, grouping them as a single person. The detection and connection method is a known method using neural networks and other machine learning models and will not be described in detail here, although some details will be mentioned later.

[0027] The object detection unit 252 detects and stores specific objects in the imaging signal input from the image input controller 203, as shown in Figure 4(a), as bounding boxes, with coordinates and length / width, as shown in Figure 4(b) 411. The detection method is a known method using neural networks and other machine learning models and will not be described in detail here, although some details will be mentioned later.

[0028] The behavior detection unit 253 receives detection information such as the coordinates and size of each joint and object output from the joint detection unit 251 and the object detection unit 252 in the image, and infers the behavior of the target person. Behavior refers to suspicious behavior to be detected by a surveillance camera, or the action of kicking a ball in soccer filming. The detection method uses a neural network, decision tree, other machine learning model, or a simple rule-based algorithm. The learning and implementation methods for these will be known methods and will not be described in detail.

[0029] Furthermore, the action detection unit 253 may be multiple in number to support multiple modes for shooting multiple sports events or to support multiple shooting environments.

[0030] The camera control unit 250 controls various parts of the camera body 20 while exchanging information with them. Furthermore, the camera control unit 250 executes various processes corresponding to user operations, such as turning the power on / off, changing various settings, image capture, AF processing, and playback of recorded images, in response to input from the camera operation unit 213 based on user operations. In addition, the camera control unit 250 transmits control commands to the lens unit 10 (lens control unit 106) and information from the camera body 20 to the lens control unit 106, and obtains information from the lens unit 10 from the lens control unit 106. The camera control unit 250 is composed of a microcomputer and controls the entire camera system, including the interchangeable lens 10, by executing computer programs stored in the ROM 210.

[0031] The following describes the processing performed by the camera body 20. The camera control unit 250 performs the following processing according to the imaging processing program, which is a computer program. Figure 2 is a flowchart showing the procedure for detecting a person's actions by the camera body 20. S stands for step.

[0032] In S201, the joint detection unit 251, which includes a machine learning model, detects the joints and their positions in the human body. In S202, the processing between S204 and S204 is repeated for each joint connection in the human body. In this embodiment, there are 13 joints: the connection from the neck to each joint, the head, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0033] In S203, the joints are connected and the person detection result is output to the action detection unit 253. In S205, the object detection unit 252, which includes a machine learning model, detects an object and outputs the detection result to the action detection unit 253. In S206, the coordinates of the person's joints detected up to S204 and the coordinates of the object detected in S205 are input to the action detection unit 253, and the probability of a specific action is inferred.

[0034] In S207, check if the actions of all people on the screen have been detected. For example, if two heads are detected, it is assumed that there are two people on the screen and the process is repeated twice. If there are still people whose actions have not been detected, proceed to S202; otherwise, proceed to S208.

[0035] S208 displays the action probabilities of all characters on the screen. Depending on your use case, you can either select the character with the highest action probability from all characters, or list characters whose action probability is above a certain threshold.

[0036] If different behavior detection units 253 are selected by different individuals, and the scale and bias of the behavior probabilities differ, normalization or the like will be used to enable direct comparison.

[0037] This completes the process of detecting the action.

[0038] Figures 3 and 4 illustrate the operation of the joint detection unit 251 and the object detection unit 252.

[0039] For posture estimation, 14 points are detected, as shown in Figure 3(a): the head, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0040] In this embodiment, to avoid making the diagram too complex and difficult to see, we have used six joints as shown in Figure 3(b): the head, center of gravity, left and right hands, and left and right feet. However, if you were to implement this, you could increase the number of joints to 14 or more, or reduce them depending on the amount of computation required for detection and connection.

[0041] For example, suppose an image like Figure 3(c) is input to the joint detection unit 251. Then, the head's response will appear in the head's peak map as shown in Figure 3(d). Similarly, the center of gravity will appear as shown in Figure 3(e), the right hand as shown in Figure 3(f), the left hand as shown in Figure 3(g), the right foot as shown in Figure 3(h), and the left foot as shown in Figure 3(i).

[0042] As shown in Figure 3(b), five connections are required to connect the joints: the head to the center of gravity 301, the right hand to the center of gravity 302, the left hand to the center of gravity 303, the right foot to the center of gravity 253, and the left foot to the center of gravity 305.

[0043] For example, if there is a response with two heads as shown in Figure 3(d), then there are two people on the screen, so the loop S202-S204 in Figure 2 will be processed twice.

[0044] When connecting from head 311 to the center of gravity in the first loop, there are two possible points, 312 and 313, as shown in Figure 3(e). By selecting 312 using some known algorithm, such as the shorter distance, one connection between the head and the center of gravity is completed. Similarly, for 6 joints including the center of gravity, the process is repeated for 5 connections until the connections are completed as shown in Figure 3(b).

[0045] Furthermore, joints detected by the joint detection unit 251 are detected as shown in Figures 3(d) to (i). These can be determined by the presence or absence of peak detection as shown in Figure 3(d)311, or a certain threshold can be set, and if the peak value is above the threshold (meets the criteria), it is detected, and if it is below the threshold, it is not detected (missing, does not meet the criteria).

[0046] Furthermore, each person is assumed to have 6 joints. In Figure 4(a), both feet are detected, and the left foot 401 is assumed to have been detected as 402 in the peak map for left foot detection, Figure 4(e). In the next frame, Figure 4(c), the left foot 404 is hidden by clothing, and similarly, the peak value is low, below the threshold, as in the peak map for left foot detection, Figure 4(f) 403, and therefore it could not be detected. In this case, as in Figure 4(d), the left foot coordinates may be omitted and connected at 5 joints, leaving the left foot coordinates as missing values, and joint information and ball information may be input to the action detection unit 253. However, the coordinates of the left foot 402, which was detected in the previous frame Figure 4(a), may be used to connect 6 joints and perform action detection. How many frames back to go will be discussed later.

[0047] In the example in Figure 4, there is only one person on the screen, but if there are multiple people, care must be taken not to connect the joints of different people in Figures 4(a) and (c). For example, if the number of frames per second is sufficiently large and the time difference between frames is small, the distance the head moves between frames should not be very large, so you can confirm that it is the same person between frames and use the joint coordinates that were not detected.

[0048] Similar to joints, the bounding box detected by the object detection unit 252 in Figure 4(a) is detected as shown in Figure 4(b) 411. However, these can also be simply marked as detected or not detected, or a certain threshold can be set so that if the detected value is above the threshold, it is detected, and if it is below the threshold, it is not detected.

[0049] Similarly, in Figure 4(a), the writer's ball 404 is detected with coordinates and size as shown in Figure 4(b) using a bounding box. In the next frame, Figure 4(c), ball 405 is half-visible outside the frame and could not be detected. In this case, the joint information and ball information may be input to the action detection unit 253 as missing values, or the action may be detected using the coordinates and size 411 of the ball that was detected in the previous frame, Figure 4(a). The number of frames prior to which reference is possible will be discussed later.

[0050] In most ball games, only one ball is used during a match, so frame-to-frame matching, similar to that required for human joints, is not necessary. However, in situations like practice where multiple balls are scattered around, frame-to-frame matching can be performed before using the coordinates from the previous frame.

[0051] Using the table in Figure 5(a), the processing in the first embodiment when joints or objects are not detected will be described.

[0052] When the action detection unit 253 infers the actions of a target person from the information of each joint and object output from the joint detection unit 251 and the object detection unit 252, it is naturally advantageous to have all the information available. For example, having information on all the joints of the body makes it easier to infer the actions the person is about to take, and the positions of balls, shuttlecocks, rackets, etc. in sports provide important clues.

[0053] However, detection may fail due to limitations in image analysis accuracy or occlusion. In such cases, using information from past frames can be considered.

[0054] In the case of a human, if the coordinate information of joints that could not be detected by the joint detection unit 251 is interpolated using past frames, referring to past frames that are too far removed in time when the human body is moving significantly will result in a distorted and unnatural posture for a human body. In other words, when attempting to infer behavior from a person's posture, a decrease in detection accuracy is likely to occur.

[0055] In the case of objects, the coordinate and size information of objects that could not be detected by the object detection unit 252 is interpolated using past frames. If the object is a ball and the action detection target is a sporting action, even if the position is slightly off, there is no problem such as the human body being in an unnatural posture. Therefore, referring to past frames from a certain distance in time does not cause much of a problem.

[0056] In the table in Figure 5(a), frame number 501 is the number of each frame in the video, and is assigned in increasing order over time. 502 indicates the detection status of one joint A (for simplicity) among the joints detected by the joint detection unit 251 on a specific person in the image. The processing is the same even if there are multiple joints. 503 indicates the object detection status output from the object detector 252.

[0057] In this case, the joint detection unit 251 uses the coordinates of the missing joints up to n frames prior for interpolation, so n=1. The object detection unit 252 uses the coordinates and size of the missing objects up to m frames prior for interpolation, so m=2. Here, m>n.

[0058] Figure 5(a) illustrates this with an example.

[0059] In frame number 1, both joint A and the object have been detected, so the joint and object information is input to the action detection unit 253, and the action is inferred.

[0060] In frame number 2, neither joint A nor the object is detected. Therefore, tracing back to the frames in which they are detected, both joint A and the object were detected in the previous frame, frame number 1. The coordinates of joint A and the coordinates and size of the object from frame number 1 are input to the action detection unit 253, and the action is inferred.

[0061] In frame number 3, neither joint A nor the object is detected. Therefore, tracing back to the frames where detection is possible, both joint A and the object were detected in frame number 1, two frames prior. Since joint A is set to n=1, this is not used and is treated as a missing value. The coordinates and size of the object are m=2, so the values ​​from frame number 1 are used, and both are input to the action detection unit 253 to infer the action.

[0062] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0063] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention.

[0064] As described above, according to the first embodiment of the present invention, when a part of the human body, or associated objects, belongings, or sports equipment used in a particular sport, is obscured or cannot be detected, past detection information can be effectively utilized while suppressing a decrease in detection accuracy.

[0065] [Second Embodiment] (If the difference between ball coordinates and size exceeds a threshold, it will not be accepted.) Except for the processes described below, the configuration and control will be the same as in the first embodiment. In this embodiment, when detecting human action, if a person's joints (points) cannot be detected, the system is configured to allow referencing past frames, and it is characterized by determining whether the past frames can be reliably referenced. Specifically, if there is a change of more than a certain threshold between past frames, those past frames will not be referenced.

[0066] Using the table in Figure 5(b), the processing in the second embodiment when joints or objects are not detected will be described.

[0067] In the table in Figure 5(b), frame number 511 is the number of each frame in the video, and it increases with time. 512 indicates the X coordinate or detection status (for simplicity) of a specific object detected by the object detection unit 252. The same applies to the Y coordinate or the detected size of the object. 513 is the difference between the X coordinate of 512 and the previous frame.

[0068] At this point, the object detection unit 252 uses the missing coordinates and size of the object up to m frames prior for interpolation, so m=2. Here again, we assume that m>n.

[0069] Here, the specific object is a ball used in ball games. The speed of ball games is known depending on the sport. The speed of a volleyball spike is said to be 150 km / h. By considering the camera lens's focal length, shooting distance, image pixel count, frame rate, and a margin of error, the maximum possible movement speed within the image can be calculated. Here, we will use 100 pixels. Balls moving faster than this may be falsely detected.

[0070] Figure 5(b) illustrates this with an example.

[0071] In frame number 1, an object is detected and its X-coordinate is output. Since it is the first frame, there is no difference. As with other embodiments, the joint coordinates are also detected by the joint detection unit 251, although this will not be described in detail. The joint and object information is then input to the action detection unit 253, and the action is inferred.

[0072] In frame number 2, an object is detected and its X-coordinate is output. The difference of X-coordinates of the object, which is 10, is recorded as the difference between frame numbers 1 and 2. This frame is also input to the action detection unit 253 with joint and object information, and the action is inferred.

[0073] In frame number 3, an object is detected and its X-coordinate is output. The difference in the object's X-coordinate between frames 2 and 3, 490, is recorded. This frame is also input to the action detection unit 253 with joint and object information to infer action.

[0074] In frame number 4, the object is not detected and its X coordinate is not output. The difference in the object's X coordinate is also not calculated. Checking the difference in the object's X coordinate in frame number 3, we find it to be 490. As mentioned earlier, the maximum movement speed of the ball is used as the difference threshold, so the X coordinate difference threshold is 100. Therefore, the X coordinate difference in frame number 3 is greater than the X coordinate difference threshold. Consequently, the X coordinate of 600 in frame number 3 is not used as the position coordinate of the ball in frame number 4. Therefore, in frame number 4, the coordinates of the object (ball) are output to the action detection unit 253 as missing values, and the action detection unit 253 infers the person's actions using the joint point information.

[0075] The above process checks for the possibility of false detection of an object based on its movement speed. If there is a possibility of false detection, it is not used to interpolate missing values ​​when an object is not detected. Ultimately, the value that may have been falsely detected is used in frame number 3, but this has the effect of preventing the false detection from affecting multiple frames. This effect is even greater if the action detection unit 253 includes time-direction filtering or similar processing.

[0076] In this embodiment, the difference with a simple past frame was compared with a threshold, but the position of the current frame may be predicted from past frames, and the difference between the predicted position and the detected coordinates may be used instead of the X-coordinate difference 513 of the object for processing.

[0077] In this embodiment, the constants included m, n, the X-coordinate difference threshold, the Y-coordinate difference threshold, and the diameter difference threshold of the object. Multiple sets of these constants may be used depending on the camera environment, the behavior to be inferred, the sport, etc.

[0078] As described above, according to the second embodiment of the present invention, when a part of the human body, or associated objects, belongings, or sports equipment used in a particular sport, is obscured or cannot be detected, past detection information can be effectively utilized while suppressing a decrease in detection accuracy. Furthermore, in this embodiment, a decrease in detection accuracy can be suppressed even when there is a high possibility of false detection of objects.

[0079] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0080] (Other embodiments) The object of the present invention can also be achieved as follows: a storage medium containing program code for software describing the procedures for realizing the functions of each embodiment described above is supplied to a system or device. The computer (or CPU, MPU, etc.) of that system or device then reads and executes the program code stored on the storage medium.

[0081] In this case, the program code read from the storage medium itself realizes the novel function of the present invention, and the storage medium and program that store that program code constitute the present invention.

[0082] Furthermore, storage media for supplying program code include, for example, flexible disks, hard disks, optical disks, and magneto-optical disks. CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, and ROMs can also be used.

[0083] Furthermore, the functions of each of the embodiments described above are realized by making the program code read by the computer executable. In addition, this also includes cases in which the OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program code, and the functions of each of the embodiments described above are realized through that processing.

[0084] Furthermore, the following cases are also included: First, program code read from a storage medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of that program code, the CPU or other components of that function expansion board or function expansion unit perform some or all of the actual processing.

[0085] This embodiment includes the following configurations, methods, and programs.

[0086] (Composition 1) A first detection means for detecting a first object from an image using a first learning model based on machine learning, A second detection means that uses a second learning model based on machine learning to detect a second object different from the first object from an image, The system includes an action detection means that detects a specific action of a subject based on the detection results of the first and second detection means in a video image consisting of multiple input frames. The first detection means can refer to detection results up to n frames prior (n≧0) of the target frame in order to output the detection result of the first object in the target frame. The image processing apparatus is characterized in that the second detection means can refer to detection results up to m frames prior (m>n) of the target frame in order to output the detection result of the second object in the target frame.

[0087] (Configuration 2) The image processing apparatus according to claim 1, characterized in that the first detection means and the second detection means detect a specific action of a subject in a target frame by interpolating by referring to a frame earlier than the target frame when the inference result for the input of the target frame does not meet the criteria.

[0088] (Composition 3) The image processing apparatus according to claim 1, characterized in that the first object is a joint of a person and the second object is a ball.

[0089] (Composition 4) The image processing apparatus according to claim 1, characterized in that the second detection means does not refer to past frames prior to the target frame if at least one of the difference in coordinates or size of the detected second object is greater than a standard.

[0090] (Composition 5) The image processing apparatus according to claim 1, wherein the first and second detection means have multiple modes suitable for multiple different sports, and the values ​​of n and m are switched for each mode.

[0091] (Composition 6) An image processing apparatus according to any one of claims 1 to 5, An imaging means for capturing the aforementioned image, An imaging device characterized by including

[0092] (Method 1) A first detection step involves detecting a first object from an image using a first learning model based on machine learning, A second detection step involves using a second machine learning-based learning model to detect a second object from an image that is different from the first object, The system includes an action detection step in which a specific action of a subject is detected in a video image consisting of multiple input frames based on the detection results of the first and second detection means, In the first detection step, in order to output the detection result of the first object in the target frame, it is possible to refer to the detection results up to n frames prior (n≧0) to the target frame. The image processing method is characterized in that, in the second detection step, it is possible to refer to detection results up to m frames prior (m>n) of the target frame in order to output the detection result of the second object in the target frame.

[0093] (Program 1) A computer-executable program describing the steps of the image processing method described in claim 7. [Explanation of symbols]

[0094] 10 Lens Units 106 Lens control unit 107 Lens control section 20 cameras 201 Image sensor 203 Image Input Controller 204 AF signal processing unit 205 Display Control Unit 206 Display section 207 Recording medium control unit 208 recording media 250 Camera Control Unit 251 Joint detection unit 252 Object detection unit 253 Action Detection Unit

Claims

1. A first detection means for detecting a first object from an image using a first learning model based on machine learning, A second detection means for detecting a second object different from the first object from an image using a second learning model based on machine learning, The system includes an action detection means that detects a specific action of a subject based on the detection results of the first and second detection means in a video image consisting of multiple input frames. The first detection means can refer to detection results up to n frames earlier (n≧0) of the target frame showing the same subject in order to output the detection result of the first object in the target frame. The second detection means can refer to detection results up to m frames prior (m > n) of the target frame in order to output the detection result of the second object in the target frame. An image processing apparatus characterized in that the subject is a person, the first object is the joint of the person, and the second object is a ball.

2. The image processing apparatus according to claim 1, characterized in that n is 1 or less.

3. The image processing apparatus according to Claim 1, wherein the first detection means determines whether or not to use the joint detection result in a past frame that is a reference candidate when the joint cannot be detected in the target frame, based on the identity of the person between the target frame and the past frame, and uses the joint detection result in the past frame only if it is determined that they are the same.

4. The image processing apparatus according to claim 1, characterized in that the first detection means and the second detection means detect a specific action of a subject in a target frame by interpolating by referring to a frame earlier than the target frame when the detection result for the input of the target frame does not meet the criteria.

5. The image processing apparatus according to claim 1, characterized in that the first object is a joint of a person and the second object is a ball.

6. The image processing apparatus according to claim 1, characterized in that the second detection means does not refer to past frames prior to the target frame if at least one of the difference in coordinates or size of the detected second object is greater than a standard.

7. The image processing apparatus according to claim 1, wherein the first and second detection means have multiple modes suitable for multiple different sports, and the values ​​of n and m are switched for each mode.

8. An imaging apparatus comprising an image processing apparatus according to any one of claims 1 to 7, and an imaging means for capturing the image.

9. A first detection step in which a first object is detected from an image using a first learning model based on machine learning, A second detection step involves using a second machine learning-based learning model to detect a second object from an image that is different from the first object, The system includes an action detection step in which a specific action of a subject is detected in a video image consisting of multiple input frames based on the detection results of the first and second detection means, In the first detection step, in order to output the detection result of the first object in the target frame, it is possible to refer to the detection results up to n frames earlier (n≧0) of the target frame that show the same subject. In the second detection step, in order to output the detection result of the second object in the target frame, it is possible to refer to the detection results up to m frames prior to the target frame (m > n). An image processing method characterized in that the subject is a person, the first object is the joint of the person, and the second object is a ball.

10. A computer-executable program describing the procedure for the image processing method described in claim 9.

Images