Image processing method, image processing program, and image processing device

Abstract

Realizing a highly accurate skeletal detection technology with low processing load [Solution] Likelihood information indicating that a keypoint exists at each coordinate point on each of the X and Y axes of the image coordinate axes is obtained from the image to be processed. A window of a predetermined width w is set at each coordinate point of the likelihood information, and a score is calculated based on the likelihood change information for w surrounding coordinates toward the origin, the likelihood change information for w surrounding coordinates toward the end point, and the likelihood at each coordinate point. A score is obtained for each coordinate, and the coordinate point in the likelihood information where the score is maximum is estimated as the likelihood peak point. The intersection of the peak points determined on the X and Y axes is estimated to be the area where a keypoint exists, and the keypoint in the image to be processed is determined from the estimated area.

Description

[Technical Field]

[0001] The present invention relates to a program, a method, and an apparatus. [Background technology]

[0002] Skeleton detection and skeletal estimation are known techniques for extracting a human body region from a captured image, calculating feature values ​​for each of multiple key points on the human body from that region, and estimating the posture and physique of the human body in the image based on the calculated feature values. Non-Patent Document 1 discloses a skeleton detection and pose estimation technique that uses a CNN (convolutional neural network) learning model called OpenPose. Skeleton detection technology has also been used in in-vehicle devices; for example, Patent Document 1 discloses a technique for estimating the physique of an occupant using skeleton detection technology from captured images inside a vehicle. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] Non-patent document 1: Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, P. 7291-7299 [Patent Document 1] Patent Publication No. 2020-104680 Summary of the Invention [Problem to be solved by the invention]

[0004] Skeleton detection technology using CNNs such as OpenPose has the advantage of being able to robustly detect even in situations where there is severe occlusion between human parts, and can process multiple people. However, the processing load is large, and when trying to use it in embedded software for devices such as in-vehicle equipment, it is unavoidable that the software will become larger and costlier in order to operate properly.

[0005] Therefore, in skeleton detection technology that uses a CNN learning model, it is possible to reduce the computational load by quantizing and processing the features obtained from the CNN learning model or by using a lightweight CNN learning model, but this can lead to problems such as errors in estimating the position of the keypoint.For example, Figure 6 is a graph showing the likelihood (reliability) that a certain keypoint exists at a position on the X axis of the image, and shows an example of estimating the position of the keypoint by estimating the point at which the likelihood is maximum.However, a problem occurred in which a different point, PeakF, was estimated as the point where the keypoint exists, rather than PeakT on the graph, which is originally considered to be the correct answer. [Means for solving the problem]

[0006] According to the present invention, an image acquisition step of acquiring an image; a likelihood information generation step of acquiring, using a trained model, first likelihood information which is the likelihood that a keypoint exists at each coordinate point on the X axis of an image coordinate system of the image, and second likelihood information which is the likelihood that the keypoint exists at each coordinate point on the Y axis of the image coordinate system; a peak point estimation step of calculating a score for each of the first likelihood information and the second likelihood information based on the likelihood at each coordinate point and a change in likelihood of neighboring coordinates, and estimating the coordinate point at which the score is maximum as a peak point; a keypoint determination step of determining coordinate points of the keypoints in the image from the peak points of the first likelihood information and the peak points of the second likelihood information; An image processing method is provided, comprising: [Effects of the Invention]

[0007] According to the image processing method of the present invention, a keypoint detection technique with a low processing load and high accuracy can be realized. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing the hardware configuration of an image processing device 1. FIG. [Figure 2] FIG. 10 is a flowchart showing the flow of skeleton recognition processing. [Figure 3] 10 is a flowchart showing the flow of processing for peak point estimation by a peak point estimation unit 1013. FIG. [Figure 4] 10A and 10B are graphs and diagrams illustrating likelihood information. [Figure 5] 10A and 10B are diagrams illustrating and explaining calculations when detecting peak points; [Figure 6] FIG. 1 is a diagram illustrating an example of a conventional technique. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, identical or equivalent components, parts, processes, and signals shown in each drawing will be assigned the same reference numerals, and duplicate explanations will be omitted where appropriate. Furthermore, some components that are not important for the explanation will be omitted in each drawing. The following embodiments do not unduly limit the content of the present disclosure described in the claims, and not all of the components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0010] (Hardware configuration of image processing device 1) The image processing device 1 according to this embodiment includes a control unit 101, an imaging unit 102, and a storage unit 103. The control unit 101 is configured with a processor, and reads out an application program and data required for processing stored in the storage unit 103, executes the processing, and realizes the functions of the application program. As a result, the image processing device 1 is realized as an information processing device that performs image processing. Note that, in this embodiment, the image processing device 1 is assumed to be an integrated hardware device such as an in-vehicle device, but is not limited thereto. The image processing program of this embodiment may be executed on a personal computer or a server, or may be realized on a cloud server or a mobile terminal such as a smartphone.

[0011] (Configuration of the control unit 101 of the image processing device 1) The control unit 101 is composed of the following functional units: an image acquisition unit 1011, a likelihood information generation unit 1012, a peak point estimation unit 1013, a keypoint determination unit 1014, and a skeleton information generation unit 1015. The control unit 101 reads out an application program stored in the storage unit 103 and executes processing to realize each functional unit.

[0012] (Functional configuration of the imaging unit 102 of the image processing device 1) The imaging unit 102 is a digital camera having an imaging element such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor). The imaging unit 102 is capable of capturing images of the surroundings of the image processing device 1 at a predetermined frame rate, and outputs the captured images to the storage unit 103. The imaging unit 102 may be configured integrally with the image processing device 1, or may be configured as an external device capable of communicating with the image processing device 1.

[0013] (Functional configuration of the storage unit 103 of the image processing device 1) The memory unit 103 is a storage device for permanently or temporarily storing data such as image processing programs, learning models, and images captured by the imaging unit 102, and is composed of volatile memory such as RAM, ROM, a hard disk, and an SSD.

[0014] (Image acquisition unit 1011) The image acquisition unit 1011 acquires a video or still image captured by the imaging unit 102 as an input image, generates an image to be processed by the likelihood information generation unit 1012 (hereinafter referred to as a processing target image) from the input image, and stores the processing target image in the storage unit 103. In the case of a still image, the input image is the entire still image captured by the imaging unit 102 or a portion thereof. In the case of a video, the input image is the entire still image or a portion thereof of one of multiple frame images included in the video. When the image acquisition unit 1011 uses a frame of a video or a portion of a still image captured by the imaging unit 102 as the input image, the image acquisition unit 1011 can determine a rectangular area of ​​a person as a target subject using a known learning model such as YOLO or an object detection function such as pattern matching, and clip the image of the rectangular area to use as the input image. Alternatively, an image within a fixed area within the image may be clipped and used as the input image. Furthermore, when multiple people are present in a still image or frame, an input image is generated for each person, but the generation of input images may be restricted according to arbitrary conditions.

[0015] The image acquisition unit 1011 generates a processing target image by resizing the input image to a predetermined size. The predetermined resized size is the size of the image to be processed by the trained model, and may be, for example, 512 pixels vertically and 512 pixels horizontally. The image size to be resized corresponds to the standardized image size input to the trained model when training the trained model. The input image is resized using a known image reduction method such as a bilinear method, or an image enlargement method using pixel value interpolation processing. In other words, the image acquisition unit 1011 generates a processing target image by converting the input image to a standardized size for processing by the likelihood information generation unit 1012.

[0016] (Likelihood information generation unit 1012) The likelihood information generation unit 1012 acquires the processing target image from the storage unit 103 and generates likelihood information indicating the likelihood (reliability) of the presence of keypoints in the processing target image using a trained model based on a CNN (convolutional neural network). Keypoints are nodes of skeletal information in skeletal detection, i.e., body parts such as joints, for example, nose joint points, neck joint points, shoulder joint points, elbow joint points, waist joint points, wrist joint points, knee joint points, and ankle joint points. The trained model is a model that has trained in advance the positions of each keypoint and a set of images, and outputs the likelihood of the presence of a keypoint at each coordinate in the processing target image as likelihood information. The trained model can generate likelihood information for each keypoint for the processing target image. That is, it generates likelihood information for each keypoint, such as nose joint points, neck joint points, shoulder joint points, elbow joint points, waist joint points, wrist joint points, knee joint points, and ankle joint points.

[0017] The likelihood information will be described with reference to Fig. 4. Fig. 4(a) shows the processing target image img1, and for convenience, the X-axis and Y-axis are shown together as image coordinate axes. The likelihood information generation unit 1012 inputs the processing target image img1 to a trained model and obtains likelihood information in the X-axis direction and likelihood information in the Y-axis direction as outputs of the trained model. The likelihood information in the X-axis direction is information indicating the likelihood that a keypoint exists at each coordinate point on the X-axis of the processing target image img1, and the likelihood information in the Y-axis direction is information indicating the likelihood that a keypoint exists at each coordinate point on the Y-axis of the processing target image img1.

[0018] The graph in FIG. 4(b) shows an example of likelihood information in the X-axis direction. The horizontal axis of the graph in FIG. 4(b) corresponds to the X-axis coordinate of the image to be processed. The vertical axis of the graph in FIG. 4(b) shows the likelihood value, which in this embodiment takes integer values ​​from 0 to 255. The range of the likelihood value may differ depending on the trained model, and the data type of the value may be not only integer values ​​but also floating point values. However, in this embodiment, the likelihood is quantized to reduce the calculation load, and takes integer values ​​in the aforementioned range. In the example of FIG. 4, the likelihood at coordinate p on the horizontal axis is Fp, which indicates the likelihood that a keypoint exists at coordinate p on the X-axis of the image to be processed (img1).

[0019] 4(b) has been described regarding likelihood information for the X-axis direction of the image to be processed, but likelihood information is also calculated for the Y-axis direction. The graph in FIG. 4(b) can also be applied to likelihood information for the Y-axis direction, with the horizontal axis of the graph representing position information corresponding to the Y-axis direction of the image to be processed, and the vertical axis representing the likelihood at that coordinate point. The likelihood information generating unit 1012 stores the generated likelihood information in the storage unit 103.

[0020] (Peak point estimation unit 1013) The peak point estimation unit 1013 acquires likelihood information of an arbitrary keypoint from the storage unit 103 and estimates a peak point. A peak point is a coordinate point where a keypoint is estimated to exist in the likelihood information. In conventional technology, simply estimating a point with a high likelihood as a peak point based on the quantized likelihood value results in an erroneous estimation, as shown in the example of Figure 6. Therefore, the peak point is determined by taking into account the change in likelihood around the target coordinate point. Taking the graph of Figure 6 as an example, when observing the change in likelihood around the coordinate point PeakT to be determined as the actual peak point, it is effective to estimate as the peak point a point whose graph shape is upwardly convex and has a high likelihood value. Therefore, the peak point estimation unit 1013 calculates a score for each coordinate point in the likelihood information based on the change in the likelihood around the coordinate point and the likelihood value, and estimates the coordinate point with the largest score as the peak point. A specific method for calculating the score will be described later.

[0021] The peak point estimation unit 1013 estimates the peak point for any key point independently for the likelihood information in the X-axis direction and the Y-axis direction of the image to be processed, and stores the obtained peak point information in the memory unit 103.

[0022] (Keypoint determination unit 1014) The keypoint determination unit 1014 acquires peak point information from the storage unit 103 and determines the coordinates of the keypoints in the image to be processed. As described above, the peak point information is information about coordinate points on the X and Y axes in the image to be processed. The coordinate points of intersection on the image to be processed are obtained from the peak points obtained from the likelihood information in the X and Y axes of the image to be processed. The keypoint determination unit 1014 estimates these coordinate points as keypoints in the image to be processed. The keypoint determination unit 1014 determines the coordinates of keypoints on the input image from the keypoints in the image to be processed. If the image to be processed is a resized image, the coordinate points of the image to be processed correspond to areas on the input image, so the center of gravity of the area is used as the coordinate points of the keypoints in the input image. The coordinate points of the keypoints in the input image can be determined from the coordinate points of the keypoints in the image to be processed using a method other than the above. The keypoint determination unit 1014 stores the information of the determined keypoints in the storage unit 103.

[0023] (Skeletal information generation unit 1015) The skeletal information generation unit 1015 acquires key point information from the storage unit 103 and generates skeletal information. As described above, in skeleton detection, key points are body parts, and key point information is generated for each of them. Therefore, a combination of key point type and coordinates on the input image can be obtained from the key point information, and the skeletal information generation unit 1015 estimates the skeletal structure from the connection relationships between predetermined key point types and generates skeletal information.

[0024] (Operation of image processing device 1) Next, we will explain each process of the image processing device 1. Figures 2 and 3 are flowcharts showing the operation of the processes. Figure 2 shows the flow of the process of generating skeleton information by the image processing device 1, and Figure 3 shows the flow of the process of estimating peak points from likelihood information.

[0025] (Step S01: Generation of processing target image) The image acquisition unit 1011 acquires a moving image or a still image captured by the imaging unit 102 as an input image, generates a processing target image from the input image, and stores the processing target image in the storage unit 103. As described above, the processing target image is an input image, which is all or part of the image captured by the imaging unit 102, with the image size standardized for processing by the likelihood information generation unit 1012.

[0026] (Step S02: Feature extraction using CNN) The likelihood information generation unit 1012 acquires the processing target image from the storage unit 103 and generates likelihood information for the processing target image using a trained model based on a CNN (convolutional neural network). As explained above, likelihood information is generated for each key point in the X-axis and Y-axis directions of the processing target image. The likelihood information generation unit 1012 stores the generated likelihood information in the storage unit 103.

[0027] (Step S03: Start of process for determining each key point) The peak point estimation unit 1013 starts the process of determining key points for each body part in the skeleton detection. As shown in the loop in the flowchart of Figure 2, the process of steps S04 to S06 is performed for each body part, and the process of determining key points is performed.

[0028] (Step S04: Estimating the peak point in the X-axis direction) The peak point estimation unit 1013 acquires likelihood information about the X-axis direction of the image to be processed from the likelihood information of the keypoint to be detected from the storage unit 103. The peak point estimation unit 1013 obtains a score calculated from the change in likelihood around each coordinate point of the likelihood information and the likelihood value, and estimates the coordinate point with the maximum score as the peak point. A specific method for calculating the score will be described later using Figure 3.

[0029] (Step S05: Estimation of peak point in Y-axis direction) The peak point estimation unit 1013 acquires likelihood information about the Y-axis direction of the image to be processed from the likelihood information of the keypoint to be detected from the storage unit 103. The peak point estimation unit 1013 obtains a score calculated from the change in likelihood around each coordinate point of the likelihood information and the likelihood value, and estimates the coordinate point with the maximum score as the peak point. A specific method for calculating the score will be described later using Figure 3.

[0030] (Step S06: Determining key points) The keypoint determination unit 1014 acquires peak point information from the storage unit 103 and determines the coordinates of the keypoints in the image to be processed. As described above, the peak point information is information about coordinate points on the X and Y axes in the image to be processed. The coordinate points of intersection on the image to be processed are obtained from the peak points obtained from the likelihood information in the X and Y axes of the image to be processed. The keypoint determination unit 1014 estimates these coordinate points as keypoints in the image to be processed. The keypoint determination unit 1014 determines the coordinates of keypoints on the input image from the keypoints in the image to be processed. If the image to be processed is a resized image, the coordinate points of the image to be processed correspond to areas on the input image, so the center of gravity of the area is used as the coordinate points of the keypoints in the input image. The coordinate points of the keypoints in the input image can be determined from the coordinate points of the keypoints in the image to be processed using a method other than the above. The keypoint determination unit 1014 stores the information of the determined keypoints in the storage unit 103.

[0031] (Step S07: End of process for determining each key point) When the key point determination unit 1014 has completed the process of determining each key point, the control unit 101 advances the process to step S08.

[0032] (Step S08: Generate skeletal information) The skeletal information generation unit 1015 acquires key point information from the storage unit 103 and generates skeletal information. As described above, in skeleton detection, key points are body parts, and key point information is generated for each of them. Therefore, a combination of key point type and coordinates on the input image can be obtained from the key point information, and the skeletal information generation unit 1015 estimates the skeletal structure from the connection relationships between predetermined key point types and generates skeletal information.

[0033] The generated skeletal information is used for known posture recognition processing, physique recognition processing, etc. by a functional unit of the control unit 101 (not shown) or a device external to the image processing device 1. When a moving image is to be processed by the imaging unit 102 of the image processing device 1, a predetermined target image is generated for each frame of the moving image, and skeletal information is generated, thereby generating chronological skeletal information. This allows chronological processing such as action recognition processing and subject tracking processing to be performed by a functional unit of the control unit 101 (not shown) or a device external to the image processing device 1.

[0034] Next, the peak point estimation process in steps S04 and S05 will be described with reference to Fig. 3 and Fig. 5(a). Fig. 3 is a flow diagram showing the flow of the peak point estimation process by the peak point estimation unit 1013. Fig. 5(a) is a graph and illustration for explaining the process of calculating the score from the likelihood information in this embodiment.

[0035] First, the peak point estimation unit 1013 acquires likelihood information of the keypoint to be processed from the storage unit 102. The likelihood information acquired at this time is likelihood information about the X-axis direction of the image to be processed in the process at step S04, and likelihood information about the Y-axis direction of the image to be processed in the process at step S05. In the following explanation of steps S11 to S16, the likelihood information about the X-axis direction of the image to be processed will be described, but the processing is similar for the Y-axis direction.

[0036] (Step S11: Parameter initialization) The peak point estimation unit 1013 initializes parameters for processing. In the graph in Fig. 5(a), the score is calculated for each coordinate point on the horizontal axis, so the coordinate point p that is the target of score calculation is initialized to 0. In addition, the maximum score Ms is initialized to 0, and the coordinate point mp where the score is maximum is initialized to 0.

[0037] (Step S12: Start processing for each coordinate point) The processes from steps S13 to S16 are repeated until p becomes the end point of the coordinate axis. The value of p is determined by the number of pixels on the X axis of the image to be processed. For example, if the X axis of the image to be processed is 512 pixels, p will take values ​​from 0 to 511 as the process proceeds.

[0038] (Step S13: Calculating the score at the coordinate point) The peak point estimation unit 1013 calculates a score T at coordinate p. Here, the likelihood value at an arbitrary coordinate i in the likelihood information is assumed to be F(i). Referring to FIG. 5(a) as an example, the likelihood at coordinate p is F(p). At an arbitrary coordinate i, the difference in likelihood from coordinate i-1 is assumed to be Dl(i), and is defined as Dl(i) = F(i) - F(i-1). Furthermore, at an arbitrary coordinate point i, the difference in likelihood from coordinate point i+1 is assumed to be Dr(i), and is defined as Dr(i) = F(i) - F(i+1).

[0039] Here, the width of the window to be processed is r (r=4 in this embodiment and in FIG. 5(a)), and the sum of Dl at r coordinate points from coordinate point p toward the origin (left side of FIG. 5(a)) is defined as Sl. Also, Dr at r coordinate points from coordinate point p toward the end point (right side of FIG. 5(a)) is obtained, and the sum of these is defined as Sr. The formula for calculating Sl is shown in Equation 1, and the formula for calculating Sr is shown in Equation 2.

[0040]

number

[0041]

number

[0042] When calculating Sl and Sr, if the reference coordinate point due to the processing window width r falls outside the range of the coordinate axes, Dl and Dr at the coordinates outside the range are ignored. For example, when calculating Sl when p=1, Dl outside the range cannot be calculated, so it is ignored as a target for adding Sl, and only the points within the range of the coordinate axes are added.

[0043] Furthermore, Sl and Sr are compared, and the smaller of them is set as T'.

[0044] The peak point estimation unit 1013 calculates the score T at the coordinate point p from T'. The score T is calculated by weighting the likelihood F(p) and T' with a predetermined weight w (0<=w<=1) and taking the sum as shown in Equation 3. The peak point estimation unit 1013 stores the calculated score T in the storage unit 103. In calculating the score T, the weight w may be set to 1 and the score T may be calculated using only T'.

[0045]

number

[0046] In this embodiment, when calculating Sl and Sr, the processing window width r is set to the same number and is calculated so as to be symmetrical on the left and right sides of the coordinate p on the position coordinate axis, but when calculating Sl and Sr, the processing window width r may be set to different numbers.

[0047] Furthermore, in step S13, the processing target window width r may be set to different values ​​when processing likelihood information about the X-axis direction of the image to be processed and when processing likelihood information about the Y-axis direction. As described above, the processing target window width r can also be set to different values ​​when calculating Sl and Sr, so that predetermined values ​​can be set for a total of four values: the processing target window width r for Sl and Sr when processing likelihood information about the X-axis direction, and the processing target window width r used to calculate Sl and Sr when processing likelihood information about the Y-axis direction. This allows processing to be performed taking into account changes in likelihood in different ranges in each of the X-axis and Y-axis directions of the image to be processed.

[0048] (Step S14: Update the maximum score and its coordinates) The peak point estimation unit 1013 compares the score T in the storage unit 103 with Ms, which is the current maximum score, and if T>Ms, updates Ms to T. Also, the coordinate point mp at which the score is maximum is updated based on the value of p when Ms is updated. When T=Ms, mp may be updated to a new coordinate point, or the original value of mp may be maintained without being updated.

[0049] (Step S15: End processing for each coordinate point) When the processing of steps S13 and S14 is completed, p is incremented, and if p is not the end point of the position coordinate axis of the likelihood information, in this embodiment, if p<=512, the processing returns to step S12 and is repeated. When p exceeds 512, the loop is terminated and the processing proceeds to step S16.

[0050] (Step S16: Determining the peak point) The peak point estimation unit 1013 determines the coordinate point mp at which the likelihood of the likelihood information is maximum as the peak point, stores the information in the storage unit 103, and ends the process.

[0051] The peak point estimation process described above can estimate peak points by taking into account the likelihood as well as the score, which is a value indicating whether the graph shape around the coordinate point is convex upward, as shown in Figure 5(a). When Sl at coordinate point p is positive and large, it is an index of the proportion of positive slopes (rising upward to the right on the graph) toward coordinate point p. When Sr is positive and large, it is an index of the proportion of negative slopes (falling downward to the right on the graph) from coordinate point p to the right (toward the coordinate end point). Therefore, a positive T' value indicates that the shape around the graph at coordinate point p is convex upward. On the other hand, when the shape around the graph at coordinate point p is a downward convex valley or monotonically increasing or decreasing, T' is negative. Furthermore, when the shape around the graph at coordinate point p is nearly horizontal, T' is close to 0. This can improve the peak point estimation error that was previously a problem.

[0052] Furthermore, since the calculation processing in each loop can be performed using only simple addition and subtraction, the calculation order is small and the amount of calculation processing can be reduced. This means that the load of keypoint detection processing in skeleton detection, etc. can be reduced even in environments with limited hardware resources, such as embedded software in automotive equipment.

[0053] (Another example of score calculation) Another example of the score calculation method in step S13 will be described with reference to FIG. 5(b). In this example, the calculation method for Sl and Sr differs from the previous example. The width of the processing window is r (r=4 in this example and FIG. 5(b)). Sl is the difference between the likelihood at coordinate point p and the smallest likelihood among r coordinate points moving from coordinate point p toward the origin (left side of FIG. 5(a)). Sr is the difference between the likelihood at coordinate point p and the smallest likelihood among r coordinate points moving from coordinate point p toward the end point (right side of FIG. 5(b)). In the example of FIG. 5(b), Sl is the difference between the likelihood at coordinate p and the likelihood at coordinate p-2, and Sr is the difference between the likelihood at coordinate p and the likelihood at coordinate p+2. This method reduces the amount of addition and subtraction required for score calculation compared to the calculation method of the previous example, enabling peak point estimation processing with a lower processing load.

[0054] As in the previous embodiment, the processing window width r may be set to different values ​​when calculating Sl and Sr. Furthermore, in step S13, the processing window width r may be set to different values ​​when processing likelihood information about the X-axis direction of the processing target image and when processing likelihood information about the Y-axis direction.

[0055] As described above, the image processing method and program disclosed herein can provide a keypoint detection technique with a low processing load and high accuracy.

[0056] The invention made by the present inventor has been specifically described based on the embodiments, but the key points detected by the present invention are not limited to joint points in skeletal detection, and can be used to detect any region of interest (ROI) or feature points in still images or videos. These can be realized by using all or part of the present invention. Furthermore, while the learning model in this embodiment was based on a CNN (convolutional neural network), the present invention is not limited to this and can be applied to features obtained by various learning models.

[0057] Furthermore, the present invention is not limited to the above-described embodiments. The above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. It is possible to add, delete, or replace part of the configuration of the above-described embodiments with other configurations.

[0058] The above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a storage device such as a memory, hard disk, or SSD, or in a storage medium such as an IC card, SD card, or DVD. [Explanation of symbols]

[0059] 1. Image processing device 101 Control section 1011 Image acquisition unit 1012 Likelihood information generation unit 1013 Peak point estimation unit 1014 Keypoint determination unit 1015 Skeleton information generation unit 102 Imaging unit 103 Storage section

Claims

1. A method implemented on a computer having a processor and a memory, comprising: an image acquisition step of acquiring an image; a likelihood information generation step of acquiring, using a trained model, first likelihood information which is the likelihood that a keypoint exists at each coordinate point on the X axis of an image coordinate system of the image, and second likelihood information which is the likelihood that the keypoint exists at each coordinate point on the Y axis of the image coordinate system; a peak point estimation step of calculating a score for each of the first likelihood information and the second likelihood information based on the likelihood at each coordinate point and a change in likelihood of neighboring coordinates, and estimating the coordinate point at which the score is maximum as a peak point; a keypoint determination step of determining coordinate points of the keypoints in the image from peak points of the first likelihood information and peak points of the second likelihood information; It consists of The calculation of the score in the peak point estimation step is Each of the first likelihood information and the second likelihood information is defined as target likelihood information, the difference between the likelihood at an arbitrary coordinate i of the target likelihood information and the likelihood at coordinate i-1 is defined as dl, the difference between the likelihood at coordinate i and the likelihood at coordinate i+1 is defined as dr, at an arbitrary coordinate p of the target likelihood information, the sum of dl at a first predetermined number of coordinate points from coordinate p toward the origin of the coordinate axis is defined as Sl, the sum of dr at a second predetermined number of coordinate points from coordinate p toward the end point of the coordinate axis is defined as Sr, the smaller of Sl and Sr is defined as T, the likelihood at coordinate p is defined as Fp, and the score is calculated by performing a weighted addition of T and Fp. Image processing methods.

2. In the weighted addition in the peak point estimation step, T is taken as the score; The image processing method according to claim 1 .

3. A method executed on a computer having a processor and a memory, comprising: an image acquisition step of acquiring an image; a likelihood information generation step of acquiring, using a trained model, first likelihood information which is the likelihood that a keypoint exists at each coordinate point on the X axis of an image coordinate system of the image, and second likelihood information which is the likelihood that the keypoint exists at each coordinate point on the Y axis of the image coordinate system; a peak point estimation step of calculating a score for each of the first likelihood information and the second likelihood information based on the likelihood at each coordinate point and a change in likelihood of neighboring coordinates, and estimating the coordinate point at which the score is maximum as a peak point; a keypoint determination step of determining coordinate points of the keypoints in the image from peak points of the first likelihood information and peak points of the second likelihood information; It consists of The calculation of the score in the peak point estimation step is Each of the first likelihood information and the second likelihood information is defined as target likelihood information, and the difference between the likelihood Fp at an arbitrary coordinate p of the target likelihood information and the smallest likelihood among the likelihoods at a first predetermined number of coordinate points from the coordinate p toward the origin of the coordinate axis is defined as Sl, and the difference between the likelihood Fp and the smallest likelihood among the likelihoods at a second predetermined number of coordinate points from the coordinate p toward the end point of the coordinate axis is defined as Sr, and the smaller of Sl and Sr is defined as T, and the score is calculated by performing a weighted addition of T and Fp. Image processing methods.

4. In the weighted addition in the peak point estimation step, T is taken as the score; The image processing method according to claim 3 .

5. An image processing program that causes a computer to execute the image processing method described in any one of claims 1 to 4.

6. An image processing device that executes the image processing method described in any one of claims 1 to 4 using a processor.

Images